6: Storing Resources In ZODB

Store and retrieve resource tree containers and items in a database.

Background

We now have a resource tree that can go infinitely deep, adding items and subcontainers along the way. We obviously need a database, one that can support hierarchies. ZODB is a transaction-based Python database that supports transparent persistence. We will modify our application to work with the ZODB.

Along the way we will add the use of pyramid_tm, a system for adding transaction awareness to our code. With this we don't need to manually manage our transaction begin/commit cycles in our application code. Instead, transactions are setup transparently on request/response boundaries, outside our application code.

Objectives

  • Create a CRUD app that adds records to persistent storage.
  • Setup pyramid_tm and pyramid_zodbconn.
  • Make our "content" classes inherit from Persistent.
  • Set up a database connection string in our application.
  • Set up a root factory that serves the root from ZODB rather than from memory.

Steps

  1. We are going to use the previous step as our starting point:

    $ cd ..; cp -r addcontent zodb; cd zodb
    
  2. Introduce some new dependencies in zodb/setup.py:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    from setuptools import setup
    
    requires = [
        'pyramid',
        'pyramid_jinja2',
        'ZODB3',
        'pyramid_zodbconn',
        'pyramid_tm',
        'pyramid_debugtoolbar'
    ]
    
    setup(name='tutorial',
          install_requires=requires,
          entry_points="""\
          [paste.app_factory]
          main = tutorial:main
          """,
    )
    
  3. We can now install our project:

    $ $VENV/bin/python setup.py develop
    
  4. Modify our zodb/development.ini to include some configuration and give database connection parameters:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    [app:main]
    use = egg:tutorial
    pyramid.reload_templates = true
    pyramid.includes =
        pyramid_debugtoolbar
        pyramid_zodbconn
        pyramid_tm
    zodbconn.uri = file://%(here)s/Data.fs?connection_cache_size=20000
    
    [server:main]
    use = egg:pyramid#wsgiref
    host = 0.0.0.0
    port = 6543
    
    # Begin logging configuration
    
    [loggers]
    keys = root, tutorial
    
    [logger_tutorial]
    level = DEBUG
    handlers =
    qualname = tutorial
    
    [handlers]
    keys = console
    
    [formatters]
    keys = generic
    
    [logger_root]
    level = INFO
    handlers = console
    
    [handler_console]
    class = StreamHandler
    args = (sys.stderr,)
    level = NOTSET
    formatter = generic
    
    [formatter_generic]
    format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s
    
    # End logging configuration
    
  5. Our startup code in zodb/tutorial/__init__.py gets some bootstrapping changes:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    from pyramid.config import Configurator
    from pyramid_zodbconn import get_connection
    
    from .resources import bootstrap
    
    
    def root_factory(request):
        conn = get_connection(request)
        return bootstrap(conn.root())
    
    def main(global_config, **settings):
        config = Configurator(settings=settings,
                              root_factory=root_factory)
        config.include('pyramid_jinja2')
        config.scan('.views')
        return config.make_wsgi_app()
    
  6. Our views in zodb/tutorial/views.py have modest changes in add_folder and add_content for how new instances are made and put into a container:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    from random import randint
    
    from pyramid.httpexceptions import HTTPFound
    from pyramid.location import lineage
    from pyramid.view import view_config
    
    from .resources import (
        Root,
        Folder,
        Document
        )
    
    
    class TutorialViews(object):
        def __init__(self, context, request):
            self.context = context
            self.request = request
            self.parents = reversed(list(lineage(context)))
    
        @view_config(renderer='templates/root.jinja2',
                     context=Root)
        def root(self):
            page_title = 'Quick Tutorial: Root'
            return dict(page_title=page_title)
    
        @view_config(renderer='templates/folder.jinja2',
                     context=Folder)
        def folder(self):
            page_title = 'Quick Tutorial: Folder'
            return dict(page_title=page_title)
    
        @view_config(name='add_folder', context=Folder)
        def add_folder(self):
            # Make a new Folder
            title = self.request.POST['folder_title']
            name = str(randint(0, 999999))
            new_folder = Folder(title)
            new_folder.__name__ = name
            new_folder.__parent__ = self.context
            self.context[name] = new_folder
    
            # Redirect to the new folder
            url = self.request.resource_url(new_folder)
            return HTTPFound(location=url)
    
        @view_config(name='add_document', context=Folder)
        def add_document(self):
            # Make a new Document
            title = self.request.POST['document_title']
            name = str(randint(0, 999999))
            new_document = Document(title)
            new_document.__name__ = name
            new_document.__parent__ = self.context
            self.context[name] = new_document
    
            # Redirect to the new document
            url = self.request.resource_url(new_document)
            return HTTPFound(location=url)
    
        @view_config(renderer='templates/document.jinja2',
                     context=Document)
        def document(self):
            page_title = 'Quick Tutorial: Document'
            return dict(page_title=page_title)
    
  7. Make our resources persistent in zodb/tutorial/resources.py:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    from persistent import Persistent
    from persistent.mapping import PersistentMapping
    import transaction
    
    
    class Folder(PersistentMapping):
        def __init__(self, title):
            PersistentMapping.__init__(self)
            self.title = title
    
    
    class Root(Folder):
        __name__ = None
        __parent__ = None
    
    
    class Document(Persistent):
        def __init__(self, title):
            Persistent.__init__(self)
            self.title = title
    
    
    def bootstrap(zodb_root):
        if not 'tutorial' in zodb_root:
            root = Root('My Site')
            zodb_root['tutorial'] = root
            transaction.commit()
        return zodb_root['tutorial']
    
  8. No changes to any templates!

  9. Run your Pyramid application with:

    $ $VENV/bin/pserve development.ini --reload
    
  10. Open http://localhost:6543/ in your browser.

Analysis

We install pyramid_zodbconn to handle database connections to ZODB. This pulls the ZODB3 package as well.

To enable pyramid_zodbconn:

  • We activate the package configuration using pyramid.includes.
  • We define a zodbconn.uri setting with the path to the Data.fs file.

In the root factory, instead of using our old root object, we now get a connection to the ZODB and create the object using that.

Our resources need a couple of small changes. Folders now inherit from persistent.PersistentMapping and document from persistent.Persistent. Note that Folder now needs to call super() on the __init__ method, or the mapping will not initialize properly.

On the bootstrap, note the use of transaction.commit() to commit the change. This is because on first startup, we want a root resource in place before continuing.

ZODB has many modes of deployment. For example, ZEO is a pure-Python object storage service across multiple processes and hosts. RelStorage lets you use a RDBMS for storage/retrieval of your Python pickles.

Extra Credit

  1. Create a view that deletes a document.
  2. Remove the configuration line that includes pyramid_tm. What happens when you restart the application? Are your changes persisted across restarts?
  3. What happens if you delete the files named Data.fs*?