├── zodb.png ├── .static └── zodb.ico ├── documentation ├── guide │ ├── TODO │ ├── admin.rst │ ├── README │ ├── index.rst │ ├── links.rst │ ├── convert_zodb_guide.py │ ├── chatter.py │ ├── transactions.rst │ ├── introduction.rst │ ├── zeo.rst │ ├── gfdl.rst │ ├── modules.rst │ └── prog-zodb.rst ├── articles │ ├── images │ │ └── zeo-diagram.png │ ├── index.rst │ ├── multi-zodb-gc.rst │ ├── ZODB-overview.rst │ ├── ZODB2.rst │ └── ZODB1.rst └── tutorial.rst ├── .travis.yml ├── ISSUE_TEMPLATE.md ├── PULL_REQUEST_TEMPLATE.md ├── README.rst ├── zodb.svg ├── Makefile ├── bootstrap.py ├── conf.py └── index.rst /zodb.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zopefoundation/zodbdocs/HEAD/zodb.png -------------------------------------------------------------------------------- /.static/zodb.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zopefoundation/zodbdocs/HEAD/.static/zodb.ico -------------------------------------------------------------------------------- /documentation/guide/TODO: -------------------------------------------------------------------------------- 1 | Write section on __setstate__ 2 | Continue working on it 3 | Suppress the full GFDL text in the PDF/PS versions 4 | 5 | -------------------------------------------------------------------------------- /documentation/articles/images/zeo-diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zopefoundation/zodbdocs/HEAD/documentation/articles/images/zeo-diagram.png -------------------------------------------------------------------------------- /documentation/guide/admin.rst: -------------------------------------------------------------------------------- 1 | .. % Administration 2 | .. % Importing and exporting data 3 | .. % Disaster recovery/avoidance 4 | .. % Security 5 | 6 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | install: 3 | - python bootstrap.py 4 | - bin/buildout 5 | script: 6 | - make html 7 | notifications: 8 | email: false 9 | -------------------------------------------------------------------------------- /ISSUE_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | ## This repository is no longer active 2 | 3 | ZODB documentation has moved to the [ZODB repository](/zopefoundation/ZODB). 4 | 5 | Please report new issues [there](/zopefoundation/ZODB/issues/new). 6 | -------------------------------------------------------------------------------- /PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | ## This repository is no longer active 2 | 3 | ZODB documentation has moved to the [ZODB repository](/zopefoundation/ZODB). 4 | 5 | Please open new pull requests [there](/zopefoundation/ZODB/issues/compare). 6 | -------------------------------------------------------------------------------- /documentation/guide/README: -------------------------------------------------------------------------------- 1 | This directory contains Andrew Kuchling's programmer's guide to ZODB 2 | and ZEO. The tex source was not being updated in the ZODB docs directory 3 | 4 | It was originally taken from Andrew's zodb.sf.net project on 5 | SourceForge. Because the original version is no longer updated, this 6 | version [in the zodb docs dir] is best viewed as an independent fork now. 7 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | .. caution:: 2 | 3 | This repository has been archived. If you want to work on it please open a ticket in https://github.com/zopefoundation/meta/issues requesting its unarchival. 4 | 5 | This package is deprecated 6 | ========================== 7 | 8 | ZODB documentation is moving to the ZODB repository. 9 | 10 | When the new documentation reaches parity with the old (ancient) 11 | developers's guide, then zodb.org will be redirectied to the new location. 12 | -------------------------------------------------------------------------------- /documentation/guide/index.rst: -------------------------------------------------------------------------------- 1 | ====================== 2 | ZODB programming guide 3 | ====================== 4 | 5 | 6 | This guide is based heavily on the work of A. M. Kuchling who wrote the 7 | original guide back in 2002 and which was published under the GNU Free 8 | Documentation License, Version 1.1. See the appendix entitled "GNU Free 9 | Documentation License" for more information. 10 | 11 | Contents 12 | -------- 13 | 14 | .. toctree:: 15 | :maxdepth: 2 16 | 17 | introduction.rst 18 | prog-zodb.rst 19 | zeo.rst 20 | transactions.rst 21 | modules.rst 22 | links.rst 23 | gfdl.rst 24 | -------------------------------------------------------------------------------- /documentation/guide/links.rst: -------------------------------------------------------------------------------- 1 | .. % links.tex 2 | .. % Collection of relevant links 3 | 4 | 5 | Resources 6 | ========= 7 | 8 | Introduction to the Zope Object Database, by Jim Fulton: --- Goes into much 9 | greater detail, explaining advanced uses of the ZODB and how it's actually 10 | implemented. A definitive reference, and highly recommended. --- 11 | ``_ 12 | 13 | Persistent Programing with ZODB, by Jeremy Hylton and Barry Warsaw: --- Slides 14 | for a tutorial presented at the 10th Python conference. Covers much of the same 15 | ground as this guide, with more details in some areas and less in others. --- 16 | ``_ 17 | 18 | -------------------------------------------------------------------------------- /documentation/guide/convert_zodb_guide.py: -------------------------------------------------------------------------------- 1 | # Use the python docs converter to convert to rst 2 | # Requires http://svn.python.org/projects/doctools/converter 3 | 4 | from converter import restwriter, convert_file 5 | 6 | import sys 7 | import os 8 | 9 | 10 | if __name__ == '__main__': 11 | try: 12 | rootdir = sys.argv[1] 13 | destdir = os.path.abspath(sys.argv[2]) 14 | except IndexError: 15 | print "usage: convert.py docrootdir destdir" 16 | sys.exit() 17 | 18 | os.chdir(rootdir) 19 | 20 | class IncludeRewrite: 21 | def get(self, a, b=None): 22 | if os.path.exists(a + '.tex'): 23 | return a + '.rst' 24 | print "UNKNOWN FILE %s" % a 25 | return a 26 | restwriter.includes_mapping = IncludeRewrite() 27 | 28 | for infile in os.listdir('.'): 29 | if infile.endswith('.tex'): 30 | convert_file(infile, os.path.join(destdir, infile[:-3]+'rst')) 31 | -------------------------------------------------------------------------------- /documentation/articles/index.rst: -------------------------------------------------------------------------------- 1 | ZODB articles 2 | ============= 3 | 4 | Contents 5 | -------- 6 | 7 | .. toctree:: 8 | :maxdepth: 2 9 | 10 | ZODB-overview.rst 11 | ZODB1.rst 12 | ZODB2.rst 13 | multi-zodb-gc.rst 14 | 15 | 16 | Other ZODB Resources 17 | -------------------- 18 | 19 | - `ZODB/ZEO Programming Guide <../zodbguide/index.html>`_ 20 | 21 | - Jim Fulton's `Introduction to the Zope Object Database `_ 22 | 23 | - IBM developerWorks `Example-driven ZODB 24 | `_ 25 | 26 | - Martin Faassen's `A misconception about the ZODB 27 | `_ 28 | 29 | - `How To Love ZODB and Forget RDBMS 30 | `_ 31 | 32 | - `ZODB wiki `_ and `Documentation 33 | page `_ and `Historical pages 34 | `_ 35 | 36 | - `Very old ZODB wiki `_ 37 | 38 | - `ZODB-dev `_ mailing list 39 | and `archive `_ 40 | 41 | - `Administering Zope 2 - ZODB 42 | `_ 43 | 44 | - `Add-on software for ZODB `_ 45 | -------------------------------------------------------------------------------- /zodb.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | ZODB_logo 5 | Created with Sketch. 6 | 7 | 8 | 12 | 13 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = bin/sphinx-build 7 | PAPER = 8 | 9 | # Internal variables. 10 | PAPEROPT_a4 = -D latex_paper_size=a4 11 | PAPEROPT_letter = -D latex_paper_size=letter 12 | ALLSPHINXOPTS = -d build/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 13 | 14 | .PHONY: help clean html web pickle htmlhelp latex changes linkcheck 15 | 16 | help: 17 | @echo "Please use \`make ' where is one of" 18 | @echo " html to make standalone HTML files" 19 | @echo " pickle to make pickle files" 20 | @echo " json to make JSON files" 21 | @echo " htmlhelp to make HTML files and a HTML help project" 22 | @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" 23 | @echo " changes to make an overview over all changed/added/deprecated items" 24 | @echo " linkcheck to check all external links for integrity" 25 | 26 | clean: 27 | -rm -rf build/* 28 | 29 | html: 30 | mkdir -p build/html build/doctrees 31 | $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) build/html 32 | @echo 33 | @echo "Build finished. The HTML pages are in build/html." 34 | 35 | pickle: 36 | mkdir -p build/pickle build/doctrees 37 | $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) build/pickle 38 | @echo 39 | @echo "Build finished; now you can process the pickle files." 40 | 41 | web: pickle 42 | 43 | json: 44 | mkdir -p build/json build/doctrees 45 | $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) build/json 46 | @echo 47 | @echo "Build finished; now you can process the JSON files." 48 | 49 | htmlhelp: 50 | mkdir -p build/htmlhelp build/doctrees 51 | $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) build/htmlhelp 52 | @echo 53 | @echo "Build finished; now you can run HTML Help Workshop with the" \ 54 | ".hhp project file in build/htmlhelp." 55 | 56 | latex: 57 | mkdir -p build/latex build/doctrees 58 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) build/latex 59 | @echo 60 | @echo "Build finished; the LaTeX files are in build/latex." 61 | @echo "Run \`make all-pdf' or \`make all-ps' in that directory to" \ 62 | "run these through (pdf)latex." 63 | 64 | changes: 65 | mkdir -p build/changes build/doctrees 66 | $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) build/changes 67 | @echo 68 | @echo "The overview file is in build/changes." 69 | 70 | linkcheck: 71 | mkdir -p build/linkcheck build/doctrees 72 | $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) build/linkcheck 73 | @echo 74 | @echo "Link check complete; look for any errors in the above output " \ 75 | "or in build/linkcheck/output.txt." 76 | -------------------------------------------------------------------------------- /bootstrap.py: -------------------------------------------------------------------------------- 1 | ############################################################################## 2 | # 3 | # Copyright (c) 2006 Zope Corporation and Contributors. 4 | # All Rights Reserved. 5 | # 6 | # This software is subject to the provisions of the Zope Public License, 7 | # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. 8 | # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED 9 | # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 10 | # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS 11 | # FOR A PARTICULAR PURPOSE. 12 | # 13 | ############################################################################## 14 | """Bootstrap a buildout-based project 15 | 16 | Simply run this script in a directory containing a buildout.cfg. 17 | The script accepts buildout command-line options, so you can 18 | use the -c option to specify an alternate configuration file. 19 | 20 | $Id: bootstrap.py 90478 2008-08-27 22:44:46Z georgyberdyshev $ 21 | """ 22 | 23 | import os, shutil, sys, tempfile, urllib2 24 | 25 | tmpeggs = tempfile.mkdtemp() 26 | 27 | is_jython = sys.platform.startswith('java') 28 | 29 | try: 30 | import pkg_resources 31 | except ImportError: 32 | ez = {} 33 | exec urllib2.urlopen('http://peak.telecommunity.com/dist/ez_setup.py' 34 | ).read() in ez 35 | ez['use_setuptools'](to_dir=tmpeggs, download_delay=0) 36 | 37 | import pkg_resources 38 | 39 | if sys.platform == 'win32': 40 | def quote(c): 41 | if ' ' in c: 42 | return '"%s"' % c # work around spawn lamosity on windows 43 | else: 44 | return c 45 | else: 46 | def quote (c): 47 | return c 48 | 49 | cmd = 'from setuptools.command.easy_install import main; main()' 50 | ws = pkg_resources.working_set 51 | 52 | if is_jython: 53 | import subprocess 54 | 55 | assert subprocess.Popen([sys.executable] + ['-c', quote(cmd), '-mqNxd', 56 | quote(tmpeggs), 'zc.buildout'], 57 | env=dict(os.environ, 58 | PYTHONPATH= 59 | ws.find(pkg_resources.Requirement.parse('setuptools')).location 60 | ), 61 | ).wait() == 0 62 | 63 | else: 64 | assert os.spawnle( 65 | os.P_WAIT, sys.executable, quote (sys.executable), 66 | '-c', quote (cmd), '-mqNxd', quote (tmpeggs), 'zc.buildout', 67 | dict(os.environ, 68 | PYTHONPATH= 69 | ws.find(pkg_resources.Requirement.parse('setuptools')).location 70 | ), 71 | ) == 0 72 | 73 | ws.add_entry(tmpeggs) 74 | ws.require('zc.buildout') 75 | import zc.buildout.buildout 76 | zc.buildout.buildout.main(sys.argv[1:] + ['bootstrap']) 77 | shutil.rmtree(tmpeggs) 78 | -------------------------------------------------------------------------------- /documentation/guide/chatter.py: -------------------------------------------------------------------------------- 1 | 2 | import sys, time, os, random 3 | 4 | import transaction 5 | from persistent import Persistent 6 | 7 | from ZEO import ClientStorage 8 | import ZODB 9 | from ZODB.POSException import ConflictError 10 | from BTrees import OOBTree 11 | 12 | class ChatSession(Persistent): 13 | 14 | """Class for a chat session. 15 | Messages are stored in a B-tree, indexed by the time the message 16 | was created. (Eventually we'd want to throw messages out, 17 | 18 | add_message(message) -- add a message to the channel 19 | new_messages() -- return new messages since the last call to 20 | this method 21 | 22 | 23 | """ 24 | 25 | def __init__(self, name): 26 | """Initialize new chat session. 27 | name -- the channel's name 28 | """ 29 | 30 | self.name = name 31 | 32 | # Internal attribute: _messages holds all the chat messages. 33 | self._messages = OOBTree.OOBTree() 34 | 35 | 36 | def new_messages(self): 37 | "Return new messages." 38 | 39 | # self._v_last_time is the time of the most recent message 40 | # returned to the user of this class. 41 | if not hasattr(self, '_v_last_time'): 42 | self._v_last_time = 0 43 | 44 | new = [] 45 | T = self._v_last_time 46 | 47 | for T2, message in self._messages.items(): 48 | if T2 > T: 49 | new.append( message ) 50 | self._v_last_time = T2 51 | 52 | return new 53 | 54 | def add_message(self, message): 55 | """Add a message to the channel. 56 | message -- text of the message to be added 57 | """ 58 | 59 | while 1: 60 | try: 61 | now = time.time() 62 | self._messages[ now ] = message 63 | transaction.commit() 64 | except ConflictError: 65 | # Conflict occurred; this process should abort, 66 | # wait for a little bit, then try again. 67 | transaction.abort() 68 | time.sleep(.2) 69 | else: 70 | # No ConflictError exception raised, so break 71 | # out of the enclosing while loop. 72 | break 73 | # end while 74 | 75 | def get_chat_session(conn, channelname): 76 | """Return the chat session for a given channel, creating the session 77 | if required.""" 78 | 79 | # We'll keep a B-tree of sessions, mapping channel names to 80 | # session objects. The B-tree is stored at the ZODB's root under 81 | # the key 'chat_sessions'. 82 | root = conn.root() 83 | if not root.has_key('chat_sessions'): 84 | print 'Creating chat_sessions B-tree' 85 | root['chat_sessions'] = OOBTree.OOBTree() 86 | transaction.commit() 87 | 88 | sessions = root['chat_sessions'] 89 | 90 | # Get a session object corresponding to the channel name, creating 91 | # it if necessary. 92 | if not sessions.has_key( channelname ): 93 | print 'Creating new session:', channelname 94 | sessions[ channelname ] = ChatSession(channelname) 95 | transaction.commit() 96 | 97 | session = sessions[ channelname ] 98 | return session 99 | 100 | 101 | if __name__ == '__main__': 102 | if len(sys.argv) != 2: 103 | print 'Usage: %s ' % sys.argv[0] 104 | sys.exit(0) 105 | 106 | storage = ClientStorage.ClientStorage( ('localhost', 9672) ) 107 | db = ZODB.DB( storage ) 108 | conn = db.open() 109 | 110 | s = session = get_chat_session(conn, sys.argv[1]) 111 | 112 | messages = ['Hi.', 'Hello', 'Me too', "I'M 3L33T!!!!"] 113 | 114 | while 1: 115 | # Send a random message 116 | msg = random.choice(messages) 117 | session.add_message( '%s: pid %i' % (msg,os.getpid() )) 118 | 119 | # Display new messages 120 | for msg in session.new_messages(): 121 | print msg 122 | 123 | # Wait for a few seconds 124 | pause = random.randint( 1, 4 ) 125 | time.sleep( pause ) 126 | -------------------------------------------------------------------------------- /conf.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # ZODB documentation and articles documentation build configuration file, created by 4 | # sphinx-quickstart on Sat Feb 21 09:17:33 2009. 5 | # 6 | # This file is execfile()d with the current directory set to its containing dir. 7 | # 8 | # The contents of this file are pickled, so don't put values in the namespace 9 | # that aren't pickleable (module imports are okay, they're removed automatically). 10 | # 11 | # Note that not all possible configuration values are present in this 12 | # autogenerated file. 13 | # 14 | # All configuration values have a default; values that are commented out 15 | # serve to show the default. 16 | 17 | # If your extensions are in another directory, add it here. If the directory 18 | # is relative to the documentation root, use os.path.abspath to make it 19 | # absolute, like shown here. 20 | #sys.path.append(os.path.abspath('.')) 21 | 22 | # General configuration 23 | # --------------------- 24 | 25 | # Add any Sphinx extension module names here, as strings. They can be extensions 26 | # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. 27 | extensions = [] 28 | 29 | # Add any paths that contain templates here, relative to this directory. 30 | templates_path = ['.templates'] 31 | 32 | # The suffix of source filenames. 33 | source_suffix = '.rst' 34 | 35 | # The encoding of source files. 36 | #source_encoding = 'utf-8' 37 | 38 | # The master toctree document. 39 | master_doc = 'index' 40 | 41 | # General information about the project. 42 | project = u'ZODB' 43 | copyright = u'2009-2011, Zope Foundation' 44 | 45 | # The version info for the project you're documenting, acts as replacement for 46 | # |version| and |release|, also used in various other places throughout the 47 | # built documents. 48 | # 49 | # The short X.Y version. 50 | version = '3.10' 51 | # The full version, including alpha/beta/rc tags. 52 | release = '3.10.3' 53 | 54 | # The language for content autogenerated by Sphinx. Refer to documentation 55 | # for a list of supported languages. 56 | #language = None 57 | 58 | # There are two options for replacing |today|: either, you set today to some 59 | # non-false value, then it is used: 60 | #today = '' 61 | # Else, today_fmt is used as the format for a strftime call. 62 | #today_fmt = '%B %d, %Y' 63 | 64 | # List of documents that shouldn't be included in the build. 65 | #unused_docs = [] 66 | 67 | # List of directories, relative to source directory, that shouldn't be searched 68 | # for source files. 69 | exclude_trees = ['build'] 70 | 71 | # The reST default role (used for this markup: `text`) to use for all documents. 72 | #default_role = None 73 | 74 | # If true, '()' will be appended to :func: etc. cross-reference text. 75 | #add_function_parentheses = True 76 | 77 | # If true, the current module name will be prepended to all description 78 | # unit titles (such as .. function::). 79 | #add_module_names = True 80 | 81 | # If true, sectionauthor and moduleauthor directives will be shown in the 82 | # output. They are ignored by default. 83 | #show_authors = False 84 | 85 | # The name of the Pygments (syntax highlighting) style to use. 86 | pygments_style = 'sphinx' 87 | 88 | 89 | # Options for HTML output 90 | # ----------------------- 91 | 92 | # The style sheet to use for HTML and HTML Help pages. A file of that name 93 | # must exist either in Sphinx' static/ path, or in one of the custom paths 94 | # given in html_static_path. 95 | #html_style = 'default.css' 96 | 97 | # The name for this set of Sphinx documents. If None, it defaults to 98 | # " v documentation". 99 | #html_title = None 100 | 101 | # A shorter title for the navigation bar. Default is the same as html_title. 102 | #html_short_title = None 103 | 104 | # The name of an image file (relative to this directory) to place at the top 105 | # of the sidebar. 106 | html_logo = 'zodb.png' 107 | 108 | # The name of an image file (within the static path) to use as favicon of the 109 | # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 110 | # pixels large. 111 | html_favicon = 'zodb.ico' 112 | 113 | # Add any paths that contain custom static files (such as style sheets) here, 114 | # relative to this directory. They are copied after the builtin static files, 115 | # so a file named "default.css" will overwrite the builtin "default.css". 116 | html_static_path = ['.static'] 117 | 118 | # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, 119 | # using the given strftime format. 120 | #html_last_updated_fmt = '%b %d, %Y' 121 | 122 | # If true, SmartyPants will be used to convert quotes and dashes to 123 | # typographically correct entities. 124 | #html_use_smartypants = True 125 | 126 | # Custom sidebar templates, maps document names to template names. 127 | #html_sidebars = {} 128 | 129 | # Additional templates that should be rendered to pages, maps page names to 130 | # template names. 131 | #html_additional_pages = {} 132 | 133 | # If false, no module index is generated. 134 | #html_use_modindex = True 135 | 136 | # If false, no index is generated. 137 | #html_use_index = True 138 | 139 | # If true, the index is split into individual pages for each letter. 140 | #html_split_index = False 141 | 142 | # If true, the reST sources are included in the HTML build as _sources/. 143 | #html_copy_source = True 144 | 145 | # If true, an OpenSearch description file will be output, and all pages will 146 | # contain a tag referring to it. The value of this option must be the 147 | # base URL from which the finished HTML is served. 148 | #html_use_opensearch = '' 149 | 150 | # If nonempty, this is the file name suffix for HTML files (e.g. ".xhtml"). 151 | #html_file_suffix = '' 152 | 153 | # Output file base name for HTML help builder. 154 | htmlhelp_basename = 'ZODBdocumentationandarticlesdoc' 155 | 156 | 157 | # Options for LaTeX output 158 | # ------------------------ 159 | 160 | # The paper size ('letter' or 'a4'). 161 | #latex_paper_size = 'letter' 162 | 163 | # The font size ('10pt', '11pt' or '12pt'). 164 | #latex_font_size = '10pt' 165 | 166 | # Grouping the document tree into LaTeX files. List of tuples 167 | # (source start file, target name, title, author, document class [howto/manual]). 168 | latex_documents = [ 169 | ('index', 'ZODBdocumentationandarticles.tex', ur'ZODB documentation and articles', 170 | ur'Zope Developer Community', 'manual'), 171 | ] 172 | 173 | # The name of an image file (relative to this directory) to place at the top of 174 | # the title page. 175 | #latex_logo = None 176 | 177 | # For "manual" documents, if this is true, then toplevel headings are parts, 178 | # not chapters. 179 | #latex_use_parts = False 180 | 181 | # Additional stuff for the LaTeX preamble. 182 | #latex_preamble = '' 183 | 184 | # Documents to append as an appendix to all manuals. 185 | #latex_appendices = [] 186 | 187 | # If false, no module index is generated. 188 | #latex_use_modindex = True 189 | -------------------------------------------------------------------------------- /documentation/articles/multi-zodb-gc.rst: -------------------------------------------------------------------------------- 1 | Using zc.zodbdgc (fix PosKeyError's) 2 | ==================================== 3 | 4 | *This article was written by Hanno Schlichting* 5 | 6 | The `zc.zodbdgc `_ library contains two 7 | useful features. On the one hand it supports advanced ZODB packing and garbage 8 | collection approaches and on the other hand it includes the ability to create a 9 | database of all persistent references. 10 | 11 | The second feature allows us to debug and repair PosKeyErrors by finding the 12 | persistent object(s) that point to the lost object. 13 | 14 | Note: This documentation applies to ZODB 3.9 and later. Earlier versions of the 15 | ZODB are not supported, as they lack the fast storage iteration API's required 16 | by `zc.zodbdgc`. 17 | 18 | This documentation does not apply to 19 | `RelStorage `_ which has the same 20 | features built-in, but accessible in different ways. Look at the options for 21 | the `zodbpack` script. The `--prepack` option creates a table containing the 22 | same information as we are creating in the reference database. 23 | 24 | Setup 25 | ----- 26 | 27 | We'll assume you are familiar with a buildout setup. A typical config might 28 | look like this:: 29 | 30 | [buildout] 31 | parts = 32 | zeo 33 | zeopy 34 | zeo-conf 35 | zodbdgc 36 | refdb-conf 37 | 38 | [zeo] 39 | recipe = plone.recipe.zeoserver 40 | zeo-address = 127.0.0.1:8100 41 | blob-storage = ${buildout:directory}/var/blobstorage 42 | pack-gc = false 43 | pack-keep-old = false 44 | 45 | [zeopy] 46 | recipe = zc.recipe.egg 47 | eggs = 48 | ZODB3 49 | zc.zodbdgc 50 | interpreter = zeopy 51 | scripts = zeopy 52 | 53 | [zeo-conf] 54 | recipe = collective.recipe.template 55 | input = inline: 56 | 57 | 58 | blob-dir ${buildout:directory}/var/blobstorage 59 | shared-blob-dir yes 60 | server ${zeo:zeo-address} 61 | storage 1 62 | name zeostorage 63 | var ${buildout:directory}/var 64 | 65 | 66 | output = ${buildout:directory}/etc/zeo.conf 67 | 68 | [zodbdgc] 69 | recipe = zc.recipe.egg 70 | eggs = zc.zodbdgc 71 | 72 | [refdb-conf] 73 | recipe = collective.recipe.template 74 | input = inline: 75 | 76 | 77 | path ${buildout:directory}/var/refdb.fs 78 | 79 | 80 | output = ${buildout:directory}/etc/refdb.conf 81 | 82 | 83 | Garbage collection 84 | ------------------ 85 | 86 | We configured the ZEO server to skip garbage collection as part of the normal 87 | pack in the above config (`pack-gc = false`). Instead we use explicit garbage 88 | collection via a different job:: 89 | 90 | bin/multi-zodb-gc etc/zeo.conf 91 | 92 | On larger databases garbage collection can take a couple hours. We can run this 93 | only once a week or even less frequent. All explicitly deleted objects will 94 | still be packed away by the normal pack, so the database doesn't grow 95 | out-of-bound. We can also run the analysis against a database copy, taking away 96 | load from the live database and only write the resulting deletions to the 97 | production database. 98 | 99 | 100 | Packing 101 | ------- 102 | 103 | We can do regular packing every day while the ZEO server is running, via:: 104 | 105 | bin/zeopack 106 | 107 | Packing without garbage collection is much faster. 108 | 109 | 110 | Reference analysis and POSKeyErrors 111 | ----------------------------------- 112 | 113 | If our database has any POSKeyErrors, we can find and repair those. 114 | 115 | Either we already have the oids of lost objects, or we can check the entire 116 | database for any errors. To check everything we run the following command:: 117 | 118 | $ bin/multi-zodb-check-refs etc/zeo.conf 119 | 120 | This can take about 15 to 30 minutes on moderately sized databases of up to 121 | 10gb, dependent on disk speed. We'll write down the reported errors, as we'll 122 | need them later on to analyze them. 123 | 124 | If there are any lost objects, we can create a reference database to make it 125 | easier to debug and find those lost objects:: 126 | 127 | $ bin/multi-zodb-check-refs -r var/refdb.fs etc/zeo.conf 128 | 129 | This is significantly slower and can take several hours to complete. Once this 130 | is complete we can open the generated database via our interpreter:: 131 | 132 | $ bin/zeopy 133 | 134 | >>> import ZODB.config 135 | >>> db = ZODB.config.databaseFromFile(open('./etc/refdb.conf')) 136 | >>> conn = db.open() 137 | >>> refs = conn.root()['references'] 138 | 139 | If we've gotten this error report:: 140 | 141 | !!! main 13184375 ? 142 | POSKeyError: 0xc92d77 143 | 144 | We can look up the persistent oid it was referenced from via:: 145 | 146 | >>> parent = list(refs['main'][13184375]) 147 | >>> parent 148 | [13178389] 149 | 150 | We can also get the hex representation:: 151 | 152 | >>> from ZODB.utils import p64 153 | >>> p64(parent[0]) 154 | '\x00\x00\x00\x00\x00\xc9\x16\x15' 155 | 156 | With this information, we should get back to our actual database and look 157 | up this object. We'll leave the ref db open, as we might need to recursively 158 | look up some more objects, until we get one we can identify and work on. 159 | 160 | We could load the parent. In a debug prompt we could do something like:: 161 | 162 | >>> app._p_jar.get('\x00\x00\x00\x00\x00\xc9\x16\x15') 163 | 2010-04-28 14:28:28 ERROR ZODB.Connection Couldn't load state for 0xc91615 164 | Traceback (most recent call last): 165 | ... 166 | ZODB.POSException.POSKeyError: 0xc92d77 167 | 168 | Gah, this gives us the POSKeyError of course. But we can load the actual data 169 | of the parent, to get an idea of what this is:: 170 | 171 | >>> app._p_jar.db()._storage.load('\x00\x00\x00\x00\x00\xc9\x16\x15', '') 172 | ('cBTrees.IOBTree 173 | IOBucket 174 | q\x01.((J$KT\x02ccopy_reg 175 | _reconstructor 176 | q\x02(cfive.intid.keyreference 177 | KeyReferenceToPersistent 178 | ... 179 | 180 | Now we can be real evil and create a new fake object in place of the missing 181 | one:: 182 | 183 | >>> import transaction 184 | >>> transaction.begin() 185 | 186 | The persistent oid that was reported missing was ``13184375``:: 187 | 188 | >>> from ZODB.utils import p64 189 | >>> p64(13184375) 190 | '\x00\x00\x00\x00\x00\xc9-w' 191 | 192 | >>> from persistent import Persistent 193 | >>> a = Persistent() 194 | >>> a._p_oid = '\x00\x00\x00\x00\x00\xc9-w' 195 | 196 | We cannot use the ``add`` method of the connection, as this would assign the 197 | object a new persistent oid. So we replicate its internals here:: 198 | 199 | >>> a._p_jar = app._p_jar 200 | >>> app._p_jar._register(a) 201 | >>> app._p_jar._added[a._p_oid] = a 202 | 203 | >>> transaction.commit() 204 | 205 | Both getting the object as well as its parent will work now:: 206 | 207 | >>> app._p_jar.get('\x00\x00\x00\x00\x00\xc9-w') 208 | 209 | 210 | >>> app._p_jar.get('\x00\x00\x00\x00\x00\xc9\x16\x15') 211 | BTrees.IOBTree.IOBucket([(39078692, >> conn.close() 216 | >>> db.close() 217 | 218 | Depending on the class of object that went missing, we might need to use a 219 | different persistent class, like a persistent mapping or a BTree bucket. 220 | 221 | In general it's best to remove the parent object and thus our fake object from 222 | the database and rebuild the data structure again via the proper application 223 | level API's. 224 | -------------------------------------------------------------------------------- /documentation/articles/ZODB-overview.rst: -------------------------------------------------------------------------------- 1 | An overview of the ZODB (by Laurence Rowe) 2 | ========================================== 3 | 4 | ZODB in comparison to relational databases, transactions, scalability and best 5 | practice. Originally delivered to the Plone Conference 2007, Naples. 6 | 7 | Comparison to other database types 8 | ---------------------------------- 9 | 10 | **Relational Databases** are great at handling large quantities of homogenous 11 | data. If you’re building a ledger system a Relational Database is a great fit. 12 | But Relational Databases only support hierarchical data structures to a 13 | limited degree. Using foreign-key relationships must refer to a single table, 14 | so only a single type can be contained. 15 | 16 | **Hierarchical databases** (such as LDAP or a filesystem) are much more 17 | suitable for modelling the flexible containment hierarchies required for 18 | content management applications. But most of these systems do not support 19 | transactional semantics. ORMs such as `SQLAlchemy 20 | `_. make working with Relational Databases in an 21 | object orientated manner much more pleasant. But they don’t overcome the 22 | restrictions inherent in a relational model. 23 | 24 | The **ZODB** is an (almost) transparent python object persistence system, 25 | heavily influenced by Smalltalk. As an Object-Orientated Database it gives you 26 | the flexibility to build a data model fit your application. For the most part 27 | you don’t have to worry about persistency - you only work with python objects 28 | and it just happens in the background. 29 | 30 | Of course this power comes at a price. While changing the methods your classes 31 | provide is not a problem, changing attributes can necessitate writing a 32 | migration script, as you would with a relational schema change. With ZODB 33 | obejcts though explicit schema migrations are not enforced, which can bite you 34 | later. 35 | 36 | Transactions 37 | ------------ 38 | 39 | The ZODB has a transactional support at its core. Transactions provide 40 | concurrency control and atomicity. Transactions are executed as if they have 41 | exclusive access to the data, so as an application developer you don’t have to 42 | worry about threading. Of course there is nothing to prevent two simultaneous 43 | conflicting requests, So checks are made at transaction commit time to ensure 44 | consistency. 45 | 46 | Since Zope 2.8 ZODB has implemented **Multi Version Concurrency Control**. 47 | This means no more ReadConflictErrors, each transaction is guaranteed to be 48 | able to load any object as it was when the transaction begun. 49 | 50 | You may still see (Write) **ConflictErrors**. These can be minimised using 51 | data structures that support conflict resolution, primarily B-Trees in the 52 | BTrees library. These scalable data structures are used in Large Plone Folders 53 | and many parts of Zope. One downside is that they don’t support user definable 54 | ordering. 55 | 56 | The hot points for ConflictErrors are the catalogue indexes. Some of the 57 | indexes do not support conflict resolution and you will see ConflictErrors 58 | under write-intensive loads. On solution is to defer catalogue updates using 59 | `QueueCatalog `_ 60 | (`PloneQueueCatalog 61 | `_), which allows 62 | indexing operations to be serialized using a seperate ZEO client. This can 63 | bring big performance benefits as request retries are reduced, but the 64 | downside is that index updates are no longer reflected immediately in the 65 | application. Another alternative is to offload text indexing to a dedicated 66 | search engine using `collective.solr 67 | `_. 68 | 69 | This brings us to **Atomicity**, the other key feature of ZODB transactions. A 70 | transaction will either succeed or fail, your data is never left in an 71 | inconsistent state if an error occurs. This makes Zope a forgiving system to 72 | work with. 73 | 74 | You must though be careful with interactions with external systems. If a 75 | ConflictError occurs Zope will attempt to replay a transaction up to three 76 | times. Interactions with an external system should be made through a Data 77 | Manager that participates in the transaction. If you’re talking to a database 78 | use a Zope DA or a SQLAlchemy wrapper like `zope.sqlalchemy 79 | `_. 80 | 81 | Unfortunately the default MailHost implementation used by Plone is not 82 | transaction aware. With it you can see duplicate emails sent. If this is a 83 | problem use TransactionalMailHost. 84 | 85 | Scalability Python is limited to a single CPU by the Global Interpreter Lock, 86 | but that’s ok, ZEO lets us run multiple Zope Application servers sharing a 87 | single database. You should run one Zope client for each processor on your 88 | server. ZEO also lets you connect a debug session to your database at the same 89 | time as your Zope web server, invaluable for debugging. 90 | 91 | ZEO tends to be IO bound, so the GIL is not an issue. 92 | 93 | ZODB also supports **partitioning**, allowing you to spread data over multiple 94 | storages. However you should be careful about cross database references 95 | (especially when copying and pasting between two databases) as they can be 96 | problematic. 97 | 98 | Another common reason to use partitioning is because the ZODB in memory cache 99 | settings are made per database. Separating the catalogue into another storage 100 | lets you set a higher target cache size for catalogue objects than for your 101 | content objects. As much of the Plone interface is catalogue driven this can 102 | have a significant performance benefit, especially on a large site. 103 | 104 | .. image:: images/zeo-diagram.png 105 | 106 | Storage Options 107 | --------------- 108 | 109 | **FileStorage** is the default. Everything in one big Data.fs file, which is 110 | essentially a transaction log. Use this unless you have a very good reason not 111 | to. 112 | 113 | **DirectoryStorage** (`site `_) stores one 114 | file per object revision. Does not require the Data.fs.index to be rebuilt on 115 | an unclean shutdown (which can take a significant time for a large database). 116 | Small number of users. 117 | 118 | **RelStorage** (`pypi `_) stores 119 | pickles in a relational database. PostgreSQL, MySQL and Oracle are supported 120 | and no ZEO server is required. You benefit from the faster network layers of 121 | these database adapters. However, conflict resolution is moved to the 122 | application server, which can be bad for worst case performance when you have 123 | high network latency. 124 | 125 | BDBStorage, OracleStorage, PGStorage and APE have now fallen by the wayside. 126 | 127 | Other features 128 | -------------- 129 | 130 | **Savepoints** (previously sub-transactions) allow fine grained error control 131 | and objects to be garbage collected during a transaction, saving memory. 132 | 133 | Versions are deprecated (and will be removed in ZODB 3.9). The application 134 | layer is responsible for versioning, e.g. CMFEditions / ZopeVersionControl. 135 | 136 | **Undo**, don’t rely on it! If your object is indexed it may prove impossible 137 | to undo the transaction (independently) if a later transaction has changed the 138 | same index. Undo is only performed on a single database, so if you have 139 | separated out your catalogue it will get out of sync. Fine for undoing in 140 | portal_skins/custom though. 141 | 142 | **BLOBs** are new in ZODB 3.8 / Zope 2.11, bringing efficient large file 143 | support. Great for document management applications. 144 | 145 | **Packing** removes old revisions of objects. Similar to `Routine Vacuuming 146 | `_ in 147 | PostgreSQL. 148 | 149 | Some best practice 150 | ------------------ 151 | 152 | **Don’t write on read**. Your Data.fs should not grow on a read. Beware of 153 | setDefault and avoid inplace migration. 154 | 155 | **Keep your code on the filesystem**. Too much stuff in the custom folder will 156 | just lead to pain further down the track. Though this can be very convenient 157 | for getting things done when they are needed yesterday... 158 | 159 | **Use scalable data structures** such as BTrees. Keep your content objects 160 | simple, add functionality with adapters and views. 161 | -------------------------------------------------------------------------------- /documentation/guide/transactions.rst: -------------------------------------------------------------------------------- 1 | .. % Transactions and Versioning 2 | .. % Committing and Aborting 3 | .. % Subtransactions 4 | .. % Undoing 5 | .. % Versions 6 | .. % Multithreaded ZODB Programs 7 | 8 | 9 | Transactions and Versioning 10 | =========================== 11 | 12 | 13 | Committing and Aborting 14 | ----------------------- 15 | 16 | Changes made during a transaction don't appear in the database until the 17 | transaction commits. This is done by calling the :meth:`commit` method of the 18 | current :class:`Transaction` object, where the latter is obtained from the 19 | :meth:`get` method of the current transaction manager. If the default thread 20 | transaction manager is being used, then ``transaction.commit()`` suffices. 21 | 22 | Similarly, a transaction can be explicitly aborted (all changes within the 23 | transaction thrown away) by invoking the :meth:`abort` method of the current 24 | :class:`Transaction` object, or simply ``transaction.abort()`` if using the 25 | default thread transaction manager. 26 | 27 | Prior to ZODB 3.3, if a commit failed (meaning the ``commit()`` call raised an 28 | exception), the transaction was implicitly aborted and a new transaction was 29 | implicitly started. This could be very surprising if the exception was 30 | suppressed, and especially if the failing commit was one in a sequence of 31 | subtransaction commits. 32 | 33 | So, starting with ZODB 3.3, if a commit fails, all further attempts to commit, 34 | join, or register with the transaction raise 35 | :exc:`ZODB.POSException.TransactionFailedError`. You must explicitly start a 36 | new transaction then, either by calling the :meth:`abort` method of the current 37 | transaction, or by calling the :meth:`begin` method of the current transaction's 38 | transaction manager. 39 | 40 | 41 | Subtransactions 42 | --------------- 43 | 44 | Subtransactions can be created within a transaction. Each subtransaction can be 45 | individually committed and aborted, but the changes within a subtransaction are 46 | not truly committed until the containing transaction is committed. 47 | 48 | The primary purpose of subtransactions is to decrease the memory usage of 49 | transactions that touch a very large number of objects. Consider a transaction 50 | during which 200,000 objects are modified. All the objects that are modified in 51 | a single transaction have to remain in memory until the transaction is 52 | committed, because the ZODB can't discard them from the object cache. This can 53 | potentially make the memory usage quite large. With subtransactions, a commit 54 | can be be performed at intervals, say, every 10,000 objects. Those 10,000 55 | objects are then written to permanent storage and can be purged from the cache 56 | to free more space. 57 | 58 | To commit a subtransaction instead of a full transaction, pass a true value to 59 | the :meth:`commit` or :meth:`abort` method of the :class:`Transaction` object. 60 | :: 61 | 62 | # Commit a subtransaction 63 | transaction.commit(True) 64 | 65 | # Abort a subtransaction 66 | transaction.abort(True) 67 | 68 | A new subtransaction is automatically started upon successful committing or 69 | aborting the previous subtransaction. 70 | 71 | 72 | Undoing Changes 73 | --------------- 74 | 75 | Some types of :class:`Storage` support undoing a transaction even after it's 76 | been committed. You can tell if this is the case by calling the 77 | :meth:`supportsUndo` method of the :class:`DB` instance, which returns true if 78 | the underlying storage supports undo. Alternatively you can call the 79 | :meth:`supportsUndo` method on the underlying storage instance. 80 | 81 | If a database supports undo, then the :meth:`undoLog(start, end[, func])` method 82 | on the :class:`DB` instance returns the log of past transactions, returning 83 | transactions between the times *start* and *end*, measured in seconds from the 84 | epoch. If present, *func* is a function that acts as a filter on the 85 | transactions to be returned; it's passed a dictionary representing each 86 | transaction, and only transactions for which *func* returns true will be 87 | included in the list of transactions returned to the caller of :meth:`undoLog`. 88 | The dictionary contains keys for various properties of the transaction. The 89 | most important keys are ``id``, for the transaction ID, and ``time``, for the 90 | time at which the transaction was committed. :: 91 | 92 | >>> print storage.undoLog(0, sys.maxint) 93 | [{'description': '', 94 | 'id': 'AzpGEGqU/0QAAAAAAAAGMA', 95 | 'time': 981126744.98, 96 | 'user_name': ''}, 97 | {'description': '', 98 | 'id': 'AzpGC/hUOKoAAAAAAAAFDQ', 99 | 'time': 981126478.202, 100 | 'user_name': ''} 101 | ... 102 | 103 | To store a description and a user name on a commit, get the current transaction 104 | and call the :meth:`note(text)` method to store a description, and the 105 | :meth:`setUser(user_name)` method to store the user name. While :meth:`setUser` 106 | overwrites the current user name and replaces it with the new value, the 107 | :meth:`note` method always adds the text to the transaction's description, so it 108 | can be called several times to log several different changes made in the course 109 | of a single transaction. :: 110 | 111 | transaction.get().setUser('amk') 112 | transaction.get().note('Change ownership') 113 | 114 | To undo a transaction, call the :meth:`DB.undo(id)` method, passing it the ID of 115 | the transaction to undo. If the transaction can't be undone, a 116 | :exc:`ZODB.POSException.UndoError` exception will be raised, with the message 117 | "non-undoable transaction". Usually this will happen because later transactions 118 | modified the objects affected by the transaction you're trying to undo. 119 | 120 | After you call :meth:`undo` you must commit the transaction for the undo to 121 | actually be applied. [#]_ There is one glitch in the undo process. The thread 122 | that calls undo may not see the changes to the object until it calls 123 | :meth:`Connection.sync` or commits another transaction. 124 | 125 | 126 | Versions 127 | -------- 128 | 129 | .. warning:: 130 | 131 | Versions should be avoided. They're going to be deprecated, replaced by better 132 | approaches to long-running transactions. 133 | 134 | While many subtransactions can be contained within a single regular transaction, 135 | it's also possible to contain many regular transactions within a long-running 136 | transaction, called a version in ZODB terminology. Inside a version, any number 137 | of transactions can be created and committed or rolled back, but the changes 138 | within a version are not made visible to other connections to the same ZODB. 139 | 140 | Not all storages support versions, but you can test for versioning ability by 141 | calling :meth:`supportsVersions` method of the :class:`DB` instance, which 142 | returns true if the underlying storage supports versioning. 143 | 144 | A version can be selected when creating the :class:`Connection` instance using 145 | the :meth:`DB.open([*version*])` method. The *version* argument must be a string 146 | that will be used as the name of the version. :: 147 | 148 | vers_conn = db.open(version='Working version') 149 | 150 | Transactions can then be committed and aborted using this versioned connection. 151 | Other connections that don't specify a version, or provide a different version 152 | name, will not see changes committed within the version named ``Working 153 | version``. To commit or abort a version, which will either make the changes 154 | visible to all clients or roll them back, call the :meth:`DB.commitVersion` or 155 | :meth:`DB.abortVersion` methods. XXX what are the source and dest arguments for? 156 | 157 | The ZODB makes no attempt to reconcile changes between different versions. 158 | Instead, the first version which modifies an object will gain a lock on that 159 | object. Attempting to modify the object from a different version or from an 160 | unversioned connection will cause a :exc:`ZODB.POSException.VersionLockError` to 161 | be raised:: 162 | 163 | from ZODB.POSException import VersionLockError 164 | 165 | try: 166 | transaction.commit() 167 | except VersionLockError, (obj_id, version): 168 | print ('Cannot commit; object %s ' 169 | 'locked by version %s' % (obj_id, version)) 170 | 171 | The exception provides the ID of the locked object, and the name of the version 172 | having a lock on it. 173 | 174 | 175 | Multithreaded ZODB Programs 176 | --------------------------- 177 | 178 | ZODB databases can be accessed from multithreaded Python programs. The 179 | :class:`Storage` and :class:`DB` instances can be shared among several threads, 180 | as long as individual :class:`Connection` instances are created for each thread. 181 | 182 | .. rubric:: Footnotes 183 | 184 | .. [#] There are actually two different ways a storage can implement the undo feature. 185 | Most of the storages that ship with ZODB use the transactional form of undo 186 | described in the main text. Some storages may use a non-transactional undo 187 | makes changes visible immediately. 188 | 189 | -------------------------------------------------------------------------------- /documentation/guide/introduction.rst: -------------------------------------------------------------------------------- 1 | .. % Introduction 2 | .. % What is ZODB? 3 | .. % What is ZEO? 4 | .. % OODBs vs. Relational DBs 5 | .. % Other OODBs 6 | 7 | 8 | Introduction 9 | ============ 10 | 11 | This guide explains how to write Python programs that use the Z Object Database 12 | (ZODB) and Zope Enterprise Objects (ZEO). The latest version of the guide is 13 | always available at ``_. 14 | 15 | 16 | What is the ZODB? 17 | ----------------- 18 | 19 | The ZODB is a persistence system for Python objects. Persistent programming 20 | languages provide facilities that automatically write objects to disk and read 21 | them in again when they're required by a running program. By installing the 22 | ZODB, you add such facilities to Python. 23 | 24 | It's certainly possible to build your own system for making Python objects 25 | persistent. The usual starting points are the :mod:`pickle` module, for 26 | converting objects into a string representation, and various database modules, 27 | such as the :mod:`gdbm` or :mod:`bsddb` modules, that provide ways to write 28 | strings to disk and read them back. It's straightforward to combine the 29 | :mod:`pickle` module and a database module to store and retrieve objects, and in 30 | fact the :mod:`shelve` module, included in Python's standard library, does this. 31 | 32 | The downside is that the programmer has to explicitly manage objects, reading an 33 | object when it's needed and writing it out to disk when the object is no longer 34 | required. The ZODB manages objects for you, keeping them in a cache, writing 35 | them out to disk when they are modified, and dropping them from the cache if 36 | they haven't been used in a while. 37 | 38 | 39 | OODBs vs. Relational DBs 40 | ------------------------ 41 | 42 | Another way to look at it is that the ZODB is a Python-specific object-oriented 43 | database (OODB). Commercial object databases for C++ or Java often require that 44 | you jump through some hoops, such as using a special preprocessor or avoiding 45 | certain data types. As we'll see, the ZODB has some hoops of its own to jump 46 | through, but in comparison the naturalness of the ZODB is astonishing. 47 | 48 | Relational databases (RDBs) are far more common than OODBs. Relational databases 49 | store information in tables; a table consists of any number of rows, each row 50 | containing several columns of information. (Rows are more formally called 51 | relations, which is where the term "relational database" originates.) 52 | 53 | Let's look at a concrete example. The example comes from my day job working for 54 | the MEMS Exchange, in a greatly simplified version. The job is to track process 55 | runs, which are lists of manufacturing steps to be performed in a semiconductor 56 | fab. A run is owned by a particular user, and has a name and assigned ID 57 | number. Runs consist of a number of operations; an operation is a single step 58 | to be performed, such as depositing something on a wafer or etching something 59 | off it. 60 | 61 | Operations may have parameters, which are additional information required to 62 | perform an operation. For example, if you're depositing something on a wafer, 63 | you need to know two things: 1) what you're depositing, and 2) how much should 64 | be deposited. You might deposit 100 microns of silicon oxide, or 1 micron of 65 | copper. 66 | 67 | Mapping these structures to a relational database is straightforward:: 68 | 69 | CREATE TABLE runs ( 70 | int run_id, 71 | varchar owner, 72 | varchar title, 73 | int acct_num, 74 | primary key(run_id) 75 | ); 76 | 77 | CREATE TABLE operations ( 78 | int run_id, 79 | int step_num, 80 | varchar process_id, 81 | PRIMARY KEY(run_id, step_num), 82 | FOREIGN KEY(run_id) REFERENCES runs(run_id), 83 | ); 84 | 85 | CREATE TABLE parameters ( 86 | int run_id, 87 | int step_num, 88 | varchar param_name, 89 | varchar param_value, 90 | PRIMARY KEY(run_id, step_num, param_name) 91 | FOREIGN KEY(run_id, step_num) 92 | REFERENCES operations(run_id, step_num), 93 | ); 94 | 95 | In Python, you would write three classes named :class:`Run`, :class:`Operation`, 96 | and :class:`Parameter`. I won't present code for defining these classes, since 97 | that code is uninteresting at this point. Each class would contain a single 98 | method to begin with, an :meth:`__init__` method that assigns default values, 99 | such as 0 or ``None``, to each attribute of the class. 100 | 101 | It's not difficult to write Python code that will create a :class:`Run` instance 102 | and populate it with the data from the relational tables; with a little more 103 | effort, you can build a straightforward tool, usually called an object- 104 | relational mapper, to do this automatically. (See 105 | ``_ for a quick hack at a 106 | Python object-relational mapper, and 107 | ``_ for Joel 108 | Shprentz's more successful implementation of the same idea; Unlike mine, 109 | Shprentz's system has been used for actual work.) 110 | 111 | However, it is difficult to make an object-relational mapper reasonably quick; a 112 | simple-minded implementation like mine is quite slow because it has to do 113 | several queries to access all of an object's data. Higher performance object- 114 | relational mappers cache objects to improve performance, only performing SQL 115 | queries when they actually need to. 116 | 117 | That helps if you want to access run number 123 all of a sudden. But what if 118 | you want to find all runs where a step has a parameter named 'thickness' with a 119 | value of 2.0? In the relational version, you have two unappealing choices: 120 | 121 | #. Write a specialized SQL query for this case: ``SELECT run_id FROM operations 122 | WHERE param_name = 'thickness' AND param_value = 2.0`` 123 | 124 | If such queries are common, you can end up with lots of specialized queries. 125 | When the database tables get rearranged, all these queries will need to be 126 | modified. 127 | 128 | #. An object-relational mapper doesn't help much. Scanning through the runs 129 | means that the the mapper will perform the required SQL queries to read run #1, 130 | and then a simple Python loop can check whether any of its steps have the 131 | parameter you're looking for. Repeat for run #2, 3, and so forth. This does a 132 | vast number of SQL queries, and therefore is incredibly slow. 133 | 134 | An object database such as ZODB simply stores internal pointers from object to 135 | object, so reading in a single object is much faster than doing a bunch of SQL 136 | queries and assembling the results. Scanning all runs, therefore, is still 137 | inefficient, but not grossly inefficient. 138 | 139 | 140 | What is ZEO? 141 | ------------ 142 | 143 | The ZODB comes with a few different classes that implement the :class:`Storage` 144 | interface. Such classes handle the job of writing out Python objects to a 145 | physical storage medium, which can be a disk file (the :class:`FileStorage` 146 | class), a BerkeleyDB file (:class:`BDBFullStorage`), a relational database 147 | (:class:`DCOracleStorage`), or some other medium. ZEO adds 148 | :class:`ClientStorage`, a new :class:`Storage` that doesn't write to physical 149 | media but just forwards all requests across a network to a server. The server, 150 | which is running an instance of the :class:`StorageServer` class, simply acts as 151 | a front-end for some physical :class:`Storage` class. It's a fairly simple 152 | idea, but as we'll see later on in this document, it opens up many 153 | possibilities. 154 | 155 | 156 | About this guide 157 | ---------------- 158 | 159 | The primary author of this guide works on a project which uses the ZODB and ZEO 160 | as its primary storage technology. We use the ZODB to store process runs and 161 | operations, a catalog of available processes, user information, accounting 162 | information, and other data. Part of the goal of writing this document is to 163 | make our experience more widely available. A few times we've spent hours or 164 | even days trying to figure out a problem, and this guide is an attempt to gather 165 | up the knowledge we've gained so that others don't have to make the same 166 | mistakes we did while learning. 167 | 168 | The author's ZODB project is described in a paper available here, 169 | ``_ 170 | 171 | This document will always be a work in progress. If you wish to suggest 172 | clarifications or additional topics, please send your comments to the 173 | `ZODB-dev mailing list `_. 174 | 175 | 176 | Acknowledgements 177 | ---------------- 178 | 179 | Andrew Kuchling wrote the original version of this guide, which provided some of 180 | the first ZODB documentation for Python programmers. His initial version has 181 | been updated over time by Jeremy Hylton and Tim Peters. 182 | 183 | I'd like to thank the people who've pointed out inaccuracies and bugs, offered 184 | suggestions on the text, or proposed new topics that should be covered: Jeff 185 | Bauer, Willem Broekema, Thomas Guettler, Chris McDonough, George Runyan. 186 | 187 | -------------------------------------------------------------------------------- /documentation/tutorial.rst: -------------------------------------------------------------------------------- 1 | ======== 2 | Tutorial 3 | ======== 4 | 5 | This tutorial is intended to guide developers with a step-by-step introduction 6 | of how to develop an application which stores its data in the ZODB. 7 | 8 | Introduction 9 | ============ 10 | 11 | To save application data in ZODB, you'll generally define classes that 12 | subclass ``persistent.Persistent``:: 13 | 14 | # account.py 15 | 16 | import persistent 17 | 18 | class Account(persistent.Persistent): 19 | 20 | def __init__(self): 21 | self.balance = 0.0 22 | 23 | def deposit(self, amount): 24 | self.balance += amount 25 | 26 | def cash(self, amount): 27 | assert amount < self.balance 28 | self.balance -= amount 29 | 30 | This code defines a simple class that holds the balance of a bank 31 | account and provides two methods to manipulate the balance: deposit 32 | and cash. 33 | 34 | Subclassing ``Persistent`` provides a number of features: 35 | 36 | - The database will automatically track object changes made by setting 37 | attributes [#changed]_. 38 | 39 | - Data will be saved in its own database record. 40 | 41 | You can save data that doesn't subclass ``Persistent``, but it will be 42 | stored in the database record of whatever persistent object 43 | references it. 44 | 45 | - Objects will have unique persistent identity. 46 | 47 | Multiple objects can refer to the same persistent object and they'll 48 | continue to refer to the same object even after being saved 49 | and loaded from the database. 50 | 51 | Non-persistent objects are essentially owned by their containing 52 | persistent object and if multiple persistent objects refer to the 53 | same non-persistent subobject, they'll (eventually) get their own 54 | copies. 55 | 56 | Note that we put the class in a named module. Classes aren't stored 57 | in the ZODB [#persistentclasses]_. They exist on the file system and 58 | their names, consisting of their class and module names, are stored in 59 | the database. It's sometimes tempting to create persistent classes in 60 | scripts or in interactive sessions, but if you do, then their module 61 | name will be ``'__main__'`` and you'll always have to define them that 62 | way. 63 | 64 | Installation 65 | ============ 66 | 67 | Before being able to use ZODB we have to install it. A common way to 68 | do this is with pip:: 69 | 70 | $ pip install ZODB 71 | 72 | Creating Databases 73 | ================== 74 | 75 | When a program wants to use the ZODB it has to establish a connection, 76 | like any other database. For the ZODB we need 3 different parts: a 77 | storage, a database and finally a connection:: 78 | 79 | import ZODB, ZODB.FileStorage 80 | 81 | storage = ZODB.FileStorage.FileStorage('mydata.fs') 82 | db = ZODB.DB(storage) 83 | connection = db.open() 84 | root = connection.root 85 | 86 | ZODB has a pluggable storage framework. This means there are a 87 | variety of storage implementations to meet different needs, from 88 | in-memory databases, to databases stored in local files, to databases 89 | on remote database servers, and specialized databases for compression, 90 | encryption, and so on. In the example above, we created a database 91 | that stores its data in a local file, using the ``FileStorage`` 92 | class. 93 | 94 | Having a storage, we then use it to instantiate a database, which we 95 | then connect to by calling ``open()``. A process with multiple 96 | threads will often have multiple connections to the same database, 97 | with different threads having different connections. 98 | 99 | There are a number of convenient shortcuts you can use for some of the 100 | commonly used storages: 101 | 102 | - You can pass a file name to the ``DB`` constructor to have it construct 103 | a FileStorage for you:: 104 | 105 | db = ZODB.DB('mydata.fs') 106 | 107 | You can pass None to create an in-memory database:: 108 | 109 | memory_db = ZODB.DB(None) 110 | 111 | - If you're only going to use one connection, you can call the 112 | ``connection`` function:: 113 | 114 | connection = ZODB.connection('mydata.fs') 115 | memory_connection = ZODB.connection(None) 116 | 117 | Storing objects 118 | =============== 119 | 120 | To store an object in the ZODB we simply attach it to any other object 121 | that already lives in the database. Hence, the root object functions 122 | as a boot-strapping point. The root object is meant to serve as a 123 | namespace for top-level objects in your database. We could store 124 | account objects directly on the root object:: 125 | 126 | import account 127 | 128 | # Probably a bad idea: 129 | root.account1 = account.Account() 130 | 131 | But if you're going to store many objects, you'll want to use a 132 | collection object [#root]_:: 133 | 134 | import account, BTrees.OOBTree 135 | 136 | root.accounts = BTrees.OOBTree.BTree() 137 | root.accounts['account-1'] = Account() 138 | 139 | Another common practice is to store a persistent object in the root of 140 | the database that provides an application-specific root:: 141 | 142 | root.accounts = AccountManagementApplication() 143 | 144 | That can facilitate encapsulation of an application that shares a 145 | database with other applications. This is a little bit like using 146 | modules to avoid namespace colisions in Python programs. 147 | 148 | Containers and search 149 | ===================== 150 | 151 | BTrees provide the core scalable containers and indexing facility for 152 | ZODB. There are different families of BTrees. The most general are 153 | OOBTrees, which have object keys and values. There are specialized 154 | BTrees that support integer keys and values. Integers can be stored 155 | more efficiently, and compared more quickly than objects and they're 156 | often used as application-level object identifiers. It's critical, 157 | when using BTrees, to make sure that its keys have a stable ordering. 158 | 159 | ZODB doesn't provide a query engine. The primary way to access 160 | objects in ZODB is by traversing (accessing attributes or items, or 161 | calling methods) other objects. Object traversal is typically much 162 | faster than search. 163 | 164 | You can use BTrees to build indexes for efficient search, when 165 | necessary. If your application is search centric, or if you prefer to 166 | approach data access that way, then ZODB might not be the best 167 | technology for you. 168 | 169 | Transactions 170 | ============ 171 | 172 | You now have objects in your root object and in your database. 173 | However, they are not permanently stored yet. The ZODB uses 174 | transactions and to make your changes permanent, you have to commit 175 | the transaction:: 176 | 177 | import transaction 178 | 179 | transaction.commit() 180 | 181 | Now you can stop and start your application and look at the root object again, 182 | and you will find the data you saved. 183 | 184 | If your application makes changes during a transaction and finds that it does 185 | not want to commit those changes, then you can abort the transaction and have 186 | the changes rolled back [#rollback]_ for you:: 187 | 188 | transaction.abort() 189 | 190 | Transactions are a very powerful way to protect the integrity of a 191 | database. Transactions have the property that all of the changes made 192 | in a transaction are saved, or none of them are. If in the midst of a 193 | program, there's an error after making changes, you can simply abort 194 | the transaction (or not commit it) and all of the intermediate changes 195 | you make are automatically discarded. 196 | 197 | Memory Management 198 | ================= 199 | 200 | ZODB manages moving objects in and out of memory for you. The unit of 201 | storage is the persistent object. When you access attributes of a 202 | persistent objects, it's loaded from the database automatically, if 203 | necessary. If too many objects are in memory, then objects used least 204 | recently are evicted [#eviction]_. The maximum number of objects or 205 | bytes in memory is configurable. 206 | 207 | Summary 208 | ======= 209 | 210 | You have seen how to install ZODB and how to open a database in your 211 | application and to start storing objects in it. We also touched the 212 | two simple transaction commands: ``commit`` and ``abort``. The 213 | reference documentation contains sections with more information on the 214 | individual topics. 215 | 216 | .. [#changed] 217 | You can manually mark an object as changed by setting its 218 | ``_p_changed__`` attribute to ``True``. You might do this if you 219 | update a subobject, such as a standard Python ``list`` or ``set``, 220 | that doesn't subclass ``Persistent``. 221 | 222 | .. [#persistentclasses] 223 | Actually, there is semi-experimental support for storing classes in 224 | the database, but applications rarely do this. 225 | 226 | .. [#root] 227 | The root object is a fairy simple persistent object that's stored 228 | in a single database record. If you stored many objects in it, 229 | its database record would become very large, causing updates to be 230 | inefficient and causing memory to be used ineffeciently. 231 | 232 | Another reason not to store items directly in the root object is 233 | that doing so would make adding a second collection of objects 234 | later awkward. 235 | 236 | .. [#rollback] 237 | A caveat is that ZODB can only roll back changes to objects that 238 | have been stored and committed to the database. Objects not 239 | previously committed can't be rolled back because there's no 240 | previous state to roll back to. 241 | 242 | .. [#eviction] 243 | Objects aren't actually evicted, but their state is released, so 244 | they take up much less memory and any objects they referenced can 245 | be removed from memory. 246 | -------------------------------------------------------------------------------- /index.rst: -------------------------------------------------------------------------------- 1 | ========================================== 2 | ZODB - a native object database for Python 3 | ========================================== 4 | 5 | Because ZODB is an object database: 6 | 7 | - no separate language for database operations 8 | 9 | - very little impact on your code to make objects persistent 10 | 11 | - no database mapper that partially hides the database. 12 | 13 | Using an object-relational mapping **is not** like using an object database. 14 | 15 | - almost no seam between code and database. 16 | 17 | Check out the :doc:`documentation/tutorial`! 18 | 19 | Transactions 20 | ============ 21 | 22 | Make programs easier to reason about. 23 | 24 | Transactions are atomic 25 | Changes made in a transaction are either saved in their entirety or 26 | not at all. 27 | 28 | This makes error handling a lot easier. If you have an error, you 29 | just abort the current transaction. You don't have to worry about 30 | undoing previous database changes. 31 | 32 | Transactions provide isolation 33 | Transactions allow multiple logical threads (threads or processes) 34 | to access databases and the database prevents the threads from 35 | making conflicting changes. 36 | 37 | This allows you to scale your application across multiple threads, 38 | processes or machines without having to use low-level locking 39 | primitives. 40 | 41 | You still have to deal with concurrency on some level. For 42 | timestamp-based systems like ZODB, you may have to retry conflicting 43 | transactions. With locking-based systems, you have to deal with 44 | possible deadlocks. 45 | 46 | Transactions affect multiple objects 47 | Most NoSQL databases don't have transactions. Their notions of 48 | consistency are much weaker, typically applying to single documents. 49 | There can be good reasons to use NoSQL databases for their extreme 50 | scalability, but otherwise, think hard about giving up the benefits 51 | of transactions. 52 | 53 | ZODB transaction support: 54 | 55 | - `ACID `_ transactions with 56 | `snapshot isolation 57 | `_ 58 | 59 | - Distributed transaction support using two-phase commit 60 | 61 | This allows transactions to span multiple ZODB databases and to span 62 | ZODB and non-ZODB databases. 63 | 64 | Other notable ZODB features 65 | =========================== 66 | 67 | Pluggable layered storage 68 | ZODB has a pluggable storage architecture. This allows a variety of 69 | storage schemes including memory-based, file-based and distributed 70 | (client-server) storage. Through storage layering, storage 71 | components provide compression, encryption, replication and more. 72 | 73 | Database caching with invalidation 74 | Every database connection has a cache that is a consistent partial database 75 | replica. When accessing database objects, data already in the cache 76 | is accessed without any database interactions. When data are 77 | modified, invalidations are sent to clients causing cached objects 78 | to be invalidated. The next time invalidated objects are accessed 79 | they'll be loaded from the database. 80 | 81 | This makes caching extremely efficient, but provides some limit to 82 | the number of clients. The server has to send an invalidation 83 | message to each client for each write. 84 | 85 | Easy testing 86 | ZODB provides in-memory storage implementations as well as 87 | copy-on-write layered "demo storage" implementations that make testing 88 | database-related code very easy. 89 | 90 | Time travel 91 | ZODB storages typically add new records on write and remove old 92 | records on "pack" operations. This allows limited time travel, back 93 | to the last pack time. This can be very useful for forensic 94 | analysis. 95 | 96 | Binary large objects, Blobs 97 | Many databases have these, but so does ZODB. 98 | 99 | In applications, Blobs are files, so they can be treated as files in 100 | many ways. This can be especially useful when serving media. If you 101 | use AWS, there's a Blob implementation that stores blobs in S3 and 102 | caches them on disk. 103 | 104 | When should you use ZODB? 105 | ========================= 106 | 107 | You want to focus on your application without writing a lot of database code. 108 | Even if find you need to incorporate or switch to another database 109 | later, you can use ZODB in the early part of your project to make 110 | initial discovery and learning much quicker. 111 | 112 | Your application has complex relationships and data structures. 113 | In relational databases you have to join tables to model complex 114 | data structures and these joins can be tedious and expensive. You 115 | can mitigate this to some extent in databases like Postgres by using 116 | more powerful data types like arrays and JSON columns, but when 117 | relationships extend across rows, you still have to do joins. 118 | 119 | In NoSQL databases, you can model complex data structures with 120 | documents, but if you have relationships across documents, then you 121 | have to do joins and join capabilities in NoSQL databases are 122 | typically far less powerful and transactional semantics typically don't 123 | cross documents, if they exist at all. 124 | 125 | In ZODB, you can make objects as complex as you want and cross 126 | object relationships are handled with Python object references. 127 | 128 | You access data through object attributes and methods. 129 | If your primary object access is search, then other database 130 | technologies might be a better fit. 131 | 132 | ZODB has no query language other than Python. It's primary support 133 | for search is through mapping objects called BTrees. People have 134 | build higher-level search APIs on top of ZODB. These work well 135 | enough to support some search. 136 | 137 | You read data a lot more than you write it. 138 | ZODB caches aggressively, and if you're working set fits (or mostly 139 | fits) in memory, performance is very good because it rarely has to 140 | touch the database server. 141 | 142 | If your application is very write heavy (e.g. logging), then you're 143 | better off using something else. Sometimes, you can use a database 144 | suitable for heavy writes in combination with ZODB. 145 | 146 | Need to test logic that uses your database. 147 | ZODB has a number of storage implementations, including layered 148 | in-memory implementations that make testing very easy. 149 | 150 | A database without an in-memory storage option can make testing very 151 | complicated. 152 | 153 | When should you *not* use ZODB? 154 | =============================== 155 | 156 | - Search is a dominant data access path 157 | 158 | - You have high write volume 159 | 160 | - Caching is unlikely to benefit you 161 | 162 | This can be the case when write volume is high, or when you tend to 163 | access small amounts of data from a working set way too large to fit in 164 | memory and when there's no good mechanism for dividing the working 165 | set across application servers. 166 | 167 | - You need to use non-Python tools to access your database. 168 | 169 | especially tools designed to work with relational databases 170 | 171 | How does ZODB scale? 172 | ==================== 173 | 174 | Not as well as many technologies, but some fairly large applications 175 | have been built on ZODB. 176 | 177 | At Zope Corporation, several hundred newspaper content-management 178 | systems and web sites were hosted using a multi-database configuration 179 | with most data in a main database and a catalog database. The 180 | databases had several hundred gigabytes of ordinary database records 181 | plus multiple terabytes of blob data. 182 | 183 | ZODB is mature 184 | ============== 185 | 186 | ZODB is very mature. Development started in 1996 and it has been used 187 | in production in thousands of applications for many years. 188 | 189 | ZODB is in heavy use in the `Pyramid `_ 190 | and `Plone `_ communities and in many other 191 | applications. 192 | 193 | Learning more 194 | ============= 195 | 196 | .. toctree:: 197 | :maxdepth: 1 198 | 199 | documentation/tutorial 200 | documentation/guide/index 201 | documentation/articles/index 202 | 203 | * `The ZODB Book (in progress) `_ 204 | 205 | Downloads 206 | ========= 207 | 208 | ZODB is distributed through the `Python Package Index 209 | `_. 210 | 211 | You can install the ZODB using pip command:: 212 | 213 | $ pip install ZODB 214 | 215 | Community and contributing 216 | ========================== 217 | 218 | Discussion occurs on the `ZODB mailing list 219 | `_. (And for the 220 | transaction system on the `transaction list 221 | `_) 222 | 223 | Bug reporting and feature requests are submitted through github issue 224 | trackers for various ZODB components: 225 | 226 | - `ZODB `_ 227 | 228 | - `persistent `_ 229 | 230 | - `transaction `_ 231 | 232 | - `BTrees `_ 233 | 234 | - `ZEO (client-server framework) `_ 235 | 236 | If you'd like to contribute then we'll gladly accept work on documentation, 237 | helping out other developers and users at the mailing list, submitting bugs, 238 | creating proposals and writing code. 239 | 240 | ZODB is a project managed by the Zope Foundation so you can get write access 241 | for contributing directly - check out the foundation's `Zope Developer Information `_. 242 | -------------------------------------------------------------------------------- /documentation/guide/zeo.rst: -------------------------------------------------------------------------------- 1 | .. % ZEO 2 | .. % Installing ZEO 3 | .. % How ZEO works (ClientStorage) 4 | .. % Configuring ZEO 5 | 6 | 7 | .. _zeo: 8 | 9 | ZEO 10 | === 11 | 12 | 13 | How ZEO Works 14 | ------------- 15 | 16 | The ZODB, as I've described it so far, can only be used within a single Python 17 | process (though perhaps with multiple threads). ZEO, Zope Enterprise Objects, 18 | extends the ZODB machinery to provide access to objects over a network. The 19 | name "Zope Enterprise Objects" is a bit misleading; ZEO can be used to store 20 | Python objects and access them in a distributed fashion without Zope ever 21 | entering the picture. The combination of ZEO and ZODB is essentially a Python- 22 | specific object database. 23 | 24 | ZEO consists of about 12,000 lines of Python code, excluding tests. The code is 25 | relatively small because it contains only code for a TCP/IP server, and for a 26 | new type of Storage, :class:`ClientStorage`. :class:`ClientStorage` simply makes 27 | remote procedure calls to the server, which then passes them on a regular 28 | :class:`Storage` class such as :class:`FileStorage`. The following diagram lays 29 | out the system: 30 | 31 | XXX insert diagram here later 32 | 33 | Any number of processes can create a :class:`ClientStorage` instance, and any 34 | number of threads in each process can be using that instance. 35 | :class:`ClientStorage` aggressively caches objects locally, so in order to avoid 36 | using stale data the ZEO server sends an invalidation message to all the 37 | connected :class:`ClientStorage` instances on every write operation. The 38 | invalidation message contains the object ID for each object that's been 39 | modified, letting the :class:`ClientStorage` instances delete the old data for 40 | the given object from their caches. 41 | 42 | This design decision has some consequences you should be aware of. First, while 43 | ZEO isn't tied to Zope, it was first written for use with Zope, which stores 44 | HTML, images, and program code in the database. As a result, reads from the 45 | database are *far* more frequent than writes, and ZEO is therefore better suited 46 | for read-intensive applications. If every :class:`ClientStorage` is writing to 47 | the database all the time, this will result in a storm of invalidate messages 48 | being sent, and this might take up more processing time than the actual database 49 | operations themselves. These messages are small and sent in batches, so there 50 | would need to be a lot of writes before it became a problem. 51 | 52 | On the other hand, for applications that have few writes in comparison to the 53 | number of read accesses, this aggressive caching can be a major win. Consider a 54 | Slashdot-like discussion forum that divides the load among several Web servers. 55 | If news items and postings are represented by objects and accessed through ZEO, 56 | then the most heavily accessed objects -- the most recent or most popular 57 | postings -- will very quickly wind up in the caches of the 58 | :class:`ClientStorage` instances on the front-end servers. The back-end ZEO 59 | server will do relatively little work, only being called upon to return the 60 | occasional older posting that's requested, and to send the occasional invalidate 61 | message when a new posting is added. The ZEO server isn't going to be contacted 62 | for every single request, so its workload will remain manageable. 63 | 64 | 65 | Installing ZEO 66 | -------------- 67 | 68 | This section covers how to install the ZEO package, and how to configure and 69 | run a ZEO Storage Server on a machine. 70 | 71 | 72 | Requirements 73 | ^^^^^^^^^^^^ 74 | 75 | The ZEO server software is included in ZODB3. As with the rest of ZODB3, you'll 76 | need Python 2.3 or higher. 77 | 78 | 79 | Running a server 80 | ^^^^^^^^^^^^^^^^ 81 | 82 | The runzeo.py script in the ZEO directory can be used to start a server. Run it 83 | with the -h option to see the various values. If you're just experimenting, a 84 | good choise is to use ``python ZEO/runzeo.py -a /tmp/zeosocket -f 85 | /tmp/test.fs`` to run ZEO with a Unix domain socket and a :class:`FileStorage`. 86 | 87 | 88 | Testing the ZEO Installation 89 | ---------------------------- 90 | 91 | Once a ZEO server is up and running, using it is just like using ZODB with a 92 | more conventional disk-based storage; no new programming details are introduced 93 | by using a remote server. The only difference is that programs must create a 94 | :class:`ClientStorage` instance instead of a :class:`FileStorage` instance. 95 | From that point onward, ZODB-based code is happily unaware that objects are 96 | being retrieved from a ZEO server, and not from the local disk. 97 | 98 | As an example, and to test whether ZEO is working correctly, try running the 99 | following lines of code, which will connect to the server, add some bits of data 100 | to the root of the ZODB, and commits the transaction:: 101 | 102 | from ZEO import ClientStorage 103 | from ZODB import DB 104 | import transaction 105 | 106 | # Change next line to connect to your ZEO server 107 | addr = 'kronos.example.com', 1975 108 | storage = ClientStorage.ClientStorage(addr) 109 | db = DB(storage) 110 | conn = db.open() 111 | root = conn.root() 112 | 113 | # Store some things in the root 114 | root['list'] = ['a', 'b', 1.0, 3] 115 | root['dict'] = {'a':1, 'b':4} 116 | 117 | # Commit the transaction 118 | transaction.commit() 119 | 120 | If this code runs properly, then your ZEO server is working correctly. 121 | 122 | You can also use a configuration file. :: 123 | 124 | 125 | 126 | server localhost:9100 127 | 128 | 129 | 130 | One nice feature of the configuration file is that you don't need to specify 131 | imports for a specific storage. That makes the code a little shorter and allows 132 | you to change storages without changing the code. :: 133 | 134 | import ZODB.config 135 | 136 | db = ZODB.config.databaseFromURL('/tmp/zeo.conf') 137 | 138 | 139 | ZEO Programming Notes 140 | --------------------- 141 | 142 | ZEO is written using :mod:`asyncore`, from the Python standard library. It 143 | assumes that some part of the user application is running an :mod:`asyncore` 144 | mainloop. For example, Zope run the loop in a separate thread and ZEO uses 145 | that. If your application does not have a mainloop, ZEO will not process 146 | incoming invalidation messages until you make some call into ZEO. The 147 | :meth:`Connection.sync` method can be used to process pending invalidation 148 | messages. You can call it when you want to make sure the :class:`Connection` 149 | has the most recent version of every object, but you don't have any other work 150 | for ZEO to do. 151 | 152 | 153 | Sample Application: chatter.py 154 | ------------------------------ 155 | 156 | For an example application, we'll build a little chat application. What's 157 | interesting is that none of the application's code deals with network 158 | programming at all; instead, an object will hold chat messages, and be 159 | magically shared between all the clients through ZEO. I won't present the 160 | complete script here; you can download it from :download:`chatter.py 161 | `. Only the interesting portions of the code will be covered here. 162 | 163 | The basic data structure is the :class:`ChatSession` object, which provides an 164 | :meth:`add_message` method that adds a message, and a :meth:`new_messages` 165 | method that returns a list of new messages that have accumulated since the last 166 | call to :meth:`new_messages`. Internally, :class:`ChatSession` maintains a 167 | B-tree that uses the time as the key, and stores the message as the 168 | corresponding value. 169 | 170 | The constructor for :class:`ChatSession` is pretty simple; it simply creates an 171 | attribute containing a B-tree:: 172 | 173 | class ChatSession(Persistent): 174 | def __init__(self, name): 175 | self.name = name 176 | # Internal attribute: _messages holds all the chat messages. 177 | self._messages = BTrees.OOBTree.OOBTree() 178 | 179 | :meth:`add_message` has to add a message to the ``_messages`` B-tree. A 180 | complication is that it's possible that some other client is trying to add a 181 | message at the same time; when this happens, the client that commits first wins, 182 | and the second client will get a :exc:`ConflictError` exception when it tries to 183 | commit. For this application, :exc:`ConflictError` isn't serious but simply 184 | means that the operation has to be retried; other applications might treat it as 185 | a fatal error. The code uses ``try...except...else`` inside a ``while`` loop, 186 | breaking out of the loop when the commit works without raising an exception. :: 187 | 188 | def add_message(self, message): 189 | """Add a message to the channel. 190 | message -- text of the message to be added 191 | """ 192 | 193 | while 1: 194 | try: 195 | now = time.time() 196 | self._messages[now] = message 197 | get_transaction().commit() 198 | except ConflictError: 199 | # Conflict occurred; this process should abort, 200 | # wait for a little bit, then try again. 201 | transaction.abort() 202 | time.sleep(.2) 203 | else: 204 | # No ConflictError exception raised, so break 205 | # out of the enclosing while loop. 206 | break 207 | # end while 208 | 209 | :meth:`new_messages` introduces the use of *volatile* attributes. Attributes of 210 | a persistent object that begin with ``_v_`` are considered volatile and are 211 | never stored in the database. :meth:`new_messages` needs to store the last time 212 | the method was called, but if the time was stored as a regular attribute, its 213 | value would be committed to the database and shared with all the other clients. 214 | :meth:`new_messages` would then return the new messages accumulated since any 215 | other client called :meth:`new_messages`, which isn't what we want. :: 216 | 217 | def new_messages(self): 218 | "Return new messages." 219 | 220 | # self._v_last_time is the time of the most recent message 221 | # returned to the user of this class. 222 | if not hasattr(self, '_v_last_time'): 223 | self._v_last_time = 0 224 | 225 | new = [] 226 | T = self._v_last_time 227 | 228 | for T2, message in self._messages.items(): 229 | if T2 > T: 230 | new.append(message) 231 | self._v_last_time = T2 232 | 233 | return new 234 | 235 | This application is interesting because it uses ZEO to easily share a data 236 | structure; ZEO and ZODB are being used for their networking ability, not 237 | primarily for their data storage ability. I can foresee many interesting 238 | applications using ZEO in this way: 239 | 240 | * With a Tkinter front-end, and a cleverer, more scalable data structure, you 241 | could build a shared whiteboard using the same technique. 242 | 243 | * A shared chessboard object would make writing a networked chess game easy. 244 | 245 | * You could create a Python class containing a CD's title and track information. 246 | To make a CD database, a read-only ZEO server could be opened to the world, or 247 | an HTTP or XML-RPC interface could be written on top of the ZODB. 248 | 249 | * A program like Quicken could use a ZODB on the local disk to store its data. 250 | This avoids the need to write and maintain specialized I/O code that reads in 251 | your objects and writes them out; instead you can concentrate on the problem 252 | domain, writing objects that represent cheques, stock portfolios, or whatever. 253 | 254 | -------------------------------------------------------------------------------- /documentation/articles/ZODB2.rst: -------------------------------------------------------------------------------- 1 | Advanced ZODB for Python Programmers 2 | ==================================== 3 | 4 | In the first article in this series, "ZODB for Python 5 | Programmers":ZODB1 I covered some of the simpler aspects of Python 6 | object persistence. In this article, I'll go over some of the more 7 | advanced features of ZODB. 8 | 9 | In addition to simple persistence, ZODB offers some very useful 10 | extras for the advanced Python application. Specificly, we'll cover 11 | the following advanced features in this article: 12 | 13 | - Persistent-Aware Types -- ZODB comes with some special, 14 | "persistent-aware" data types for storing data in a ZODB. The 15 | most useful of these is the "BTree", which is a fast, efficient 16 | storage object for lots of data. 17 | 18 | - Voalitile Data -- Not all your data is meant to be stored in the 19 | database, ZODB let's you have volatile data on your objects that 20 | does not get saved. 21 | 22 | - Pluggable Storages -- ZODB offers you the ability to use many 23 | different storage back-ends to store your object data, including 24 | files, relational databases and a special client-server storage 25 | that stores objects on a remote server. 26 | 27 | - Conflict Resolution -- When many threads try to write to the same 28 | object at the same time, you can get conflicts. ZODB offers a 29 | conflict resolution protocol that allows you to mitigate most 30 | conflicting writes to your data. 31 | 32 | - Transactions -- When you want your changes to be "all or nothing" 33 | transactions come to the rescue. 34 | 35 | Persistent-Aware Types 36 | ---------------------- 37 | 38 | You can also get around the mutable attribute problem discussed in 39 | the first article by using special types that are "persistent 40 | aware". ZODB comes with the following persistent aware mutable 41 | object types: 42 | 43 | - PersistentList -- This type works just like a list, except that 44 | changing it does not require setting _p_changed or explicitly 45 | re-assigning the attribute. 46 | 47 | - PersistentMapping -- A persistent aware dictionary, much like 48 | PersistentList. 49 | 50 | - BTree -- A dictionary-like object that can hold large 51 | collections of objects in an ordered, fast, efficient way. 52 | 53 | BTrees offer a very powerful facility to the Python programmer: 54 | 55 | - BTrees can hold a large collection of information in an 56 | efficient way; more objects than your computer has enough 57 | memory to hold at one time. 58 | 59 | - BTrees are integrated into the persistence machinery to work 60 | effectively with ZODB's object cache. Recently, or heavily 61 | used objects are kept in a memory cache for speed. 62 | 63 | - BTrees can be searched very quickly, because they are stored 64 | in an fast, balanced tree data structure. 65 | 66 | - BTrees come in three flavors, OOBTrees, IOBTrees, OIBTrees, and 67 | IIBTrees. The last three are optimized for integer keys, values, 68 | and key-value pairs, respectively. This means that, for example, 69 | an IOBTree is meant to map an integer to an object, and is 70 | optimized for having integers keys. 71 | 72 | Using BTrees 73 | ------------ 74 | 75 | Suppose you track the movement of all your employees with 76 | heat-seeking cameras hidden in the ceiling tiles. Since your 77 | employees tend to frequently congregate against you, all of the 78 | tracking information could end up to be a lot of data, possibly 79 | thousands of coordinates per day per employee. Further, you want 80 | to key the coordinate on the time that it was taken, so that you 81 | can only look at where your employees were during certain times:: 82 | 83 | from BTrees import IOBTree 84 | from time import time 85 | 86 | class Employee(Persistent): 87 | 88 | def __init__(self): 89 | self.movements = IOBTree() 90 | 91 | def fix(self, coords): 92 | "get a fix on the employee" 93 | self.movements[int(time())] = coords 94 | 95 | def trackToday(self): 96 | "return all the movements of the 97 | employee in the last 24 hours" 98 | current_time = int(time()) 99 | return self.movements.items(current_time - 86400, 100 | current_time) 101 | 102 | 103 | In this example, the 'fix' method is called every time one of your 104 | cameras sees that employee. This information is then stored in a 105 | BTree, with the current 'time()' as the key and the 'coordinates' 106 | as the value. 107 | 108 | Because BTrees store their information is a ordered structure, 109 | they can be quickly searched for a range of key values. The 110 | 'trackToday' method uses this feature to return a sequence of 111 | coordinates from 24 hours hence to the present. 112 | 113 | This example shows how BTrees can be quickly searched for a range 114 | of values from a minimum to a maximum, and how you can use this 115 | technique to oppress your workforce. BTrees have a very rich API, 116 | including doing unions and intersections of result sets. 117 | 118 | Not All Objects are Persistent 119 | ------------------------------ 120 | 121 | You don't have to make all of your objects persistent. 122 | Non-persistent objects are often useful to represent either 123 | "canned" behavior (classes that define methods but no state), or 124 | objects that are useful only as a "cache" that can be thrown away 125 | when your persistent object is deactivated (removed from memory 126 | when not used). 127 | 128 | ZODB provides you with the ability to have *volatile* attributes. 129 | Volatile attributes are attributes of persistent objects that are 130 | never saved in the database, even if they are capable of being 131 | persistent. Volatile attributes begin with '_v_' are good for 132 | keeping cached information around for optimization. ZODB also 133 | provides you with access to special pickling hooks that allow you 134 | to set volatile information when an object is activated. 135 | 136 | Imagine you had a class that stored a complex image that you 137 | needed to calculate. This calculation is expensive. Instead of 138 | calculating the image every time you called a method, it would be 139 | better to calculate it *once* and then cache the result in a 140 | volatile attribute:: 141 | 142 | def image(self): 143 | "a large and complex image of the terrain" 144 | if hasattr(self, '_v_image'): 145 | return self._v_image 146 | image=expensive_calculation() 147 | self._v_image=image 148 | return image 149 | 150 | Here, calling 'image' the first time the object is activated will 151 | cause the method to do the expensive calculation. After the first 152 | call, the image will be cached in a volatile attribute. If the 153 | object is removed from memory, the '_v_image' attribute is not 154 | saved, so the cached image is thrown away, only to be recalculated 155 | the next time you call 'image'. 156 | 157 | ZODB and Concurrency 158 | -------------------- 159 | 160 | Different, threads, processes, and computers on a network can open 161 | connections to a single ZODB object database. Each of these 162 | different processes keeps its own copy of the objects that it uses 163 | in memory. 164 | 165 | The problem with allowing concurrent access is that conflicts can 166 | occur. If different threads try to commit changes to the same 167 | objects at the same time, one of the threads will raise a 168 | ConflictError. If you want, you can write your application to 169 | either resolve or retry conflicts a reasonable number of times. 170 | 171 | Zope will retry a conflicting ZODB operation three times. This is 172 | usually pretty reasonable behavior. Because conflicts only happen 173 | when two threads write to the same object, retrying a conflict 174 | means that one thread will win the conflict and write itself, and 175 | the other thread will retry a few seconds later. 176 | 177 | Pluggable Storages 178 | ------------------ 179 | 180 | Different processes and computers can connection to the same 181 | database using a special kind of storage called a 'ClientStorage'. 182 | A 'ClientStorage' connects to a 'StorageServer' over a network. 183 | 184 | In the very beginning, you created a connection to the database by 185 | first creating a storage. This was of the type 'FileStorage'. 186 | Zope comes with several different back end storage objects, but 187 | one of the most interesting is the 'ClientStorage' from the Zope 188 | Enterprise Objects product (ZEO). 189 | 190 | The 'ClientStorage' storage makes a TCP/IP connection to a 191 | 'StorageServer' (also provided with ZEO). This allows many 192 | different processes on one or machines to work with the same 193 | object database and, hence, the same objects. Each process gets a 194 | cached "copy" of a particular object for speed. All of the 195 | 'ClientStorages' connected to a 'StorageServer' speak a special 196 | object transport and cache invalidation protocol to keep all of 197 | your computers synchronized. 198 | 199 | Opening a 'ClientStorage' connection is simple. The following 200 | code creates a database connection and gets the root object for a 201 | 'StorageServer' listening on "localhost:12345":: 202 | 203 | from ZODB import DB 204 | from ZEO import ClientStorage 205 | storage = ClientStorage.ClientStorage('localhost', 12345) 206 | db = DB( storage ) 207 | connection = db.open() 208 | root = connection.root() 209 | 210 | In the rare event that two processes (or threads) modify the same 211 | object at the same time, ZODB provides you with the ability to 212 | retry or resolve these conflicts yourself. 213 | 214 | Resolving Conflicts 215 | ------------------- 216 | 217 | If a conflict happens, you have two choices. The first choice is 218 | that you live with the error and you try again. Statistically, 219 | conflicts are going to happen, but only in situations where objects 220 | are "hot-spots". Most problems like this can be "designed away"; 221 | if you can redesign your application so that the changes get 222 | spread around to many different objects then you can usually get 223 | rid of the hot spot. 224 | 225 | Your second choice is to try and *resolve* the conflict. In many 226 | situations, this can be done. For example, consider the following 227 | persistent object:: 228 | 229 | class Counter(Persistent): 230 | 231 | self.count = 0 232 | 233 | def hit(self): 234 | self.count = self.count + 1 235 | 236 | This is a simple counter. If you hit this counter with a lot of 237 | requests though, it will cause conflict errors as different threads 238 | try to change the count attribute simultaneously. 239 | 240 | But resolving the conflict between conflicting threads in this 241 | case is easy. Both threads want to increment the self.count 242 | attribute by a value, so the resolution is to increment the 243 | attribute by the sum of the two values and make both commits 244 | happy. 245 | 246 | To resolve a conflict, a class should define an 247 | '_p_resolveConflict' method. This method takes three arguments: 248 | 249 | - 'oldState' -- The state of the object that the changes made by 250 | the current transaction were based on. The method is permitted 251 | to modify this value. 252 | 253 | - 'savedState' -- The state of the object that is currently 254 | stored in the database. This state was written after 'oldState' 255 | and reflects changes made by a transaction that committed 256 | before the current transaction. The method is permitted to 257 | modify this value. 258 | 259 | - 'newState' -- The state after changes made by the current 260 | transaction. The method is *not* permitted to modify this 261 | value. This method should compute a new state by merging 262 | changes reflected in 'savedState' and 'newState', relative to 263 | 'oldState'. 264 | 265 | The method should return the state of the object after resolving 266 | the differences. 267 | 268 | Here is an example of a '_p_resolveConflict' in the 'Counter' 269 | class:: 270 | 271 | class Counter(Persistent): 272 | 273 | self.count = 0 274 | 275 | def hit(self): 276 | self.count = self.count + 1 277 | 278 | def _p_resolveConflict(self, oldState, savedState, newState): 279 | 280 | # Figure out how each state is different: 281 | savedDiff= savedState['count'] - oldState['count'] 282 | newDiff= newState['count']- oldState['count'] 283 | 284 | # Apply both sets of changes to old state: 285 | return oldState['count'] + savedDiff + newDiff 286 | 287 | In the above example, '_p_resolveConflict' resolves the difference 288 | between the two conflicting transactions. 289 | 290 | Transactions and Subtransactions 291 | -------------------------------- 292 | 293 | Transactions are a very powerful concept in databases. 294 | Transactions let you make many changes to your information as if 295 | they were all one big change. Imagine software that did online 296 | banking and allowed you to transfer money from one account to 297 | another. You would do this by deducting the amount of the 298 | transfer from one account, and adding that amount onto the 299 | other. 300 | 301 | If an error happened while you were adding the money to the 302 | receiving account (say, the bank's computers were unavailable), 303 | then you would want to abort the transaction so that the state of 304 | the accounts went back to the way they were before you changed 305 | anything. 306 | 307 | To abort a transaction, you need to call the 'abort' method of the 308 | transactions object:: 309 | 310 | >>> import transaction 311 | >>> transaction.abort() 312 | 313 | This will throw away all the currently changed objects and start a 314 | new, empty transaction. 315 | 316 | Subtransactions, sometimes called "inner transactions", are 317 | transactions that happen inside another transaction. 318 | Subtransactions can be commited and aborted like regular "outer" 319 | transactions. Subtransactions mostly provide you with an 320 | optimization technique. 321 | 322 | Subtransactions can be commited and aborted. Commiting or 323 | aborting a subtransaction does not commit or abort its outer 324 | transaction, just the subtransaction. This lets you use many, 325 | fine-grained transactions within one big transaction. 326 | 327 | Why is this important? Well, in order for a transaction to be 328 | "rolled back" the changes in the transaction must be stored in 329 | memory until commit time. By commiting a subtransaction, you are 330 | telling Zope that "I'm pretty sure what I've done so far is 331 | permenant, you can store this subtransaction somewhere other than 332 | in memory". For very, very large transactions, this can be a big 333 | memory win for you. 334 | 335 | If you abort an outer transaction, then all of its inner 336 | subtransactions will also be aborted and not saved. If you abort 337 | an inner subtransaction, then only the changes made during that 338 | subtransaction are aborted, and the outer transaction is *not* 339 | aborted and more changes can be made and commited, including more 340 | subtransactions. 341 | 342 | You can commit or abort a subtransaction by calling either 343 | commit() or abort() with an argument of 1:: 344 | 345 | transaction.commit(1) # or 346 | transaction.abort(1) 347 | 348 | Subtransactions offer you a nice way to "batch" all of your "all 349 | or none" actions into smaller "all or none" actions while still 350 | keeping the outer level "all or none" transaction intact. As a 351 | bonus, they also give you much better memory resource performance. 352 | 353 | Conclusion 354 | ---------- 355 | 356 | ZODB offers many advanced features to help you develop simple, but 357 | powerful python programs. In this article, you used some of the 358 | more advanced features of ZODB to handle different application 359 | needs, like storing information in large sets, using the database 360 | concurrently, and maintaining transactional integrity. For more 361 | information on ZODB, join the discussion list at zodb-dev@zope.org 362 | where you can find out more about this powerful component of Zope. 363 | 364 | 365 | 366 | -------------------------------------------------------------------------------- /documentation/articles/ZODB1.rst: -------------------------------------------------------------------------------- 1 | Introduction to the ZODB (by Michel Pelletier) 2 | ============================================== 3 | 4 | In this article, we cover the very basics of the Zope Object 5 | Database (ZODB) for Python programmers. This short article 6 | documents almost everything you need to know about using this 7 | powerful object database in Python. In a later article, I will 8 | cover some of the more advanced features of ZODB for Python 9 | programmers. 10 | 11 | ZODB is a database for Python objects that comes with 12 | `Zope `_. If you've ever worked with a 13 | relational database, like PostgreSQL, MySQL, or Oracle, than you 14 | should be familiar with the role of a database. It's a long term 15 | or short term storage for your application data. 16 | 17 | For many tasks, relational databases are clearly a good solution, 18 | but sometimes relational databases don't fit well with your object 19 | model. If you have lots of different kinds of interconnected 20 | objects with complex relationships, and changing schemas then ZODB 21 | might be worth giving a try. 22 | 23 | A major feature of ZODB is transparency. You do not need to write 24 | any code to explicitly read or write your objects to or from a 25 | database. You just put your *persistent* objects into a container 26 | that works just like a Python dictionary. Everything inside this 27 | dictionary is saved in the database. This dictionary is said to 28 | be the "root" of the database. It's like a magic bag; any Python 29 | object that you put inside it becomes persistent. 30 | 31 | Actually there are a few restrictions on what you can store in the 32 | ZODB. You can store any objects that can be "pickled" into a 33 | standard, cross-platform serial format. Objects like lists, 34 | dictionaries, and numbers can be pickled. Objects like files, 35 | sockets, and Python code objects, cannot be stored in the database 36 | because they cannot be pickled. For more information on 37 | "pickling", see the Python pickle module documentation at 38 | http://www.python.org/doc/current/lib/module-pickle.html 39 | 40 | A Simple Example 41 | ---------------- 42 | 43 | The first thing you need to do to start working with ZODB is to 44 | create a "root object". This process involves first opening a 45 | connection to a "storage", which is the actual back-end that stores 46 | your data. 47 | 48 | ZODB supports many pluggable storage back-ends, but for the 49 | purposes of this article I'm going to show you how to use the 50 | 'FileStorage' back-end storage, which stores your object data in a 51 | file. Other storages include storing objects in relational 52 | databases, Berkeley databases, and a client to server storage that 53 | stores objects on a remote storage server. 54 | 55 | To set up a ZODB, you must first install it. ZODB comes with 56 | Zope, so the easiest way to install ZODB is to install Zope and 57 | use the ZODB that comes with your Zope installation. For those of 58 | you who don't want all of Zope, but just ZODB, see the 59 | instructions for downloading StandaloneZODB from the `ZODB web 60 | page `_. 61 | 62 | StandaloneZODB can be installed into your system's Python 63 | libraries using the standard 'distutils' Python module. 64 | 65 | After installing ZODB, you can start to experiment with it right 66 | from the Python command line interpreter. For example, try the 67 | following python code in your interpreter:: 68 | 69 | >>> from ZODB import FileStorage, DB 70 | >>> storage = FileStorage.FileStorage('mydatabase.fs') 71 | >>> db = DB(storage) 72 | >>> connection = db.open() 73 | >>> root = connection.root() 74 | 75 | Here, you create storage and use the 'mydatabse.fs' file to store 76 | the object information. Then, you create a database that uses 77 | that storage. 78 | 79 | Next, the database needs to be "opened" by calling the 'open()' 80 | method. This will return a connection object to the database. 81 | The connection object then gives you access to the 'root' of the 82 | database with the 'root()' method. 83 | 84 | The 'root' object is the dictionary that holds all of your 85 | persistent objects. For example, you can store a simple list of 86 | strings in the root object:: 87 | 88 | >>> root['employees'] = ['Mary', 'Jo', 'Bob'] 89 | 90 | Now, you have changed the persistent database by adding a new 91 | object, but this change is so far only temporary. In order to 92 | make the change permanent, you must commit the current 93 | transaction:: 94 | 95 | >>> import transaction 96 | >>> transaction.commit() 97 | 98 | Transactions group of lots of changes in one atomic operation. In 99 | a later article, I'll show you how this is a very powerful 100 | feature. For now, you can think of committing transactions as 101 | "checkpoints" where you save the changes you've made to your 102 | objects so far. Later on, I'll show you how to abort those 103 | changes, and how to undo them after they are committed. 104 | 105 | Now let's find out if our data was actually saved. First close the 106 | database connection:: 107 | 108 | >>> connection.close() 109 | 110 | Then quit Python. Now start the Python interpreter up again, and 111 | connect to the database you just created:: 112 | 113 | >>> from ZODB import FileStorage, DB 114 | >>> storage = FileStorage.FileStorage('mydatabase.fs') 115 | >>> db = DB(storage) 116 | >>> connection = db.open() 117 | >>> root = connection.root() 118 | 119 | Now, let's see what's in the root:: 120 | 121 | >>> root.items() 122 | [('employees', ['Mary', 'Jo', 'Bob'])] 123 | 124 | There's your list. If you had used a relational database, you 125 | would have had to issue a SQL query to save even a simple Python 126 | list like the above example. You would have also needed some code 127 | to convert a SQL query back into the list when you wanted to use 128 | it again. You don't have to do any of this work when using ZODB. 129 | Using ZODB is almost completely transparent, in fact, ZODB based 130 | programs often look suspiciously simple! 131 | 132 | Keep in mind that ZODB's persistent dictionary is just the tip of 133 | the persistent iceberg. Persistent objects can have attributes 134 | that are themselves persistent. In other words, even though you 135 | may have only one or two "top level" persistent objects as values 136 | in the persistent dictionary, you can still have thousands of 137 | sub-objects below them. This is, in fact, how Zope does it. In 138 | Zope, there is only *one* top level object that is the root 139 | "application" object for all other objects in Zope. 140 | 141 | Detecting Changes 142 | ----------------- 143 | 144 | One thing that makes ZODB so easy to use is that it doesn't 145 | require you to keep track of your changes. All you have to do is 146 | to make changes to persistent objects and then commit a 147 | transaction. Anything that has changed will be stored in the 148 | database. 149 | 150 | There is one exception to this rule when it comes to simple 151 | mutable Python types like lists and dictionaries. If you change a 152 | list or dictionary that is already stored in the database, then 153 | the change will *not* take effect. Consider this example:: 154 | 155 | >>> root['employees'].append('Bill') 156 | >>> transaction.commit() 157 | 158 | You would expect this to work, but it doesn't. The reason for 159 | this is that ZODB cannot detect that the 'employees' list 160 | changed. The 'employees' list is a mutable object that does not 161 | notify ZODB when it changes. 162 | 163 | There are a couple of very simple ways around this problem. The 164 | simplest is to re-assign the changed object:: 165 | 166 | >>> employees = root['employees'] 167 | >>> employees.append('Bill') 168 | >>> root['employees'] = employees 169 | >>> transaction.commit() 170 | 171 | Here, you move the employees list to a local variable, change the 172 | list, and then *reassign* the list back into the database and 173 | commit the transaction. This reassignment notifies the database 174 | that the list changed and needs to be saved to the database. 175 | 176 | Later in this article, we'll show you another technique for 177 | notifying the ZODB that your objects have changed. Also, in a 178 | later article, we'll show you how to use simple, ZODB-aware list 179 | and dictionary classes that come pre-packaged with ZODB for your 180 | convenience. 181 | 182 | Persistent Classes 183 | ------------------ 184 | 185 | The easiest way to create mutable objects that notify the ZODB of 186 | changes is to create a persistent class. Persistent classes let 187 | you store your own kinds of objects in the database. For example, 188 | consider a class that represents a employee:: 189 | 190 | import ZODB 191 | from Persistence import Persistent 192 | 193 | class Employee(Persistent): 194 | 195 | def setName(self, name): 196 | self.name = name 197 | 198 | To create a persistent class, simply subclass from 199 | 'Persistent.Persistent'. Because of some special magic that ZODB 200 | does, you must first import ZODB before you can import Persistent. 201 | The 'Persistent' module is actually *created* when you import 202 | 'ZODB'. 203 | 204 | Now, you can put Employee objects in your database:: 205 | 206 | >>> employees=[] 207 | >>> for name in ['Mary', 'Joe', 'Bob']: 208 | ... employee = Employee() 209 | ... employee.setName(name) 210 | ... employees.append(employee) 211 | >>> root['employees']=employees 212 | >>> transaction.commit() 213 | 214 | Don't forget to call 'commit()', so that the changes you have made 215 | so far are committed to the database, and a new transaction is 216 | begun. 217 | 218 | Now you can change your employees and they will be saved in the 219 | database. For example you can change Bob's name to "Robert":: 220 | 221 | >>> bob=root['employees'][2] 222 | >>> bob.setName('Robert') 223 | >>> transaction.commit() 224 | 225 | You can even change attributes of persistent instaces without 226 | calling methods:: 227 | 228 | >>> bob=root['employees'][2] 229 | >>> bob._coffee_prefs=('Cream', 'Sugar') 230 | >>> transaction.commit() 231 | 232 | It doesn't matter whether you change an attribute directly, or 233 | whether it's changed by a method. As you can tell, all of the 234 | normal Python language rules still work as you'd expect. 235 | 236 | Mutable Attributes 237 | ------------------ 238 | 239 | Earlier you saw how ZODB can't detect changes to normal mutable 240 | objects like Python lists. This issue still affects you when using 241 | persistent instances. This is because persistent instances can 242 | have attributes which are normal mutable objects. For example, 243 | consider this class:: 244 | 245 | class Employee(Persistent): 246 | 247 | def __init__(self): 248 | self.tasks = [] 249 | 250 | def setName(self, name): 251 | self.name = name 252 | 253 | def addTask(self, task): 254 | self.task.append(task) 255 | 256 | When you call 'addTask', the ZODB won't know that the mutable 257 | attribute 'self.tasks' has changed. As you saw earlier, you can 258 | reassign 'self.tasks' after you change it to get around this 259 | problem. However, when you're using persistent instances, you have 260 | another choice. You can signal the ZODB that your instance has 261 | changed with the '_p_changed' attribute:: 262 | 263 | class Employee(Persistent): 264 | ... 265 | 266 | def addTask(self, task): 267 | self.task.append(task) 268 | self._p_changed = 1 269 | 270 | To signal that this object has change, set the '_p_changed' 271 | attribute to 1. You only need to signal ZODB once, even if you 272 | change many mutable attributes. 273 | 274 | The '_p_changed' flag leads us to one of the few rules of you must 275 | follow when creating persistent classes: your instances *cannot* 276 | have attributes that begin with '_p_', those names are reserved 277 | for use by the ZODB. 278 | 279 | A Complete Example 280 | ------------------ 281 | 282 | Here's a complete example program. It builds on the employee 283 | examples used so far:: 284 | 285 | from ZODB import DB 286 | from ZODB.FileStorage import FileStorage 287 | from ZODB.PersistentMapping import PersistentMapping 288 | from Persistence import Persistent 289 | import transaction 290 | 291 | class Employee(Persistent): 292 | """An employee""" 293 | 294 | def __init__(self, name, manager=None): 295 | self.name=name 296 | self.manager=manager 297 | 298 | # setup the database 299 | storage=FileStorage("employees.fs") 300 | db=DB(storage) 301 | connection=db.open() 302 | root=connection.root() 303 | 304 | # get the employees mapping, creating an empty mapping if 305 | # necessary 306 | if not root.has_key("employees"): 307 | root["employees"] = {} 308 | employees=root["employees"] 309 | 310 | 311 | def listEmployees(): 312 | if len(employees.values())==0: 313 | print "There are no employees." 314 | print 315 | return 316 | for employee in employees.values(): 317 | print "Name: %s" % employee.name 318 | if employee.manager is not None: 319 | print "Manager's name: %s" % employee.manager.name 320 | print 321 | 322 | def addEmployee(name, manager_name=None): 323 | if employees.has_key(name): 324 | print "There is already an employee with this name." 325 | return 326 | if manager_name: 327 | try: 328 | manager=employees[manager_name] 329 | except KeyError: 330 | print 331 | print "No such manager" 332 | print 333 | return 334 | employees[name]=Employee(name, manager) 335 | else: 336 | employees[name]=Employee(name) 337 | 338 | root['employees'] = employees # reassign to change 339 | transaction.commit() 340 | print "Employee %s added." % name 341 | print 342 | 343 | 344 | if __name__=="__main__": 345 | while 1: 346 | choice=raw_input("Press 'L' to list employees, 'A' to add" 347 | "an employee, or 'Q' to quit:") 348 | choice=choice.lower() 349 | if choice=="l": 350 | listEmployees() 351 | elif choice=="a": 352 | name=raw_input("Employee name:") 353 | manager_name=raw_input("Manager name:") 354 | addEmployee(name, manager_name) 355 | elif choice=="q": 356 | break 357 | 358 | # close database 359 | connection.close() 360 | 361 | This program demonstrates a couple interesting things. First, this 362 | program shows how persistent objects can refer to each other. The 363 | 'self.manger' attribute of 'Employee' instances can refer to other 364 | 'Employee' instances. Unlike a relational database, there is no 365 | need to use indirection such as object ids when referring from one 366 | persistent object to another. You can just use normal Python 367 | references. In fact, you can even use circular references. 368 | 369 | A final trick used by this program is to look for a persistent 370 | object and create it if it is not present. This allows you to just 371 | run this program without having to run a setup script to build the 372 | database first. If there is not database present, the program will 373 | create one and initialize it. 374 | 375 | Conclusion 376 | ---------- 377 | 378 | ZODB is a very simple, transparent object database for Python that 379 | is a freely available component of the Zope application server. 380 | As these examples illustrate, only a few lines of code are needed 381 | to start storing Python objects in ZODB, with no need to write SQL 382 | queries. In the next article on ZODB, we'll show you some more 383 | advanced techniques for using ZODB, like using ZODB's distributed 384 | object protocol to distribute your persistent objects across many 385 | machines. 386 | 387 | ZODB Resources 388 | 389 | - `Andrew Kuchling's "ZODB pages" `_ (archived) 390 | 391 | - `Zope.org "ZODB Wiki" `_ 392 | 393 | - `Jim Fulton's "Introduction to the Zope Object Database" `_ 394 | 395 | 396 | 397 | 398 | 399 | 400 | 401 | 402 | -------------------------------------------------------------------------------- /documentation/guide/gfdl.rst: -------------------------------------------------------------------------------- 1 | .. % gfdl.tex 2 | .. % This file is a chapter. It must be included in a larger document to work 3 | .. % properly. 4 | 5 | 6 | GNU Free Documentation License 7 | ============================== 8 | 9 | Version 1.1, March 2000 --- 10 | 11 | Copyright 2000 Free Software Foundation, Inc. --- 59 Temple 12 | Place, Suite 330, Boston, MA 02111-1307 USA --- Everyone is permitted to copy 13 | and distribute verbatim copies of this license document, but changing it is not 14 | allowed. 15 | 16 | Preamble 17 | 18 | The purpose of this License is to make a manual, textbook, or other written 19 | document "free" in the sense of freedom: to assure everyone the effective 20 | freedom to copy and redistribute it, with or without modifying it, either 21 | commercially or noncommercially. Secondarily, this License preserves for the 22 | author and publisher a way to get credit for their work, while not being 23 | considered responsible for modifications made by others. 24 | 25 | This License is a kind of "copyleft", which means that derivative works of the 26 | document must themselves be free in the same sense. It complements the GNU 27 | General Public License, which is a copyleft license designed for free software. 28 | 29 | We have designed this License in order to use it for manuals for free software, 30 | because free software needs free documentation: a free program should come with 31 | manuals providing the same freedoms that the software does. But this License is 32 | not limited to software manuals; it can be used for any textual work, regardless 33 | of subject matter or whether it is published as a printed book. We recommend 34 | this License principally for works whose purpose is instruction or reference. 35 | 36 | 37 | Applicability and Definitions 38 | ----------------------------- 39 | 40 | This License applies to any manual or other work that contains a notice placed 41 | by the copyright holder saying it can be distributed under the terms of this 42 | License. The "Document", below, refers to any such manual or work. Any member 43 | of the public is a licensee, and is addressed as "you". 44 | 45 | A "Modified Version" of the Document means any work containing the Document or a 46 | portion of it, either copied verbatim, or with modifications and/or translated 47 | into another language. 48 | 49 | A "Secondary Section" is a named appendix or a front-matter section of the 50 | Document that deals exclusively with the relationship of the publishers or 51 | authors of the Document to the Document's overall subject (or to related 52 | matters) and contains nothing that could fall directly within that overall 53 | subject. (For example, if the Document is in part a textbook of mathematics, a 54 | Secondary Section may not explain any mathematics.) The relationship could be a 55 | matter of historical connection with the subject or with related matters, or of 56 | legal, commercial, philosophical, ethical or political position regarding them. 57 | 58 | The "Invariant Sections" are certain Secondary Sections whose titles are 59 | designated, as being those of Invariant Sections, in the notice that says that 60 | the Document is released under this License. 61 | 62 | The "Cover Texts" are certain short passages of text that are listed, as Front- 63 | Cover Texts or Back-Cover Texts, in the notice that says that the Document is 64 | released under this License. 65 | 66 | A "Transparent" copy of the Document means a machine-readable copy, represented 67 | in a format whose specification is available to the general public, whose 68 | contents can be viewed and edited directly and straightforwardly with generic 69 | text editors or (for images composed of pixels) generic paint programs or (for 70 | drawings) some widely available drawing editor, and that is suitable for input 71 | to text formatters or for automatic translation to a variety of formats suitable 72 | for input to text formatters. A copy made in an otherwise Transparent file 73 | format whose markup has been designed to thwart or discourage subsequent 74 | modification by readers is not Transparent. A copy that is not "Transparent" is 75 | called "Opaque". 76 | 77 | Examples of suitable formats for Transparent copies include plain ASCII without 78 | markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly 79 | available DTD, and standard-conforming simple HTML designed for human 80 | modification. Opaque formats include PostScript, PDF, proprietary formats that 81 | can be read and edited only by proprietary word processors, SGML or XML for 82 | which the DTD and/or processing tools are not generally available, and the 83 | machine-generated HTML produced by some word processors for output purposes 84 | only. 85 | 86 | The "Title Page" means, for a printed book, the title page itself, plus such 87 | following pages as are needed to hold, legibly, the material this License 88 | requires to appear in the title page. For works in formats which do not have 89 | any title page as such, "Title Page" means the text near the most prominent 90 | appearance of the work's title, preceding the beginning of the body of the text. 91 | 92 | 93 | Verbatim Copying 94 | ---------------- 95 | 96 | You may copy and distribute the Document in any medium, either commercially or 97 | noncommercially, provided that this License, the copyright notices, and the 98 | license notice saying this License applies to the Document are reproduced in all 99 | copies, and that you add no other conditions whatsoever to those of this 100 | License. You may not use technical measures to obstruct or control the reading 101 | or further copying of the copies you make or distribute. However, you may 102 | accept compensation in exchange for copies. If you distribute a large enough 103 | number of copies you must also follow the conditions in section 3. 104 | 105 | You may also lend copies, under the same conditions stated above, and you may 106 | publicly display copies. 107 | 108 | 109 | Copying in Quantity 110 | ------------------- 111 | 112 | If you publish printed copies of the Document numbering more than 100, and the 113 | Document's license notice requires Cover Texts, you must enclose the copies in 114 | covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts 115 | on the front cover, and Back-Cover Texts on the back cover. Both covers must 116 | also clearly and legibly identify you as the publisher of these copies. The 117 | front cover must present the full title with all words of the title equally 118 | prominent and visible. You may add other material on the covers in addition. 119 | Copying with changes limited to the covers, as long as they preserve the title 120 | of the Document and satisfy these conditions, can be treated as verbatim copying 121 | in other respects. 122 | 123 | If the required texts for either cover are too voluminous to fit legibly, you 124 | should put the first ones listed (as many as fit reasonably) on the actual 125 | cover, and continue the rest onto adjacent pages. 126 | 127 | If you publish or distribute Opaque copies of the Document numbering more than 128 | 100, you must either include a machine-readable Transparent copy along with each 129 | Opaque copy, or state in or with each Opaque copy a publicly-accessible 130 | computer-network location containing a complete Transparent copy of the 131 | Document, free of added material, which the general network-using public has 132 | access to download anonymously at no charge using public-standard network 133 | protocols. If you use the latter option, you must take reasonably prudent 134 | steps, when you begin distribution of Opaque copies in quantity, to ensure that 135 | this Transparent copy will remain thus accessible at the stated location until 136 | at least one year after the last time you distribute an Opaque copy (directly or 137 | through your agents or retailers) of that edition to the public. 138 | 139 | It is requested, but not required, that you contact the authors of the Document 140 | well before redistributing any large number of copies, to give them a chance to 141 | provide you with an updated version of the Document. 142 | 143 | 144 | Modifications 145 | ------------- 146 | 147 | You may copy and distribute a Modified Version of the Document under the 148 | conditions of sections 2 and 3 above, provided that you release the Modified 149 | Version under precisely this License, with the Modified Version filling the role 150 | of the Document, thus licensing distribution and modification of the Modified 151 | Version to whoever possesses a copy of it. In addition, you must do these 152 | things in the Modified Version: 153 | 154 | * Use in the Title Page (and on the covers, if any) a title distinct from that 155 | of the Document, and from those of previous versions (which should, if there 156 | were any, be listed in the History section of the Document). You may use the 157 | same title as a previous version if the original publisher of that version gives 158 | permission. 159 | 160 | * List on the Title Page, as authors, one or more persons or entities 161 | responsible for authorship of the modifications in the Modified Version, 162 | together with at least five of the principal authors of the Document (all of its 163 | principal authors, if it has less than five). 164 | 165 | * State on the Title page the name of the publisher of the Modified Version, as 166 | the publisher. 167 | 168 | * Preserve all the copyright notices of the Document. 169 | 170 | * Add an appropriate copyright notice for your modifications adjacent to the 171 | other copyright notices. 172 | 173 | * Include, immediately after the copyright notices, a license notice giving the 174 | public permission to use the Modified Version under the terms of this License, 175 | in the form shown in the Addendum below. 176 | 177 | * Preserve in that license notice the full lists of Invariant Sections and 178 | required Cover Texts given in the Document's license notice. 179 | 180 | * Include an unaltered copy of this License. 181 | 182 | * Preserve the section entitled "History", and its title, and add to it an item 183 | stating at least the title, year, new authors, and publisher of the Modified 184 | Version as given on the Title Page. If there is no section entitled "History" 185 | in the Document, create one stating the title, year, authors, and publisher of 186 | the Document as given on its Title Page, then add an item describing the 187 | Modified Version as stated in the previous sentence. 188 | 189 | * Preserve the network location, if any, given in the Document for public access 190 | to a Transparent copy of the Document, and likewise the network locations given 191 | in the Document for previous versions it was based on. These may be placed in 192 | the "History" section. You may omit a network location for a work that was 193 | published at least four years before the Document itself, or if the original 194 | publisher of the version it refers to gives permission. 195 | 196 | * In any section entitled "Acknowledgements" or "Dedications", preserve the 197 | section's title, and preserve in the section all the substance and tone of each 198 | of the contributor acknowledgements and/or dedications given therein. 199 | 200 | * Preserve all the Invariant Sections of the Document, unaltered in their text 201 | and in their titles. Section numbers or the equivalent are not considered part 202 | of the section titles. 203 | 204 | * Delete any section entitled "Endorsements". Such a section may not be 205 | included in the Modified Version. 206 | 207 | * Do not retitle any existing section as "Endorsements" or to conflict in title 208 | with any Invariant Section. 209 | 210 | If the Modified Version includes new front-matter sections or appendices that 211 | qualify as Secondary Sections and contain no material copied from the Document, 212 | you may at your option designate some or all of these sections as invariant. To 213 | do this, add their titles to the list of Invariant Sections in the Modified 214 | Version's license notice. These titles must be distinct from any other section 215 | titles. 216 | 217 | You may add a section entitled "Endorsements", provided it contains nothing but 218 | endorsements of your Modified Version by various parties -- for example, 219 | statements of peer review or that the text has been approved by an organization 220 | as the authoritative definition of a standard. 221 | 222 | You may add a passage of up to five words as a Front-Cover Text, and a passage 223 | of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in 224 | the Modified Version. Only one passage of Front-Cover Text and one of Back- 225 | Cover Text may be added by (or through arrangements made by) any one entity. If 226 | the Document already includes a cover text for the same cover, previously added 227 | by you or by arrangement made by the same entity you are acting on behalf of, 228 | you may not add another; but you may replace the old one, on explicit permission 229 | from the previous publisher that added the old one. 230 | 231 | The author(s) and publisher(s) of the Document do not by this License give 232 | permission to use their names for publicity for or to assert or imply 233 | endorsement of any Modified Version. 234 | 235 | 236 | Combining Documents 237 | ------------------- 238 | 239 | You may combine the Document with other documents released under this License, 240 | under the terms defined in section 4 above for modified versions, provided that 241 | you include in the combination all of the Invariant Sections of all of the 242 | original documents, unmodified, and list them all as Invariant Sections of your 243 | combined work in its license notice. 244 | 245 | The combined work need only contain one copy of this License, and multiple 246 | identical Invariant Sections may be replaced with a single copy. If there are 247 | multiple Invariant Sections with the same name but different contents, make the 248 | title of each such section unique by adding at the end of it, in parentheses, 249 | the name of the original author or publisher of that section if known, or else a 250 | unique number. Make the same adjustment to the section titles in the list of 251 | Invariant Sections in the license notice of the combined work. 252 | 253 | In the combination, you must combine any sections entitled "History" in the 254 | various original documents, forming one section entitled "History"; likewise 255 | combine any sections entitled "Acknowledgements", and any sections entitled 256 | "Dedications". You must delete all sections entitled "Endorsements." 257 | 258 | 259 | Collections of Documents 260 | ------------------------ 261 | 262 | You may make a collection consisting of the Document and other documents 263 | released under this License, and replace the individual copies of this License 264 | in the various documents with a single copy that is included in the collection, 265 | provided that you follow the rules of this License for verbatim copying of each 266 | of the documents in all other respects. 267 | 268 | You may extract a single document from such a collection, and distribute it 269 | individually under this License, provided you insert a copy of this License into 270 | the extracted document, and follow this License in all other respects regarding 271 | verbatim copying of that document. 272 | 273 | 274 | Aggregation With Independent Works 275 | ---------------------------------- 276 | 277 | A compilation of the Document or its derivatives with other separate and 278 | independent documents or works, in or on a volume of a storage or distribution 279 | medium, does not as a whole count as a Modified Version of the Document, 280 | provided no compilation copyright is claimed for the compilation. Such a 281 | compilation is called an "aggregate", and this License does not apply to the 282 | other self-contained works thus compiled with the Document, on account of their 283 | being thus compiled, if they are not themselves derivative works of the 284 | Document. 285 | 286 | If the Cover Text requirement of section 3 is applicable to these copies of the 287 | Document, then if the Document is less than one quarter of the entire aggregate, 288 | the Document's Cover Texts may be placed on covers that surround only the 289 | Document within the aggregate. Otherwise they must appear on covers around the 290 | whole aggregate. 291 | 292 | 293 | Translation 294 | ----------- 295 | 296 | Translation is considered a kind of modification, so you may distribute 297 | translations of the Document under the terms of section 4. Replacing Invariant 298 | Sections with translations requires special permission from their copyright 299 | holders, but you may include translations of some or all Invariant Sections in 300 | addition to the original versions of these Invariant Sections. You may include 301 | a translation of this License provided that you also include the original 302 | English version of this License. In case of a disagreement between the 303 | translation and the original English version of this License, the original 304 | English version will prevail. 305 | 306 | 307 | Termination 308 | ----------- 309 | 310 | You may not copy, modify, sublicense, or distribute the Document except as 311 | expressly provided for under this License. Any other attempt to copy, modify, 312 | sublicense or distribute the Document is void, and will automatically terminate 313 | your rights under this License. However, parties who have received copies, or 314 | rights, from you under this License will not have their licenses terminated so 315 | long as such parties remain in full compliance. 316 | 317 | 318 | Future Revisions of This Licence 319 | -------------------------------- 320 | 321 | The Free Software Foundation may publish new, revised versions of the GNU Free 322 | Documentation License from time to time. Such new versions will be similar in 323 | spirit to the present version, but may differ in detail to address new problems 324 | or concerns. See ``_. 325 | 326 | Each version of the License is given a distinguishing version number. If the 327 | Document specifies that a particular numbered version of this License "or any 328 | later version" applies to it, you have the option of following the terms and 329 | conditions either of that specified version or of any later version that has 330 | been published (not as a draft) by the Free Software Foundation. If the 331 | Document does not specify a version number of this License, you may choose any 332 | version ever published (not as a draft) by the Free Software Foundation. 333 | 334 | ADDENDUM: How to use this License for your documents 335 | 336 | To use this License in a document you have written, include a copy of the 337 | License in the document and put the following copyright and license notices just 338 | after the title page: 339 | 340 | Copyright YEAR YOUR NAME. Permission is granted to copy, 341 | distribute and/or modify this document under the terms of the GNU Free 342 | Documentation License, Version 1.1 or any later version published by the Free 343 | Software Foundation; with the Invariant Sections being LIST THEIR TITLES, with 344 | the Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST. A 345 | copy of the license is included in the section entitled "GNU Free Documentation 346 | License". 347 | 348 | 349 | If you have no Invariant Sections, write "with no Invariant Sections" instead of 350 | saying which ones are invariant. If you have no Front-Cover Texts, write "no 351 | Front-Cover Texts" instead of "Front-Cover Texts being LIST"; likewise for Back- 352 | Cover Texts. 353 | 354 | If your document contains nontrivial examples of program code, we recommend 355 | releasing these examples in parallel under your choice of free software license, 356 | such as the GNU General Public License, to permit their use in free software. 357 | 358 | -------------------------------------------------------------------------------- /documentation/guide/modules.rst: -------------------------------------------------------------------------------- 1 | .. % Related Modules 2 | .. % PersistentMapping 3 | .. % PersistentList 4 | .. % BTrees 5 | .. % Total Ordering and Persistence 6 | .. % Iteration and Mutation 7 | .. % BTree Diagnostic Tools 8 | 9 | 10 | Related Modules 11 | =============== 12 | 13 | The ZODB package includes a number of related modules that provide useful data 14 | types such as BTrees. 15 | 16 | 17 | :mod:`persistent.mapping.PersistentMapping` 18 | ------------------------------------------- 19 | 20 | The :class:`PersistentMapping` class is a wrapper for mapping objects that will 21 | set the dirty bit when the mapping is modified by setting or deleting a key. 22 | 23 | 24 | .. function:: PersistentMapping(container = {}) 25 | 26 | Create a :class:`PersistentMapping` object that wraps the mapping object 27 | *container*. If you don't specify a value for *container*, a regular Python 28 | dictionary is used. 29 | 30 | :class:`PersistentMapping` objects support all the same methods as Python 31 | dictionaries do. 32 | 33 | 34 | :mod:`persistent.list.PersistentList` 35 | ------------------------------------- 36 | 37 | The :class:`PersistentList` class is a wrapper for mutable sequence objects, 38 | much as :class:`PersistentMapping` is a wrapper for mappings. 39 | 40 | 41 | .. function:: PersistentList(initlist = []) 42 | 43 | Create a :class:`PersistentList` object that wraps the mutable sequence object 44 | *initlist*. If you don't specify a value for *initlist*, a regular Python list 45 | is used. 46 | 47 | :class:`PersistentList` objects support all the same methods as Python lists do. 48 | 49 | 50 | BTrees Package 51 | -------------- 52 | 53 | When programming with the ZODB, Python dictionaries aren't always what you need. 54 | The most important case is where you want to store a very large mapping. When a 55 | Python dictionary is accessed in a ZODB, the whole dictionary has to be 56 | unpickled and brought into memory. If you're storing something very large, such 57 | as a 100,000-entry user database, unpickling such a large object will be slow. 58 | BTrees are a balanced tree data structure that behave like a mapping but 59 | distribute keys throughout a number of tree nodes. The nodes are stored in 60 | sorted order (this has important consequences -- see below). Nodes are then 61 | only unpickled and brought into memory as they're accessed, so the entire tree 62 | doesn't have to occupy memory (unless you really are touching every single key). 63 | 64 | The BTrees package provides a large collection of related data structures. 65 | There are variants of the data structures specialized to integers, which are 66 | faster and use less memory. There are five modules that handle the different 67 | variants. The first two letters of the module name specify the types of the 68 | keys and values in mappings -- O for any object, I for 32-bit signed integer, 69 | and (new in ZODB 3.4) F for 32-bit C float. For example, the 70 | :mod:`BTrees.IOBTree` module provides a mapping with integer keys and arbitrary 71 | objects as values. 72 | 73 | The four data structures provide by each module are a BTree, a Bucket, a 74 | TreeSet, and a Set. The BTree and Bucket types are mappings and support all the 75 | usual mapping methods, e.g. :func:`update` and :func:`keys`. The TreeSet and 76 | Set types are similar to mappings but they have no values; they support the 77 | methods that make sense for a mapping with no keys, e.g. :func:`keys` but not 78 | :func:`items`. The Bucket and Set types are the individual building blocks for 79 | BTrees and TreeSets, respectively. A Bucket or Set can be used when you are 80 | sure that it will have few elements. If the data structure will grow large, you 81 | should use a BTree or TreeSet. Like Python lists, Buckets and Sets are allocated 82 | in one contiguous piece, and insertions and deletions can take time proportional 83 | to the number of existing elements. Also like Python lists, a Bucket or Set is 84 | a single object, and is pickled and unpickled in its entirety. BTrees and 85 | TreeSets are multi-level tree structures with much better (logarithmic) worst- 86 | case time bounds, and the tree structure is built out of multiple objects, which 87 | ZODB can load individually as needed. 88 | 89 | The five modules are named :mod:`OOBTree`, :mod:`IOBTree`, :mod:`OIBTree`, 90 | :mod:`IIBTree`, and (new in ZODB 3.4) :mod:`IFBTree`. The two letter prefixes 91 | are repeated in the data types names. The :mod:`BTrees.OOBTree` module defines 92 | the following types: :class:`OOBTree`, :class:`OOBucket`, :class:`OOSet`, and 93 | :class:`OOTreeSet`. Similarly, the other four modules each define their own 94 | variants of those four types. 95 | 96 | The :func:`keys`, :func:`values`, and :func:`items` methods on BTree and TreeSet 97 | types do not materialize a list with all of the data. Instead, they return lazy 98 | sequences that fetch data from the BTree as needed. They also support optional 99 | arguments to specify the minimum and maximum values to return, often called 100 | "range searching". Because all these types are stored in sorted order, range 101 | searching is very efficient. 102 | 103 | The :func:`keys`, :func:`values`, and :func:`items` methods on Bucket and Set 104 | types do return lists with all the data. Starting in ZODB 3.3, there are also 105 | :func:`iterkeys`, :func:`itervalues`, and :func:`iteritems` methods that return 106 | iterators (in the Python 2.2 sense). Those methods also apply to BTree and 107 | TreeSet objects. 108 | 109 | A BTree object supports all the methods you would expect of a mapping, with a 110 | few extensions that exploit the fact that the keys are sorted. The example below 111 | demonstrates how some of the methods work. The extra methods are :func:`minKey` 112 | and :func:`maxKey`, which find the minimum and maximum key value subject to an 113 | optional bound argument, and :func:`byValue`, which should probably be ignored 114 | (it's hard to explain exactly what it does, and as a result it's almost never 115 | used -- best to consider it deprecated). The various methods for enumerating 116 | keys, values and items also accept minimum and maximum key arguments ("range 117 | search"), and (new in ZODB 3.3) optional Boolean arguments to control whether a 118 | range search is inclusive or exclusive of the range's endpoints. :: 119 | 120 | >>> from BTrees.OOBTree import OOBTree 121 | >>> t = OOBTree() 122 | >>> t.update({1: "red", 2: "green", 3: "blue", 4: "spades"}) 123 | >>> len(t) 124 | 4 125 | >>> t[2] 126 | 'green' 127 | >>> s = t.keys() # this is a "lazy" sequence object 128 | >>> s 129 | 130 | >>> len(s) # it acts like a Python list 131 | 4 132 | >>> s[-2] 133 | 3 134 | >>> list(s) # materialize the full list 135 | [1, 2, 3, 4] 136 | >>> list(t.values()) 137 | ['red', 'green', 'blue', 'spades'] 138 | >>> list(t.values(1, 2)) # values at keys in 1 to 2 inclusive 139 | ['red', 'green'] 140 | >>> list(t.values(2)) # values at keys >= 2 141 | ['green', 'blue', 'spades'] 142 | >>> list(t.values(min=1, max=4)) # keyword args new in ZODB 3.3 143 | ['red', 'green', 'blue', 'spades'] 144 | >>> list(t.values(min=1, max=4, excludemin=True, excludemax=True)) 145 | ['green', 'blue'] 146 | >>> t.minKey() # smallest key 147 | 1 148 | >>> t.minKey(1.5) # smallest key >= 1.5 149 | 2 150 | >>> for k in t.keys(): 151 | ... print k, 152 | 1 2 3 4 153 | >>> for k in t: # new in ZODB 3.3 154 | ... print k, 155 | 1 2 3 4 156 | >>> for pair in t.iteritems(): # new in ZODB 3.3 157 | ... print pair, 158 | ... 159 | (1, 'red') (2, 'green') (3, 'blue') (4, 'spades') 160 | >>> t.has_key(4) # returns a true value, but exactly what undefined 161 | 2 162 | >>> t.has_key(5) 163 | 0 164 | >>> 4 in t # new in ZODB 3.3 165 | True 166 | >>> 5 in t # new in ZODB 3.3 167 | False 168 | >>> 169 | 170 | 171 | Each of the modules also defines some functions that operate on BTrees -- 172 | :func:`difference`, :func:`union`, and :func:`intersection`. The 173 | :func:`difference` function returns a Bucket, while the other two methods return 174 | a Set. If the keys are integers, then the module also defines 175 | :func:`multiunion`. If the values are integers or floats, then the module also 176 | defines :func:`weightedIntersection` and :func:`weightedUnion`. The function 177 | doc strings describe each function briefly. 178 | 179 | .. % XXX I'm not sure all of the following is actually correct. The 180 | .. % XXX set functions have complicated behavior. 181 | 182 | ``BTrees/Interfaces.py`` defines the operations, and is the official 183 | documentation. Note that the interfaces don't define the concrete types 184 | returned by most operations, and you shouldn't rely on the concrete types that 185 | happen to be returned: stick to operations guaranteed by the interface. In 186 | particular, note that the interfaces don't specify anything about comparison 187 | behavior, and so nothing about it is guaranteed. In ZODB 3.3, for example, two 188 | BTrees happen to use Python's default object comparison, which amounts to 189 | comparing the (arbitrary but fixed) memory addresses of the BTrees. This may or 190 | may not be true in future releases. If the interfaces don't specify a behavior, 191 | then whether that behavior appears to work, and exactly happens if it does 192 | appear to work, are undefined and should not be relied on. 193 | 194 | 195 | Total Ordering and Persistence 196 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 197 | 198 | The BTree-based data structures differ from Python dicts in several fundamental 199 | ways. One of the most important is that while dicts require that keys support 200 | hash codes and equality comparison, the BTree-based structures don't use hash 201 | codes and require a total ordering on keys. 202 | 203 | Total ordering means three things: 204 | 205 | #. Reflexive. For each *x*, ``x == x`` is true. 206 | 207 | #. Trichotomy. For each *x* and *y*, exactly one of ``x < y``, ``x == y``, and 208 | ``x > y`` is true. 209 | 210 | #. Transitivity. Whenever ``x <= y`` and ``y <= z``, it's also true that ``x <= 211 | z``. 212 | 213 | The default comparison functions for most objects that come with Python satisfy 214 | these rules, with some crucial cautions explained later. Complex numbers are an 215 | example of an object whose default comparison function does not satisfy these 216 | rules: complex numbers only support ``==`` and ``!=`` comparisons, and raise an 217 | exception if you try to compare them in any other way. They don't satisfy the 218 | trichotomy rule, and must not be used as keys in BTree-based data structures 219 | (although note that complex numbers can be used as keys in Python dicts, which 220 | do not require a total ordering). 221 | 222 | Examples of objects that are wholly safe to use as keys in BTree-based 223 | structures include ints, longs, floats, 8-bit strings, Unicode strings, and 224 | tuples composed (possibly recursively) of objects of wholly safe types. 225 | 226 | It's important to realize that even if two types satisfy the rules on their own, 227 | mixing objects of those types may not. For example, 8-bit strings and Unicode 228 | strings both supply total orderings, but mixing the two loses trichotomy; e.g., 229 | ``'x' < chr(255)`` and ``u'x' == 'x'``, but trying to compare ``chr(255)`` to 230 | ``u'x'`` raises an exception. Partly for this reason (another is given later), 231 | it can be dangerous to use keys with multiple types in a single BTree-based 232 | structure. Don't try to do that, and you don't have to worry about it. 233 | 234 | Another potential problem is mutability: when a key is inserted in a BTree- 235 | based structure, it must retain the same order relative to the other keys over 236 | time. This is easy to run afoul of if you use mutable objects as keys. For 237 | example, lists supply a total ordering, and then :: 238 | 239 | >>> L1, L2, L3 = [1], [2], [3] 240 | >>> from BTrees.OOBTree import OOSet 241 | >>> s = OOSet((L2, L3, L1)) # this is fine, so far 242 | >>> list(s.keys()) # note that the lists are in sorted order 243 | [[1], [2], [3]] 244 | >>> s.has_key([3]) # and [3] is in the set 245 | 1 246 | >>> L2[0] = 5 # horrible -- the set is insane now 247 | >>> s.has_key([3]) # for example, it's insane this way 248 | 0 249 | >>> s 250 | OOSet([[1], [5], [3]]) 251 | >>> 252 | 253 | Key lookup relies on that the keys remain in sorted order (an efficient form of 254 | binary search is used). By mutating key L2 after inserting it, we destroyed the 255 | invariant that the OOSet is sorted. As a result, all future operations on this 256 | set are unpredictable. 257 | 258 | A subtler variant of this problem arises due to persistence: by default, Python 259 | does several kinds of comparison by comparing the memory addresses of two 260 | objects. Because Python never moves an object in memory, this does supply a 261 | usable (albeit arbitrary) total ordering across the life of a program run (an 262 | object's memory address doesn't change). But if objects compared in this way 263 | are used as keys of a BTree-based structure that's stored in a database, when 264 | the objects are loaded from the database again they will almost certainly wind 265 | up at different memory addresses. There's no guarantee then that if key K1 had 266 | a memory address smaller than the memory address of key K2 at the time K1 and K2 267 | were inserted in a BTree, K1's address will also be smaller than K2's when that 268 | BTree is loaded from a database later. The result will be an insane BTree, 269 | where various operations do and don't work as expected, seemingly at random. 270 | 271 | Now each of the types identified above as "wholly safe to use" never compares 272 | two instances of that type by memory address, so there's nothing to worry about 273 | here if you use keys of those types. The most common mistake is to use keys 274 | that are instances of a user-defined class that doesn't supply its own 275 | :meth:`__cmp__` method. Python compares such instances by memory address. This 276 | is fine if such instances are used as keys in temporary BTree-based structures 277 | used only in a single program run. It can be disastrous if that BTree-based 278 | structure is stored to a database, though. 279 | 280 | :: 281 | 282 | >>> class C: 283 | ... pass 284 | ... 285 | >>> a, b = C(), C() 286 | >>> print a < b # this may print 0 if you try it 287 | 1 288 | >>> del a, b 289 | >>> a, b = C(), C() 290 | >>> print a < b # and this may print 0 or 1 291 | 0 292 | >>> 293 | 294 | That example illustrates that comparison of instances of classes that don't 295 | define :meth:`__cmp__` yields arbitrary results (but consistent results within a 296 | single program run). 297 | 298 | Another problem occurs with instances of classes that do define :meth:`__cmp__`, 299 | but define it incorrectly. It's possible but rare for a custom :meth:`__cmp__` 300 | implementation to violate one of the three required formal properties directly. 301 | It's more common for it to "fall back" to address-based comparison by mistake. 302 | For example:: 303 | 304 | class Mine: 305 | def __cmp__(self, other): 306 | if other.__class__ is Mine: 307 | return cmp(self.data, other.data) 308 | else: 309 | return cmp(self.data, other) 310 | 311 | It's quite possible there that the ``else`` clause allows a result to be 312 | computed based on memory address. The bug won't show up until a BTree-based 313 | structure uses objects of class :class:`Mine` as keys, and also objects of other 314 | types as keys, and the structure is loaded from a database, and a sequence of 315 | comparisons happens to execute the ``else`` clause in a case where the 316 | relative order of object memory addresses happened to change. 317 | 318 | This is as difficult to track down as it sounds, so best to stay far away from 319 | the possibility. 320 | 321 | You'll stay out of trouble by follwing these rules, violating them only with 322 | great care: 323 | 324 | #. Use objects of simple immutable types as keys in BTree-based data structures. 325 | 326 | #. Within a single BTree-based data structure, use objects of a single type as 327 | keys. Don't use multiple key types in a single structure. 328 | 329 | #. If you want to use class instances as keys, and there's any possibility that 330 | the structure may be stored in a database, it's crucial that the class define a 331 | :meth:`__cmp__` method, and that the method is carefully implemented. 332 | 333 | Any part of a comparison implementation that relies (explicitly or implicitly) 334 | on an address-based comparison result will eventually cause serious failure. 335 | 336 | #. Do not use :class:`Persistent` objects as keys, or objects of a subclass of 337 | :class:`Persistent`. 338 | 339 | That last item may be surprising. It stems from details of how conflict 340 | resolution is implemented: the states passed to conflict resolution do not 341 | materialize persistent subobjects (if a persistent object P is a key in a BTree, 342 | then P is a subobject of the bucket containing P). Instead, if an object O 343 | references a persistent subobject P directly, and O is involved in a conflict, 344 | the states passed to conflict resolution contain an instance of an internal 345 | :class:`PersistentReference` stub class everywhere O references P. Two 346 | :class:`PersistentReference` instances compare equal if and only if they 347 | "represent" the same persistent object; when they're not equal, they compare by 348 | memory address, and, as explained before, memory-based comparison must never 349 | happen in a sane persistent BTree. Note that it doesn't help in this case if 350 | your :class:`Persistent` subclass defines a sane :meth:`__cmp__` method: 351 | conflict resolution doesn't know about your class, and so also doesn't know 352 | about its :meth:`__cmp__` method. It only sees instances of the internal 353 | :class:`PersistentReference` stub class. 354 | 355 | 356 | Iteration and Mutation 357 | ^^^^^^^^^^^^^^^^^^^^^^ 358 | 359 | As with a Python dictionary or list, you should not mutate a BTree-based data 360 | structure while iterating over it, except that it's fine to replace the value 361 | associated with an existing key while iterating. You won't create internal 362 | damage in the structure if you try to remove, or add new keys, while iterating, 363 | but the results are undefined and unpredictable. A weak attempt is made to 364 | raise :exc:`RuntimeError` if the size of a BTree-based structure changes while 365 | iterating, but it doesn't catch most such cases, and is also unreliable. 366 | Example:: 367 | 368 | >>> from BTrees.IIBTree import * 369 | >>> s = IISet(range(10)) 370 | >>> list(s) 371 | [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] 372 | >>> for i in s: # the output is undefined 373 | ... print i, 374 | ... s.remove(i) 375 | 0 2 4 6 8 376 | Traceback (most recent call last): 377 | File "", line 1, in ? 378 | RuntimeError: the bucket being iterated changed size 379 | >>> list(s) # this output is also undefined 380 | [1, 3, 5, 7, 9] 381 | >>> 382 | 383 | Also as with Python dictionaries and lists, the safe and predictable way to 384 | mutate a BTree-based structure while iterating over it is to iterate over a copy 385 | of the keys. Example:: 386 | 387 | >>> from BTrees.IIBTree import * 388 | >>> s = IISet(range(10)) 389 | >>> for i in list(s.keys()): # this is well defined 390 | ... print i, 391 | ... s.remove(i) 392 | 0 1 2 3 4 5 6 7 8 9 393 | >>> list(s) 394 | [] 395 | >>> 396 | 397 | 398 | BTree Diagnostic Tools 399 | ^^^^^^^^^^^^^^^^^^^^^^ 400 | 401 | A BTree (or TreeSet) is a complex data structure, really a graph of variable- 402 | size nodes, connected in multiple ways via three distinct kinds of C pointers. 403 | There are some tools available to help check internal consistency of a BTree as 404 | a whole. 405 | 406 | Most generally useful is the :mod:`BTrees.check` module. The 407 | :func:`check.check` function examines a BTree (or Bucket, Set, or TreeSet) for 408 | value-based consistency, such as that the keys are in strictly increasing order. 409 | See the function docstring for details. The :func:`check.display` function 410 | displays the internal structure of a BTree. 411 | 412 | BTrees and TreeSets also have a :meth:`_check` method. This verifies that the 413 | (possibly many) internal pointers in a BTree or TreeSet are mutually consistent, 414 | and raises :exc:`AssertionError` if they're not. 415 | 416 | If a :func:`check.check` or :meth:`_check` call fails, it may point to a bug in 417 | the implementation of BTrees or conflict resolution, or may point to database 418 | corruption. 419 | 420 | Repairing a damaged BTree is usually best done by making a copy of it. For 421 | example, if *self.data* is bound to a corrupted IOBTree, :: 422 | 423 | self.data = IOBTree(self.data) 424 | 425 | usually suffices. If object identity needs to be preserved, :: 426 | 427 | acopy = IOBTree(self.data) 428 | self.data.clear() 429 | self.data.update(acopy) 430 | 431 | does the same, but leaves *self.data* bound to the same object. 432 | 433 | -------------------------------------------------------------------------------- /documentation/guide/prog-zodb.rst: -------------------------------------------------------------------------------- 1 | .. % ZODB Programming 2 | .. % How ZODB works (ExtensionClass, dirty bits) 3 | .. % Installing ZODB 4 | .. % Rules for Writing Persistent Classes 5 | 6 | 7 | ZODB Programming 8 | ================ 9 | 10 | 11 | Installing ZODB 12 | --------------- 13 | 14 | ZODB is packaged using the standard distutils tools. 15 | 16 | 17 | Requirements 18 | ^^^^^^^^^^^^ 19 | 20 | You will need Python 2.3 or higher. Since the code is packaged using distutils, 21 | it is simply a matter of untarring or unzipping the release package, and then 22 | running ``python setup.py install``. 23 | 24 | You'll need a C compiler to build the packages, because there are various C 25 | extension modules. Binary installers are provided for Windows users. 26 | 27 | 28 | Installing the Packages 29 | ^^^^^^^^^^^^^^^^^^^^^^^ 30 | 31 | Download the ZODB tarball containing all the packages for both ZODB and ZEO from 32 | ``_. See the :file:`README.txt` file in 33 | the top level of the release directory for details on building, testing, and 34 | installing. 35 | 36 | You can find information about ZODB and the most current releases in the ZODB 37 | Wiki at ``_. 38 | 39 | 40 | How ZODB Works 41 | -------------- 42 | 43 | The ZODB is conceptually simple. Python classes subclass a 44 | :class:`persistent.Persistent` class to become ZODB-aware. Instances of 45 | persistent objects are brought in from a permanent storage medium, such as a 46 | disk file, when the program needs them, and remain cached in RAM. The ZODB 47 | traps modifications to objects, so that when a statement such as ``obj.size = 48 | 1`` is executed, the modified object is marked as "dirty." On request, any 49 | dirty objects are written out to permanent storage; this is called committing a 50 | transaction. Transactions can also be aborted or rolled back, which results in 51 | any changes being discarded, dirty objects reverting to their initial state 52 | before the transaction began. 53 | 54 | The term "transaction" has a specific technical meaning in computer science. 55 | It's extremely important that the contents of a database don't get corrupted by 56 | software or hardware crashes, and most database software offers protection 57 | against such corruption by supporting four useful properties, Atomicity, 58 | Consistency, Isolation, and Durability. In computer science jargon these four 59 | terms are collectively dubbed the ACID properties, forming an acronym from their 60 | names. 61 | 62 | The ZODB provides all of the ACID properties. Definitions of the ACID 63 | properties are: 64 | 65 | Atomicity 66 | means that any changes to data made during a transaction are all-or-nothing. 67 | Either all the changes are applied, or none of them are. If a program makes a 68 | bunch of modifications and then crashes, the database won't be partially 69 | modified, potentially leaving the data in an inconsistent state; instead all the 70 | changes will be forgotten. That's bad, but it's better than having a partially- 71 | applied modification put the database into an inconsistent state. 72 | 73 | Consistency 74 | means that each transaction executes a valid transformation of the database 75 | state. Some databases, but not ZODB, provide a variety of consistency checks in 76 | the database or language; for example, a relational database constraint columns 77 | to be of particular types and can enforce relations across tables. Viewed more 78 | generally, atomicity and isolation make it possible for applications to provide 79 | consistency. 80 | 81 | Isolation 82 | means that two programs or threads running in two different transactions cannot 83 | see each other's changes until they commit their transactions. 84 | 85 | Durability 86 | means that once a transaction has been committed, a subsequent crash will not 87 | cause any data to be lost or corrupted. 88 | 89 | 90 | Opening a ZODB 91 | -------------- 92 | 93 | There are 3 main interfaces supplied by the ZODB: :class:`Storage`, :class:`DB`, 94 | and :class:`Connection` classes. The :class:`DB` and :class:`Connection` 95 | interfaces both have single implementations, but there are several different 96 | classes that implement the :class:`Storage` interface. 97 | 98 | * :class:`Storage` classes are the lowest layer, and handle storing and 99 | retrieving objects from some form of long-term storage. A few different types of 100 | Storage have been written, such as :class:`FileStorage`, which uses regular disk 101 | files, and :class:`BDBFullStorage`, which uses Sleepycat Software's BerkeleyDB 102 | database. You could write a new Storage that stored objects in a relational 103 | database, for example, if that would better suit your application. Two example 104 | storages, :class:`DemoStorage` and :class:`MappingStorage`, are available to use 105 | as models if you want to write a new Storage. 106 | 107 | * The :class:`DB` class sits on top of a storage, and mediates the interaction 108 | between several connections. One :class:`DB` instance is created per process. 109 | 110 | * Finally, the :class:`Connection` class caches objects, and moves them into and 111 | out of object storage. A multi-threaded program should open a separate 112 | :class:`Connection` instance for each thread. Different threads can then modify 113 | objects and commit their modifications independently. 114 | 115 | Preparing to use a ZODB requires 3 steps: you have to open the :class:`Storage`, 116 | then create a :class:`DB` instance that uses the :class:`Storage`, and then get 117 | a :class:`Connection` from the :class:`DB instance`. All this is only a few 118 | lines of code:: 119 | 120 | from ZODB import FileStorage, DB 121 | 122 | storage = FileStorage.FileStorage('/tmp/test-filestorage.fs') 123 | db = DB(storage) 124 | conn = db.open() 125 | 126 | Note that you can use a completely different data storage mechanism by changing 127 | the first line that opens a :class:`Storage`; the above example uses a 128 | :class:`FileStorage`. In section :ref:`zeo`, "How ZEO Works", you'll see how 129 | ZEO uses this flexibility to good effect. 130 | 131 | 132 | Using a ZODB Configuration File 133 | ------------------------------- 134 | 135 | ZODB also supports configuration files written in the ZConfig format. A 136 | configuration file can be used to separate the configuration logic from the 137 | application logic. The storages classes and the :class:`DB` class support a 138 | variety of keyword arguments; all these options can be specified in a config 139 | file. 140 | 141 | The configuration file is simple. The example in the previous section could use 142 | the following example:: 143 | 144 | 145 | 146 | path /tmp/test-filestorage.fs 147 | 148 | 149 | 150 | The :mod:`ZODB.config` module includes several functions for opening database 151 | and storages from configuration files. :: 152 | 153 | import ZODB.config 154 | 155 | db = ZODB.config.databaseFromURL('/tmp/test.conf') 156 | conn = db.open() 157 | 158 | The ZConfig documentation, included in the ZODB3 release, explains the format in 159 | detail. Each configuration file is described by a schema, by convention stored 160 | in a :file:`component.xml` file. ZODB, ZEO, zLOG, and zdaemon all have schemas. 161 | 162 | 163 | Writing a Persistent Class 164 | -------------------------- 165 | 166 | Making a Python class persistent is quite simple; it simply needs to subclass 167 | from the :class:`Persistent` class, as shown in this example:: 168 | 169 | from persistent import Persistent 170 | 171 | class User(Persistent): 172 | pass 173 | 174 | The :class:`Persistent` base class is a new-style class implemented in C. 175 | 176 | For simplicity, in the examples the :class:`User` class will simply be used as a 177 | holder for a bunch of attributes. Normally the class would define various 178 | methods that add functionality, but that has no impact on the ZODB's treatment 179 | of the class. 180 | 181 | The ZODB uses persistence by reachability; starting from a set of root objects, 182 | all the attributes of those objects are made persistent, whether they're simple 183 | Python data types or class instances. There's no method to explicitly store 184 | objects in a ZODB database; simply assign them as an attribute of an object, or 185 | store them in a mapping, that's already in the database. This chain of 186 | containment must eventually reach back to the root object of the database. 187 | 188 | As an example, we'll create a simple database of users that allows retrieving a 189 | :class:`User` object given the user's ID. First, we retrieve the primary root 190 | object of the ZODB using the :meth:`root` method of the :class:`Connection` 191 | instance. The root object behaves like a Python dictionary, so you can just add 192 | a new key/value pair for your application's root object. We'll insert an 193 | :class:`OOBTree` object that will contain all the :class:`User` objects. (The 194 | :class:`BTree` module is also included as part of Zope.) :: 195 | 196 | dbroot = conn.root() 197 | 198 | # Ensure that a 'userdb' key is present 199 | # in the root 200 | if not dbroot.has_key('userdb'): 201 | from BTrees.OOBTree import OOBTree 202 | dbroot['userdb'] = OOBTree() 203 | 204 | userdb = dbroot['userdb'] 205 | 206 | Inserting a new user is simple: create the :class:`User` object, fill it with 207 | data, insert it into the :class:`BTree` instance, and commit this transaction. 208 | :: 209 | 210 | # Create new User instance 211 | import transaction 212 | 213 | newuser = User() 214 | 215 | # Add whatever attributes you want to track 216 | newuser.id = 'amk' 217 | newuser.first_name = 'Andrew' ; newuser.last_name = 'Kuchling' 218 | ... 219 | 220 | # Add object to the BTree, keyed on the ID 221 | userdb[newuser.id] = newuser 222 | 223 | # Commit the change 224 | transaction.commit() 225 | 226 | The :mod:`transaction` module defines a few top-level functions for working with 227 | transactions. :func:`commit` writes any modified objects to disk, making the 228 | changes permanent. :func:`abort` rolls back any changes that have been made, 229 | restoring the original state of the objects. If you're familiar with database 230 | transactional semantics, this is all what you'd expect. :func:`get` returns a 231 | :class:`Transaction` object that has additional methods like :meth:`note`, to 232 | add a note to the transaction metadata. 233 | 234 | More precisely, the :mod:`transaction` module exposes an instance of the 235 | :class:`ThreadTransactionManager` transaction manager class as 236 | ``transaction.manager``, and the :mod:`transaction` functions :func:`get` and 237 | :func:`begin` redirect to the same-named methods of ``transaction.manager``. 238 | The :func:`commit` and :func:`abort` functions apply the methods of the same 239 | names to the :class:`Transaction` object returned by 240 | ``transaction.manager.get()``. This is for convenience. It's also possible to 241 | create your own transaction manager instances, and to tell ``DB.open()`` to use 242 | your transaction manager instead. 243 | 244 | Because the integration with Python is so complete, it's a lot like having 245 | transactional semantics for your program's variables, and you can experiment 246 | with transactions at the Python interpreter's prompt:: 247 | 248 | >>> newuser 249 | 250 | >>> newuser.first_name # Print initial value 251 | 'Andrew' 252 | >>> newuser.first_name = 'Bob' # Change first name 253 | >>> newuser.first_name # Verify the change 254 | 'Bob' 255 | >>> transaction.abort() # Abort transaction 256 | >>> newuser.first_name # The value has changed back 257 | 'Andrew' 258 | 259 | 260 | Rules for Writing Persistent Classes 261 | ------------------------------------ 262 | 263 | Practically all persistent languages impose some restrictions on programming 264 | style, warning against constructs they can't handle or adding subtle semantic 265 | changes, and the ZODB is no exception. Happily, the ZODB's restrictions are 266 | fairly simple to understand, and in practice it isn't too painful to work around 267 | them. 268 | 269 | The summary of rules is as follows: 270 | 271 | * If you modify a mutable object that's the value of an object's attribute, the 272 | ZODB can't catch that, and won't mark the object as dirty. The solution is to 273 | either set the dirty bit yourself when you modify mutable objects, or use a 274 | wrapper for Python's lists and dictionaries (:class:`PersistentList`, 275 | :class:`PersistentMapping`) that will set the dirty bit properly. 276 | 277 | * Recent versions of the ZODB allow writing a class with :meth:`__setattr__` , 278 | :meth:`__getattr__`, or :meth:`__delattr__` methods. (Older versions didn't 279 | support this at all.) If you write such a :meth:`__setattr__` or 280 | :meth:`__delattr__` method, its code has to set the dirty bit manually. 281 | 282 | * A persistent class should not have a :meth:`__del__` method. The database 283 | moves objects freely between memory and storage. If an object has not been used 284 | in a while, it may be released and its contents loaded from storage the next 285 | time it is used. Since the Python interpreter is unaware of persistence, it 286 | would call :meth:`__del__` each time the object was freed. 287 | 288 | Let's look at each of these rules in detail. 289 | 290 | 291 | Modifying Mutable Objects 292 | ^^^^^^^^^^^^^^^^^^^^^^^^^ 293 | 294 | The ZODB uses various Python hooks to catch attribute accesses, and can trap 295 | most of the ways of modifying an object, but not all of them. If you modify a 296 | :class:`User` object by assigning to one of its attributes, as in 297 | ``userobj.first_name = 'Andrew'``, the ZODB will mark the object as having been 298 | changed, and it'll be written out on the following :meth:`commit`. 299 | 300 | The most common idiom that *isn't* caught by the ZODB is mutating a list or 301 | dictionary. If :class:`User` objects have a attribute named ``friends`` 302 | containing a list, calling ``userobj.friends.append(otherUser)`` doesn't mark 303 | ``userobj`` as modified; from the ZODB's point of view, ``userobj.friends`` was 304 | only read, and its value, which happened to be an ordinary Python list, was 305 | returned. The ZODB isn't aware that the object returned was subsequently 306 | modified. 307 | 308 | This is one of the few quirks you'll have to remember when using the ZODB; if 309 | you modify a mutable attribute of an object in place, you have to manually mark 310 | the object as having been modified by setting its dirty bit to true. This is 311 | done by setting the :attr:`_p_changed` attribute of the object to true:: 312 | 313 | userobj.friends.append(otherUser) 314 | userobj._p_changed = True 315 | 316 | You can hide the implementation detail of having to mark objects as dirty by 317 | designing your class's API to not use direct attribute access; instead, you can 318 | use the Java-style approach of accessor methods for everything, and then set the 319 | dirty bit within the accessor method. For example, you might forbid accessing 320 | the ``friends`` attribute directly, and add a :meth:`get_friend_list` accessor 321 | and an :meth:`add_friend` modifier method to the class. :meth:`add_friend` 322 | would then look like this:: 323 | 324 | def add_friend(self, friend): 325 | self.friends.append(otherUser) 326 | self._p_changed = True 327 | 328 | Alternatively, you could use a ZODB-aware list or mapping type that handles the 329 | dirty bit for you. The ZODB comes with a :class:`PersistentMapping` class, and 330 | I've contributed a :class:`PersistentList` class that's included in my ZODB 331 | distribution, and may make it into a future upstream release of Zope. 332 | 333 | .. % XXX It'd be nice to discuss what happens when an object is ``ghosted'' (e.g. 334 | .. % you set an object's _p_changed = None). The __p_deactivate__ method should 335 | .. % not be used (it's also obsolete). 336 | 337 | 338 | :meth:`__getattr__`, :meth:`__delattr__`, and :meth:`__setattr__` 339 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 340 | 341 | ZODB allows persistent classes to have hook methods like :meth:`__getattr__` and 342 | :meth:`__setattr__`. There are four special methods that control attribute 343 | access; the rules for each are a little different. 344 | 345 | The :meth:`__getattr__` method works pretty much the same for persistent classes 346 | as it does for other classes. No special handling is needed. If an object is a 347 | ghost, then it will be activated before :meth:`__getattr__` is called. 348 | 349 | The other methods are more delicate. They will override the hooks provided by 350 | :class:`Persistent`, so user code must call special methods to invoke those 351 | hooks anyway. 352 | 353 | The :meth:`__getattribute__` method will be called for all attribute access; it 354 | overrides the attribute access support inherited from :class:`Persistent`. A 355 | user-defined :meth:`__getattribute__` must always give the :class:`Persistent` 356 | base class a chance to handle special attribute, as well as :attr:`__dict__` or 357 | :attr:`__class__`. The user code should call :meth:`_p_getattr`, passing the 358 | name of the attribute as the only argument. If it returns True, the user code 359 | should call :class:`Persistent`'s :meth:`__getattribute__` to get the value. If 360 | not, the custom user code can run. 361 | 362 | A :meth:`__setattr__` hook will also override the :class:`Persistent` 363 | :meth:`__setattr__` hook. User code must treat it much like 364 | :meth:`__getattribute__`. The user-defined code must call :meth:`_p_setattr` 365 | first to all :class:`Persistent` to handle special attributes; 366 | :meth:`_p_setattr` takes the attribute name and value. If it returns True, 367 | :class:`Persistent` handled the attribute. If not, the user code can run. If 368 | the user code modifies the object's state, it must assigned to 369 | :attr:`_p_changed`. 370 | 371 | A :meth:`__delattr__` hooks must be implemented the same was as a the last two 372 | hooks. The user code must call :meth:`_p_delattr`, passing the name of the 373 | attribute as an argument. If the call returns True, :class:`Persistent` handled 374 | the attribute; if not, the user code can run. 375 | 376 | 377 | :meth:`__del__` methods 378 | ^^^^^^^^^^^^^^^^^^^^^^^ 379 | 380 | A :meth:`__del__` method is invoked just before the memory occupied by an 381 | unreferenced Python object is freed. Because ZODB may materialize, and 382 | dematerialize, a given persistent object in memory any number of times, there 383 | isn't a meaningful relationship between when a persistent object's 384 | :meth:`__del__` method gets invoked and any natural aspect of a persistent 385 | object's life cycle. For example, it is emphatically not the case that a 386 | persistent object's :meth:`__del__` method gets invoked only when the object is 387 | no longer referenced by other objects in the database. :meth:`__del__` is only 388 | concerned with reachability from objects in memory. 389 | 390 | Worse, a :meth:`__del__` method can interfere with the persistence machinery's 391 | goals. For example, some number of persistent objects reside in a 392 | :class:`Connection`'s memory cache. At various times, to reduce memory burden, 393 | objects that haven't been referenced recently are removed from the cache. If a 394 | persistent object with a :meth:`__del___` method is so removed, and the cache 395 | was holding the last memory reference to the object, the object's 396 | :meth:`__del__` method will be invoked. If the :meth:`__del__` method then 397 | references any attribute of the object, ZODB needs to load the object from the 398 | database again, in order to satisfy the attribute reference. This puts the 399 | object back into the cache again: such an object is effectively immortal, 400 | occupying space in the memory cache forever, as every attempt to remove it from 401 | cache puts it back into the cache. In ZODB versions prior to 3.2.2, this could 402 | even cause the cache reduction code to fall into an infinite loop. The infinite 403 | loop no longer occurs, but such objects continue to live in the memory cache 404 | forever. 405 | 406 | Because :meth:`__del__` methods don't make good sense for persistent objects, 407 | and can create problems, persistent classes should not define :meth:`__del__` 408 | methods. 409 | 410 | 411 | Writing Persistent Classes 412 | -------------------------- 413 | 414 | Now that we've looked at the basics of programming using the ZODB, we'll turn to 415 | some more subtle tasks that are likely to come up for anyone using the ZODB in a 416 | production system. 417 | 418 | 419 | Changing Instance Attributes 420 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 421 | 422 | Ideally, before making a class persistent you would get its interface right the 423 | first time, so that no attributes would ever need to be added, removed, or have 424 | their interpretation change over time. It's a worthy goal, but also an 425 | impractical one unless you're gifted with perfect knowledge of the future. Such 426 | unnatural foresight can't be required of any person, so you therefore have to be 427 | prepared to handle such structural changes gracefully. In object-oriented 428 | database terminology, this is a schema update. The ZODB doesn't have an actual 429 | schema specification, but you're changing the software's expectations of the 430 | data contained by an object, so you're implicitly changing the schema. 431 | 432 | One way to handle such a change is to write a one-time conversion program that 433 | will loop over every single object in the database and update them to match the 434 | new schema. This can be easy if your network of object references is quite 435 | structured, making it easy to find all the instances of the class being 436 | modified. For example, if all :class:`User` objects can be found inside a 437 | single dictionary or BTree, then it would be a simple matter to loop over every 438 | :class:`User` instance with a :keyword:`for` statement. This is more difficult 439 | if your object graph is less structured; if :class:`User` objects can be found 440 | as attributes of any number of different class instances, then there's no longer 441 | any easy way to find them all, short of writing a generalized object traversal 442 | function that would walk over every single object in a ZODB, checking each one 443 | to see if it's an instance of :class:`User`. 444 | 445 | Some OODBs support a feature called extents, which allow quickly finding all the 446 | instances of a given class, no matter where they are in the object graph; 447 | unfortunately the ZODB doesn't offer extents as a feature. 448 | 449 | .. % XXX Rest of section not written yet: __getstate__/__setstate__ 450 | 451 | --------------------------------------------------------------------------------