├── .hgignore ├── .travis.yml ├── CHANGELOG.txt ├── DESIGN.txt ├── README.rst ├── doc ├── conf.py ├── index.rst ├── sqlalchemy.rst └── theory.rst ├── setup.cfg ├── setup.py └── vdm ├── __init__.py └── sqlalchemy ├── __init__.py ├── base.py ├── demo.py ├── demo_meta.py ├── demo_simple.py ├── sqla.py ├── stateful.py ├── test_demo.py ├── test_demo_misc.py ├── test_stateful.py ├── test_stateful_collections.py ├── test_tools.py └── tools.py /.hgignore: -------------------------------------------------------------------------------- 1 | syntax: glob 2 | *.pyc 3 | *.egg-info/* 4 | build/* 5 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | python: 3 | - "2.7" 4 | install: 5 | - pip install nose 6 | - pip install sqlalchemy==1.1 7 | - pip install psycopg2==2.4.5 8 | - python setup.py develop 9 | - sudo -u postgres psql -c "CREATE USER ckan_default WITH PASSWORD 'pass';" 10 | - sudo -u postgres psql -c 'CREATE DATABASE vdmtest WITH OWNER ckan_default;' 11 | script: nosetests vdm/sqlalchemy 12 | -------------------------------------------------------------------------------- /CHANGELOG.txt: -------------------------------------------------------------------------------- 1 | HEAD 2 | ==== 3 | 4 | v0.13 2014-08-12 5 | ================ 6 | * Support for SQLAlchemy 0.9 7 | 8 | v0.12 2014-01-20 9 | ================ 10 | * Support for SQLAlchemy 0.8 11 | 12 | v0.10 2011-10-26 13 | ================ 14 | * Support for SQLAlchemy 0.7 15 | * Fix timestamps to be UTC. 16 | 17 | v0.9 2010-01-11 18 | =============== 19 | 20 | * Support for SQLAlchemy 0.6 and SQLite (in addition to PostgreSQL) (#901) 21 | * Minor bugfixes (#898 #899 #900) 22 | * Remove (long-deprecated) SQLObject code from source tree 23 | 24 | v0.8 2010-01-10 25 | =============== 26 | * Recalled 2010-01-11 27 | 28 | v0.7 2010-05-11 29 | =============== 30 | 31 | * Support for SQLALchemy v0.5 (ticket:81) 32 | * Improved diff support: diff on object creation (ticket:267) and reorganised 33 | diff interface to allow diffing of related domain objects 34 | * Minor bugfixes (cset:984d053fc15a/vdm, cset:0a7d889c5c1a/vdm) 35 | 36 | v0.6 2010-01-25 37 | =============== 38 | 39 | * Support for non-integer primary keys 40 | * Diffing revisioned/versioned objects via diff(to_revision, from_revision) method 41 | * **BREAKING CHANGE**: revision id changed to UUID from integer 42 | * For an example upgrade script for client systems see 43 | http://knowledgeforge.net/ckan/hg/file/tip/ckan/migration/versions/008_update_vdm_ids.py 44 | * **BREAKING CHANGE**: Convert State from object to an enumeration 45 | * For an example upgrade script for client systems see 46 | http://knowledgeforge.net/ckan/hg/file/d3a25bb4eb1b/ckan/migration/versions/015_remove_state_object.py 47 | * Various minor bugfixes and improvements especially to `Repository` 48 | 49 | v0.5 2009-09-17 50 | =============== 51 | 52 | (All for vdm.sqlalchemy) 53 | 54 | * Support stateful dict-like collections (already support list-like ones) 55 | * Identity-map for stateful lists which eliminates subtle problems when 56 | adding "existing" objects to a stateful list (or moving them within the 57 | list) 58 | 59 | v0.4 2009-04-10 60 | =============== 61 | 62 | (All for vdm.sqlalchemy) 63 | 64 | * New Repository object (tools.py) to encapsulate repo-wide operations 65 | * Repo create/clean/init operations 66 | * Purge revision functionality (r408) 67 | * List all objects changed in a revision (r408) 68 | * Put stateful system under proper test and fix several major bugs (r418) 69 | 70 | v0.3 2008-10-31 71 | =============== 72 | 73 | * No substantial new features but lots of bugfixes 74 | * Better compatability with SQLAlchemy v0.5 75 | * Revision has several new convenience methods and utilizes State 76 | 77 | v0.2 2008-05-30 78 | =============== 79 | 80 | * SQLAlchemy implementation of versioned domain model (major) 81 | * Customized to sqlalchemy with major performance improvements 82 | * Utilizes 'cache-head-on-continuity' model 83 | * Partial Elixir implementation. 84 | * Various (minor) bugfixes and improvments for SQLObject version 85 | * Greatly improved documentation 86 | * See announce: 87 | 88 | v0.1 2007-02 89 | ============ 90 | 91 | * Fully functional SQLObject implementation of versioned domain model. 92 | * Support for basic versioned objects 93 | * Support for m2m 94 | * Atomic commits and Revision object 95 | -------------------------------------------------------------------------------- /DESIGN.txt: -------------------------------------------------------------------------------- 1 | For an introduction to vdm and a basic overview of its design please see the 2 | main package docstring: vdm/__init__.py. 3 | 4 | ## Issues -- 2007-12-23 5 | 6 | (in rough order of priority) 7 | 8 | ### 1. Support for versioned many to many 9 | 10 | ### 2. Model traversal (for 'old' objects) works correctly 11 | 12 | * this is very closely related to versioned many to many 13 | * minor subissue: ensure one gets continuity object not a 'version' 14 | when you do changes 15 | * in terms of use case below what happens if we do: 16 | some_other_object.movie = m1r1 17 | * not a problem if you can only do changes using HEAD 18 | 19 | ### Support delete and undeleting (State) 20 | 21 | Current versioning model despite being versioned does not allow deleting 22 | and undeleting of objects. 23 | 24 | ### 4. Conflicts, merging and locking 25 | 26 | Once we have versioning we have conflicts, merging and locking ... 27 | 28 | ### 5. Revisions should be 'hidden' -- i.e. created automatically 29 | 30 | Decision: wontfix 31 | 32 | Why: 33 | 34 | * problem with this is where do you set up the revision itself (e.g. 35 | log_message, author) 36 | * thus think it is inevitable that you create some kind of Revision 37 | object, however no need to have to call commit on revision (just use 38 | session.flush stuff) 39 | * simplest approach is simply to require the user to create a revision 40 | and attach it to session before calling flush (if absolutely require 41 | could provide a backup where this revision is auto created if 42 | nonexistent) 43 | * alternative is to override flush on extension to take an argument 44 | (or have a SessionExtension and override before_flush in some way) 45 | -- but don't like this much 46 | * TODO: without explicit call to commit how do we set timestamp on 47 | revision -- could just let it be as it was when revision was 48 | created or just override before_update on Revision 49 | 50 | ## Analysis and Use Cases 51 | 52 | Here's the basis use case setup we will refer to below: 53 | 54 | class Movie(Entity): 55 | id = Field(Integer, primary_key=True) 56 | title = Field(String(60), primary_key=True) 57 | description = Field(String(512)) 58 | ignoreme = Field(Integer, default=0) 59 | director = ManyToOne('Director', inverse='movies') 60 | actors = ManyToMany('Genre', inverse='movies', 61 | tablename='movie_2_genre') 62 | using_options(tablename='movies') 63 | acts_as_versioned(ignore=['ignoreme']) 64 | 65 | class Director(Entity): 66 | name = Field(String(60)) 67 | movies = OneToMany('Movie', inverse='director') 68 | using_options(tablename='directors') 69 | 70 | class Genre(Entity): 71 | name = Field(String(60)) 72 | movies = ManyToMany('Movie', inverse='genres', 73 | tablename='movie_2_genre') 74 | using_options(tablename='actors') 75 | 76 | 77 | ### Versioned Many to Many 78 | 79 | This is fairly trivial conceptually: just turn the intermediate object 80 | in a many to many into a versioned object in its own right but also 81 | adding a state attribute. 82 | 83 | I.e. we implement ManyToMany explicitly using: 84 | 85 | class Movie2Genre 86 | movie = ManyToOne('Movie') 87 | actor = ManyToOne('Actor') 88 | state = Field(Integer) 89 | acts_as_versioned() 90 | 91 | and the association_proxy provided by sqlalchemy. 92 | 93 | However to get this to work 'nicely' with elixir involves: 94 | 95 | * setting up a new m2m property which is aware of state 96 | * proper model traversal (o/w we still have the problem that we only 97 | ever get the HEAD value for movie.genres even when using an old 98 | version of movie). 99 | 100 | 101 | ### Model Traversal 102 | 103 | # revision 1 104 | m1 = Movie(id=1) 105 | d1 = Director(id=1, name='Blogs') 106 | m1.director = d1 107 | flush() 108 | ts1 = datetime.now() 109 | 110 | # revision 2 111 | m1 = Movie.get(1) 112 | d1 = Director.get(2) 113 | d1.name = 'Jones' 114 | g1 = Genre(...) 115 | d2 = Director(...) 116 | m1.genres.append(g1) 117 | m1.director = d2 118 | flush() 119 | ts2 = datetime.now() 120 | # basic domain model traversal (requires remembering revision/timestamp) 121 | # in elixir 122 | # assert m1r1.director.name == 'Jones' 123 | assert m1r1.director.name == 'Blogs' 124 | 125 | 126 | m1 = Movie.get(1) 127 | m1r1 = m1.get_as_of(ts1) 128 | assert len(m1r1.genres) == 0 129 | # in elixir this results in an error as m1r1 does not have attribute genres 130 | # (or any m2m versioning support) 131 | # with broken traversal but m2m attribute we get 132 | # assert len(m1r1.genres) == 1 133 | # note that traversal would include proper m2m versioning 134 | 135 | #### The Issue 136 | 137 | How do we pass around information about what the current reference 138 | timestamp/revision this is because doing many to many involves 139 | *traversing* the domain model i.e. we have moved from the Movie object 140 | to the implicit Movie2Genre object and then on to the Genre object. 141 | 142 | The key question in resolving this is deciding what we get back when we 143 | do: 144 | 145 | m1r1 = m1.get_as_of(ts1) 146 | 147 | Solution 1 148 | 149 | At present the elixir approach is that this returns a MovieVersion 150 | object appropriate at ts1. The problem with this is that: 151 | 152 | 1. This does not behave like the continuity object (i.e. Movie) in 153 | important respects most notably in terms of 'special' properties such as 154 | m2m lists (and any other properties you've specially added). 155 | 156 | 2. Even if it did it would be unclear what the m2m links would work with 157 | (i.e. point to)? Should they point to GenreVersion or to Genre (more 158 | explicitly should we have MovieVersion2GenreVersion objects or 159 | MovieVersion2Genre objects or ...) 160 | 161 | To put this formally what happens when one does: 162 | 163 | # does this return d1r1 or just d1 in elixir this returns d1 164 | thedirector = m1r1.director 165 | # again do we get g1 or g1r1 and what list do we get (as it was at 166 | # r1 or now) 167 | thegenres = m1r1.genres 168 | # and this really gets difficult if genres were to have some foreign 169 | # key or even another m2m 170 | mylist = m1r1.genres.some_other_m2m 171 | 172 | Solution 2 173 | 174 | m1r1 is m1 (i.e. the continuity object) but with some information 175 | telling it to return information on attribute calls as if it was at ts1. 176 | [This was approach taken with sqlobject code]. 177 | 178 | However this is problematic because it means overriding all property 179 | read accesses to ensure (if necessary) the call is passed down to the 180 | relevant history object. 181 | 182 | To be continued ... 183 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | .. image:: https://travis-ci.org/okfn/vdm.svg?branch=master 2 | :target: https://travis-ci.org/okfn/vdm 3 | 4 | To install do:: 5 | 6 | $ easy_install vdm 7 | 8 | Or checkout from our git repository:: 9 | 10 | $ git clone https://github.com/okfn/vdm 11 | 12 | For more information see the main package docstring. To view this either just 13 | open vdm/__init__.py or do (after installation):: 14 | 15 | $ pydoc vdm 16 | 17 | 18 | For Developers 19 | ============== 20 | 21 | Tests currently pass against postgres or sqlite (see 'TEST_ENGINE' setting 22 | in vdm/sqlalchemy/demo.py). 23 | 24 | To run tests with postgres you will need to have a set up a postgresql 25 | database with user 'tester' and password 'pass' (see settings in 26 | vdm/sqlalchemy/demo.py). 27 | 28 | Run the tests using nosetests:: 29 | 30 | $ nosetests vdm/sqlalchemy 31 | 32 | -------------------------------------------------------------------------------- /doc/conf.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # Versioned Domain Model documentation build configuration file, created by 4 | # sphinx-quickstart on Wed Aug 4 08:56:11 2010. 5 | # 6 | # This file is execfile()d with the current directory set to its containing dir. 7 | # 8 | # The contents of this file are pickled, so don't put values in the namespace 9 | # that aren't pickleable (module imports are okay, they're removed automatically). 10 | # 11 | # Note that not all possible configuration values are present in this 12 | # autogenerated file. 13 | # 14 | # All configuration values have a default; values that are commented out 15 | # serve to show the default. 16 | 17 | import sys, os 18 | 19 | # If your extensions (or modules documented by autodoc) are in another directory, 20 | # add these directories to sys.path here. If the directory is relative to the 21 | # documentation root, use os.path.abspath to make it absolute, like shown here. 22 | #sys.path.append(os.path.abspath('.')) 23 | 24 | # General configuration 25 | # --------------------- 26 | 27 | # Add any Sphinx extension module names here, as strings. They can be extensions 28 | # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. 29 | extensions = ['sphinx.ext.autodoc', 'sphinx.ext.intersphinx'] 30 | 31 | # Add any paths that contain templates here, relative to this directory. 32 | templates_path = ['.templates'] 33 | 34 | # The suffix of source filenames. 35 | source_suffix = '.rst' 36 | 37 | # The encoding of source files. 38 | #source_encoding = 'utf-8' 39 | 40 | # The master toctree document. 41 | master_doc = 'index' 42 | 43 | # General information about the project. 44 | project = u'Versioned Domain Model' 45 | copyright = u'2010, Rufus Pollock (Open Knowledge Foundation)' 46 | 47 | # The version info for the project you're documenting, acts as replacement for 48 | # |version| and |release|, also used in various other places throughout the 49 | # built documents. 50 | # 51 | # The short X.Y version. 52 | version = '0.11' 53 | # The full version, including alpha/beta/rc tags. 54 | release = '0.11' 55 | 56 | # The language for content autogenerated by Sphinx. Refer to documentation 57 | # for a list of supported languages. 58 | #language = None 59 | 60 | # There are two options for replacing |today|: either, you set today to some 61 | # non-false value, then it is used: 62 | #today = '' 63 | # Else, today_fmt is used as the format for a strftime call. 64 | #today_fmt = '%B %d, %Y' 65 | 66 | # List of documents that shouldn't be included in the build. 67 | #unused_docs = [] 68 | 69 | # List of directories, relative to source directory, that shouldn't be searched 70 | # for source files. 71 | exclude_trees = [] 72 | 73 | # The reST default role (used for this markup: `text`) to use for all documents. 74 | #default_role = None 75 | 76 | # If true, '()' will be appended to :func: etc. cross-reference text. 77 | #add_function_parentheses = True 78 | 79 | # If true, the current module name will be prepended to all description 80 | # unit titles (such as .. function::). 81 | #add_module_names = True 82 | 83 | # If true, sectionauthor and moduleauthor directives will be shown in the 84 | # output. They are ignored by default. 85 | #show_authors = False 86 | 87 | # The name of the Pygments (syntax highlighting) style to use. 88 | pygments_style = 'sphinx' 89 | 90 | 91 | # Options for HTML output 92 | # ----------------------- 93 | 94 | # The style sheet to use for HTML and HTML Help pages. A file of that name 95 | # must exist either in Sphinx' static/ path, or in one of the custom paths 96 | # given in html_static_path. 97 | html_style = 'default.css' 98 | 99 | # The name for this set of Sphinx documents. If None, it defaults to 100 | # " v documentation". 101 | #html_title = None 102 | 103 | # A shorter title for the navigation bar. Default is the same as html_title. 104 | #html_short_title = None 105 | 106 | # The name of an image file (relative to this directory) to place at the top 107 | # of the sidebar. 108 | #html_logo = None 109 | 110 | # The name of an image file (within the static path) to use as favicon of the 111 | # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 112 | # pixels large. 113 | #html_favicon = None 114 | 115 | # Add any paths that contain custom static files (such as style sheets) here, 116 | # relative to this directory. They are copied after the builtin static files, 117 | # so a file named "default.css" will overwrite the builtin "default.css". 118 | html_static_path = ['.static'] 119 | 120 | # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, 121 | # using the given strftime format. 122 | #html_last_updated_fmt = '%b %d, %Y' 123 | 124 | # If true, SmartyPants will be used to convert quotes and dashes to 125 | # typographically correct entities. 126 | #html_use_smartypants = True 127 | 128 | # Custom sidebar templates, maps document names to template names. 129 | #html_sidebars = {} 130 | 131 | # Additional templates that should be rendered to pages, maps page names to 132 | # template names. 133 | #html_additional_pages = {} 134 | 135 | # If false, no module index is generated. 136 | #html_use_modindex = True 137 | 138 | # If false, no index is generated. 139 | #html_use_index = True 140 | 141 | # If true, the index is split into individual pages for each letter. 142 | #html_split_index = False 143 | 144 | # If true, the reST sources are included in the HTML build as _sources/. 145 | #html_copy_source = True 146 | 147 | # If true, an OpenSearch description file will be output, and all pages will 148 | # contain a tag referring to it. The value of this option must be the 149 | # base URL from which the finished HTML is served. 150 | #html_use_opensearch = '' 151 | 152 | # If nonempty, this is the file name suffix for HTML files (e.g. ".xhtml"). 153 | #html_file_suffix = '' 154 | 155 | # Output file base name for HTML help builder. 156 | htmlhelp_basename = 'VersionedDomainModeldoc' 157 | 158 | 159 | # Options for LaTeX output 160 | # ------------------------ 161 | 162 | # The paper size ('letter' or 'a4'). 163 | #latex_paper_size = 'letter' 164 | 165 | # The font size ('10pt', '11pt' or '12pt'). 166 | #latex_font_size = '10pt' 167 | 168 | # Grouping the document tree into LaTeX files. List of tuples 169 | # (source start file, target name, title, author, document class [howto/manual]). 170 | latex_documents = [ 171 | ('index', 'VersionedDomainModel.tex', ur'Versioned Domain Model Documentation', 172 | ur'Rufus Pollock (Open Knowledge Foundation)', 'manual'), 173 | ] 174 | 175 | # The name of an image file (relative to this directory) to place at the top of 176 | # the title page. 177 | #latex_logo = None 178 | 179 | # For "manual" documents, if this is true, then toplevel headings are parts, 180 | # not chapters. 181 | #latex_use_parts = False 182 | 183 | # Additional stuff for the LaTeX preamble. 184 | #latex_preamble = '' 185 | 186 | # Documents to append as an appendix to all manuals. 187 | #latex_appendices = [] 188 | 189 | # If false, no module index is generated. 190 | #latex_use_modindex = True 191 | 192 | 193 | # Example configuration for intersphinx: refer to the Python standard library. 194 | intersphinx_mapping = {'http://docs.python.org/dev': None} 195 | -------------------------------------------------------------------------------- /doc/index.rst: -------------------------------------------------------------------------------- 1 | Welcome to Versioned Domain Model's documentation! 2 | ================================================== 3 | 4 | Contents: 5 | 6 | .. toctree:: 7 | :maxdepth: 2 8 | 9 | sqlalchemy 10 | theory 11 | 12 | .. automodule:: vdm 13 | 14 | Indices and tables 15 | ================== 16 | 17 | * :ref:`genindex` 18 | * :ref:`modindex` 19 | * :ref:`search` 20 | 21 | -------------------------------------------------------------------------------- /doc/sqlalchemy.rst: -------------------------------------------------------------------------------- 1 | ========================= 2 | Using VDM with SQLAlchemy 3 | ========================= 4 | 5 | The Revision object is used to encapsulate changes to the domain model/database. It also allows changes to multiple objects/rows to be part of a single 'revision': 6 | 7 | .. autoclass:: vdm.sqlalchemy.Revision 8 | 9 | .. autoclass:: vdm.sqlalchemy.Revisioner 10 | 11 | .. autofunction:: vdm.sqlalchemy.modify_base_object_mapper 12 | 13 | .. autofunction:: vdm.sqlalchemy.add_stateful_m2m 14 | 15 | Example 16 | ======= 17 | 18 | Here is a full demonstration of using vdm which can be found in vdm/sqlalchemy/demo.py: 19 | 20 | .. literalinclude:: ../vdm/sqlalchemy/demo.py 21 | 22 | -------------------------------------------------------------------------------- /doc/theory.rst: -------------------------------------------------------------------------------- 1 | ==================================================== 2 | Versioning / Revisioning for Domain Models: Concepts 3 | ==================================================== 4 | 5 | There are several ways to *implement* revisioning (versioning) of domain model (and DBs/data generally): 6 | 7 | * Copy on write - so one has a 'full' copy of the model/DB at each version. 8 | * Diffs: store diffs between versions (plus, usually, a full version of the model at a given point in time e.g. store HEAD) 9 | 10 | In both cases one will usually want an explicit Revision/Changeset object to which : 11 | 12 | * timestamp 13 | * author of change 14 | * log message 15 | 16 | In more complex revisioning models this metadata may also be used to store key data relevant to the revisioning structure (e.g. revision parents) 17 | 18 | 19 | Copy on write 20 | ============= 21 | 22 | In its simplest form copy-on-write (CoW) would copy entire DB on each change. However, this is cleary very inefficient and hence one usually restricts the copy-on-write to relevant changed "objects". The advantage of doing this is that it limits the the changes we have to store (in essence objects unchanged between revision X and revision Y get "merged" into a single object). 23 | 24 | For example, if our domain model had Person, Address, Job, a change to Person X would only require a copy of Person X record (an even more standard example is wiki pages). Obviously, for this to work, one needs to able to partition the data (domain model). With normal domain model this is trivial: pick the object types e.g. Person, Address, Job etc. However, for a graph setup (as with RDF) this is not so trivial. 25 | 26 | Why? In essence, for copy on write to work we need: 27 | 28 | a) a way to reference entities/records 29 | b) support for putting objects in a deleted state 30 | 31 | The (RDF) graph model has poor way for referencing triples (we could use named graphs, quads or reification but none are great). We could move to the object level and only work with groups of triples (e.g. those corresponding to a "Person"). You'd also need to add a state triple to every base entity (be that a triple or named graph) and add that to every query statement. This seems painful. 32 | 33 | Diffs 34 | ===== 35 | 36 | The diff models involves computing diffs (forward or backward) for each change. A given version of the model is then computed by composing diffs. 37 | 38 | Usually for performance reasons full representations of the model/DB at a given version are cached -- most commonly HEAD is kept available. It is also possible to cache more frequently and, like copy-on-write, to cache selectively (i.e. only cache items which have change since the last cache period). 39 | 40 | The disadvantage of the diff model is the need (and cost) of creating and composing diffs (CoW is, generally, easier to implement and use). However, it is more efficient in storage terms and works better with general data (one can always compute diffs), especially that which doesn't have such a clear domain model -- e.g. the RDF case discussed above. 41 | 42 | Usage 43 | ===== 44 | 45 | * Wikis: Many wikis implement a full copy-on-write model with a full copy of each page being made on each write. 46 | * Source control: diff model (usually with HEAD cached and backwards diffs) 47 | * vdm: copy-on-write using SQL tables as core 'domain objects' 48 | * ordf (http://packages.python.org/ordf): (RDF) diffs (with HEAD caching) 49 | 50 | Todo 51 | ==== 52 | 53 | Discuss application of tree algorithms to structured data (such as XML). 54 | 55 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/okfn/vdm/ebe7b58da843770b5bc39391517b37b8aa598497/setup.cfg -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup, find_packages 2 | 3 | from vdm import __version__ 4 | from vdm import __description__ 5 | from vdm import __doc__ as __long_description__ 6 | 7 | setup( 8 | name = 'vdm', 9 | version = __version__, 10 | packages = find_packages(), 11 | install_requires = [ ], 12 | 13 | # metadata for upload to PyPI 14 | author = "Rufus Pollock (Open Knowledge Foundation)", 15 | author_email = "info@okfn.org", 16 | description = __description__, 17 | long_description = __long_description__, 18 | license = "MIT", 19 | keywords = "versioning sqlobject sqlalchemy orm", 20 | url = "http://www.okfn.org/vdm/", 21 | download_url = "https://github.com/okfn/vdm", 22 | zip_safe = False, 23 | classifiers = [ 24 | 'Development Status :: 4 - Beta', 25 | 'Environment :: Console', 26 | 'Intended Audience :: Developers', 27 | 'License :: OSI Approved :: MIT License', 28 | 'Operating System :: OS Independent', 29 | 'Programming Language :: Python', 30 | 'Topic :: Software Development :: Libraries :: Python Modules'], 31 | ) 32 | -------------------------------------------------------------------------------- /vdm/__init__.py: -------------------------------------------------------------------------------- 1 | '''About 2 | ===== 3 | 4 | Versioned Domain Model (vdm) is a package which allows you to 'version' your 5 | domain model in the same way that source code version control systems such as 6 | subversion allow you version your code. In particular, versioned domain model 7 | versions a complete model and not just individual domain objects (for more on 8 | this distinction see below). 9 | 10 | At present the package is provided as an extension to SQLAlchemy (tested 11 | against v0.4-v0.8). 12 | 13 | The library is pretty stable and has been used by the authors in production 14 | systems since v0.2 (May 2008). 15 | 16 | 17 | Copyright and License 18 | ===================== 19 | 20 | Copyright (c) 2007-2010 The Open Knowledge Foundation 21 | 22 | Licensed under the MIT license: http://www.opensource.org/licenses/mit-license.php 23 | 24 | 25 | Authors 26 | ======= 27 | 28 | Rufus Pollock (Open Knowledge Foundation) - http://rufuspollock.org/ 29 | http://www.okfn.org/ 30 | 31 | 32 | A Full Versioned Domain Model 33 | ============================= 34 | 35 | To permit 'atomic' changes involving multiple objects at once as well as to 36 | facilitate domain object traversal it is necessary to introduce an explicit 37 | 'Revision' object to represent a single changeset to the domain model. 38 | 39 | One also needs to introduce the concept of 'State'. This allows us to make 40 | (some) domain objects stateful, in particular those which are to be versioned 41 | (State is necessary to support delete/undelete functionality as well as to 42 | implement versioned many-to-many relationships). 43 | 44 | For each original domain object that comes versioned we end up with 2 domain 45 | objects: 46 | 47 | * The 'continuity': the original domain object. 48 | * The 'version/revision': the versions/revisions of that domain object. 49 | 50 | Often a user will never need to be concerned (explicitly) with the 51 | version/revision object as they will just interact with the original domain 52 | object, which will, where necessary, 'proxy' requests down to the 53 | 'version/revision'. 54 | 55 | To give a flavour of all of this here is a pseudo-code example:: 56 | 57 | # We need a session of some kind to track which objects have been changed 58 | # In SQLAlchemy can use its Session object 59 | session = get_session_in_some_way() 60 | 61 | # Our Revision object 62 | rev1 = Revision(author='me') 63 | # Associate revision with session 64 | # Any given session will have a single associated revision 65 | session.revision = rev1 66 | 67 | # Book and Author are domain objects which has been made versioned using this library 68 | # Note the typo! 69 | b1 = Book(name='warandpeace', title='War and Peacee') 70 | b2 = Book(name='annakarenina', title='Anna') 71 | # Note the duplicate! 72 | b3 = Book(name='warandpeace') 73 | a1 = Author(name='tolstoy') 74 | 75 | # this is just shorthand for ending this revision and saving all changes 76 | # this may vary depending on the implementation 77 | rev1.commit() 78 | timestamp1 = rev1.timestamp 79 | 80 | # some time later 81 | rev2 = Revision(author='me') 82 | session.revision = rev2 83 | 84 | b1 = Book.get(name='warandpeace') 85 | # correct typo 86 | b1.title = 'War and Peace' 87 | # add the author 88 | a1 = Author.get(name='tolstoy') 89 | b1.authors.append(a1) 90 | # duplicate item so delete 91 | b3.delete() 92 | rev2.commit() 93 | 94 | # some time even later 95 | rev1 = Revision.get(timestamp=timestamp1) 96 | b1 = Book.get(name='warandpeace') 97 | b1 = b1.get_as_of(rev1) 98 | assert b1.title == 'War and Peacee' 99 | assert b1.authors == [] 100 | # etc 101 | 102 | 103 | Code in Action 104 | -------------- 105 | 106 | To see some real code in action take a look at, for SQLAlchemy:: 107 | 108 | vdm/sqlalchemy/demo.py 109 | vdm/sqlalchemy/demo_test.py 110 | 111 | 112 | General Conceptual Documentation 113 | ================================ 114 | 115 | A great starting point is Fowler's *Patterns for things that change with time*: 116 | 117 | http://martinfowler.com/eaaDev/timeNarrative.html 118 | 119 | In particular Temporal Object: 120 | 121 | http://martinfowler.com/eaaDev/TemporalObject.html 122 | 123 | Two possible approaches: 124 | 125 | 1. (simpler) Versioned domain objects are versioned independently (like a 126 | wiki). This is less of a versioned 'domain model' and more of plain 127 | versioned domain objects. 128 | 2. (more complex) Have explicit 'Revision' object and multiple objects can be 129 | changed simultaneously in each revision (atomicity). This is a proper 130 | versioned *domain model*. 131 | 132 | Remark: using the first approach it is: 133 | 134 | * Impossible to support versioning of many-to-many links between versioned 135 | domain objects. 136 | * Impossible to change multiple objects 'at once' -- that is as part of 137 | one atomic change 138 | * Difficult to support domain model traversal, that is the ability to 139 | navigate around the domain model at a particular 'revision'/point-in-time. 140 | * More discussions of limitations can be found in this thread [1]. 141 | 142 | [1]: 143 | 144 | The versioned domain model (vdm) package focuses on supporting the second case 145 | (this obviously includes the first one as a subcase) -- hence the name. 146 | 147 | 148 | Use Cases 149 | --------- 150 | 151 | SA = Implemented in SQLAlchemy 152 | 153 | 1. (SA) CRUD for a simple versioned object (no references other than HasA) 154 | 155 | 2. (SA) Versioning of Many-2-Many and many-2-one relationships where one or 156 | both of the related objects are versioned. 157 | 158 | 3. (SA) Undelete for the above. 159 | 160 | 4. (SA) Purge for the above. 161 | 162 | 5. (SA) Support for changing multiple objects in a single commit. 163 | 164 | 6. (SA) Consistent object traversal both at HEAD and "in the past" 165 | 166 | 7. (SA) Diffing support on versioning objects and listing of changes for a 167 | given Revision. 168 | 169 | 8. Concurrency checking: 170 | 171 | 1. Simultaneous edits of different parts of the domain model 172 | 2. Simultaneous edits of same parts of domain model (conflict resolution or 173 | locking) 174 | 175 | 1. Alice and Bob both get object X 176 | 2. Bob updates object X and commits (A's X is now out of date) 177 | 3. Alice updates object X and commits 178 | 4. Conflict!! 179 | 180 | This can be resolved in the following ways: 181 | 182 | 1. Locking 183 | 2. Merging 184 | 185 | Rather than summarize all situations just see Fowler on concurrency 186 | 187 | 9. Support for pending updates (so updates must be approved before being visible) 188 | 189 | 1. A non-approved user makes a change 190 | 2. This change is marked as pending 191 | 3. This change is notified to a moderator 192 | 4. A moderator either allows or disallows the change 193 | ''' 194 | __version__ = '0.15' 195 | __description__ = 'A versioned domain model framework.' 196 | -------------------------------------------------------------------------------- /vdm/sqlalchemy/__init__.py: -------------------------------------------------------------------------------- 1 | '''SQLAlchemy versioned domain model extension. 2 | 3 | For general information about versioned domain models see the root vdm package 4 | docstring. 5 | 6 | Implementation Notes 7 | ==================== 8 | 9 | SQLAlchemy conveniently provides its own Session object which can be used as 10 | the 'session' for the vdm (i.e. the object which holds the current revision). 11 | 12 | Some useful links: 13 | 14 | http://blog.discorporate.us/2008/02/sqlalchemy-partitioned-collections-1/ 15 | http://groups.google.com/group/sqlalchemy/browse_thread/thread/ecd515c8c1c013b1 16 | http://www.sqlalchemy.org/trac/browser/sqlalchemy/trunk/lib/sqlalchemy/ext/associationproxy.py 17 | http://www.sqlalchemy.org/docs/04/plugins.html#plugins_associationproxy 18 | 19 | 20 | TODO 21 | ==== 22 | 23 | 1. How do we commit revisions (do we need to). 24 | * At very least do we not need to update timestamps? 25 | * support for state of revision (active, deleted (spam), in-progress etc) 26 | 27 | 2. Support for composite primary keys. 28 | 29 | 4. Support for m2m collections other than lists. 30 | ''' 31 | from base import * 32 | from tools import Repository 33 | 34 | __all__ = [ 35 | 'set_revision', 'get_revision', 36 | 'make_state_table', 'make_revision_table', 37 | 'make_table_stateful', 'make_table_revisioned', 38 | 'make_State', 'make_Revision', 39 | 'StatefulObjectMixin', 'RevisionedObjectMixin', 40 | 'Revisioner', 'modify_base_object_mapper', 'create_object_version', 41 | 'add_stateful_versioned_m2m', 'add_stateful_versioned_m2m_on_version', 42 | 'Repository' 43 | ] 44 | 45 | -------------------------------------------------------------------------------- /vdm/sqlalchemy/base.py: -------------------------------------------------------------------------------- 1 | from datetime import datetime 2 | import difflib 3 | import uuid 4 | import logging 5 | import weakref 6 | 7 | from sqlalchemy import * 8 | from sqlalchemy.orm.attributes import get_history, PASSIVE_OFF 9 | from sqlalchemy import __version__ as sqav 10 | 11 | from sqla import SQLAlchemyMixin 12 | from sqla import copy_column, copy_table_columns, copy_table 13 | 14 | make_uuid = lambda: unicode(uuid.uuid4()) 15 | logger = logging.getLogger('vdm') 16 | 17 | ## ------------------------------------- 18 | class SQLAlchemySession(object): 19 | '''Handle setting/getting attributes on the SQLAlchemy session. 20 | 21 | TODO: update all methods so they can take an object as well as session 22 | object. 23 | ''' 24 | 25 | @classmethod 26 | def setattr(self, session, attr, value): 27 | setattr(session, attr, value) 28 | # check if we are being given the Session class (threadlocal case) 29 | # if so set on both class and instance 30 | # this is important because sqlalchemy's object_session (used below) seems 31 | # to return a Session() not Session 32 | if isinstance(session, sqlalchemy.orm.scoping.ScopedSession): 33 | sess = session() 34 | setattr(sess, attr, value) 35 | 36 | @classmethod 37 | def getattr(self, session, attr): 38 | return getattr(session, attr) 39 | 40 | # make explicit to avoid errors from typos (no attribute defns in python!) 41 | @classmethod 42 | def set_revision(self, session, revision): 43 | self.setattr(session, 'HEAD', True) 44 | self.setattr(session, 'revision', revision) 45 | if revision.id is None: 46 | # make uuid here so that if other objects in this session are flushed 47 | # at the same time they know thier revision id 48 | revision.id = make_uuid() 49 | # there was a begin_nested here but that just caused flush anyway. 50 | session.add(revision) 51 | session.flush() 52 | 53 | @classmethod 54 | def get_revision(self, session): 55 | '''Get revision on current Session/session. 56 | 57 | NB: will return None if not set 58 | ''' 59 | return getattr(session, 'revision', None) 60 | 61 | @classmethod 62 | def set_not_at_HEAD(self, session): 63 | self.setattr(session, 'HEAD', False) 64 | 65 | @classmethod 66 | def at_HEAD(self, session): 67 | return getattr(session, 'HEAD', True) 68 | 69 | 70 | ## -------------------------------------------------------- 71 | ## VDM-Specific Domain Objects and Tables 72 | 73 | # Enumeration 74 | class State(object): 75 | ACTIVE = u'active' 76 | DELETED = u'deleted' 77 | PENDING = u'pending' 78 | all = (ACTIVE, DELETED, PENDING) 79 | 80 | def make_revision_table(metadata): 81 | revision_table = Table('revision', metadata, 82 | Column('id', UnicodeText, primary_key=True, default=make_uuid), 83 | Column('timestamp', DateTime, default=datetime.utcnow), 84 | Column('author', String(200)), 85 | Column('message', UnicodeText), 86 | Column('state', UnicodeText, default=State.ACTIVE) 87 | ) 88 | return revision_table 89 | 90 | 91 | class Revision(SQLAlchemyMixin): 92 | '''A Revision to the Database/Domain Model. 93 | 94 | All versioned objects have an associated Revision which can be accessed via 95 | the revision attribute. 96 | ''' 97 | # TODO:? set timestamp in ctor ... (maybe not as good to have undefined 98 | # until actual save ...) 99 | @property 100 | def __id__(self): 101 | if self.id is None: 102 | self.id = make_uuid() 103 | return self.id 104 | @classmethod 105 | def youngest(self, session): 106 | '''Get the youngest (most recent) revision. 107 | 108 | If session is not provided assume there is a contextual session. 109 | ''' 110 | q = session.query(self) 111 | return q.first() 112 | 113 | 114 | def make_Revision(mapper, revision_table): 115 | mapper(Revision, revision_table, properties={ 116 | }, 117 | order_by=revision_table.c.timestamp.desc()) 118 | return Revision 119 | 120 | ## -------------------------------------------------------- 121 | ## Table Helpers 122 | 123 | def make_table_stateful(base_table): 124 | '''Make a table 'stateful' by adding appropriate state column.''' 125 | base_table.append_column( 126 | Column('state', UnicodeText, default=State.ACTIVE) 127 | ) 128 | 129 | def make_table_revisioned(base_table): 130 | logger.warn('make_table_revisioned is deprecated: use make_revisioned_table') 131 | return make_revisioned_table(base_table) 132 | 133 | def make_revisioned_table(base_table): 134 | '''Modify base_table and create correponding revision table. 135 | 136 | # TODO: (complex) support for complex primary keys on continuity. 137 | # Search for "composite foreign key sqlalchemy" for helpful info 138 | 139 | @return revision table. 140 | ''' 141 | base_table.append_column( 142 | Column('revision_id', UnicodeText, ForeignKey('revision.id')) 143 | ) 144 | newtable = Table(base_table.name + '_revision', base_table.metadata, 145 | ) 146 | copy_table(base_table, newtable) 147 | 148 | # create foreign key 'continuity' constraint 149 | # remember base table primary cols have been exactly duplicated onto our table 150 | pkcols = [] 151 | for col in base_table.c: 152 | if col.primary_key: 153 | pkcols.append(col) 154 | if len(pkcols) > 1: 155 | msg = 'Do not support versioning objects with multiple primary keys' 156 | raise ValueError(msg) 157 | fk_name = base_table.name + '.' + pkcols[0].name 158 | newtable.append_column( 159 | Column('continuity_id', pkcols[0].type, ForeignKey(fk_name)) 160 | ) 161 | # TODO: a start on composite primary key stuff 162 | # newtable.append_constraint( 163 | # ForeignKeyConstraint( 164 | # [c.name for c in pkcols], 165 | # [base_table.name + '.' + c.name for c in pkcols ] 166 | # )) 167 | 168 | # TODO: why do we iterate all the way through rather than just using dict 169 | # functionality ...? Surely we always have a revision here ... 170 | for col in newtable.c: 171 | if col.name == 'revision_id': 172 | col.primary_key = True 173 | newtable.primary_key.columns.add(col) 174 | return newtable 175 | 176 | 177 | ## -------------------------------------------------------- 178 | ## Object Helpers 179 | 180 | class StatefulObjectMixin(object): 181 | __stateful__ = True 182 | 183 | def delete(self): 184 | logger.debug('Running delete on %s', self) 185 | self.state = State.DELETED 186 | 187 | def undelete(self): 188 | self.state = State.ACTIVE 189 | 190 | def is_active(self): 191 | # also support None in case this object is not yet refreshed ... 192 | return self.state is None or self.state == State.ACTIVE 193 | 194 | 195 | class RevisionedObjectMixin(object): 196 | __ignored_fields__ = ['revision_id'] 197 | __revisioned__ = True 198 | 199 | @classmethod 200 | def revisioned_fields(cls): 201 | table = sqlalchemy.orm.class_mapper(cls).mapped_table 202 | fields = [ col.name for col in table.c if col.name not in 203 | cls.__ignored_fields__ ] 204 | return fields 205 | 206 | def get_as_of(self, revision=None): 207 | '''Get this domain object at the specified revision. 208 | 209 | If no revision is specified revision will be looked up on the global 210 | session object. If that not found return head. 211 | 212 | get_as_of does most of the crucial work in supporting the 213 | versioning. 214 | ''' 215 | sess = object_session(self) 216 | if revision: # set revision on the session so dom traversal works 217 | # TODO: should we test for overwriting current session? 218 | # if rev != revision: 219 | # msg = 'The revision on the session does not match the one you' + \ 220 | # 'requesting.' 221 | # raise Exception(msg) 222 | logger.debug('get_as_of: setting revision and not_as_HEAD: %s', 223 | revision) 224 | SQLAlchemySession.set_revision(sess, revision) 225 | SQLAlchemySession.set_not_at_HEAD(sess) 226 | else: 227 | revision = SQLAlchemySession.get_revision(sess) 228 | 229 | if SQLAlchemySession.at_HEAD(sess): 230 | return self 231 | else: 232 | revision_class = self.__revision_class__ 233 | # TODO: when dealing with multi-col pks will need to update this 234 | # (or just use continuity) 235 | out = sess.query(revision_class).join('revision').\ 236 | filter( 237 | Revision.timestamp <= revision.timestamp 238 | ).\ 239 | filter( 240 | revision_class.id == self.id 241 | ).\ 242 | order_by( 243 | Revision.timestamp.desc() 244 | ) 245 | return out.first() 246 | 247 | @property 248 | def all_revisions(self): 249 | allrevs = self.all_revisions_unordered 250 | ourcmp = lambda revobj1, revobj2: cmp(revobj1.revision.timestamp, 251 | revobj2.revision.timestamp) 252 | sorted_revobjs = sorted(allrevs, cmp=ourcmp, reverse=True) 253 | return sorted_revobjs 254 | 255 | def diff(self, to_revision=None, from_revision=None): 256 | '''Diff this object returning changes between `from_revision` and 257 | `to_revision`. 258 | 259 | @param to_revision: revision to diff to (defaults to the youngest rev) 260 | @param from_revision: revision to diff from (defaults to one revision 261 | older than to_revision) 262 | @return: dict of diffs keyed by field name 263 | 264 | e.g. diff(HEAD, HEAD-2) will show diff of changes made in last 2 265 | commits (NB: no changes may have occurred to *this* object in those 266 | commits). 267 | ''' 268 | obj_rev_class = self.__revision_class__ 269 | sess = object_session(self) 270 | obj_rev_query = sess.query(obj_rev_class).join('revision').\ 271 | filter(obj_rev_class.id==self.id).\ 272 | order_by(Revision.timestamp.desc()) 273 | obj_class = self 274 | to_obj_rev, from_obj_rev = self.get_obj_revisions_to_diff(\ 275 | obj_rev_query, 276 | to_revision=to_revision, 277 | from_revision=from_revision) 278 | return self.diff_revisioned_fields(to_obj_rev, from_obj_rev, 279 | obj_class) 280 | 281 | 282 | def get_obj_revisions_to_diff(self, obj_revision_query, to_revision=None, 283 | from_revision=None): 284 | '''Diff this object returning changes between `from_revision` and 285 | `to_revision`. 286 | 287 | @param obj_revision_query: query of all object revisions related to 288 | the object being diffed. e.g. all PackageRevision objects with 289 | @param to_revision: revision to diff to (defaults to the youngest rev) 290 | @param from_revision: revision to diff from (defaults to one revision 291 | older than to_revision) 292 | @return: dict of diffs keyed by field name 293 | 294 | e.g. diff(HEAD, HEAD-2) will show diff of changes made in last 2 295 | commits (NB: no changes may have occurred to *this* object in those 296 | commits). 297 | ''' 298 | sess = object_session(self) 299 | if to_revision is None: 300 | to_revision = Revision.youngest(sess) 301 | out = obj_revision_query.\ 302 | filter(Revision.timestamp<=to_revision.timestamp) 303 | to_obj_rev = out.first() 304 | if not from_revision: 305 | from_revision = sess.query(Revision).\ 306 | filter(Revision.timestamp 0 592 | 593 | if revision_already: 594 | logger.debug('Updating version of %s: %s', instance, colvalues) 595 | connection.execute(self.revision_table.update(existing_revision_clause).values(colvalues)) 596 | else: 597 | logger.debug('Creating version of %s: %s', instance, colvalues) 598 | ins = self.revision_table.insert().values(colvalues) 599 | connection.execute(ins) 600 | 601 | # set to None to avoid accidental reuse 602 | # ERROR: cannot do this as after_* is called per object and may be run 603 | # before_update on other objects ... 604 | # probably need a SessionExtension to deal with this properly 605 | # object_session(instance).revision = None 606 | 607 | def before_update(self, mapper, connection, instance): 608 | self._is_changed[instance] = self.check_real_change(instance, mapper, connection) 609 | if not self.revisioning_disabled(instance) and self._is_changed[instance]: 610 | logger.debug('before_update: %s', instance) 611 | self.set_revision(instance) 612 | self._is_changed[instance] = self.check_real_change( 613 | instance, mapper, connection) 614 | return EXT_CONTINUE 615 | 616 | # We do most of the work in after_insert/after_update as at that point 617 | # instance has been properly created (which means e.g. instance.id is 618 | # available ...) 619 | def before_insert(self, mapper, connection, instance): 620 | self._is_changed[instance] = self.check_real_change(instance, mapper, connection) 621 | if not self.revisioning_disabled(instance) and self._is_changed[instance]: 622 | logger.debug('before_insert: %s', instance) 623 | self.set_revision(instance) 624 | return EXT_CONTINUE 625 | 626 | def after_update(self, mapper, connection, instance): 627 | if not self.revisioning_disabled(instance) and self._is_changed[instance]: 628 | logger.debug('after_update: %s', instance) 629 | self.make_revision(instance, mapper, connection) 630 | return EXT_CONTINUE 631 | 632 | def after_insert(self, mapper, connection, instance): 633 | if not self.revisioning_disabled(instance) and self._is_changed[instance]: 634 | logger.debug('after_insert: %s', instance) 635 | self.make_revision(instance, mapper, connection) 636 | return EXT_CONTINUE 637 | 638 | def append_result(self, mapper, selectcontext, row, instance, result, 639 | **flags): 640 | # TODO: 2009-02-13 why is this needed? Can we remove this? 641 | return EXT_CONTINUE 642 | 643 | -------------------------------------------------------------------------------- /vdm/sqlalchemy/demo.py: -------------------------------------------------------------------------------- 1 | '''Demo of vdm for SQLAlchemy. 2 | 3 | This module sets up a small domain model with some versioned objects. Code 4 | that then uses these objects can be found in demo_test.py. 5 | ''' 6 | from datetime import datetime 7 | import logging 8 | logger = logging.getLogger('vdm') 9 | 10 | from sqlalchemy import * 11 | from sqlalchemy import __version__ as sqla_version 12 | # from sqlalchemy import create_engine 13 | 14 | import vdm.sqlalchemy 15 | 16 | TEST_ENGINE = "postgres" # or "sqlite" 17 | 18 | if TEST_ENGINE == "postgres": 19 | engine = create_engine('postgres://ckan_default:pass@localhost/vdmtest', 20 | pool_threadlocal=True) 21 | else: 22 | # setting the isolation_level is a hack required for sqlite support 23 | # until http://code.google.com/p/pysqlite/issues/detail?id=24 is 24 | # fixed. 25 | engine = create_engine('sqlite:///:memory:', 26 | connect_args={'isolation_level': None}) 27 | 28 | metadata = MetaData(bind=engine) 29 | 30 | ## VDM-specific tables 31 | 32 | revision_table = vdm.sqlalchemy.make_revision_table(metadata) 33 | 34 | ## Demo tables 35 | 36 | license_table = Table('license', metadata, 37 | Column('id', Integer, primary_key=True), 38 | Column('name', String(100)), 39 | Column('open', Boolean), 40 | ) 41 | 42 | import uuid 43 | def uuidstr(): return str(uuid.uuid4()) 44 | package_table = Table('package', metadata, 45 | # Column('id', Integer, primary_key=True), 46 | Column('id', String(36), default=uuidstr, primary_key=True), 47 | Column('name', String(100), unique=True), 48 | Column('title', String(100)), 49 | Column('license_id', Integer, ForeignKey('license.id')), 50 | Column('notes', UnicodeText), 51 | ) 52 | 53 | tag_table = Table('tag', metadata, 54 | Column('id', Integer, primary_key=True), 55 | Column('name', String(100)), 56 | ) 57 | 58 | package_tag_table = Table('package_tag', metadata, 59 | Column('id', Integer, primary_key=True), 60 | # Column('package_id', Integer, ForeignKey('package.id')), 61 | Column('package_id', String(36), ForeignKey('package.id')), 62 | Column('tag_id', Integer, ForeignKey('tag.id')), 63 | ) 64 | 65 | 66 | vdm.sqlalchemy.make_table_stateful(license_table) 67 | vdm.sqlalchemy.make_table_stateful(package_table) 68 | vdm.sqlalchemy.make_table_stateful(tag_table) 69 | vdm.sqlalchemy.make_table_stateful(package_tag_table) 70 | license_revision_table = vdm.sqlalchemy.make_revisioned_table(license_table) 71 | package_revision_table = vdm.sqlalchemy.make_revisioned_table(package_table) 72 | # TODO: this has a composite primary key ... 73 | package_tag_revision_table = vdm.sqlalchemy.make_revisioned_table(package_tag_table) 74 | 75 | 76 | 77 | ## ------------------- 78 | ## Mapped classes 79 | 80 | 81 | class License(vdm.sqlalchemy.RevisionedObjectMixin, 82 | vdm.sqlalchemy.StatefulObjectMixin, 83 | vdm.sqlalchemy.SQLAlchemyMixin 84 | ): 85 | def __init__(self, **kwargs): 86 | for k,v in kwargs.items(): 87 | setattr(self, k, v) 88 | 89 | class Package(vdm.sqlalchemy.RevisionedObjectMixin, 90 | vdm.sqlalchemy.StatefulObjectMixin, 91 | vdm.sqlalchemy.SQLAlchemyMixin 92 | ): 93 | 94 | def __init__(self, **kwargs): 95 | for k,v in kwargs.items(): 96 | setattr(self, k, v) 97 | 98 | 99 | class Tag(vdm.sqlalchemy.SQLAlchemyMixin): 100 | def __init__(self, name): 101 | self.name = name 102 | 103 | 104 | class PackageTag(vdm.sqlalchemy.RevisionedObjectMixin, 105 | vdm.sqlalchemy.StatefulObjectMixin, 106 | vdm.sqlalchemy.SQLAlchemyMixin 107 | ): 108 | def __init__(self, package=None, tag=None, state=None, **kwargs): 109 | logger.debug('PackageTag.__init__: %s, %s' % (package, tag)) 110 | self.package = package 111 | self.tag = tag 112 | self.state = state 113 | for k,v in kwargs.items(): 114 | setattr(self, k, v) 115 | 116 | 117 | ## -------------------------------------------------------- 118 | ## Mapper Stuff 119 | 120 | from sqlalchemy.orm import scoped_session, sessionmaker, create_session 121 | from sqlalchemy.orm import relation, backref 122 | # both options now work 123 | # Session = scoped_session(sessionmaker(autoflush=False, transactional=True)) 124 | # this is the more testing one ... 125 | if sqla_version <= '0.4.99': 126 | Session = scoped_session(sessionmaker(autoflush=True, transactional=True)) 127 | else: 128 | Session = scoped_session(sessionmaker(autoflush=True, 129 | expire_on_commit=False, 130 | autocommit=False)) 131 | 132 | # mapper = Session.mapper 133 | from sqlalchemy.orm import mapper 134 | 135 | # VDM-specific domain objects 136 | State = vdm.sqlalchemy.State 137 | Revision = vdm.sqlalchemy.make_Revision(mapper, revision_table) 138 | 139 | mapper(License, license_table, properties={ 140 | }, 141 | extension=vdm.sqlalchemy.Revisioner(license_revision_table) 142 | ) 143 | 144 | mapper(Package, package_table, properties={ 145 | 'license':relation(License), 146 | # delete-orphan on cascade does NOT work! 147 | # Why? Answer: because of way SQLAlchemy/our code works there are points 148 | # where PackageTag object is created *and* flushed but does not yet have 149 | # the package_id set (this cause us other problems ...). Some time later a 150 | # second commit happens in which the package_id is correctly set. 151 | # However after first commit PackageTag does not have Package and 152 | # delete-orphan kicks in to remove it! 153 | # 154 | # do we want lazy=False here? used in: 155 | # 156 | 'package_tags':relation(PackageTag, backref='package', cascade='all'), #, delete-orphan'), 157 | }, 158 | extension = vdm.sqlalchemy.Revisioner(package_revision_table) 159 | ) 160 | 161 | mapper(Tag, tag_table) 162 | 163 | mapper(PackageTag, package_tag_table, properties={ 164 | 'tag':relation(Tag), 165 | }, 166 | extension = vdm.sqlalchemy.Revisioner(package_tag_revision_table) 167 | ) 168 | 169 | vdm.sqlalchemy.modify_base_object_mapper(Package, Revision, State) 170 | vdm.sqlalchemy.modify_base_object_mapper(License, Revision, State) 171 | vdm.sqlalchemy.modify_base_object_mapper(PackageTag, Revision, State) 172 | PackageRevision = vdm.sqlalchemy.create_object_version(mapper, Package, 173 | package_revision_table) 174 | LicenseRevision = vdm.sqlalchemy.create_object_version(mapper, License, 175 | license_revision_table) 176 | PackageTagRevision = vdm.sqlalchemy.create_object_version(mapper, PackageTag, 177 | package_tag_revision_table) 178 | 179 | from base import add_stateful_versioned_m2m 180 | vdm.sqlalchemy.add_stateful_versioned_m2m(Package, PackageTag, 'tags', 'tag', 181 | 'package_tags') 182 | vdm.sqlalchemy.add_stateful_versioned_m2m_on_version(PackageRevision, 'tags') 183 | 184 | ## ------------------------ 185 | ## Repository helper object 186 | 187 | from tools import Repository 188 | repo = Repository(metadata, Session, 189 | versioned_objects = [ Package, License, PackageTag ] 190 | ) 191 | 192 | -------------------------------------------------------------------------------- /vdm/sqlalchemy/demo_meta.py: -------------------------------------------------------------------------------- 1 | '''SQLAlchemy Metadata and Session object''' 2 | from sqlalchemy import MetaData 3 | from sqlalchemy.orm import scoped_session, sessionmaker 4 | 5 | __all__ = ['Session', 'engine', 'metadata', 'init_with_engine' ] 6 | 7 | engine = None 8 | 9 | # IMPORTANT NOTE for vdm 10 | # You cannot use autoflush=True, autocommit=False 11 | # This is because flushes to the DB may then happen at 'random' times when no 12 | # Revision has yet been set which may result in errors 13 | Session = scoped_session(sessionmaker( 14 | autoflush=True, 15 | # autocommit=False, 16 | transactional=True, 17 | )) 18 | 19 | # Global metadata. If you have multiple databases with overlapping table 20 | # names, you'll need a metadata for each database 21 | metadata = MetaData() 22 | 23 | def init_with_engine(engine_): 24 | metadata.bind = engine_ 25 | Session.configure(bind=engine_) 26 | engine = engine_ 27 | 28 | -------------------------------------------------------------------------------- /vdm/sqlalchemy/demo_simple.py: -------------------------------------------------------------------------------- 1 | '''A simple demo of vdm. 2 | 3 | This demo shows how to use vdm for simple versioning of individual domain 4 | objects (without any versioning of the relations between objects). For more 5 | complex example see demo.py 6 | ''' 7 | 8 | from sqlalchemy import * 9 | # SQLite is not as reliable/demanding as postgres but you can use it 10 | engine = create_engine('postgres://tester:pass@localhost/vdmtest') 11 | 12 | from demo_meta import Session, metadata, init_with_engine 13 | init_with_engine(engine) 14 | 15 | # import the versioned domain model package 16 | import vdm.sqlalchemy 17 | 18 | ## ----------------------------- 19 | ## Our Tables 20 | 21 | # NB: you really need to set up your tables and domain object separately for 22 | # vdm 23 | 24 | wikipage = Table('wikipage', metadata, 25 | Column('id', Integer, primary_key=True), 26 | Column('name', Unicode(200)), 27 | Column('body', UnicodeText), 28 | ) 29 | 30 | # ----------------------------- 31 | # VDM stuff 32 | 33 | # Now we need some standard VDM tables 34 | state_table = vdm.sqlalchemy.make_state_table(metadata) 35 | revision_table = vdm.sqlalchemy.make_revision_table(metadata) 36 | 37 | 38 | # Make our original table vdm-ready by adding state to it 39 | vdm.sqlalchemy.make_table_stateful(wikipage) 40 | # And create the table for the wikipage revisions/versions 41 | wikipage_revision = vdm.sqlalchemy.make_table_revisioned(wikipage) 42 | 43 | 44 | ## ------------------------------ 45 | ## Our Domain Objects 46 | 47 | # Suppose your class started out as 48 | # Class WikiPage(object): 49 | # pass 50 | # 51 | # then to make it versioned you just need to add in some Mixins 52 | 53 | class WikiPage( 54 | vdm.sqlalchemy.StatefulObjectMixin, # make it state aware 55 | vdm.sqlalchemy.RevisionedObjectMixin, # make it versioned aware 56 | vdm.sqlalchemy.SQLAlchemyMixin # this is optional (provides nice __str__) 57 | ): 58 | 59 | pass 60 | 61 | 62 | ## Let's map the tables to the domain objects 63 | mapper = Session.mapper 64 | 65 | # VDM-specific domain objects 66 | State = vdm.sqlalchemy.make_State(mapper, state_table) 67 | Revision = vdm.sqlalchemy.make_Revision(mapper, revision_table) 68 | 69 | 70 | # Now our domain object. 71 | # This is just like any standard sqlalchemy mapper setup 72 | # The only addition is the mapper extension 73 | mapper(WikiPage, wikipage, properties={ 74 | }, 75 | # mapper extension which handles automatically versioning the object 76 | extension=vdm.sqlalchemy.Revisioner(wikipage_revision) 77 | ) 78 | # add the revision and state attributes into WikiPage 79 | vdm.sqlalchemy.modify_base_object_mapper(WikiPage, Revision, State) 80 | 81 | # Last: create domain object corresponding to the Revision/Version of the main 82 | # object 83 | WikiPageRevision = vdm.sqlalchemy.create_object_version( 84 | mapper, 85 | WikiPage, 86 | wikipage_revision 87 | ) 88 | 89 | # We recommend you use the Repository object to manage your versioned domain 90 | # objects 91 | # This isn't required but it provides extra useful features such as purging 92 | # See the module for full details 93 | from vdm.sqlalchemy.tools import Repository 94 | repo = Repository(metadata, Session, versioned_objects=[WikiPage]) 95 | 96 | 97 | # Let's try it out 98 | def test_it(): 99 | # clean out the db so we start clean 100 | repo.rebuild_db() 101 | 102 | # you need to set up a Revision for versioned objects to use 103 | # this is set on the SQLAlchemy session so as to be available generally 104 | # It can be set up any time before you commit but it is usually best to do 105 | # it before you start creating or modifying versioned objects 106 | 107 | # You can set up the revision directly e.g. 108 | # rev = Revision() 109 | # SQLAlchemySession.set_revision(rev) 110 | # (or even just Session.revision = rev) 111 | # However this will do the same and is simpler 112 | rev = repo.new_revision() 113 | 114 | # now make some changes 115 | mypage = WikiPage(name=u'Home', body=u'Some text') 116 | mypage2 = WikiPage(name=u'MyPage', body=u'') 117 | # let's add a log message to these changes 118 | rev.message = u'My first revision' 119 | # Just encapsulates Session.commit() + Session.remove() 120 | # (with some try/excepts) 121 | repo.commit_and_remove() 122 | 123 | last_revision = repo.youngest_revision() 124 | assert last_revision.message == u'My first revision' 125 | outpage = WikiPage.query.filter_by(name=u'Home').first() 126 | assert outpage and outpage.body == u'Some text' 127 | 128 | # let's make some more changes 129 | 130 | -------------------------------------------------------------------------------- /vdm/sqlalchemy/sqla.py: -------------------------------------------------------------------------------- 1 | '''Generic sqlalchemy code (not specifically related to vdm). 2 | ''' 3 | import sqlalchemy 4 | 5 | class SQLAlchemyMixin(object): 6 | def __init__(self, **kw): 7 | for k, v in kw.iteritems(): 8 | setattr(self, k, v) 9 | 10 | def __str__(self): 11 | return self.__unicode__().encode('utf8') 12 | 13 | def __unicode__(self): 14 | repr = u'<%s' % self.__class__.__name__ 15 | table = sqlalchemy.orm.class_mapper(self.__class__).mapped_table 16 | for col in table.c: 17 | repr += u' %s=%s' % (col.name, getattr(self, col.name)) 18 | repr += '>' 19 | return repr 20 | 21 | def __repr__(self): 22 | return self.__str__() 23 | 24 | ## -------------------------------------------------------- 25 | ## Table Helpers 26 | 27 | def copy_column(name, src_table, dest_table): 28 | ''' 29 | Note you cannot just copy columns standalone e.g. 30 | 31 | col = table.c['xyz'] 32 | col.copy() 33 | 34 | This will only copy basic info while more complex properties (such as fks, 35 | constraints) to work must be set when the Column has a parent table. 36 | 37 | TODO: stuff other than fks (e.g. constraints such as uniqueness) 38 | ''' 39 | col = src_table.c[name] 40 | if col.unique == True: 41 | # don't copy across unique constraints, as different versions 42 | # of an object may have identical column values 43 | col.unique = False 44 | dest_table.append_column(col.copy()) 45 | # only get it once we have a parent table 46 | newcol = dest_table.c[name] 47 | if len(col.foreign_keys) > 0: 48 | for fk in col.foreign_keys: 49 | newcol.append_foreign_key(fk.copy()) 50 | 51 | def copy_table_columns(table): 52 | columns = [] 53 | for col in table.c: 54 | newcol = col.copy() 55 | if len(col.foreign_keys) > 0: 56 | for fk in col.foreign_keys: 57 | newcol.foreign_keys.add(fk.copy()) 58 | columns.append(newcol) 59 | return columns 60 | 61 | def copy_table(table, newtable): 62 | for key in table.c.keys(): 63 | copy_column(key, table, newtable) 64 | 65 | -------------------------------------------------------------------------------- /vdm/sqlalchemy/stateful.py: -------------------------------------------------------------------------------- 1 | '''Support for stateful collections. 2 | 3 | Stateful collections are essential to the functioning of VDM's m2m support. 4 | m2m relationships become stateful when versioned and the associated collection must 5 | become state-aware. 6 | 7 | There are several subtleties in doing this, the most significant of which is 8 | how one copes with adding an "existing" object to a stateful list which already 9 | contains that object in deleted form (or, similarly, when moving an item within 10 | a list which normally corresponds to a delete and an insert). 11 | 12 | 13 | Stateful Lists and "Existing" Objects Problem 14 | ============================================= 15 | 16 | The problem here is that the "existing" object and the object being added are 17 | not literally the same object in the python sense. Why? First, because their 18 | state may differ and the ORM is not aware that state is irrelevant to identity. 19 | Second, and more significantly because the ORM often does not fully "create" 20 | the object until a flush which is too late - we already have duplicates in the 21 | list. Here's a concrete example:: 22 | 23 | # Package, Tag, PackageTag objects with Package.package_tags_active being 24 | # StatefulList and Package.tags a proxy to this showing the tags 25 | 26 | pkg.tags = [ tag1 ] 27 | # pkg.package_tags now contains 2 items 28 | # PackageTag(pkg= 29 | 30 | pkg1 = Package('abc') 31 | tag1 = Tag(name='x') 32 | pkg1.tags.append(tag1) 33 | # pkg1.package_tags contains one PackageTag 34 | 35 | # delete tag1 36 | del pkg.tags[0] 37 | # so PackageTag(pkg=pkg1, tag=tag1) is now in deleted state 38 | pkg.tags.append(tag1) 39 | # now pkg.package_tags has length 2! 40 | # Why? Really we want to undelete PackageTag(pkg=pkg1, tag=tag1) 41 | # however for association proxy what happens is that 42 | # we get a new PackageTag(pkg=None, tag=tag1) created and this is not 43 | # identified with existing PackageTag(pkg=pkg1, tag=tag1) because pkg=None 44 | # on new PackageTag (pkg only set on flush) 45 | # Thus a new item is appended rather than existing being undeleted 46 | 47 | # even more seriously suppose pkg.tags is [tag1] 48 | # what happens if we do 49 | pkg.tags = [tag1] 50 | # does *not* result in nothing happen 51 | # instead existing PackageTag(pkg=pkg1, tag=tag1) is put in deleted state and 52 | # new PackageTag(pkg=None, tag=tag1) is appended with this being changed to 53 | # PackageTag(pkg=pkg1, tag=tag1) on commit (remember sqlalchemy does not 54 | # resolve m2m objects foreign key for owner object until flush time) 55 | 56 | How do we solve this? The only real answer is implement an identity map in the 57 | stateful list based on a supplier identifier function. This is done in the code 58 | below. It has the effect of restoring expected behaviour in the above examples 59 | (further demonstrations can be found in the tests). 60 | 61 | 62 | TODO: create some proper tests for base_modifier stuff. 63 | TODO: move stateful material from base.py here? 64 | ''' 65 | import logging 66 | logger = logging.getLogger('vdm.stateful') 67 | 68 | import itertools 69 | 70 | 71 | class StatefulProxy(object): 72 | '''A proxy to an underlying collection which contains stateful objects. 73 | 74 | The proxy only shows objects in a particular state (e.g. active ones) and 75 | will also transform standard collection operations to make them 'stateful' 76 | -- for example deleting from the list will not delete the object but simply 77 | place it in a deleted state. 78 | ''' 79 | def __init__(self, target, **kwargs): 80 | ''' 81 | @param target: the target (underlying) collection (list, dict, etc) 82 | 83 | Possible kwargs: 84 | 85 | is_active, delete, undelete: a method performing the relevant operation 86 | on the underlying stateful objects. If these are not provided they 87 | will be created on the basis that there is a corresponding method 88 | on the stateful object (e.g. one can do obj.is_active() 89 | obj.delete() etc. 90 | 91 | base_modifier: function to operate on base objects before any 92 | processing. e.g. could have function: 93 | 94 | def get_as_of(x): 95 | return x.get_as_of(revision) 96 | 97 | WARNING: if base_modifier is not trivial (i.e. does not equal the 98 | identity function: lambda x: x) then only read operations should be 99 | performed on this proxy (this is because you will be operating on 100 | modified rather than original objects). 101 | 102 | WARNING: when using base_modifier the objects returned from this list will 103 | not be list objects themselves but base_modifier(object). 104 | In particular, this means that when base_modifier is turned on 105 | operations that change the list (e.g. deletions) will operate on 106 | modified objects not the base objects!! 107 | ''' 108 | self.target = target 109 | 110 | extra_args = ['is_active', 'delete', 'undelete', 'base_modifier'] 111 | for argname in extra_args: 112 | setattr(self, argname, kwargs.get(argname, None)) 113 | if self.is_active is None: 114 | # object may not exist (e.g. with get_as_of in which case it will 115 | # be None 116 | self.is_active = lambda x: not(x is None) and x.is_active() 117 | if self.delete is None: 118 | self.delete = lambda x: x.delete() 119 | if self.undelete is None: 120 | self.undelete = lambda x: x.undelete() 121 | if self.base_modifier is None: 122 | self.base_modifier = lambda x: x 123 | self._set_stateful_operators() 124 | 125 | def _set_stateful_operators(self): 126 | self._is_active = lambda x: self.is_active(self.base_modifier(x)) 127 | self._delete = self.delete 128 | self._undelete = self.undelete 129 | 130 | 131 | class StatefulList(StatefulProxy): 132 | '''A list which is 'state' aware. 133 | 134 | NB: there is some subtlety as to behaviour when adding an "existing" object 135 | to the list -- see the main module docstring for details. 136 | 137 | # TODO: should we have self.base_modifier(obj) more frequently used? (e.g. 138 | # in __iter__, append, etc 139 | ''' 140 | def __init__(self, target, **kwargs): 141 | '''Same as for StatefulProxy but with additional kwarg: 142 | 143 | @param identifier: a function which takes an object and return a key 144 | identifying that object for use in an internal identity map. (See 145 | discussion in main docstring for why this is required). 146 | 147 | @param unneeded_deleter: function to delete objects which have 148 | been added to the list but turn out to be unneeded (because existing 149 | deleted object already exists). Reason for existence: (going back to 150 | example in main docstring) suppose we pkg1.package_tags already 151 | contains tag1 ('geo') but in deleted state. We then do 152 | pkg1.tags.append(tag1). This results in a new PackageTag pkgtag2 being 153 | created by creator function in assoc proxy and passed on to stateful 154 | list. Thanks to the identifier we notice this already exists and 155 | undelete the existing PackageTag rather than adding this new one. But 156 | what do we with this pkgtag2? We need to 'get rid of it' so it is not 157 | committed into the the db. 158 | ''' 159 | super(StatefulList, self).__init__(target, **kwargs) 160 | identifier = kwargs.get('identifier', lambda x: x) 161 | unneeded_deleter = kwargs.get('unneeded_deleter', lambda x: None) 162 | self._identifier = identifier 163 | self._unneeded_deleter = unneeded_deleter 164 | self._identity_map = {} 165 | for obj in self.target: 166 | self._add_to_identity_map(obj) 167 | 168 | def _get_base_index(self, idx): 169 | # if we knew items were unique could do 170 | # return self.target.index(self[myindex]) 171 | count = -1 172 | basecount = -1 173 | if idx < 0: 174 | myindex = -(idx) - 1 175 | tbaselist = reversed(self.target) 176 | else: 177 | myindex = idx 178 | tbaselist = self.target 179 | for item in tbaselist: 180 | basecount += 1 181 | if self._is_active(item): 182 | count += 1 183 | if count == myindex: 184 | if idx < 0: 185 | return -(basecount + 1) 186 | else: 187 | return basecount 188 | raise IndexError 189 | 190 | def _add_to_identity_map(self, obj): 191 | objkey = self._identifier(obj) 192 | current = self._identity_map.get(objkey, []) 193 | current.append(obj) 194 | self._identity_map[objkey] = current 195 | 196 | def _existing_deleted_obj(self, objkey): 197 | for existing_obj in self._identity_map.get(objkey, []): 198 | if not self._is_active(existing_obj): # return 1st we find 199 | return existing_obj 200 | 201 | def _check_for_existing_on_add(self, obj): 202 | objkey = self._identifier(obj) 203 | out_obj = self._existing_deleted_obj(objkey) 204 | if out_obj is None: # no existing deleted object in list 205 | out_obj = obj 206 | self._add_to_identity_map(out_obj) 207 | else: # deleted object already in list 208 | # we are about to re-add (in active state) so must remove it first 209 | idx = self.target.index(out_obj) 210 | del self.target[idx] 211 | # We now have have to deal with original `obj` that was passed in 212 | self._unneeded_deleter(obj) 213 | 214 | self._undelete(out_obj) 215 | return out_obj 216 | 217 | def append(self, in_obj): 218 | obj = self._check_for_existing_on_add(in_obj) 219 | self.target.append(obj) 220 | 221 | def insert(self, index, value): 222 | # have some choice here so just for go for first place 223 | our_obj = self._check_for_existing_on_add(value) 224 | try: 225 | baseindex = self._get_base_index(index) 226 | except IndexError: # may be list is empty ... 227 | baseindex = len(self) 228 | self.target.insert(baseindex, our_obj) 229 | 230 | def __getitem__(self, index): 231 | baseindex = self._get_base_index(index) 232 | return self.base_modifier(self.target[baseindex]) 233 | 234 | def __item__(self, index): 235 | return self.__getitem__(self, index) 236 | 237 | def __delitem__(self, index): 238 | if not isinstance(index, slice): 239 | self._delete(self[index]) 240 | else: 241 | start = index.start 242 | end = index.stop 243 | rng = range(start, end) 244 | for ii in rng: 245 | del self[start] 246 | 247 | def __setitem__(self, index, value): 248 | if not isinstance(index, slice): 249 | del self[index] 250 | self.insert(index, value) 251 | else: 252 | if index.stop is None: 253 | stop = len(self) 254 | elif index.stop < 0: 255 | stop = len(self) + index.stop 256 | # avoid weird MemoryError when doing OurList[:] = ... 257 | elif index.stop > len(self): 258 | stop = len(self) 259 | else: 260 | stop = index.stop 261 | step = index.step or 1 262 | 263 | rng = range(index.start or 0, stop, step) 264 | if step == 1: 265 | # delete first then insert to avoid problems with indices and 266 | # statefulness 267 | for ii in rng: 268 | start = rng[0] 269 | del self[start] 270 | ii = index.start or 0 271 | for item in value: 272 | self.insert(ii, item) 273 | ii += 1 274 | else: 275 | if len(value) != len(rng): 276 | raise ValueError( 277 | 'attempt to assign sequence of size %s to ' 278 | 'extended slice of size %s' % (len(value), 279 | len(rng))) 280 | for ii, item in zip(rng, value): 281 | self[ii] = item 282 | 283 | # def __setslice__(self, start, end, values): 284 | # for ii in range(start, end): 285 | # self[ii] = values[ii-start] 286 | 287 | def __iter__(self): 288 | mytest = lambda x: self._is_active(x) 289 | myiter = itertools.ifilter(mytest, iter(self.target)) 290 | return myiter 291 | 292 | def __len__(self): 293 | return sum([1 for _ in self]) 294 | 295 | def count(self, item): 296 | myiter = itertools.ifilter(lambda v: v == item, iter(self)) 297 | counter = [1 for _ in myiter] 298 | return sum(counter) 299 | 300 | def extend(self, values): 301 | for val in values: 302 | self.append(val) 303 | 304 | def copy(self): 305 | return list(self) 306 | 307 | def clear(self): 308 | del self[0:len(self)] 309 | 310 | def pop(self, index=None): 311 | raise NotImplementedError 312 | 313 | def reverse(self): 314 | raise NotImplementedError 315 | 316 | def __repr__(self): 317 | return repr(self.target) 318 | 319 | 320 | class StatefulListDeleted(StatefulList): 321 | 322 | def _set_stateful_operators(self): 323 | self._is_active = lambda x: not self.is_active(self.base_modifier(x)) 324 | self._delete = self.undelete 325 | self._undelete = self.delete 326 | 327 | 328 | class StatefulDict(StatefulProxy): 329 | '''A stateful dictionary which only shows object in underlying dictionary 330 | which are in active state. 331 | ''' 332 | 333 | # sqlalchemy assoc proxy fails to guess this is a dictionary w/o prompting 334 | # (util.duck_type_collection appears to identify dict by looking for a set 335 | # method but dicts don't have this method!) 336 | __emulates__ = dict 337 | 338 | def __contains__(self, k): 339 | return k in self.target and self._is_active(self.target[k]) 340 | 341 | def __delitem__(self, k): 342 | # will raise KeyError if not there (which is what we want) 343 | val = self.target[k] 344 | if self._is_active(val): 345 | self._delete(val) 346 | else: 347 | raise KeyError(k) 348 | # should we raise KeyError if already deleted? 349 | 350 | def __getitem__(self, k): 351 | out = self.target[k] 352 | if self._is_active(out): 353 | return self.base_modifier(out) 354 | else: 355 | raise KeyError(k) 356 | 357 | def __iter__(self): 358 | myiter = itertools.ifilter(lambda x: self._is_active(self.target[x]), 359 | iter(self.target)) 360 | return myiter 361 | 362 | def __setitem__(self, k, v): 363 | self.target[k] = v 364 | 365 | def __len__(self): 366 | return sum([1 for _ in self]) 367 | 368 | def clear(self): 369 | for k in self: 370 | del self[k] 371 | 372 | def copy(self): 373 | # return self.__class__(self.target, base_modifier=self.base_modifier) 374 | return dict(self) 375 | 376 | def get(self, k, d=None): 377 | if k in self: 378 | return self[k] 379 | else: 380 | return d 381 | 382 | def has_key(self, k): 383 | return k in self 384 | 385 | def items(self): 386 | return [ x for x in self.iteritems() ] 387 | 388 | def iteritems(self): 389 | for k in self: 390 | yield k,self[k] 391 | 392 | def keys(self): 393 | return [ k for k in self ] 394 | 395 | def iterkeys(self): 396 | for k in self: 397 | yield k 398 | 399 | def __repr__(self): 400 | return repr(self.target) 401 | 402 | 403 | 404 | class DeferredProperty(object): 405 | def __init__(self, target_collection_name, stateful_class, **kwargs): 406 | '''Turn StatefulList into a property to allowed for deferred access 407 | (important as collections to which they are proxying may also be 408 | deferred). 409 | 410 | @param target_collection_name: name of attribute on object to which 411 | instance of stateful_class is being attached which stateful_class will 412 | 'wrap'. 413 | @param kwargs: additional arguments to stateful_class (if any) 414 | 415 | For details of other args see L{StatefulList}. 416 | ''' 417 | self.target_collection_name = target_collection_name 418 | self.stateful_class = stateful_class 419 | self.cached_kwargs = kwargs 420 | self.cached_instance_key = '_%s_%s_%s' % (type(self).__name__, 421 | self.target_collection_name, id(self)) 422 | 423 | def __get__(self, obj, class_): 424 | try: 425 | # return cached instance 426 | return getattr(obj, self.cached_instance_key) 427 | except AttributeError: 428 | # probably should do this using lazy_collections a la assoc proxy 429 | target_collection = getattr(obj, self.target_collection_name) 430 | stateful_list = self.stateful_class(target_collection, **self.cached_kwargs) 431 | # cache 432 | setattr(obj, self.cached_instance_key, stateful_list) 433 | return stateful_list 434 | 435 | def __set__(self, obj, values): 436 | # Must not replace the StatefulList object with a list, 437 | # so instead replace the values in the Stateful list with 438 | # the list values passed on. The existing values are accessed 439 | # using [:] invoking __setitem__. 440 | self.__get__(obj, None)[:] = values 441 | 442 | 443 | from sqlalchemy import __version__ as sqla_version 444 | import sqlalchemy.ext.associationproxy 445 | import weakref 446 | # write our own assoc proxy which excludes scalar support and therefore calls 447 | # which will not work since underlying list is a StatefuList not a normal 448 | # collection 449 | class OurAssociationProxy(sqlalchemy.ext.associationproxy.AssociationProxy): 450 | 451 | scalar = False 452 | def _target_is_scalar(self): 453 | return False 454 | 455 | 456 | # TODO: 2009-07-24 support dict collections 457 | def add_stateful_m2m(object_to_alter, m2m_object, m2m_property_name, 458 | attr, basic_m2m_name, **kwargs): 459 | '''Attach active and deleted stateful lists along with the association 460 | proxy based on the active list to original object (object_to_alter). 461 | 462 | To illustrate if one has:: 463 | 464 | class Package(object): 465 | 466 | # package_licenses is the basic_m2m_name attribute 467 | # 468 | # it should come from a simple relation pointing to PackageLicense 469 | # and returns PackageLicense objects (so do *not* use secondary 470 | # keyword) 471 | # 472 | # It will usually not be defined here but in the Package mapper: 473 | # 474 | # 'package_licenses':relation(License) ... 475 | 476 | package_licenses = ... from_mapper ... 477 | 478 | Then after running:: 479 | 480 | add_stateful_m2m(Package, PackageLicense, 'licenses', 'license', 481 | 'package_licenses') 482 | 483 | there will be additional properties: 484 | 485 | # NB: licenses_active and licenses_deleted are lists of PackageLicense 486 | # objects while licenses (being an assoc proxy) is a list of Licenses 487 | licenses_active # these are active PackageLicenses 488 | licenses_deleted # these are deleted PackageLicenses 489 | licenses # these are active *Licenses* 490 | 491 | @param attr: the name of the attribute on the Join object corresponding to 492 | the target (e.g. in this case 'license' on PackageLicense). 493 | @arg **kwargs: these are passed on to the DeferredProperty. 494 | ''' 495 | active_name = m2m_property_name + '_active' 496 | # in the join object (e.g. PackageLicense) the License object accessible by 497 | # the license attribute will be what we need for our identity map 498 | if not 'identifier' in kwargs: 499 | def identifier(joinobj): 500 | return getattr(joinobj, attr) 501 | kwargs['identifier'] = identifier 502 | if not 'unneeded_deleter' in kwargs: 503 | # For sqlalchemy this means expunging the object ... 504 | # (Note we can assume object is 'new' since it didn't match any 505 | # existing one in -- see _check_for_existing_on_add in StatefulList) 506 | # (NB: this assumption turns out to be false in some special cases -- 507 | # see comments in test in test_demo.py:TestVersioning2.test_2) 508 | from sqlalchemy.orm import object_session 509 | def _f(obj_to_delete): 510 | sess = object_session(obj_to_delete) 511 | if sess: # for tests at least must support obj not being sqlalchemy 512 | sess.expunge(obj_to_delete) 513 | kwargs['unneeded_deleter'] = _f 514 | 515 | active_prop = DeferredProperty(basic_m2m_name, StatefulList, **kwargs) 516 | deleted_name = m2m_property_name + '_deleted' 517 | deleted_prop = DeferredProperty(basic_m2m_name, StatefulListDeleted, 518 | **kwargs) 519 | setattr(object_to_alter, active_name, active_prop) 520 | setattr(object_to_alter, deleted_name, deleted_prop) 521 | create_m2m = make_m2m_creator_for_assocproxy(m2m_object, attr) 522 | setattr(object_to_alter, m2m_property_name, 523 | OurAssociationProxy(active_name, attr, creator=create_m2m) 524 | ) 525 | 526 | 527 | def make_m2m_creator_for_assocproxy(m2m_object, attrname): 528 | '''This creates a_creator function for SQLAlchemy associationproxy pattern. 529 | 530 | @param m2m_object: the m2m object underlying association proxy. 531 | @param attrname: the attrname to use for the default object passed in to m2m 532 | ''' 533 | def create_m2m(foreign, **kw): 534 | mykwargs = dict(kw) 535 | mykwargs[attrname] = foreign 536 | return m2m_object(**mykwargs) 537 | 538 | return create_m2m 539 | -------------------------------------------------------------------------------- /vdm/sqlalchemy/test_demo.py: -------------------------------------------------------------------------------- 1 | import logging 2 | # logging.basicConfig(level=logging.DEBUG) 3 | logging.basicConfig(level=logging.INFO) 4 | logger = logging.getLogger('vdm') 5 | 6 | from sqlalchemy.orm import object_session, class_mapper 7 | 8 | import vdm.sqlalchemy 9 | from demo import * 10 | 11 | from sqlalchemy import __version__ as sqav 12 | if sqav.startswith("0.4"): 13 | _clear = Session.clear 14 | else: 15 | _clear = Session.expunge_all 16 | 17 | class Test_01_SQLAlchemySession: 18 | @classmethod 19 | def setup_class(self): 20 | repo.rebuild_db() 21 | @classmethod 22 | def teardown_class(self): 23 | Session.remove() 24 | 25 | def test_1(self): 26 | assert not hasattr(Session, 'revision') 27 | assert vdm.sqlalchemy.SQLAlchemySession.at_HEAD(Session) 28 | rev = Revision() 29 | vdm.sqlalchemy.SQLAlchemySession.set_revision(Session, rev) 30 | assert vdm.sqlalchemy.SQLAlchemySession.at_HEAD(Session) 31 | assert Session.revision is not None 32 | out = vdm.sqlalchemy.SQLAlchemySession.get_revision(Session) 33 | assert out == rev 34 | out = vdm.sqlalchemy.SQLAlchemySession.get_revision(Session()) 35 | assert out == rev 36 | assert vdm.sqlalchemy.SQLAlchemySession.at_HEAD(Session) 37 | assert vdm.sqlalchemy.SQLAlchemySession.at_HEAD(Session()) 38 | Session.remove() 39 | 40 | 41 | class Test_02_Versioning: 42 | @classmethod 43 | def setup_class(self): 44 | repo.rebuild_db() 45 | 46 | logger.debug('===== STARTING REV 1') 47 | session = Session() 48 | rev1 = Revision() 49 | session.add(rev1) 50 | vdm.sqlalchemy.SQLAlchemySession.set_revision(session, rev1) 51 | 52 | self.name1 = 'anna' 53 | self.name2 = 'warandpeace' 54 | self.title1 = 'XYZ' 55 | self.title2 = 'ABC' 56 | self.notes1 = u'Here\nare some\nnotes' 57 | self.notes2 = u'Here\nare no\nnotes' 58 | lic1 = License(name='blah', open=True) 59 | lic1.revision = rev1 60 | lic2 = License(name='foo', open=True) 61 | p1 = Package(name=self.name1, title=self.title1, license=lic1, notes=self.notes1) 62 | p2 = Package(name=self.name2, title=self.title1, license=lic1) 63 | session.add_all([lic1,lic2,p1,p2]) 64 | 65 | logger.debug('***** Committing/Flushing Rev 1') 66 | session.commit() 67 | # can only get it after the flush 68 | self.rev1_id = rev1.id 69 | _clear() 70 | Session.remove() 71 | 72 | logger.debug('===== STARTING REV 2') 73 | session = Session() 74 | rev2 = Revision() 75 | session.add(rev2) 76 | vdm.sqlalchemy.SQLAlchemySession.set_revision(session, rev2) 77 | outlic1 = Session.query(License).filter_by(name='blah').first() 78 | outlic2 = Session.query(License).filter_by(name='foo').first() 79 | outlic2.open = False 80 | outp1 = Session.query(Package).filter_by(name=self.name1).one() 81 | outp2 = Session.query(Package).filter_by(name=self.name2).one() 82 | outp1.title = self.title2 83 | outp1.notes = self.notes2 84 | outp1.license = outlic2 85 | t1 = Tag(name='geo') 86 | session.add_all([outp1,outp2,t1]) 87 | outp1.tags = [t1] 88 | outp2.delete() 89 | # session.flush() 90 | session.commit() 91 | # must do this after flush as timestamp not set until then 92 | self.ts2 = rev2.timestamp 93 | self.rev2_id = rev2.id 94 | Session.remove() 95 | 96 | @classmethod 97 | def teardown_class(self): 98 | Session.remove() 99 | 100 | def test_revisions_exist(self): 101 | revs = Session.query(Revision).all() 102 | assert len(revs) == 2 103 | # also check order (youngest first) 104 | assert revs[0].timestamp > revs[1].timestamp 105 | 106 | def test_revision_youngest(self): 107 | rev = Revision.youngest(Session) 108 | assert rev.timestamp == self.ts2 109 | 110 | def test_basic(self): 111 | assert Session.query(License).count() == 2, Session.query(License).count() 112 | assert Session.query(Package).count() == 2, Session.query(Package).count() 113 | assert hasattr(LicenseRevision, 'revision_id') 114 | assert Session.query(LicenseRevision).count() == 3, Session.query(LicenseRevision).count() 115 | assert Session.query(PackageRevision).count() == 4, Session.query(PackageRevision).count() 116 | 117 | def test_all_revisions(self): 118 | p1 = Session.query(Package).filter_by(name=self.name1).one() 119 | assert len(p1.all_revisions) == 2 120 | # problem here is that it might pass even if broken because ordering of 121 | # uuid ids is 'right' 122 | revs = [ pr.revision for pr in p1.all_revisions ] 123 | assert revs[0].timestamp > revs[1].timestamp, revs 124 | 125 | def test_basic_2(self): 126 | # should be at HEAD (i.e. rev2) by default 127 | p1 = Session.query(Package).filter_by(name=self.name1).one() 128 | assert p1.license.open == False 129 | assert p1.revision.timestamp == self.ts2 130 | # assert p1.tags == [] 131 | assert len(p1.tags) == 1 132 | 133 | def test_basic_continuity(self): 134 | p1 = Session.query(Package).filter_by(name=self.name1).one() 135 | pr1 = Session.query(PackageRevision).filter_by(name=self.name1).first() 136 | table = class_mapper(PackageRevision).mapped_table 137 | print table.c.keys() 138 | print pr1.continuity_id 139 | assert pr1.continuity == p1 140 | 141 | def test_basic_state(self): 142 | p1 = Session.query(Package).filter_by(name=self.name1).one() 143 | p2 = Session.query(Package).filter_by(name=self.name2).one() 144 | assert p1.state 145 | assert p1.state == State.ACTIVE 146 | assert p2.state == State.DELETED 147 | 148 | def test_versioning_0(self): 149 | p1 = Session.query(Package).filter_by(name=self.name1).one() 150 | rev1 = Session.query(Revision).get(self.rev1_id) 151 | p1r1 = p1.get_as_of(rev1) 152 | assert p1r1.continuity == p1 153 | 154 | def test_versioning_1(self): 155 | p1 = Session.query(Package).filter_by(name=self.name1).one() 156 | rev1 = Session.query(Revision).get(self.rev1_id) 157 | p1r1 = p1.get_as_of(rev1) 158 | assert p1r1.name == self.name1 159 | assert p1r1.title == self.title1 160 | 161 | def test_traversal_normal_fks_and_state_at_same_time(self): 162 | p2 = Session.query(Package).filter_by(name=self.name2).one() 163 | rev1 = Session.query(Revision).get(self.rev1_id) 164 | p2r1 = p2.get_as_of(rev1) 165 | assert p2r1.state == State.ACTIVE 166 | 167 | def test_versioning_traversal_fks(self): 168 | p1 = Session.query(Package).filter_by(name=self.name1).one() 169 | rev1 = Session.query(Revision).get(self.rev1_id) 170 | p1r1 = p1.get_as_of(rev1) 171 | assert p1r1.license.open == True 172 | 173 | def test_versioning_m2m_1(self): 174 | p1 = Session.query(Package).filter_by(name=self.name1).one() 175 | rev1 = Session.query(Revision).get(self.rev1_id) 176 | ptag = p1.package_tags[0] 177 | # does not exist 178 | assert ptag.get_as_of(rev1) == None 179 | 180 | def test_versioning_m2m(self): 181 | p1 = Session.query(Package).filter_by(name=self.name1).one() 182 | rev1 = Session.query(Revision).get(self.rev1_id) 183 | p1r1 = p1.get_as_of(rev1) 184 | assert len(p1.tags_active) == 0 185 | # NB: deleted includes tags that were non-existent 186 | assert len(p1.tags_deleted) == 1 187 | assert len(p1.tags) == 0 188 | assert len(p1r1.tags) == 0 189 | 190 | def test_revision_has_state(self): 191 | rev1 = Session.query(Revision).get(self.rev1_id) 192 | assert rev1.state == State.ACTIVE 193 | 194 | def test_diff(self): 195 | p1 = Session.query(Package).filter_by(name=self.name1).one() 196 | pr2, pr1 = p1.all_revisions 197 | # pr1, pr2 = prs[::-1] 198 | 199 | diff = p1.diff_revisioned_fields(pr2, pr1, Package) 200 | assert diff['title'] == '- XYZ\n+ ABC', diff['title'] 201 | assert diff['notes'] == ' Here\n- are some\n+ are no\n notes', diff['notes'] 202 | assert diff['license_id'] == '- 1\n+ 2', diff['license_id'] 203 | 204 | diff1 = p1.diff(pr2.revision, pr1.revision) 205 | assert diff1 == diff, (diff1, diff) 206 | 207 | diff2 = p1.diff() 208 | assert diff2 == diff, (diff2, diff) 209 | 210 | def test_diff_2(self): 211 | '''Test diffing at a revision where just created.''' 212 | p1 = Session.query(Package).filter_by(name=self.name1).one() 213 | pr2, pr1 = p1.all_revisions 214 | 215 | diff1 = p1.diff(to_revision=pr1.revision) 216 | assert diff1['title'] == u'- None\n+ XYZ', diff1 217 | 218 | 219 | class Test_03_StatefulVersioned: 220 | @classmethod 221 | def setup_class(self): 222 | repo.rebuild_db() 223 | logger.info('====== TestVersioning2: start') 224 | 225 | # create a package with some tags 226 | rev1 = repo.new_revision() 227 | self.name1 = 'anna' 228 | p1 = Package(name=self.name1) 229 | t1 = Tag(name='geo') 230 | t2 = Tag(name='geo2') 231 | p1.tags.append(t1) 232 | p1.tags.append(t2) 233 | Session.add_all([p1,t1,t2]) 234 | Session.commit() 235 | self.rev1_id = rev1.id 236 | Session.remove() 237 | 238 | # now remove those tags 239 | logger.debug('====== start Revision 2') 240 | rev2 = repo.new_revision() 241 | newp1 = Session.query(Package).filter_by(name=self.name1).one() 242 | # either one works 243 | newp1.tags = [] 244 | # newp1.tags_active.clear() 245 | assert len(newp1.tags_active) == 0 246 | Session.commit() 247 | self.rev2_id = rev2.id 248 | Session.remove() 249 | 250 | # now add one of them back 251 | logger.debug('====== start Revision 3') 252 | rev3 = repo.new_revision() 253 | newp1 = Session.query(Package).filter_by(name=self.name1).one() 254 | self.tagname1 = 'geo' 255 | t1 = Session.query(Tag).filter_by(name=self.tagname1).one() 256 | assert t1 257 | newp1.tags.append(t1) 258 | repo.commit_and_remove() 259 | 260 | @classmethod 261 | def teardown_class(self): 262 | Session.remove() 263 | 264 | def test_0_remove_and_readd_m2m(self): 265 | p1 = Session.query(Package).filter_by(name=self.name1).one() 266 | assert len(p1.package_tags) == 2, p1.package_tags 267 | assert len(p1.tags_active) == 1, p1.tags_active 268 | assert len(p1.tags) == 1 269 | Session.remove() 270 | 271 | def test_1_underlying_is_right(self): 272 | rev1 = Session.query(Revision).get(self.rev1_id) 273 | ptrevs = Session.query(PackageTagRevision).filter_by(revision_id=rev1.id).all() 274 | assert len(ptrevs) == 2 275 | for pt in ptrevs: 276 | assert pt.state == State.ACTIVE 277 | 278 | rev2 = Session.query(Revision).get(self.rev2_id) 279 | ptrevs = Session.query(PackageTagRevision).filter_by(revision_id=rev2.id).all() 280 | assert len(ptrevs) == 2 281 | for pt in ptrevs: 282 | assert pt.state == State.DELETED 283 | 284 | # test should be higher up but need at least 3 revisions for problem to 285 | # show up 286 | def test_2_get_as_of(self): 287 | p1 = Session.query(Package).filter_by(name=self.name1).one() 288 | rev2 = Session.query(Revision).get(self.rev2_id) 289 | # should be 2 deleted and 1 as None 290 | ptrevs = [ pt.get_as_of(rev2) for pt in p1.package_tags ] 291 | print ptrevs 292 | print Session.query(PackageTagRevision).all() 293 | assert ptrevs[0].revision_id == rev2.id 294 | 295 | def test_3_remove_and_readd_m2m_2(self): 296 | num_package_tags = 2 297 | rev1 = Session.query(Revision).get(self.rev1_id) 298 | p1 = Session.query(Package).filter_by(name=self.name1).one() 299 | p1rev = p1.get_as_of(rev1) 300 | # NB: relations on revision object proxy to continuity 301 | # (though with get_as_of revision set) 302 | assert len(p1rev.package_tags) == num_package_tags 303 | assert len(p1rev.tags) == 2 304 | Session.remove() 305 | 306 | rev2 = Session.query(Revision).get(self.rev2_id) 307 | p1 = Session.query(Package).filter_by(name=self.name1).one() 308 | p2rev = p1.get_as_of(rev2) 309 | assert p2rev.__class__ == PackageRevision 310 | assert len(p2rev.package_tags) == num_package_tags 311 | print rev2.id 312 | print p2rev.tags_active 313 | assert len(p2rev.tags) == 0 314 | 315 | 316 | class Test_04_StatefulVersioned2: 317 | '''Similar to previous but setting m2m list using existing objects''' 318 | 319 | def setup(self): 320 | Session.remove() 321 | repo.rebuild_db() 322 | logger.info('====== TestStatefulVersioned2: start') 323 | 324 | # create a package with some tags 325 | rev1 = repo.new_revision() 326 | self.name1 = 'anna' 327 | p1 = Package(name=self.name1) 328 | t1 = Tag(name='geo') 329 | p1.tags.append(t1) 330 | Session.add_all([p1,t1]) 331 | Session.commit() 332 | self.rev1_id = rev1.id 333 | Session.remove() 334 | 335 | def setup_method(self, name=''): 336 | self.setup() 337 | 338 | @classmethod 339 | def teardown_class(self): 340 | Session.remove() 341 | 342 | def _test_package_tags(self, check_all_pkg_tags=True): 343 | p1 = Session.query(Package).filter_by(name=self.name1).one() 344 | assert len(p1.package_tags) == 2, p1.package_tags 345 | all_pkg_tags = Session.query(PackageTag).all() 346 | if check_all_pkg_tags: 347 | assert len(all_pkg_tags) == 2 348 | 349 | def _test_tags(self): 350 | p1 = Session.query(Package).filter_by(name=self.name1).one() 351 | assert len(p1.tags) == 2, p1.tags 352 | 353 | def test_1(self): 354 | rev2 = repo.new_revision() 355 | newp1 = Session.query(Package).filter_by(name=self.name1).one() 356 | t1 = Session.query(Tag).filter_by(name='geo').one() 357 | t2 = Tag(name='geo2') 358 | newp1.tags = [ t1, t2 ] 359 | repo.commit_and_remove() 360 | 361 | self._test_package_tags() 362 | self._test_tags() 363 | 364 | def test_2(self): 365 | rev2 = repo.new_revision() 366 | newp1 = Session.query(Package).filter_by(name=self.name1).one() 367 | t1 = Session.query(Tag).filter_by(name='geo').one() 368 | t2 = Tag(name='geo2') 369 | print '**** setting tags' 370 | newp1.tags[:] = [ t1, t2 ] 371 | repo.commit_and_remove() 372 | 373 | # TODO: (?) check on No of PackageTags fails 374 | # the story is that an extra PackageTag for first tag gets constructed 375 | # even though existing in deleted state (as expected) 376 | # HOWEVER (unlike in 3 other cases in this class) this PackageTag is 377 | # *already committed* when it arrives at _check_for_existing_on_add and 378 | # therefore expunge has no effect on it (we'd need to delete and that 379 | # may start getting 'hairy' ...) 380 | self._test_package_tags(check_all_pkg_tags=False) 381 | self._test_tags() 382 | 383 | def test_3(self): 384 | rev2 = repo.new_revision() 385 | newp1 = Session.query(Package).filter_by(name=self.name1).one() 386 | t1 = Session.query(Tag).filter_by(name='geo').one() 387 | t2 = Tag(name='geo2') 388 | newp1.tags[0] = t1 389 | newp1.tags.append(t2) 390 | repo.commit_and_remove() 391 | 392 | self._test_package_tags() 393 | self._test_tags() 394 | 395 | def test_4(self): 396 | rev2 = repo.new_revision() 397 | newp1 = Session.query(Package).filter_by(name=self.name1).one() 398 | t1 = Session.query(Tag).filter_by(name='geo').one() 399 | t2 = Tag(name='geo2') 400 | newp1.tags = [ t1, t2 ] 401 | newp1.tags[0] = t1 402 | del newp1.tags[1] 403 | newp1.tags.append(t2) 404 | # NB: doing this the other way round will result in 3 PackageTags 405 | # newp1.tags.append(t2) 406 | # del newp1.tags[1] 407 | # this is because our system can't work out that we've just added and 408 | # deleted the same tag 409 | repo.commit_and_remove() 410 | 411 | self._test_package_tags() 412 | self._test_tags() 413 | 414 | 415 | class Test_05_RevertAndPurge: 416 | 417 | @classmethod 418 | def setup_class(self): 419 | Session.remove() 420 | repo.rebuild_db() 421 | 422 | rev1 = Revision() 423 | Session.add(rev1) 424 | vdm.sqlalchemy.SQLAlchemySession.set_revision(Session, rev1) 425 | 426 | self.name1 = 'anna' 427 | p1 = Package(name=self.name1) 428 | p2 = Package(name='blahblah') 429 | Session.add_all([p1,p2]) 430 | repo.commit_and_remove() 431 | 432 | self.name2 = 'warandpeace' 433 | self.lname = 'testlicense' 434 | rev2 = repo.new_revision() 435 | p1 = Session.query(Package).filter_by(name=self.name1).one() 436 | p1.name = self.name2 437 | l1 = License(name=self.lname) 438 | Session.add_all([p1,l1]) 439 | repo.commit() 440 | self.rev2id = rev2.id 441 | Session.remove() 442 | 443 | @classmethod 444 | def teardown_class(self): 445 | Session.remove() 446 | repo.rebuild_db() 447 | 448 | def test_basics(self): 449 | revs = Session.query(Revision).all() 450 | assert len(revs) == 2 451 | p1 = Session.query(Package).filter_by(name=self.name2).one() 452 | assert p1.name == self.name2 453 | assert len(Session.query(Package).all()) == 2 454 | 455 | def test_list_changes(self): 456 | rev2 = Session.query(Revision).get(self.rev2id) 457 | out = repo.list_changes(rev2) 458 | assert len(out) == 3 459 | assert len(out[Package]) == 1, out 460 | assert len(out[License]) == 1, out 461 | 462 | def test_purge_revision(self): 463 | logger.debug('BEGINNING PURGE REVISION') 464 | Session.remove() 465 | rev2 = Session.query(Revision).get(self.rev2id) 466 | repo.purge_revision(rev2) 467 | revs = Session.query(Revision).all() 468 | assert len(revs) == 1 469 | p1 = Session.query(Package).filter_by(name=self.name1).first() 470 | assert p1 is not None 471 | assert len(Session.query(License).all()) == 0 472 | pkgs = Session.query(Package).all() 473 | assert len(pkgs) == 2, pkgrevs 474 | pkgrevs = Session.query(PackageRevision).all() 475 | assert len(pkgrevs) == 2, pkgrevs 476 | 477 | -------------------------------------------------------------------------------- /vdm/sqlalchemy/test_demo_misc.py: -------------------------------------------------------------------------------- 1 | from sqlalchemy.orm import object_session 2 | from sqlalchemy import __version__ as sqav 3 | from demo import * 4 | from base import * 5 | 6 | class TestMisc: 7 | @classmethod 8 | def teardown_class(self): 9 | repo.rebuild_db() 10 | 11 | def test_copy_column(self): 12 | t1 = package_table 13 | newtable = Table('mytable', metadata) 14 | copy_column('id', t1, newtable) 15 | outcol = newtable.c['id'] 16 | assert outcol.name == 'id' 17 | assert outcol.primary_key == True 18 | # pick one with a fk 19 | name = 'license_id' 20 | copy_column(name, t1, newtable) 21 | incol = t1.c[name] 22 | outcol = newtable.c[name] 23 | assert outcol != incol 24 | assert outcol.key == incol.key 25 | assert len(incol.foreign_keys) == 1 26 | assert len(outcol.foreign_keys) == 1 27 | infk = list(incol.foreign_keys)[0] 28 | outfk = list(outcol.foreign_keys)[0] 29 | assert infk.parent is not None 30 | assert outfk.parent is not None 31 | 32 | def test_table_copy(self): 33 | t1 = package_table 34 | newtable = Table('newtable', metadata) 35 | copy_table(t1, newtable) 36 | assert len(newtable.c) == len(t1.c) 37 | # pick one with a fk 38 | incol = t1.c['license_id'] 39 | outcol = None 40 | for col in newtable.c: 41 | if col.name == 'license_id': 42 | outcol = col 43 | assert outcol != incol 44 | assert outcol.key == incol.key 45 | assert len(incol.foreign_keys) == 1 46 | assert len(outcol.foreign_keys) == 1 47 | infk = list(incol.foreign_keys)[0] 48 | outfk = list(outcol.foreign_keys)[0] 49 | assert infk.parent is not None 50 | assert outfk.parent is not None 51 | 52 | def test_package_tag_table(self): 53 | col = package_tag_table.c['tag_id'] 54 | assert len(col.foreign_keys) == 1 55 | 56 | def test_make_stateful(self): 57 | assert 'state' in package_table.c 58 | 59 | def test_make_revision_table(self): 60 | assert package_revision_table.name == 'package_revision' 61 | assert 'revision_id' in package_table.c 62 | assert 'state' in package_revision_table.c 63 | assert 'revision_id' in package_revision_table.c 64 | # very crude ... 65 | assert len(package_revision_table.c) == len(package_table.c) + 1 66 | # these tests may seem odd but they would incorporated following a bug 67 | # where this was *not* the case 68 | base = package_table 69 | rev = package_revision_table 70 | # crude (could be more specific about the fk) 71 | assert len(rev.c['revision_id'].foreign_keys) == 1 72 | assert rev.c['revision_id'].primary_key 73 | assert rev.c['id'].primary_key 74 | print rev.primary_key.columns 75 | assert len(rev.primary_key.columns) == 2 76 | 77 | def test_accessing_columns_on_object(self): 78 | table = class_mapper(Package).mapped_table 79 | print table.c.keys() 80 | assert len(table.c.keys()) > 0 81 | assert 'revision_id' in table.c.keys() 82 | 83 | -------------------------------------------------------------------------------- /vdm/sqlalchemy/test_stateful.py: -------------------------------------------------------------------------------- 1 | from stateful import * 2 | 3 | ACTIVE = 'active' 4 | DELETED = 'deleted' 5 | 6 | 7 | class Stateful(object): 8 | def __init__(me, name='', order=None, state=ACTIVE): 9 | me.name = name 10 | me.state = state 11 | me.order = order 12 | 13 | def delete(me): 14 | me.state = DELETED 15 | 16 | def undelete(me): 17 | me.state = ACTIVE 18 | 19 | def __repr__(me): 20 | return '' % (me.name, me.state) 21 | 22 | def delete(st): 23 | st.delete() 24 | 25 | def undelete(st): 26 | st.undelete() 27 | 28 | def is_active(st): 29 | return st.state == ACTIVE 30 | 31 | 32 | class TestStatefulList: 33 | active = ACTIVE 34 | deleted = DELETED 35 | 36 | def setup(self): 37 | self.sb = Stateful('b', state=self.deleted) 38 | self.baselist = [ 39 | Stateful('a'), 40 | self.sb, 41 | Stateful('c', state=self.deleted), 42 | Stateful('d'), 43 | ] 44 | self.sa = self.baselist[0] 45 | self.sc = self.baselist[2] 46 | self.se = Stateful('e') 47 | self.sf = Stateful('f') 48 | self.slist = StatefulList(self.baselist, is_active=is_active) 49 | # TODO: more testing of StatefulListDeleted 50 | self.slist_deleted = StatefulListDeleted(self.baselist, is_active=is_active) 51 | self.startlen = 2 52 | self.startlen_base = 4 53 | 54 | def setup_method(self, name=''): 55 | self.setup() 56 | 57 | def test__get_base_index(self): 58 | exp = [0, 3] 59 | out = [-1, -1] 60 | for ii in range(2): 61 | out[ii] = self.slist._get_base_index(ii) 62 | assert exp == out 63 | 64 | def test___len__(self): 65 | assert len(self.baselist) == self.startlen_base 66 | assert len(self.slist) == self.startlen 67 | assert len(self.slist_deleted) == 2 68 | 69 | def test___get_item__(self): 70 | assert self.slist[1] == self.baselist[3] 71 | 72 | def test___get_item___with_negative_index(self): 73 | assert self.slist[-1] == self.baselist[-1] 74 | 75 | def test_append(self): 76 | assert len(self.baselist) == self.startlen_base 77 | assert len(self.slist) == self.startlen 78 | 79 | # not in the list 80 | self.slist.append(self.se) 81 | assert len(self.baselist) == self.startlen_base + 1 82 | assert len(self.slist) == self.startlen + 1 83 | 84 | def test_insert(self): 85 | self.slist.insert(0, self.se) 86 | assert len(self.baselist) == 5 87 | assert len(self.slist) == 3 88 | 89 | def test_delete(self): 90 | del self.slist[0] 91 | assert len(self.baselist) == self.startlen_base 92 | assert len(self.slist) == self.startlen - 1 93 | assert self.baselist[0].state == self.deleted 94 | 95 | def test___setitem__0(self): 96 | self.slist[0] = self.sf 97 | assert len(self.slist) == self.startlen 98 | assert self.slist[0].name == 'f' 99 | 100 | assert self.baselist[0].name == 'a' 101 | assert self.baselist[0].state == self.deleted 102 | assert len(self.baselist) == self.startlen_base + 1 103 | 104 | def test___setitem__2(self): 105 | # obviously this would't work since it is setting to list object itself 106 | # self.slist = [1,2,3] 107 | # in our vdm code does not matter since OurAssociationProxy has a 108 | # special __set__ which takes of this (converts to clear() + set) 109 | self.slist[:] = [] 110 | assert len(self.baselist) == self.startlen_base 111 | assert len(self.slist) == 0 112 | for item in self.baselist: 113 | assert item.state == self.deleted 114 | 115 | def test_count(self): 116 | assert self.slist.count(self.sb) == 0 117 | assert self.slist.count(self.sa) == 1 118 | 119 | def test_extend(self): 120 | self.slist.extend([self.se, self.sf]) 121 | assert len(self.slist) == 4 122 | assert len(self.baselist) == 6 123 | 124 | def test___contains__(self): 125 | assert self.sa in self.slist 126 | assert self.sb not in self.slist 127 | 128 | def test_clear(self): 129 | self.slist.clear() 130 | assert len(self.slist) == 0 131 | 132 | def test___repr__(self): 133 | out = repr(self.slist) 134 | assert out, out 135 | 136 | class TestStatefulListComplex: 137 | active = ACTIVE 138 | deleted = DELETED 139 | 140 | def setup(self): 141 | self.sb = Stateful('b', state=self.deleted) 142 | self.baselist = [ 143 | Stateful('a'), 144 | self.sb, 145 | Stateful('c', state=self.deleted), 146 | Stateful('d'), 147 | ] 148 | self.sa = self.baselist[0] 149 | self.sc = self.baselist[2] 150 | self.se = Stateful('e') 151 | self.sf = Stateful('f') 152 | identifier = lambda statefulobj: statefulobj.name 153 | self.slist = StatefulList(self.baselist, is_active=is_active, 154 | identifier=identifier) 155 | self.startlen = 2 156 | self.startlen_base = 4 157 | 158 | # py.test 159 | def setup_method(self, name=''): 160 | self.setup() 161 | 162 | def test_append(self): 163 | # already in the list but deleted 164 | self.slist.append(self.sb) 165 | assert len(self.baselist) == self.startlen_base 166 | assert len(self.slist) == self.startlen + 1 167 | # ensure it has moved to the end ... 168 | assert self.slist[-1] == self.sb 169 | assert self.baselist[-1] == self.sb 170 | 171 | def test_append_different_obj(self): 172 | newsb = Stateful('b', order=1) 173 | self.slist.append(newsb) 174 | assert len(self.slist) == self.startlen + 1 175 | assert len(self.baselist) == self.startlen_base 176 | 177 | def _test_append_with_unique(self): 178 | # already in the list but active 179 | have_exception = False 180 | try: 181 | self.slist.append(self.sa) 182 | except: 183 | have_exception = True 184 | assert have_exception, 'Should raise exception on append of active' 185 | 186 | def test___setitem__with_same_object(self): 187 | self.slist[0] = self.sa 188 | # should have no change 189 | assert len(self.slist) == self.startlen 190 | assert len(self.baselist) == self.startlen_base 191 | 192 | 193 | class TestStatefulDict: 194 | active = ACTIVE 195 | deleted = DELETED 196 | 197 | def setup(self): 198 | self.basedict = { 199 | 'a': Stateful('a'), 200 | 'b': Stateful('b', state=self.deleted), 201 | 'c': Stateful('c', state=self.deleted), 202 | 'd': Stateful('d'), 203 | } 204 | self.sa = self.basedict['a'] 205 | self.sb = self.basedict['b'] 206 | self.sc = self.basedict['c'] 207 | self.se = Stateful('e') 208 | self.sf = Stateful('f') 209 | self.sdict = StatefulDict(self.basedict, is_active=is_active) 210 | # TODO: test deleted version 211 | 212 | # py.test compatibility 213 | def setup_method(self, name=''): 214 | self.setup() 215 | 216 | def test__contains__(self): 217 | assert 'a' in self.sdict 218 | assert not 'b' in self.sdict 219 | assert not 'fajd' in self.sdict 220 | 221 | def test___delitem__(self): 222 | del self.sdict['a'] 223 | assert 'a' not in self.sdict 224 | assert 'a' in self.basedict 225 | 226 | def test___getitem__(self): 227 | out = self.sdict['a'] 228 | assert out.state == ACTIVE 229 | assert out.name == 'a' 230 | 231 | def test___iter__(self): 232 | # tested by __len__ etc 233 | pass 234 | 235 | def test___len__(self): 236 | assert len(self.sdict) == 2 237 | 238 | def test___setitem__(self): 239 | self.sdict['e'] = self.se 240 | assert len(self.sdict) == 3 241 | self.sdict['a'] = self.sf 242 | assert self.sdict['a'].name == 'f' 243 | 244 | def test_clear(self): 245 | self.sdict.clear() 246 | assert len(self.sdict) == 0 247 | assert len(self.basedict) == 4 248 | 249 | def test_copy(self): 250 | # TODO: implement this in StatefulDict 251 | # self.sdict.copy() 252 | pass 253 | 254 | def test_get(self): 255 | out = self.sdict.get('a', None) 256 | assert out 257 | out = self.sdict.get('b', None) 258 | assert not out 259 | 260 | def test_has_key(self): 261 | assert self.sdict.has_key('a') 262 | assert not self.sdict.has_key('b') 263 | assert not self.sdict.has_key('xxxx') 264 | 265 | def test_items(self): 266 | out = self.sdict.items() 267 | assert len(out) == 2 268 | assert out[0][0] == 'a' 269 | assert out[1][0] == 'd' 270 | 271 | def test_iteritems(self): 272 | # tested in items 273 | pass 274 | 275 | def test_iterkeys(self): 276 | keys = [k for k in self.sdict.iterkeys()] 277 | assert keys == ['a', 'd'], keys 278 | 279 | def test_itervalues(self): 280 | # tested in values 281 | pass 282 | 283 | def test_keys(self): 284 | out = self.sdict.keys() 285 | assert isinstance(out, list) 286 | assert len(out) == 2 287 | 288 | def values(self): 289 | out = self.sdict.values() 290 | assert isinstance(out, list) 291 | assert len(out) == 2 292 | assert out[0].name == 'a' 293 | 294 | 295 | # not yet implemented 296 | # def pop(self): 297 | # pass 298 | # 299 | # def popitem(self): 300 | # pass 301 | 302 | -------------------------------------------------------------------------------- /vdm/sqlalchemy/test_stateful_collections.py: -------------------------------------------------------------------------------- 1 | from sqlalchemy import * 2 | from sqlalchemy.orm import * 3 | 4 | from stateful import * 5 | 6 | engine = create_engine('sqlite:///:memory:', echo=True) 7 | 8 | metadata = MetaData(bind=engine) 9 | 10 | license_table = Table('license', metadata, 11 | Column('id', Integer, primary_key=True), 12 | Column('name', String(100)), 13 | ) 14 | 15 | package_table = Table('package', metadata, 16 | Column('id', String(100), primary_key=True), 17 | ) 18 | 19 | package_license_table = Table('package_license', metadata, 20 | Column('id', Integer, primary_key=True), 21 | Column('package_id', Integer, ForeignKey('package.id')), 22 | Column('license_id', Integer, ForeignKey('license.id')), 23 | Column('state', String, default='active'), 24 | ) 25 | 26 | metadata.create_all(engine) 27 | 28 | 29 | from sqlalchemy.orm import scoped_session, sessionmaker, create_session 30 | from sqlalchemy.orm import relation, backref 31 | SessionObject = scoped_session(create_session) 32 | session = SessionObject() 33 | 34 | from sqlalchemy.orm import mapper 35 | #mapper = SessionObject.mapper 36 | 37 | from sqlalchemy.ext.associationproxy import association_proxy 38 | from sqlalchemy.orm.collections import attribute_mapped_collection 39 | 40 | from sqlalchemy import __version__ as sqav 41 | if sqav.startswith("0.4"): 42 | _clear = session.clear 43 | else: 44 | _clear = session.expunge_all 45 | 46 | class BaseObject(object): 47 | 48 | def __repr__(self): 49 | return '<%s %s>' % (self.__class__.__name__, self.id) 50 | 51 | def delete(st): 52 | st.state = 'deleted' 53 | 54 | def undelete(st): 55 | st.state = 'active' 56 | 57 | def is_active(st): 58 | return st.state == 'active' 59 | 60 | def _create_pl_by_license(license): 61 | return PackageLicense(license=license) 62 | 63 | class Package(BaseObject): 64 | 65 | def __init__(self, id): 66 | self.id = id 67 | 68 | # licenses_active = StatefulListProperty('package_licenses', is_active, 69 | # delete, undelete) 70 | # licenses_deleted = StatefulListProperty('package_licenses', lambda x: not is_active(x), 71 | # undelete, delete) 72 | # licenses = OurAssociationProxy('licenses_active', 'license', 73 | # creator=_create_pl_by_license) 74 | licenses2 = association_proxy('package_licenses', 'license', 75 | creator=_create_pl_by_license) 76 | 77 | class License(BaseObject): 78 | def __init__(self, name): 79 | self.name = name 80 | 81 | class PackageLicense(object): 82 | def __init__(self, package=None, license=None, state='active'): 83 | self.package = package 84 | self.license = license 85 | self.state = state 86 | 87 | def __repr__(self): 88 | return '' % (self.id, self.package, 89 | self.license, self.state) 90 | 91 | # for testing versioned m2m 92 | def get_as_of(self): 93 | return self 94 | 95 | add_stateful_m2m(Package, PackageLicense, 'licenses', 'license', 96 | 'package_licenses', is_active=is_active, delete=delete, 97 | undelete=undelete) 98 | from base import add_stateful_versioned_m2m 99 | add_stateful_versioned_m2m(Package, PackageLicense, 'licenses3', 'license', 100 | 'package_licenses', is_active=is_active, delete=delete, 101 | undelete=undelete) 102 | 103 | mapper(Package, package_table, properties={ 104 | 'package_licenses':relation(PackageLicense), 105 | }) 106 | mapper(License, license_table) 107 | mapper(PackageLicense, package_license_table, properties={ 108 | 'package':relation(Package), 109 | 'license':relation(License), 110 | }) 111 | 112 | 113 | class TestStatefulCollections(object): 114 | 115 | @classmethod 116 | def setup_class(self): 117 | pkg1 = Package('pkg1') 118 | session.add(pkg1) 119 | lic1 = License('a') 120 | lic2 = License('b') 121 | lic3 = License('c') 122 | lic4 = License('d') 123 | self.license_list = [ lic1, lic2, lic3, lic4 ] 124 | for li in [lic1, lic2, lic3, lic4]: 125 | pkg1.licenses_active.append(PackageLicense(pkg1, li)) 126 | del pkg1.licenses_active[3] 127 | session.flush() 128 | 129 | _clear() 130 | 131 | def test_0_package_licenses(self): 132 | pkg1 = session.query(Package).get('pkg1') 133 | assert len(pkg1.package_licenses) == 4 134 | assert pkg1.package_licenses[-1].state == 'deleted' 135 | 136 | def test_1_licenses(self): 137 | p1 = session.query(Package).get('pkg1') 138 | assert len(p1.licenses) == 3 139 | 140 | def test_2_active_deleted_and_appending(self): 141 | p1 = session.query(Package).get('pkg1') 142 | assert len(p1.licenses_active) == 3 143 | assert len(p1.licenses_deleted) == 1 144 | p1.licenses_deleted.append(PackageLicense(license=License('e'))) 145 | assert len(p1.licenses_active) == 3 146 | assert len(p1.licenses_deleted) == 2 147 | session.flush() 148 | _clear() 149 | pkg1 = session.query(Package).get('pkg1') 150 | assert len(p1.package_licenses) == 5 151 | assert len(p1.licenses_active) == 3 152 | assert len(p1.licenses_deleted) == 2 153 | _clear() 154 | 155 | def test_3_assign_etc(self): 156 | p1 = session.query(Package).get('pkg1') 157 | p1.licenses = [] 158 | assert len(p1.licenses) == 0 159 | assert len(p1.licenses_active) == 0 160 | assert len(p1.licenses_deleted) == 5 161 | session.flush() 162 | _clear() 163 | 164 | pkg1 = session.query(Package).get('pkg1') 165 | assert len(p1.licenses) == 0 166 | assert len(p1.package_licenses) == 5 167 | assert len(p1.licenses_deleted) == 5 168 | 169 | # TODO: move this test to base_test (hasslesome because of all the test 170 | # fixtures) 171 | class TestStatefulVersionedCollections(object): 172 | 173 | @classmethod 174 | def setup_class(self): 175 | pkg2 = Package('pkg2') 176 | session.add(pkg2) 177 | lic1 = License('a') 178 | lic2 = License('b') 179 | lic3 = License('c') 180 | lic4 = License('d') 181 | self.license_list = [ lic1, lic2, lic3, lic4 ] 182 | for li in [lic1, lic2, lic3, lic4]: 183 | pkg2.licenses3_active.append(PackageLicense(pkg2, li)) 184 | del pkg2.licenses3_active[3] 185 | session.flush() 186 | _clear() 187 | 188 | def test_0_package_licenses(self): 189 | pkg2 = session.query(Package).get('pkg2') 190 | assert len(pkg2.package_licenses) == 4 191 | assert pkg2.package_licenses[-1].state == 'deleted' 192 | 193 | def test_1_licenses3(self): 194 | p1 = session.query(Package).get('pkg2') 195 | assert len(p1.licenses3) == 3 196 | 197 | def test_2_active_deleted_and_appending(self): 198 | p1 = session.query(Package).get('pkg2') 199 | assert len(p1.licenses3_active) == 3 200 | assert len(p1.licenses3_deleted) == 1 201 | p1.licenses3_deleted.append(PackageLicense(license=License('e'))) 202 | assert len(p1.licenses3_active) == 3 203 | assert len(p1.licenses3_deleted) == 2 204 | session.flush() 205 | _clear() 206 | p1 = session.query(Package).get('pkg2') 207 | assert len(p1.package_licenses) == 5 208 | assert len(p1.licenses3_active) == 3 209 | assert len(p1.licenses3_deleted) == 2 210 | _clear() 211 | 212 | def test_3_assign_etc(self): 213 | p1 = session.query(Package).get('pkg2') 214 | p1.licenses3 = [] 215 | assert len(p1.licenses3) == 0 216 | assert len(p1.licenses3_active) == 0 217 | assert len(p1.licenses3_deleted) == 5 218 | session.flush() 219 | _clear() 220 | 221 | p1 = session.query(Package).get('pkg2') 222 | assert len(p1.licenses3) == 0 223 | assert len(p1.package_licenses) == 5 224 | assert len(p1.licenses3_deleted) == 5 225 | 226 | 227 | class TestSimple: 228 | def test_1(self): 229 | pkg1 = Package('pkg3') 230 | session.add(pkg1) 231 | lic1 = License('a') 232 | lic2 = License('b') 233 | lic3 = License('c') 234 | lic4 = License('d') 235 | pkg1.licenses2 = [lic1, lic2, lic3] 236 | assert len(pkg1.package_licenses) == 3 237 | assert pkg1.licenses2[0].name == 'a' 238 | pkg1.licenses2.append(lic4) 239 | pkg1.package_licenses[-1].state = 'deleted' 240 | session.flush() 241 | # must clear or other things won't behave 242 | _clear() 243 | 244 | def test_2(self): 245 | p1 = session.query(Package).get('pkg3') 246 | assert p1.package_licenses[0].package == p1 247 | 248 | -------------------------------------------------------------------------------- /vdm/sqlalchemy/test_tools.py: -------------------------------------------------------------------------------- 1 | # many of the tests are in demo_test as that sets up nice fixtures 2 | from tools import * 3 | 4 | dburi = 'postgres://tester:pass@localhost/vdmtest' 5 | from demo import * 6 | class TestRepository: 7 | repo = Repository(metadata, Session, dburi) 8 | 9 | def test_transactional(self): 10 | assert self.repo.have_scoped_session 11 | assert self.repo.transactional 12 | 13 | def test_init_vdm(self): 14 | self.repo.session.remove() 15 | self.repo.clean_db() 16 | self.repo.create_db() 17 | self.repo.init_db() 18 | # nothing to test at the moment ... 19 | 20 | def test_new_revision(self): 21 | self.repo.session.remove() 22 | rev = self.repo.new_revision() 23 | assert rev is not None 24 | 25 | def test_history(self): 26 | self.repo.session.remove() 27 | self.repo.rebuild_db() 28 | rev = self.repo.new_revision() 29 | rev.message = u'abc' 30 | self.repo.commit_and_remove() 31 | history = self.repo.history() 32 | revs = history.all() 33 | assert len(revs) == 1 34 | 35 | -------------------------------------------------------------------------------- /vdm/sqlalchemy/tools.py: -------------------------------------------------------------------------------- 1 | '''Various useful tools for working with Versioned Domain Models. 2 | 3 | Primarily organized within a `Repository` object. 4 | ''' 5 | from sqlalchemy import MetaData 6 | 7 | import logging 8 | logger = logging.getLogger('vdm') 9 | 10 | # fix up table dropping on postgres 11 | # http://blog.pythonisito.com/2008/01/cascading-drop-table-with-sqlalchemy.html 12 | from sqlalchemy import __version__ as sqav 13 | if sqav[:3] in ("0.4", "0.5"): 14 | from sqlalchemy.databases import postgres 15 | class CascadeSchemaDropper(postgres.PGSchemaDropper): 16 | def visit_table(self, table): 17 | for column in table.columns: 18 | if column.default is not None: 19 | self.traverse_single(column.default) 20 | self.append("\nDROP TABLE " + 21 | self.preparer.format_table(table) + 22 | " CASCADE") 23 | self.execute() 24 | postgres.dialect.schemadropper = CascadeSchemaDropper 25 | 26 | elif sqav[:3] in ("0.6", "0.7", "0.8", "0.9", "1.0", "1.1", "1.2", "1.3"): 27 | from sqlalchemy.dialects.postgresql import base 28 | def visit_drop_table(self, drop): 29 | return "\nDROP TABLE " + \ 30 | self.preparer.format_table(drop.element) + \ 31 | " CASCADE" 32 | base.dialect.ddl_compiler.visit_drop_table = visit_drop_table 33 | else: 34 | raise ValueError("VDM only works with SQLAlchemy versions 0.4 through 0.9, not: %s" % sqav) 35 | 36 | 37 | from sqlalchemy import create_engine 38 | try: 39 | from sqlalchemy.orm import ScopedSession as scoped_session 40 | except ImportError: 41 | from sqlalchemy.orm import scoped_session 42 | 43 | from sqlalchemy.orm import class_mapper 44 | from sqlalchemy.orm import object_session 45 | from sqlalchemy import __version__ as sqla_version 46 | 47 | from base import SQLAlchemySession, State, Revision 48 | 49 | class Repository(object): 50 | '''Manage repository-wide type changes for versioned domain models. 51 | 52 | For example: 53 | * creating, cleaning and initializing the repository (DB). 54 | * purging revisions 55 | ''' 56 | def __init__(self, our_metadata, our_session, versioned_objects=None, dburi=None): 57 | ''' 58 | @param versioned_objects: list of classes of objects which are 59 | versioned (NB: not the object *versions* but the continuity objects 60 | themselves). Needed because this will vary from vdm to vdm. 61 | @param dburi: sqlalchemy dburi. If supplied will create engine and bind 62 | it to metadata and session. 63 | ''' 64 | self.metadata = our_metadata 65 | self.session = our_session 66 | self.versioned_objects = versioned_objects 67 | self.dburi = dburi 68 | self.have_scoped_session = isinstance(self.session, scoped_session) 69 | self.transactional = False 70 | if self.have_scoped_session: 71 | tmpsess = self.session() 72 | else: 73 | tmpsess = self.session 74 | if sqla_version > '0.4.99': 75 | self.transactional = not tmpsess.autocommit 76 | else: 77 | self.transactional = tmpsess.transactional 78 | if self.dburi: 79 | engine = create_engine(dburi, pool_threadlocal=True) 80 | self.metadata.bind = engine 81 | self.session.bind = engine 82 | 83 | def create_db(self): 84 | logger.info('Creating DB') 85 | self.metadata.create_all(bind=self.metadata.bind) 86 | 87 | def clean_db(self): 88 | logger.info('Cleaning DB') 89 | self.metadata.drop_all(bind=self.metadata.bind) 90 | 91 | def rebuild_db(self): 92 | logger.info('Rebuilding DB') 93 | self.clean_db() 94 | self.session.remove() 95 | self.init_db() 96 | 97 | def init_db(self): 98 | self.create_db() 99 | logger.info('Initing DB') 100 | self.session.remove() 101 | 102 | def commit(self): 103 | '''Commit/flush (as appropriate) the Sqlalchemy session.''' 104 | # TODO: should we do something like set the revision state as well ... 105 | if self.transactional: 106 | try: 107 | self.session.commit() 108 | except: 109 | self.session.rollback() 110 | # should we remove? 111 | self.session.remove() 112 | raise 113 | else: 114 | self.session.flush() 115 | 116 | def commit_and_remove(self): 117 | self.commit() 118 | self.session.remove() 119 | 120 | def new_revision(self): 121 | '''Convenience method to create new revision and set it on session. 122 | 123 | NB: if in transactional mode do *not* need to call `begin` as we are 124 | automatically within a transaction at all times if session was set up 125 | as transactional (every commit is paired with a begin) 126 | 127 | ''' 128 | rev = Revision() 129 | self.session.add(rev) 130 | SQLAlchemySession.set_revision(self.session, rev) 131 | return rev 132 | 133 | def youngest_revision(self): 134 | '''Get the youngest (most recent) revision.''' 135 | q = self.history() 136 | q = q.order_by(Revision.timestamp.desc()) 137 | return q.first() 138 | 139 | def history(self): 140 | '''Return a history of the repository as a query giving all active revisions. 141 | 142 | @return: sqlalchemy query object. 143 | ''' 144 | return self.session.query(Revision).filter_by(state=State.ACTIVE) 145 | 146 | def list_changes(self, revision): 147 | '''List all objects changed by this `revision`. 148 | 149 | @return: dictionary of changed instances keyed by object class. 150 | ''' 151 | results = {} 152 | for o in self.versioned_objects: 153 | revobj = o.__revision_class__ 154 | items = self.session.query(revobj).filter_by(revision=revision).all() 155 | results[o] = items 156 | return results 157 | 158 | def purge_revision(self, revision, leave_record=False): 159 | '''Purge all changes associated with a revision. 160 | 161 | @param leave_record: if True leave revision in existence but change message 162 | to "PURGED: {date-time-of-purge}". If false delete revision object as 163 | well. 164 | 165 | Summary of the Algorithm 166 | ------------------------ 167 | 168 | 1. list all RevisionObjects affected by this revision 169 | 2. check continuity objects and cascade on everything else ? 170 | 1. crudely get all object revisions associated with this 171 | 2. then check whether this is the only revision and delete the 172 | continuity object 173 | 174 | 3. ALTERNATIVELY delete all associated object revisions then do a 175 | select on continutity to check which have zero associated revisions 176 | (should only be these ...) 177 | ''' 178 | logger.debug('Purging revision: %s' % revision.id) 179 | to_purge = [] 180 | SQLAlchemySession.setattr(self.session, 'revisioning_disabled', True) 181 | self.session.autoflush = False 182 | for o in self.versioned_objects: 183 | revobj = o.__revision_class__ 184 | items = self.session.query(revobj).filter_by(revision=revision).all() 185 | for item in items: 186 | continuity = item.continuity 187 | 188 | if continuity.revision == revision: # need to change continuity 189 | trevobjs = self.session.query(revobj).join('revision'). filter( 190 | revobj.continuity==continuity 191 | ).order_by(Revision.timestamp.desc()).limit(2).all() 192 | if len(trevobjs) == 0: 193 | raise Exception('Should have at least one revision.') 194 | if len(trevobjs) == 1: 195 | to_purge.append(continuity) 196 | else: 197 | new_correct_revobj = trevobjs[1] # older one 198 | self.revert(continuity, new_correct_revobj) 199 | # now delete revision object 200 | self.session.delete(item) 201 | for cont in to_purge: 202 | self.session.delete(cont) 203 | if leave_record: 204 | import datetime 205 | revision.message = u'PURGED: %s UTC' % datetime.datetime.utcnow() 206 | else: 207 | self.session.delete(revision) 208 | self.commit_and_remove() 209 | 210 | def revert(self, continuity, new_correct_revobj): 211 | '''Revert continuity object back to a particular revision_object. 212 | 213 | NB: does *not* call flush/commit. 214 | ''' 215 | logger.debug('revert: %s' % continuity) 216 | table = class_mapper(continuity.__class__).mapped_table 217 | # TODO: ? this will only set columns and not mapped attribs 218 | # TODO: need to do this directly on table or disable 219 | # revisioning behaviour ... 220 | for key in table.c.keys(): 221 | value = getattr(new_correct_revobj, key) 222 | # logger.debug('%s::%s' % (key, value)) 223 | # logger.debug('old: %s' % getattr(continuity, key)) 224 | setattr(continuity, key, value) 225 | logger.debug('revert: end: %s' % continuity) 226 | logger.debug(object_session(continuity)) 227 | logger.debug(self.session) 228 | 229 | --------------------------------------------------------------------------------