├── .coveragerc ├── .gitignore ├── .travis.yml ├── CONTRIBUTING.rst ├── LICENSE ├── MAINTAINERS ├── MANIFEST.in ├── README.rst ├── RELEASE-NOTES.rst ├── bin └── run_workflow.py ├── docs ├── Makefile ├── conf.py ├── index.rst └── requirements.txt ├── pytest.ini ├── run-tests.sh ├── setup.cfg ├── setup.py ├── tests ├── global.ini ├── local.ini ├── local2.ini ├── test_config.py ├── test_engine.py ├── test_engine_db.py ├── test_engine_interface.py └── test_patterns.py ├── tox.ini └── workflow ├── __init__.py ├── config.py ├── deprecation.py ├── engine.py ├── engine_db.py ├── errors.py ├── patterns ├── __init__.py ├── controlflow.py └── utils.py ├── signals.py ├── utils.py └── version.py /.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | source = workflow -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.egg 2 | *.egg-info 3 | *.pyc 4 | *~ 5 | *.sw? 6 | .cache 7 | .coverage 8 | .ropeproject 9 | .tox 10 | .eggs 11 | MANIFEST 12 | build 13 | build/* 14 | coverage.xml 15 | dist 16 | dist/* 17 | docs/_build 18 | docs/db 19 | docs/static 20 | AUTHORS 21 | CHANGELOG 22 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | # This file is part of Workflow. 2 | # Copyright (C) 2014, 2017 CERN. 3 | # 4 | # Workflow is free software; you can redistribute it and/or modify it 5 | # under the terms of the Revised BSD License; see LICENSE file for 6 | # more details. 7 | 8 | language: python 9 | 10 | python: 11 | - "2.7" 12 | - "3.6" 13 | 14 | install: 15 | - pip install --upgrade pip 16 | - pip install pytest pytest-pep8 pytest-cov pytest-cache 17 | - pip install coveralls 18 | - pip install -e .[docs] 19 | 20 | script: 21 | - sphinx-build -qnNW docs docs/_build/html 22 | - python setup.py test 23 | 24 | after_success: 25 | - coveralls 26 | 27 | notifications: 28 | email: false 29 | 30 | deploy: 31 | provider: pypi 32 | user: inveniosoftware 33 | password: 34 | secure: "VuWcET8lKaHnQJ/6i7ryV5F1YqegOQEvmxSe8ph8s5waowxPtnroDCtegCSOYR3865cB37UoE5hqb1DJmEQ8tuII6dGP0vG8FZJ365N1d1nH87XHFJnplCmAQyLGGMGZnw8+XV17KpzsyBR8v+n8RvY909tQd/i0Rsyrp+5/OlA=" 35 | distributions: "sdist bdist_wheel" 36 | on: 37 | python: "2.7" 38 | -------------------------------------------------------------------------------- /CONTRIBUTING.rst: -------------------------------------------------------------------------------- 1 | Contributing 2 | ============ 3 | 4 | Bug reports, feature requests, and other contributions are welcome. 5 | If you find a demonstrable problem that is caused by the code of this 6 | library, please: 7 | 8 | 1. Search for `already reported problems 9 | `_. 10 | 2. Check if the issue has been fixed or is still reproducible on the 11 | latest `master` branch. 12 | 3. Create an issue with **a test case**. 13 | 14 | If you create a feature branch, you can run the tests to ensure everything is 15 | operating correctly: 16 | 17 | .. code-block:: console 18 | 19 | $ ./run-tests.sh 20 | 21 | ... 22 | Name Stmts Miss Cover Missing 23 | ------------------------------------------------------------- 24 | workflow/__init__ 2 0 100% 25 | workflow/config 231 92 60% ... 26 | workflow/engine 321 93 71% ... 27 | workflow/patterns/__init__ 5 0 100% 28 | workflow/patterns/controlflow 159 66 58% ... 29 | workflow/patterns/utils 249 200 20% ... 30 | workflow/version 2 0 100% 31 | ------------------------------------------------------------- 32 | TOTAL 969 451 53% 33 | 34 | ... 35 | 36 | 55 passed, 1 warnings in 3.10 seconds 37 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Workflow is free software; you can redistribute it and/or modify it 2 | under the terms of the Revised BSD License quoted below. 3 | 4 | Copyright (C) 2011, 2012, 2014, 2015, 2016, 2017 CERN. 5 | 6 | All rights reserved. 7 | 8 | Redistribution and use in source and binary forms, with or without 9 | modification, are permitted provided that the following conditions are 10 | met: 11 | 12 | * Redistributions of source code must retain the above copyright 13 | notice, this list of conditions and the following disclaimer. 14 | 15 | * Redistributions in binary form must reproduce the above copyright 16 | notice, this list of conditions and the following disclaimer in the 17 | documentation and/or other materials provided with the distribution. 18 | 19 | * Neither the name of the copyright holder nor the names of its 20 | contributors may be used to endorse or promote products derived from 21 | this software without specific prior written permission. 22 | 23 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 24 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 25 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 26 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 27 | HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 28 | INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, 29 | BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS 30 | OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 31 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR 32 | TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE 33 | USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH 34 | DAMAGE. 35 | 36 | In applying this license, CERN does not waive the privileges and 37 | immunities granted to it by virtue of its status as an 38 | Intergovernmental Organization or submit itself to any jurisdiction. 39 | -------------------------------------------------------------------------------- /MAINTAINERS: -------------------------------------------------------------------------------- 1 | david-caro -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | # This file is part of Workflow. 2 | # Copyright (C) 2011, 2014 CERN. 3 | # 4 | # Workflow is free software; you can redistribute it and/or modify it 5 | # under the terms of the Revised BSD License; see LICENSE file for 6 | # more details. 7 | 8 | include docs/requirements.txt 9 | include LICENSE 10 | include tox.ini 11 | include docs/*.rst docs/*.py docs/Makefile 12 | include *.rst 13 | include tests/*.ini tests/*.py 14 | include .coveragerc .travis.yml pytest.ini 15 | include *.py *.sh 16 | include AUTHORS 17 | include CHANGELOG 18 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | ========== 2 | workflow 3 | ========== 4 | 5 | .. image:: https://travis-ci.org/inveniosoftware-contrib/workflow.png?branch=master 6 | :target: https://travis-ci.org/inveniosoftware-contrib/workflow 7 | .. image:: https://coveralls.io/repos/github/inveniosoftware-contrib/workflow/badge.svg?branch=master 8 | :target: https://coveralls.io/github/inveniosoftware-contrib/workflow?branch=master 9 | 10 | About 11 | ===== 12 | 13 | Workflow is a Finite State Machine with memory. It is used to execute 14 | set of methods in a specified order. 15 | 16 | Here is a simple example of a workflow configuration: 17 | 18 | .. code-block:: text 19 | 20 | [ 21 | check_token_is_wanted, # (run always) 22 | [ # (run conditionally) 23 | check_token_numeric, 24 | translate_numeric, 25 | next_token # (stop processing, continue with next token) 26 | ], 27 | [ # (run conditionally) 28 | check_token_proper_name, 29 | translate_proper_name, 30 | next_token # (stop processing, continue with next token) 31 | ], 32 | normalize_token, # (only for "normal" tokens) 33 | translate_token, 34 | ] 35 | 36 | Documentation 37 | ============= 38 | 39 | Documentation is readable at http://workflow.readthedocs.io or can be built using Sphinx: :: 40 | 41 | pip install Sphinx 42 | python setup.py build_sphinx 43 | 44 | Installation 45 | ============ 46 | 47 | Workflow is on PyPI so all you need is: :: 48 | 49 | pip install workflow 50 | 51 | Testing 52 | ======= 53 | 54 | Running the test suite is as simple as: :: 55 | 56 | python setup.py test 57 | 58 | or, to also show code coverage: :: 59 | 60 | ./run-tests.sh 61 | -------------------------------------------------------------------------------- /RELEASE-NOTES.rst: -------------------------------------------------------------------------------- 1 | ============================= 2 | Workflow v2.0.1 is released 3 | ============================= 4 | 5 | Workflow v2.0.1 was released on August 4, 2017. 6 | 7 | About 8 | ----- 9 | 10 | Workflow is a Finite State Machine with memory. It is used to execute 11 | set of methods in a specified order. 12 | 13 | Workflow was originally developed by Roman Chyla. It is now being 14 | maintained by the Invenio collaboration. 15 | 16 | Improved features 17 | ----------------- 18 | 19 | - Replaces some deprecated calls with their non-deprecated equivalents 20 | in order to avoid raising a `DeprecationWarning`. 21 | 22 | Installation 23 | ------------ 24 | 25 | $ pip install workflow 26 | 27 | Documentation 28 | ------------- 29 | 30 | http://workflow.readthedocs.io/en/v2.0.1 31 | 32 | Good luck and thanks for using Workflow. 33 | 34 | | Invenio Development Team 35 | | Email: info@inveniosoftware.org 36 | | IRC: #invenio on irc.freenode.net 37 | | Twitter: http://twitter.com/inveniosoftware 38 | | GitHub: https://github.com/inveniosoftware/workflow 39 | -------------------------------------------------------------------------------- /bin/run_workflow.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # 3 | # This file is part of Workflow. 4 | # Copyright (C) 2011, 2014 CERN. 5 | # 6 | # Workflow is free software; you can redistribute it and/or modify it 7 | # under the terms of the Revised BSD License; see LICENSE file for 8 | # more details. 9 | 10 | import glob 11 | import six 12 | import sys 13 | import os 14 | import imp 15 | import logging 16 | import traceback 17 | import getopt 18 | 19 | from workflow import engine as main_engine 20 | from workflow.patterns import PROFILE 21 | 22 | 23 | log = main_engine.get_logger('workflow.run-worklfow') 24 | 25 | 26 | def run(selection, 27 | listwf=None, 28 | places=None, 29 | verbose=False, 30 | profile=None, 31 | **kwargs): 32 | """ 33 | Example usage: %prog -l 34 | %prog 1 [to select first workflow to run] 35 | 36 | usage: %prog glob_pattern(s) [options] 37 | -l, --listwf: list available workflows 38 | -i, --places = places: list of glob patterns to search for workflows 39 | (separate with commas!) 40 | -p, --profile=profile: profile the workflow and save output as x 41 | -v, --verbose: makes for a lot of output 42 | """ 43 | 44 | workflows = set() 45 | 46 | for pattern in places: 47 | for file in glob.glob(pattern): 48 | if '__init__.py' not in file: 49 | workflows.add(os.path.abspath(file)) 50 | for f in selection: 51 | if os.path.exists(f) and os.path.isfile(f): 52 | workflows.add(os.path.abspath(f)) 53 | 54 | workflows = sorted(workflows) 55 | 56 | short_names = [] 57 | for w in workflows: 58 | head, tail = os.path.split(w) 59 | short_names.append('%s/%s' % (os.path.split(head)[1], tail)) 60 | 61 | if listwf: 62 | for i in range(len(workflows)): 63 | print "%d - %s" % (i, short_names[i]) 64 | if not len(workflows): 65 | log.warning( 66 | 'No workflows found using default search path: \n%s' % ( 67 | '\n'.join(places))) 68 | 69 | if workflows: 70 | for s in selection: 71 | try: 72 | id = int(s) 73 | if verbose: 74 | run_workflow(workflows[id], 75 | engine=TalkativeWorkflowEngine, 76 | profile=profile) 77 | else: 78 | run_workflow(workflows[id], profile=profile) 79 | except ValueError: 80 | ids = find_workflow(workflows, os.path.normpath(s)) 81 | if len(ids) == 0: 82 | raise Exception("I found no wf for this id: %s" % (s, )) 83 | elif len(ids) > 1: 84 | raise Exception( 85 | "There is more than one wf for this id: %s (%s)" % ( 86 | s, ids)) 87 | else: 88 | if verbose: 89 | run_workflow(workflows[ids[0]], 90 | engine=TalkativeWorkflowEngine, 91 | profile=profile) 92 | else: 93 | run_workflow(workflows[ids[0]], profile=profile) 94 | 95 | 96 | def find_workflow(workflows, name): 97 | candidates = [] 98 | i = 0 99 | for wf_name in workflows: 100 | if name in wf_name: 101 | candidates.append(i) 102 | i += 1 103 | return candidates 104 | 105 | 106 | def run_workflow(file_or_module, 107 | data=None, 108 | engine=None, 109 | profile=None): 110 | """Run the workflow 111 | @var file_or_module: you can pass string (filepath) to the 112 | workflow module, the module will be loaded as an anonymous 113 | module (from the file) and .workflow will be 114 | taken for workflow definition 115 | You can also pass definition of workflow tasks in a 116 | list. 117 | If you pass anything else than we will consider it to be 118 | an object with attribute .workflow - that will be used to 119 | run the workflow (causing error if workflow attr not exists). 120 | If this object has an attribute .data it will be understood 121 | as another workflow engine definition, and data will be 122 | executed in a separated wfe, then results sent to the first 123 | wfe - the .data wfe will receive [{}] as input. 124 | 125 | @var data: data to feed into the workflow engine. If you pass 126 | data, then data defined in the workflow module are ignored 127 | @keyword engine: class that should be used to instantiate the 128 | workflow engine, default=GenericWorkflowEngine 129 | @group callbacks: standard engine callbacks 130 | @keyword profile: filepath where to save the profile if we 131 | are requested to run the workflow in the profiling mode 132 | @return: workflow engine instance (after its workflow was executed) 133 | """ 134 | 135 | if isinstance(file_or_module, six.string_types): 136 | log.info("Loading: %s" % file_or_module) 137 | workflow = get_workflow(file_or_module) 138 | elif isinstance(file_or_module, list): 139 | workflow = WorkflowModule(file_or_module) 140 | else: 141 | workflow = file_or_module 142 | 143 | if workflow: 144 | if profile: 145 | workflow_def = PROFILE(workflow.workflow, profile) 146 | else: 147 | workflow_def = workflow.workflow 148 | we = create_workflow_engine(workflow_def, 149 | engine) 150 | if data is None: 151 | data = [{}] 152 | # there is a separate workflow engine for getting data 153 | if hasattr(workflow, 'data'): 154 | log.info('Running the special data workflow in a separate WFE') 155 | datae = create_workflow_engine(workflow.data, 156 | engine) 157 | datae.process(data) 158 | if data[0]: # get prepared data 159 | data = data[0] 160 | 161 | log.info('Running the workflow') 162 | we.process(data) 163 | return we 164 | else: 165 | raise Exception('No workfow found in: %s' % file_or_module) 166 | 167 | 168 | def create_workflow_engine(workflow, 169 | engine=None): 170 | """Instantiate engine and set the workflow and callbacks 171 | directly 172 | @var workflow: normal workflow tasks definition 173 | @keyword engine: class of the engine to create WE 174 | 175 | @return: prepared WE 176 | """ 177 | if engine is None: 178 | engine = main_engine.GenericWorkflowEngine 179 | wf = engine() 180 | wf.setWorkflow(workflow) 181 | return wf 182 | 183 | 184 | def get_workflow(file): 185 | """ Initializes module into a separate object (not included in sys) """ 186 | name = 'XXX' 187 | x = imp.new_module(name) 188 | x.__file__ = file 189 | x.__id__ = name 190 | x.__builtins__ = __builtins__ 191 | 192 | # XXX - chdir makes our life difficult, especially when 193 | # one workflow wrap another wf and relative paths are used 194 | # in the config. In such cases, the same relative path can 195 | # point to different locations just because location of the 196 | # workflow (parts) are different 197 | # The reason why I was using chdir is because python had 198 | # troubles to import files that containes non-ascii chars 199 | # in their filenames. That is important for macros, but not 200 | # here. 201 | 202 | # old_cwd = os.getcwd() 203 | 204 | try: 205 | # filedir, filename = os.path.split(file) 206 | # os.chdir(filedir) 207 | execfile(file, x.__dict__) 208 | except Exception, excp: 209 | sys.stderr.write(traceback.format_exc()) 210 | log.error(excp) 211 | log.error(traceback.format_exc()) 212 | return 213 | 214 | return x 215 | 216 | 217 | def import_workflow(workflow): 218 | """Import workflow module 219 | @var workflow: string as python import, eg: merkur.workflow.load_x""" 220 | mod = __import__(workflow) 221 | components = workflow.split('.') 222 | for comp in components[1:]: 223 | mod = getattr(mod, comp) 224 | return mod 225 | 226 | 227 | class TalkativeWorkflowEngine(main_engine.GenericWorkflowEngine): 228 | counter = 0 229 | 230 | def __init__(self, *args, **kwargs): 231 | main_engine.GenericWorkflowEngine.__init__(self, *args, **kwargs) 232 | self.log = main_engine.get_logger( 233 | 'TalkativeWFE<%d>' % TalkativeWorkflowEngine.counter) 234 | TalkativeWorkflowEngine.counter += 1 235 | 236 | def execute_callback(self, callback, obj): 237 | obj_rep = [] 238 | max_len = 60 239 | 240 | def val_format(v): 241 | return '<%s ...>' % repr(v)[:max_len] 242 | 243 | def func_format(c): 244 | return '<%s ...%s:%s>' % ( 245 | c.__name__, 246 | c.func_code.co_filename[-max_len:], 247 | c.func_code.co_firstlineno) 248 | if isinstance(obj, dict): 249 | for k, v in obj.items(): 250 | obj_rep.append('%s:%s' % (k, val_format(v))) 251 | obj_rep = '{%s}' % (', '.join(obj_rep)) 252 | elif isinstance(obj, list): 253 | for v in obj: 254 | obj_rep.append(val_format(v)) 255 | obj_rep = '[%s]' % (', '.join(obj_rep)) 256 | else: 257 | obj_rep = val_format(obj) 258 | self.log.debug('%s ( %s )' % (func_format(callback), obj_rep)) 259 | callback(obj, self) 260 | 261 | 262 | class WorkflowModule(object): 263 | 264 | """Workflow wrapper.""" 265 | 266 | def __init__(self, workflow): 267 | self.workflow = workflow 268 | 269 | 270 | def usage(): 271 | print """ 272 | usage: %(prog)s [options] 273 | 274 | examples: 275 | %(prog)s -l 276 | - to list the available workflows 277 | %(prog)s 1 278 | - to run the first workflow in the list 279 | 280 | options: 281 | -l, --list: list available workflows 282 | -p, --places = places: list of glob patterns where the workflows 283 | are searched, example: ./this-folder/*.py,./that/*.pyw 284 | (separate with commas!) 285 | -o, --profile=profile: profile the workflows, be default it saves 286 | output into tmp folder/profile.out 287 | -v, --verbose: workflows are executed as talkative 288 | -e, --level = (int): sets the verbose level, the higher the level, 289 | the less messages are printed 290 | -h, --help: this help message 291 | 292 | """ % {'prog': os.path.basename(__file__)} 293 | 294 | 295 | def main(): 296 | 297 | try: 298 | opts, args = getopt.getopt(sys.argv[1:], "lp:o:ve:h", [ 299 | 'list', 'places=', 'profile=', 'verbose', 'Vlevel=', 'help']) 300 | except getopt.GetoptError, err: 301 | # print help information and exit: 302 | print str(err) # will print something like "option -a not recognized" 303 | usage() 304 | sys.exit(2) 305 | 306 | kw_args = {} 307 | output = None 308 | verbose = False 309 | for o, a in opts: 310 | if o in ("-v", '--verbose'): 311 | verbose = True 312 | elif o in ("-h", "--help"): 313 | usage() 314 | sys.exit() 315 | elif o in ("-e", "--level"): 316 | try: 317 | level = int(a) 318 | main_engine.set_global_level(level) 319 | main_engine.reset_all_loggers(level) 320 | except: 321 | print 'The argument to verbose must be integer' 322 | sys.exit(2) 323 | elif o in ('-v', '--verbose'): 324 | kw_args['verbose'] = True 325 | elif o in ('-p', '--places'): 326 | kw_args['places'] = a.split(',') 327 | elif o in ('-l', '--list'): 328 | kw_args['listwf'] = True 329 | else: 330 | assert False, "unhandled option %s" % o 331 | 332 | if (not len(args) or not len(opts)) and 'listwf' not in kw_args: 333 | usage() 334 | sys.exit() 335 | 336 | if 'places' not in kw_args: 337 | d = os.path.dirname(os.path.abspath(__file__)) 338 | kw_args['places'] = ['%s/workflows/*.py' % d, 339 | '%s/workflows/*.pyw' % d, 340 | '%s/workflows/*.cfg' % d] 341 | 342 | run(args, **kw_args) 343 | 344 | 345 | if __name__ == "__main__": 346 | main() 347 | -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- 1 | # Makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = sphinx-build 7 | PAPER = 8 | BUILDDIR = _build 9 | 10 | # User-friendly check for sphinx-build 11 | ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) 12 | $(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) 13 | endif 14 | 15 | # Internal variables. 16 | PAPEROPT_a4 = -D latex_paper_size=a4 17 | PAPEROPT_letter = -D latex_paper_size=letter 18 | ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 19 | # the i18n builder cannot share the environment and doctrees with the others 20 | I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 21 | 22 | .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext 23 | 24 | help: 25 | @echo "Please use \`make ' where is one of" 26 | @echo " html to make standalone HTML files" 27 | @echo " dirhtml to make HTML files named index.html in directories" 28 | @echo " singlehtml to make a single large HTML file" 29 | @echo " pickle to make pickle files" 30 | @echo " json to make JSON files" 31 | @echo " htmlhelp to make HTML files and a HTML help project" 32 | @echo " qthelp to make HTML files and a qthelp project" 33 | @echo " devhelp to make HTML files and a Devhelp project" 34 | @echo " epub to make an epub" 35 | @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" 36 | @echo " latexpdf to make LaTeX files and run them through pdflatex" 37 | @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" 38 | @echo " text to make text files" 39 | @echo " man to make manual pages" 40 | @echo " texinfo to make Texinfo files" 41 | @echo " info to make Texinfo files and run them through makeinfo" 42 | @echo " gettext to make PO message catalogs" 43 | @echo " changes to make an overview of all changed/added/deprecated items" 44 | @echo " xml to make Docutils-native XML files" 45 | @echo " pseudoxml to make pseudoxml-XML files for display purposes" 46 | @echo " linkcheck to check all external links for integrity" 47 | @echo " doctest to run all doctests embedded in the documentation (if enabled)" 48 | 49 | clean: 50 | rm -rf $(BUILDDIR)/* 51 | 52 | html: 53 | $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html 54 | @echo 55 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." 56 | 57 | dirhtml: 58 | $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml 59 | @echo 60 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." 61 | 62 | singlehtml: 63 | $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml 64 | @echo 65 | @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." 66 | 67 | pickle: 68 | $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle 69 | @echo 70 | @echo "Build finished; now you can process the pickle files." 71 | 72 | json: 73 | $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json 74 | @echo 75 | @echo "Build finished; now you can process the JSON files." 76 | 77 | htmlhelp: 78 | $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp 79 | @echo 80 | @echo "Build finished; now you can run HTML Help Workshop with the" \ 81 | ".hhp project file in $(BUILDDIR)/htmlhelp." 82 | 83 | qthelp: 84 | $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp 85 | @echo 86 | @echo "Build finished; now you can run "qcollectiongenerator" with the" \ 87 | ".qhcp project file in $(BUILDDIR)/qthelp, like this:" 88 | @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Workflow.qhcp" 89 | @echo "To view the help file:" 90 | @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Workflow.qhc" 91 | 92 | devhelp: 93 | $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp 94 | @echo 95 | @echo "Build finished." 96 | @echo "To view the help file:" 97 | @echo "# mkdir -p $$HOME/.local/share/devhelp/Workflow" 98 | @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Workflow" 99 | @echo "# devhelp" 100 | 101 | epub: 102 | $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub 103 | @echo 104 | @echo "Build finished. The epub file is in $(BUILDDIR)/epub." 105 | 106 | latex: 107 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 108 | @echo 109 | @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." 110 | @echo "Run \`make' in that directory to run these through (pdf)latex" \ 111 | "(use \`make latexpdf' here to do that automatically)." 112 | 113 | latexpdf: 114 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 115 | @echo "Running LaTeX files through pdflatex..." 116 | $(MAKE) -C $(BUILDDIR)/latex all-pdf 117 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 118 | 119 | latexpdfja: 120 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 121 | @echo "Running LaTeX files through platex and dvipdfmx..." 122 | $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja 123 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 124 | 125 | text: 126 | $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text 127 | @echo 128 | @echo "Build finished. The text files are in $(BUILDDIR)/text." 129 | 130 | man: 131 | $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man 132 | @echo 133 | @echo "Build finished. The manual pages are in $(BUILDDIR)/man." 134 | 135 | texinfo: 136 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 137 | @echo 138 | @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." 139 | @echo "Run \`make' in that directory to run these through makeinfo" \ 140 | "(use \`make info' here to do that automatically)." 141 | 142 | info: 143 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 144 | @echo "Running Texinfo files through makeinfo..." 145 | make -C $(BUILDDIR)/texinfo info 146 | @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." 147 | 148 | gettext: 149 | $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale 150 | @echo 151 | @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." 152 | 153 | changes: 154 | $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes 155 | @echo 156 | @echo "The overview file is in $(BUILDDIR)/changes." 157 | 158 | linkcheck: 159 | $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck 160 | @echo 161 | @echo "Link check complete; look for any errors in the above output " \ 162 | "or in $(BUILDDIR)/linkcheck/output.txt." 163 | 164 | doctest: 165 | $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest 166 | @echo "Testing of doctests in the sources finished, look at the " \ 167 | "results in $(BUILDDIR)/doctest/output.txt." 168 | 169 | xml: 170 | $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml 171 | @echo 172 | @echo "Build finished. The XML files are in $(BUILDDIR)/xml." 173 | 174 | pseudoxml: 175 | $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml 176 | @echo 177 | @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." 178 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # Workflow documentation build configuration file, created by 4 | # sphinx-quickstart on Tue Aug 26 12:50:15 2014. 5 | # 6 | # This file is execfile()d with the current directory set to its 7 | # containing dir. 8 | # 9 | # Note that not all possible configuration values are present in this 10 | # autogenerated file. 11 | # 12 | # All configuration values have a default; values that are commented out 13 | # serve to show the default. 14 | 15 | """Sphinx configuration.""" 16 | 17 | from __future__ import print_function 18 | 19 | import os 20 | import sys 21 | 22 | from autosemver.packaging import ( 23 | get_changelog, get_current_version, get_releasenotes 24 | ) 25 | 26 | 27 | if not os.path.exists('_build/html/_static'): 28 | os.makedirs('_build/html/_static') 29 | 30 | URL = 'https://github.com/inveniosoftware-contrib/workflow' 31 | BUGTRACKER_URL = URL + '/issues/' 32 | 33 | with open('_build/html/_static/RELEASE_NOTES.txt', 'w') as changelog_fd: 34 | changelog_fd.write(get_releasenotes( 35 | project_dir='..', 36 | bugtracker_url=BUGTRACKER_URL, 37 | )) 38 | with open('_build/html/_static/CHANGELOG.txt', 'w') as changelog_fd: 39 | changelog_fd.write(get_changelog( 40 | project_dir='..', 41 | bugtracker_url=BUGTRACKER_URL, 42 | )) 43 | 44 | 45 | # If extensions (or modules to document with autodoc) are in another directory, 46 | # add these directories to sys.path here. If the directory is relative to the 47 | # documentation root, use os.path.abspath to make it absolute, like shown here. 48 | sys.path.insert(0, os.path.abspath('..')) 49 | 50 | # -- General configuration ------------------------------------------------ 51 | 52 | # If your documentation needs a minimal Sphinx version, state it here. 53 | #needs_sphinx = '1.0' 54 | 55 | # Do not warn on external images. 56 | suppress_warnings = ['image.nonlocal_uri'] 57 | 58 | # Add any Sphinx extension module names here, as strings. They can be 59 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom 60 | # ones. 61 | extensions = [ 62 | 'sphinx.ext.autodoc', 63 | 'sphinx.ext.doctest', 64 | 'sphinx.ext.intersphinx', 65 | ] 66 | 67 | # Add any paths that contain templates here, relative to this directory. 68 | templates_path = ['_templates'] 69 | 70 | # The suffix of source filenames. 71 | source_suffix = '.rst' 72 | 73 | # The encoding of source files. 74 | #source_encoding = 'utf-8-sig' 75 | 76 | # The master toctree document. 77 | master_doc = 'index' 78 | 79 | # General information about the project. 80 | project = u'Workflow' 81 | copyright = u'2014, Roman Chyla' 82 | 83 | # The version info for the project you're documenting, acts as replacement for 84 | # |version| and |release|, also used in various other places throughout the 85 | # built documents. 86 | version = get_current_version( 87 | project_name='workflow' 88 | ) 89 | 90 | # The full version, including alpha/beta/rc tags. 91 | release = version 92 | 93 | # The language for content autogenerated by Sphinx. Refer to documentation 94 | # for a list of supported languages. 95 | #language = None 96 | 97 | # There are two options for replacing |today|: either, you set today to some 98 | # non-false value, then it is used: 99 | #today = '' 100 | # Else, today_fmt is used as the format for a strftime call. 101 | #today_fmt = '%B %d, %Y' 102 | 103 | # List of patterns, relative to source directory, that match files and 104 | # directories to ignore when looking for source files. 105 | exclude_patterns = ['_build'] 106 | 107 | # The reST default role (used for this markup: `text`) to use for all 108 | # documents. 109 | #default_role = None 110 | 111 | # If true, '()' will be appended to :func: etc. cross-reference text. 112 | #add_function_parentheses = True 113 | 114 | # If true, the current module name will be prepended to all description 115 | # unit titles (such as .. function::). 116 | #add_module_names = True 117 | 118 | # If true, sectionauthor and moduleauthor directives will be shown in the 119 | # output. They are ignored by default. 120 | #show_authors = False 121 | 122 | # The name of the Pygments (syntax highlighting) style to use. 123 | pygments_style = 'sphinx' 124 | 125 | # A list of ignored prefixes for module index sorting. 126 | #modindex_common_prefix = [] 127 | 128 | # If true, keep warnings as "system message" paragraphs in the built documents. 129 | #keep_warnings = False 130 | 131 | 132 | # -- Options for HTML output ---------------------------------------------- 133 | html_theme = 'alabaster' 134 | 135 | html_theme_options = { 136 | 'description': 'Run Finite State Machines with memory', 137 | 'github_user': 'inveniosoftware', 138 | 'github_repo': 'workflow', 139 | 'github_button': False, 140 | 'github_banner': True, 141 | 'show_powered_by': False, 142 | 'extra_nav_links': { 143 | 'workflow@GitHub': 'http://github.com/inveniosoftware/workflow', 144 | 'workflow@PyPI': 'http://pypi.python.org/pypi/workflow/', 145 | } 146 | } 147 | 148 | # The theme to use for HTML and HTML Help pages. See the documentation for 149 | # a list of builtin themes. 150 | 151 | # Theme options are theme-specific and customize the look and feel of a theme 152 | # further. For a list of options available for each theme, see the 153 | # documentation. 154 | #html_theme_options = {} 155 | 156 | # Add any paths that contain custom themes here, relative to this directory. 157 | #html_theme_path = [] 158 | 159 | # The name for this set of Sphinx documents. If None, it defaults to 160 | # " v documentation". 161 | #html_title = None 162 | 163 | # A shorter title for the navigation bar. Default is the same as html_title. 164 | #html_short_title = None 165 | 166 | # The name of an image file (relative to this directory) to place at the top 167 | # of the sidebar. 168 | #html_logo = None 169 | 170 | # The name of an image file (within the static path) to use as favicon of the 171 | # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 172 | # pixels large. 173 | #html_favicon = None 174 | 175 | # Add any paths that contain custom static files (such as style sheets) here, 176 | # relative to this directory. They are copied after the builtin static files, 177 | # so a file named "default.css" will overwrite the builtin "default.css". 178 | #html_static_path = ['_static'] 179 | 180 | # Add any extra paths that contain custom files (such as robots.txt or 181 | # .htaccess) here, relative to this directory. These files are copied 182 | # directly to the root of the documentation. 183 | #html_extra_path = [] 184 | 185 | # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, 186 | # using the given strftime format. 187 | #html_last_updated_fmt = '%b %d, %Y' 188 | 189 | # If true, SmartyPants will be used to convert quotes and dashes to 190 | # typographically correct entities. 191 | #html_use_smartypants = True 192 | 193 | # Custom sidebar templates, maps document names to template names. 194 | #html_sidebars = {} 195 | 196 | # Additional templates that should be rendered to pages, maps page names to 197 | # template names. 198 | #html_additional_pages = {} 199 | 200 | # If false, no module index is generated. 201 | #html_domain_indices = True 202 | 203 | # If false, no index is generated. 204 | #html_use_index = True 205 | 206 | # If true, the index is split into individual pages for each letter. 207 | #html_split_index = False 208 | 209 | # If true, links to the reST sources are added to the pages. 210 | #html_show_sourcelink = True 211 | 212 | # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. 213 | #html_show_sphinx = True 214 | 215 | # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. 216 | #html_show_copyright = True 217 | 218 | # If true, an OpenSearch description file will be output, and all pages will 219 | # contain a tag referring to it. The value of this option must be the 220 | # base URL from which the finished HTML is served. 221 | #html_use_opensearch = '' 222 | 223 | # This is the file name suffix for HTML files (e.g. ".xhtml"). 224 | #html_file_suffix = None 225 | 226 | # Output file base name for HTML help builder. 227 | htmlhelp_basename = 'Workflowdoc' 228 | 229 | 230 | # -- Options for LaTeX output --------------------------------------------- 231 | 232 | latex_elements = { 233 | # The paper size ('letterpaper' or 'a4paper'). 234 | #'papersize': 'letterpaper', 235 | 236 | # The font size ('10pt', '11pt' or '12pt'). 237 | #'pointsize': '10pt', 238 | 239 | # Additional stuff for the LaTeX preamble. 240 | #'preamble': '', 241 | } 242 | 243 | # Grouping the document tree into LaTeX files. List of tuples 244 | # (source start file, target name, title, 245 | # author, documentclass [howto, manual, or own class]). 246 | latex_documents = [ 247 | ('index', 'Workflow.tex', u'Workflow Documentation', 248 | u'Roman Chyla', 'manual'), 249 | ] 250 | 251 | # The name of an image file (relative to this directory) to place at the top of 252 | # the title page. 253 | #latex_logo = None 254 | 255 | # For "manual" documents, if this is true, then toplevel headings are parts, 256 | # not chapters. 257 | #latex_use_parts = False 258 | 259 | # If true, show page references after internal links. 260 | #latex_show_pagerefs = False 261 | 262 | # If true, show URL addresses after external links. 263 | #latex_show_urls = False 264 | 265 | # Documents to append as an appendix to all manuals. 266 | #latex_appendices = [] 267 | 268 | # If false, no module index is generated. 269 | #latex_domain_indices = True 270 | 271 | 272 | # -- Options for manual page output --------------------------------------- 273 | 274 | # One entry per manual page. List of tuples 275 | # (source start file, name, description, authors, manual section). 276 | man_pages = [ 277 | ('index', 'workflow', u'Workflow Documentation', 278 | [u'Roman Chyla'], 1) 279 | ] 280 | 281 | # If true, show URL addresses after external links. 282 | #man_show_urls = False 283 | 284 | 285 | # -- Options for Texinfo output ------------------------------------------- 286 | 287 | # Grouping the document tree into Texinfo files. List of tuples 288 | # (source start file, target name, title, author, 289 | # dir menu entry, description, category) 290 | texinfo_documents = [ 291 | ('index', 'Workflow', u'Workflow Documentation', 292 | u'Roman Chyla', 'Workflow', 'One line description of project.', 293 | 'Miscellaneous'), 294 | ] 295 | 296 | # Documents to append as an appendix to all manuals. 297 | #texinfo_appendices = [] 298 | 299 | # If false, no module index is generated. 300 | #texinfo_domain_indices = True 301 | 302 | # How to display URL addresses: 'footnote', 'no', or 'inline'. 303 | #texinfo_show_urls = 'footnote' 304 | 305 | # If true, do not generate a @detailmenu in the "Top" node's menu. 306 | #texinfo_no_detailmenu = False 307 | 308 | intersphinx_mapping = { 309 | 'python': ('http://docs.python.org/2', None), 310 | } 311 | 312 | nitpick_ignore = [ 313 | ('py:class', 'ProcessingFactory'), 314 | ('py:class', 'DbProcessingFactory'), 315 | ('py:class', 'WorkflowStatus'), 316 | ] 317 | -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | ========== 2 | Workflow 3 | ========== 4 | .. currentmodule:: workflow 5 | 6 | What is Workflow? 7 | ================= 8 | 9 | The Workflow library provides a method of running Finite State Machines with 10 | memory. It can be used to execute a set of methods, complete with conditions 11 | and patterns. 12 | 13 | Workflow allows for a number of independent pieces of data to be processed by 14 | the same logic, while allowing for the entire process to be forwarded, 15 | backwarded, paused, inspected, re-executed, modified and stored. 16 | 17 | How do I use Workflow? 18 | ====================== 19 | 20 | In the following sections we will take a look at Workflow's features by working 21 | on examples of increasing complexity. Please keep an eye out for comments in 22 | the code as they provide crucial information. 23 | 24 | Basic workflow use 25 | ------------------ 26 | 27 | Basic use is comprised of the following steps: 28 | 29 | 1. Instantiate a **workflow engine**. For this example we will use the simplest 30 | provided one, but you may also extend it to add custom behaviour. 31 | 32 | .. code-block:: python 33 | 34 | from workflow.engine import GenericWorkflowEngine 35 | my_engine = GenericWorkflowEngine() 36 | 37 | 2. Create **tasks**. These are purpose-built function functions that the 38 | workflow engine can execute. 39 | 40 | The engine always passes **(current_token, current_engine)** as arguments 41 | to these functions, so they need to support them. Note the **add_data** 42 | function needs to be able to accept more arguments. For this we use a 43 | closure. 44 | 45 | .. code-block:: python 46 | 47 | from functools import wraps 48 | 49 | def print_data(obj, eng): 50 | """Print the data found in the token.""" 51 | print obj.data 52 | 53 | def add_data(number_to_add): 54 | """Add number_to_add to obj.data.""" 55 | @wraps(add_data) 56 | def _add_data(obj, eng): 57 | obj.data += number_to_add 58 | return _add_data 59 | 60 | 61 | 2. Create a **workflow definition** (also known as **callbacks**). This is a 62 | (sometimes nested) list of **tasks** that we wish to run. 63 | 64 | .. code-block:: python 65 | 66 | my_workflow_definition = [ 67 | add_data(1), 68 | print_data 69 | ] 70 | 71 | 3. Define **tokens**. This is the data that we wish to feed the workflow. Since 72 | the data we will deal with in this example is immutable, we need to place it 73 | in **token wrappers** . Another reason you may wish to wrap your data is to 74 | be able to store `metadata` in the object. 75 | 76 | .. code-block:: python 77 | 78 | class MyObject(object): 79 | def __init__(self, data): 80 | self.data = data 81 | 82 | my_object0 = MyObject(0) 83 | my_object1 = MyObject(1) 84 | 85 | 4. **Run** the engine on a list of such wrappers with our workflow definition. 86 | 87 | The engine passes the tokens that we give it one at a time through the 88 | workflow. 89 | 90 | .. code-block:: python 91 | 92 | my_engine.callbacks.replace(my_workflow_definition) 93 | 94 | my_engine.process([my_object0, my_object1]) 95 | # The engine prints: "1\n2" 96 | my_object0.data == 1 97 | my_object1.data == 2 98 | 99 | 5. **Bonus**! Once the engine has ran, it can be reused. 100 | 101 | .. code-block:: python 102 | 103 | my_engine.process([my_object0, my_object1]) 104 | # The engine prints: "2\n3" 105 | my_object0.data == 2 106 | my_object1.data == 3 107 | 108 | 109 | Loops and interrupts 110 | -------------------- 111 | 112 | Let's take a look at a slightly more advanced example. There are two things to 113 | note here: 114 | 115 | * How control flow is done. We provide, among others, **IF_ELSE** and 116 | **FOR** statements. They are simple functions - therefore you can make your 117 | own if you wish to. We will see examples of this in the Details section. 118 | 119 | * Control flow can reach outside the engine via exceptions. We will raise the 120 | **WorkflowHalt** exception to return the control to our code before the 121 | workflow has even finished and then even resume it. 122 | 123 | In this example, we have a series of lists composed of 0 and 1 and we want to: 124 | 125 | 1. Add [0, 1] at the end of the list. 126 | 2. Repeat a until list >= [0, 1, 0, 1]. 127 | 3. Add [1] when we are done. 128 | 129 | Here are some example transformations that describe the above: 130 | 131 | * [] --> [0, 1, 0, 1, 1] 132 | * [0, 1] --> [0, 1, 0, 1, 1] 133 | * [0, 1, 0, 1] --> [0, 1, 0, 1, 0, 1, 1] 134 | 135 | Time for some code! Let's start with the imports. Pay close attention as their 136 | arguments are explained briefly here. 137 | 138 | .. code-block:: python 139 | 140 | from workflow.engine import GenericWorkflowEngine 141 | from workflow.errors import HaltProcessing 142 | from workflow.patterns.controlflow import ( 143 | FOR, # Simple for-loop, a-la python. First argument is an iterable, 144 | # second defines where to save the current value, and the third 145 | # is the code that runs in the loop. 146 | 147 | HALT, # Halts the engine. This brings it to a state where it can be 148 | # inspected, resumed, restarted, or other. 149 | 150 | IF_ELSE, # Simple `if-else` statement that accepts 3 arguments. 151 | # (condition, tasks if true, tasks if false) 152 | 153 | CMP, # Simple function to support python comparisons directly from a 154 | # workflow engine. 155 | ) 156 | 157 | Now to define some functions of our own. 158 | 159 | Note that the first function leverages `eng.extra_data`. This is a simple 160 | dictionary that the `GenericWorkflowEngine` exposes and it acts as a shared 161 | storage that persists during the execution of the engine. 162 | 163 | The two latter functions wrap engine functionality that's already there, but 164 | add `print` statements for the example. 165 | 166 | .. code-block:: python 167 | 168 | def append_from(key): 169 | """Append data from a given `key` of the engine's `extra_data`.""" 170 | def _append_from(obj, eng): 171 | obj.append(eng.extra_data[key]) 172 | print "new data:", obj 173 | return _append_from 174 | 175 | def interrupt_workflow(obj, eng): 176 | """Raise the `HaltProcessing` exception. 177 | 178 | This is not handled by the engine and bubbles up to our code. 179 | """ 180 | print "Raising HaltProcessing" 181 | eng.halt("interrupting this workflow.") 182 | 183 | def restart_workflow(obj, eng): 184 | """Restart the engine with the current object, from the first task.""" 185 | print "Restarting the engine" 186 | eng.restart('current', 'first') 187 | 188 | We are now ready to create the workflow: 189 | 190 | .. code-block:: python 191 | 192 | my_workflow = [ 193 | FOR(range(2), "my_current_value", # For-loop, from 0 to 1, that sets 194 | # the current value to 195 | # `eng.extra_data["my_current_value"]` 196 | [ 197 | append_from("my_current_value"), # Gets the value set above 198 | # and appends it to our token 199 | ] 200 | ), # END FOR 201 | 202 | IF_ELSE( 203 | CMP((lambda o, e: o), [0, 1 ,0, 1], "<"), # Condition: 204 | # "if obj < [0,1,0,1]:" 205 | 206 | [ restart_workflow ], # Tasks to run if condition 207 | # is True: 208 | # "return back to the FOR" 209 | 210 | [ # Tasks to run if condition 211 | # is False: 212 | 213 | append_from("my_current_value"), # "append 1 (note we still 214 | # have access to it) 215 | interrupt_workflow # and interrupt" 216 | ] 217 | ) # END IF_ELSE 218 | ] 219 | 220 | Because our workflow interrupts itself, we will wrap the call to `process` and 221 | `restart`, in `try-except` statements. 222 | 223 | .. code-block:: python 224 | 225 | # Create the engine as in the previous example 226 | my_engine = GenericWorkflowEngine() 227 | my_engine.callbacks.replace(my_workflow) 228 | 229 | try: 230 | # Note how we don't need to keep a reference to our tokens - the engine 231 | # allows us to access them via `my_engine.objects` later. 232 | my_engine.process([[], [0,1], [0,1,0,1]]) 233 | except HaltProcessing: 234 | # Our engine was built to throw this exception every time an object is 235 | # completed. At this point we can inspect the object to decide what to 236 | # do next. In any case, we will ask it to move to the next object, 237 | # until it stops throwing the exception (which, in our case, means it 238 | # has finished with all objects). 239 | while True: 240 | try: 241 | # Restart the engine with the next object, starting from the 242 | # first task. 243 | my_engine.restart('next', 'first') 244 | except HaltProcessing: 245 | continue 246 | else: 247 | print "Done!", my_engine.objects 248 | break 249 | 250 | Here is what the execution prints:: 251 | 252 | new data: [0] 253 | new data: [0, 1] 254 | Restarting the engine 255 | new data: [0, 1, 0] 256 | new data: [0, 1, 0, 1] 257 | new data: [0, 1, 0, 1, 1] 258 | Raising HaltProcessing 259 | new data: [0, 1, 0] 260 | new data: [0, 1, 0, 1] 261 | new data: [0, 1, 0, 1, 1] 262 | Raising HaltProcessing 263 | new data: [0, 1, 0, 1, 0] 264 | new data: [0, 1, 0, 1, 0, 1] 265 | new data: [0, 1, 0, 1, 0, 1, 1] 266 | Raising HaltProcessing 267 | Done! [[0, 1, 0, 1, 1], [0, 1, 0, 1, 1], [0, 1, 0, 1, 0, 1, 1]] 268 | 269 | Celery support 270 | -------------- 271 | 272 | Celery is a widely used distributed task queue. The independent nature of 273 | workflows and their ability to be restarted and resumed makes it a good 274 | candidate for running in a task queue. Let's take a look at running a workflow 275 | inside celery. 276 | 277 | Assuming workflow is already installed, let's also install celery:: 278 | 279 | $ pip install 'celery[redis]' 280 | 281 | Onto the code next: 282 | 283 | .. code-block:: python 284 | 285 | from celery import Celery 286 | 287 | # `app` is required by the celery worker. 288 | app = Celery('workflow_sample', broker='redis://localhost:6379/0') 289 | 290 | # Define a couple of basic tasks. 291 | def add(obj, eng): 292 | obj["value"] += 2 293 | 294 | def print_res(obj, eng): 295 | print obj.get("value") 296 | 297 | # Create a workflow out of them. 298 | flow = [add, print_res] 299 | 300 | # Mark our execution process as a celery task with this decorator. 301 | @app.task 302 | def run_workflow(data): 303 | # Note that the imports that this function requires must be done inside 304 | # it since our code will not be running in the global context. 305 | from workflow.engine import GenericWorkflowEngine 306 | wfe = GenericWorkflowEngine() 307 | wfe.setWorkflow(flow) 308 | wfe.process(data) 309 | 310 | # Code that runs when we call this script directly. This way we can start 311 | # as many workflows as we wish and let celery handle how they are 312 | # distributed and when they run. 313 | if __name__ == "__main__": 314 | run_workflow.delay([{"value": 10}, {"value": 20}, {"value": 30}]) 315 | 316 | Time to bring celery up: 317 | 318 | 1. Save this file as `/some/path/workflow_sample.py` 319 | 320 | 2. Bring up a worker in one terminal:: 321 | 322 | $ cd /some/path 323 | $ celery -A workflow_sample worker --loglevel=info 324 | 325 | 3. Use another terminal to request `run_workflow` to be ran with the above arguments:: 326 | 327 | $ cd /some/path 328 | $ python workflow_sample.py 329 | 330 | You should see the worker working. Try running `python workflow_sample.py` 331 | again. 332 | 333 | Storage-capable engine 334 | ---------------------- 335 | 336 | The Workflow library comes with an alternative engine which is built to work 337 | with SQLAlchemy databases (`DbWorkflowEngine`). This means that one can store 338 | the state of the engine and objects for later use. This opens up new 339 | possibilities: 340 | 341 | * A front-end can be attached to the engine with ease. 342 | * Workflows can be stored for resume at a later time.. 343 | * ..or even shared between processing nodes. 344 | 345 | In this example we will see a simple implementation of such a database-stored, 346 | resumable workflow. 347 | 348 | We will reveal the problem that we will be solving much later. For now, we can 349 | start by creating a couple of SQLAlchemy schemas: 350 | 351 | * One to attach to the workflow itself (a workflow will represent a student) 352 | * and one where a single grade (a grade will be the grade of a single test) 353 | 354 | Note that the `Workflow` model below can store an element pointer. This pointer 355 | (found at `engine.state.token_pos`) indicates the object that is currently being 356 | processed and saving it is crucial so that the engine can resume at a later 357 | time from that point. 358 | 359 | .. code-block:: python 360 | 361 | from sqlalchemy.ext.declarative import declarative_base 362 | from sqlalchemy import Column, Integer, String, create_engine, ForeignKey, Boolean 363 | from sqlalchemy.orm import sessionmaker, relationship 364 | 365 | # Create an engine and a session 366 | engine = create_engine('sqlite://') 367 | Base = declarative_base(bind=engine) 368 | DBSession = sessionmaker(bind=engine) 369 | session = DBSession() 370 | 371 | 372 | class Workflow(Base): 373 | __tablename__ = 'workflow' 374 | id = Column(Integer, primary_key=True) 375 | state_token_pos = Column(Integer, default=-1) 376 | grades = relationship('Grade', backref='workflow', 377 | cascade="all, delete, delete-orphan") 378 | 379 | def save(self, token_pos): 380 | """Save object to persistent storage.""" 381 | self.state_token_pos = token_pos 382 | session.begin(subtransactions=True) 383 | try: 384 | session.add(self) 385 | session.commit() 386 | except Exception: 387 | session.rollback() 388 | raise 389 | 390 | 391 | 392 | class Grade(Base): 393 | __tablename__ = 'grade' 394 | id = Column(Integer, primary_key=True) 395 | data = Column(Integer, nullable=False, default=0) 396 | user_id = Column(Integer, ForeignKey('workflow.id')) 397 | 398 | def __init__(self, grade): 399 | self.data = grade 400 | session.add(self) 401 | 402 | Base.metadata.create_all(engine) 403 | 404 | Next, we have to tell `DbWorkflowEngine` how and when to use our storage. To do 405 | that we need to know a bit about the engine's `processing_factory` property, 406 | which is expected to provide this structure of methods and properties: 407 | 408 | * `before_processing` 409 | * `after_processing` 410 | * `before_object` 411 | * `after_object` 412 | 413 | * `action_mapper` (property) 414 | 415 | * `before_callbacks` 416 | * `after_callbacks` 417 | * `before_each_callback` 418 | * `after_each_callback` 419 | 420 | * `transition_exception_mapper` (property) 421 | 422 | * `StopProcessing` 423 | * `HaltProcessing` 424 | * `ContinueNextToken` 425 | * `JumpToken` 426 | * `Exception` 427 | * ... (Can be extended by adding any method that has the name of an 428 | expected exception) 429 | 430 | The `transition_exception_mapper` can look confusing at first. It contains not 431 | Exceptions, but methods that are called when exceptions with the same name are 432 | raised. 433 | 434 | * Some exceptions are internal to the engine only and never bubble up. (eg 435 | `JumpToken`, `ContinueNextToken`) 436 | * Others are partly handled internally and then bubbled up to the user to 437 | take action. (eg `Exception`) 438 | 439 | Let's use the above to ask our engine to: 440 | 441 | 1. Save the first objects that it is given. 442 | 2. Save to our database every time it finished processing an object and 443 | when there is an expected failure. 444 | 445 | For now, all we need to know is that in our example, `HaltProcessing` is an 446 | exception that we will intentionally raise and we want to save the engine when 447 | it occurs. Once again, follow the comments carefully to understand the code. 448 | 449 | .. code-block:: python 450 | 451 | from workflow.engine_db import DbWorkflowEngine 452 | from workflow.errors import HaltProcessing 453 | from workflow.engine import TransitionActions, ProcessingFactory 454 | 455 | class MyDbWorkflowEngine(DbWorkflowEngine): 456 | 457 | def __init__(self, db_obj): 458 | """Load an old `token_pos` from the db into the engine.""" 459 | 460 | # The reason we save token_pos _first_, is because calling `super` 461 | # will reset. 462 | token_pos = db_obj.state_token_pos 463 | 464 | self.db_obj = db_obj 465 | super(DbWorkflowEngine, self).__init__() 466 | 467 | # And now we inject it back into the engine's `state`. 468 | if token_pos is not None: 469 | self.state.token_pos = token_pos 470 | self.save() 471 | 472 | # For this example we are interested in saving `token_pos` as explained 473 | # previously, so we override `save` to do that. 474 | def save(self, token_pos=None): 475 | """Save the state of the workflow.""" 476 | if token_pos is not None: 477 | self.state.token_pos = token_pos 478 | self.db_obj.save(self.state.token_pos) 479 | 480 | # We want our own processing factory, so we tell the engine that we 481 | # have subclassed it below. 482 | @staticproperty 483 | def processing_factory(): 484 | """Provide a processing factory.""" 485 | return MyProcessingFactory 486 | 487 | 488 | class MyProcessingFactory(ProcessingFactory): 489 | """Processing factory for persistence requirements.""" 490 | 491 | # We also have our own `transition_actions` 492 | @staticproperty 493 | def transition_exception_mapper(): 494 | """Define our for handling transition exceptions.""" 495 | return MyTransitionActions 496 | 497 | # Before any processing is done, we wish to save the `objects` (tokens) 498 | # that have been passed to the engine, if they aren't already stored. 499 | @staticmethod 500 | def before_processing(eng, objects): 501 | """Make sure the engine has a relationship with 502 | its objects.""" 503 | if not eng.db_obj.grades: 504 | for obj in objects: 505 | eng.db_obj.grades.append(obj) 506 | 507 | # We wish to save on every successful completion of a token. 508 | @staticmethod 509 | def after_processing(eng, objects): 510 | """Save after we processed all the objects successfully.""" 511 | eng.save() 512 | 513 | 514 | class MyTransitionActions(TransitionActions): 515 | 516 | # But we also wish to save when `HaltProcessing` is raised, because this 517 | # is going to be an expected situation. 518 | @staticmethod 519 | def HaltProcessing(obj, eng, callbacks, e): 520 | """Save whenever HaltProcessing is raised, so 521 | that we don't lose the state.""" 522 | eng.save() 523 | raise e 524 | 525 | 526 | And now, for the problem that we want to solve itself. Imagine an fictional 527 | exam where a student has to take 6 tests in one day. The tests are processed 528 | in a specific order by a system. Whenever the system locates a failing grade, 529 | as punishment, the student is asked to take the failed test again the next day. 530 | Then the checking process is resumed until the next failing grade is located 531 | and the student must show up again the following day. 532 | 533 | Assume that a student, Zack P. Hacker, has just finished taking all 6 tests. A 534 | workflow that does the following checking can now be implemented like so: 535 | 536 | .. code-block:: python 537 | 538 | from workflow.patterns.controlflow import IF, HALT 539 | from workflow.utils import staticproperty 540 | 541 | my_workflow_instance = Workflow() 542 | my_db_engine = MyDbWorkflowEngine(my_workflow_instance) 543 | 544 | def grade_is_not_passing(obj, eng): 545 | print 'Testing grade #{0}, with data {1}'.format(obj.id, obj.data) 546 | return obj.data < 5 547 | 548 | callbacks = [ 549 | IF(grade_is_not_passing, 550 | [ 551 | HALT() 552 | ]), 553 | ] 554 | 555 | my_db_engine.callbacks.replace(callbacks) 556 | try: 557 | my_db_engine.process([ 558 | Grade(6), Grade(5), Grade(4), 559 | Grade(5), Grade(2), Grade(6) 560 | ]) 561 | except HaltProcessing: 562 | print 'The student has failed this test!' 563 | 564 | # At this point, the engine has already saved its state in the database, 565 | # regardless of the outcome. 566 | 567 | The above script prints:: 568 | 569 | Testing grade #1, with data 6 570 | Testing grade #2, with data 5 571 | Testing grade #3, with data 4 572 | The student has failed this test! 573 | 574 | "Obviously this system is terrible and something must be done", thinks Zack who 575 | was just notified about his "4" and logs onto the system, armed with a small 576 | python script: 577 | 578 | .. code-block:: python 579 | 580 | def amend_grade(obj, eng): 581 | print 'Amending this grade..' 582 | obj.data = 5 583 | 584 | evil_callbacks = [ 585 | IF(grade_is_not_passing, 586 | [ 587 | amend_grade 588 | ]), 589 | ] 590 | 591 | # Load yesterday's workflow and bring up an engine for it. 592 | revived_workflow = session.query(Workflow).one() 593 | my_db_engine = MyDbWorkflowEngine(revived_workflow) 594 | 595 | print '\nWhat Zak sees:', [grade.data for grade in revived_workflow.grades] 596 | 597 | # Let's fix that. 598 | my_db_engine.callbacks.replace(evil_callbacks) 599 | print 'Note how the engine resumes from the last failing test:' 600 | my_db_engine.restart('current', 'first', objects=revived_workflow.grades) 601 | 602 | These words are printed in Zack's terminal:: 603 | 604 | 605 | What Zak sees: [6, 5, 4, 5, 2, 6] 606 | Note how the engine resumes from the last failing test: 607 | Testing grade #3, with data 4 608 | Amending this grade.. 609 | Testing grade #4, with data 5 610 | Testing grade #5, with data 2 611 | Amending this grade.. 612 | Testing grade #6, with data 6 613 | 614 | 615 | When someone logs into the system to check how Zack did.. 616 | 617 | .. code-block:: python 618 | 619 | revived_workflow = session.query(Workflow).one() 620 | print '\nWhat the professor sees:', [grade.data for grade in revived_workflow.grades] 621 | 622 | Everything looks good:: 623 | 624 | What the professor sees: [6, 5, 5, 5, 5, 6] 625 | 626 | The moral of this story is to keep off-site logs and back-ups. Also, workflows 627 | are complex but powerful. 628 | 629 | .. DbWorkflowEngine interface 630 | .. ~~~~~~~~~~~~~~~~~~~~~~~~~~ 631 | 632 | .. `DBWorkflowEngine` provides a more extended interface out of the box. 633 | 634 | .. has a `save` method out of the box. By default it 635 | .. calls `.save()` on `db_obj`, allowing for an optional argument: 636 | .. `status`. `status` allows saving the status (eg HALTED) of the engine 637 | .. the database. For simplicity, we do not support this in our 638 | .. database. You can read more about the default status storage representations in the " 639 | 640 | Signals support 641 | =============== 642 | 643 | Adding to the exception and override-based mechanisms, Workflow supports a few 644 | signals out of the box if the `blinker` package is installed. The following 645 | exceptions are triggered by the `GenericWorkflowEngine`. 646 | 647 | ============================ ================================================= 648 | Signal Called by 649 | ============================ ================================================= 650 | `workflow_started` ProcessingFactory.before_processing 651 | `workflow_finished` ProcessingFactory.after_processing 652 | `workflow_halted` TransitionActions.HaltProcessing 653 | ============================ ================================================= 654 | 655 | Useful engine methods 656 | ===================== 657 | 658 | Other than `eng.halt`, the GenericWorkflowEngine provides more convenience 659 | methods out of the box. 660 | 661 | =============================== ============================== 662 | Sample call Description 663 | =============================== ============================== 664 | `eng.stop()` stop the workflow 665 | `eng.halt("list exhausted")` halt the workflow 666 | `eng.continue_next_token()` continue from the next token 667 | `eng.jump_token(-2)` jump `offset` tokens 668 | `eng.jump_call(3)` jump `offset` steps of a loop 669 | `eng.break_current_loop()` break out of the current loop 670 | =============================== ============================== 671 | 672 | By calling these, any **task** can influence the whole pipeline. You can read 673 | more about the methods our engines provide at the end of this document. 674 | 675 | 676 | Patterns 677 | ======== 678 | 679 | The workflow module also comes with many patterns that can be directly used in 680 | the definition of the pipeline, such as **PARALLEL_SPLIT**. 681 | 682 | Consider this example of a task: 683 | 684 | .. code-block:: python 685 | 686 | def if_else(call): 687 | def inner_call(obj, eng): 688 | if call(obj, eng): # if True, continue processing.. 689 | eng.jump_call(1) 690 | else: # ..else, skip the next step 691 | eng.jump_call(2) 692 | return inner_call 693 | 694 | We can then write a **workflow definition** like this: 695 | 696 | .. code-block:: python 697 | 698 | [ 699 | if_else(stage_submission), 700 | [ 701 | [ 702 | if_else(fulltext_available), 703 | [extract_metadata, populate_empty_fields], 704 | [] 705 | ], 706 | [ 707 | if_else(check_for_duplicates), 708 | [stop_processing], 709 | [synchronize_fields, replace_values] 710 | ], 711 | check_mandatory_fields, 712 | ], 713 | [ 714 | check_mandatory_fields, 715 | check_preferred_values, 716 | save_record 717 | ] 718 | ] 719 | 720 | Example: Parallel split 721 | ----------------------- 722 | 723 | .. raw:: html 724 | 725 | 726 | 727 | This pattern is called Parallel split (as tasks B,C,D are all started in 728 | parallel after task A). It could be implemented like this: 729 | 730 | .. code-block:: python 731 | 732 | def PARALLEL_SPLIT(*args): 733 | """ 734 | Tasks A,B,C,D... are all started in parallel 735 | @attention: tasks A,B,C,D... are not addressable, 736 | you can't use jumping to them (they are invisible to 737 | the workflow engine). Though you can jump inside the 738 | branches 739 | @attention: tasks B,C,D... will be running on their own 740 | once you have started them, and we are not waiting for 741 | them to finish. Workflow will continue executing other 742 | tasks while B,C,D... might be still running. 743 | @attention: a new engine is spawned for each branch or code, 744 | all operations works as expected, but mind that the branches 745 | know about themselves, they don't see other tasks outside. 746 | They are passed the object, but not the old workflow 747 | engine object 748 | @postcondition: eng object will contain lock (to be used 749 | by threads) 750 | """ 751 | 752 | def _parallel_split(obj, eng, calls): 753 | lock = thread.allocate_lock() 754 | eng.store['lock'] = lock 755 | for func in calls: 756 | new_eng = eng.duplicate() 757 | new_eng.setWorkflow([lambda o, e: e.store.update({'lock': lock}), func]) 758 | thread.start_new_thread(new_eng.process, ([obj], )) 759 | return lambda o, e: _parallel_split(o, e, args) 760 | 761 | 762 | Subsequently, we can use PARALLEL_SPLIT like this. 763 | 764 | .. code-block:: python 765 | 766 | from workflow.patterns import PARALLEL_SPLIT 767 | from my_module_x import task_a,task_b,task_c,task_d 768 | 769 | [ 770 | task_a, 771 | PARALLEL_SPLIT(task_b,task_c,task_d) 772 | ] 773 | 774 | Note that PARALLEL_SPLIT is already provided in 775 | `workflow.patterns.PARALLEL_SPLIT`. 776 | 777 | Example: Synchronisation 778 | ------------------------ 779 | 780 | .. raw:: html 781 | 782 | 783 | 784 | After the execution of task B, task C, and task D, task E can be executed 785 | (I will present the threaded version, as the sequential version would be 786 | dead simple). 787 | 788 | .. code-block:: python 789 | 790 | def SYNCHRONIZE(*args, **kwargs): 791 | """ 792 | After the execution of task B, task C, and task D, task E can be executed. 793 | @var *args: args can be a mix of callables and list of callables 794 | the simplest situation comes when you pass a list of callables 795 | they will be simply executed in parallel. 796 | But if you pass a list of callables (branch of callables) 797 | which is potentially a new workflow, we will first create a 798 | workflow engine with the workflows, and execute the branch in it 799 | @attention: you should never jump out of the synchronized branches 800 | """ 801 | timeout = MAX_TIMEOUT 802 | if 'timeout' in kwargs: 803 | timeout = kwargs['timeout'] 804 | 805 | if len(args) < 2: 806 | raise Exception('You must pass at least two callables') 807 | 808 | def _synchronize(obj, eng): 809 | queue = MyTimeoutQueue() 810 | #spawn a pool of threads, and pass them queue instance 811 | for i in range(len(args)-1): 812 | t = MySpecialThread(queue) 813 | t.setDaemon(True) 814 | t.start() 815 | 816 | for func in args[0:-1]: 817 | if isinstance(func, list) or isinstance(func, tuple): 818 | new_eng = duplicate_engine_instance(eng) 819 | new_eng.setWorkflow(func) 820 | queue.put(lambda: new_eng.process([obj])) 821 | else: 822 | queue.put(lambda: func(obj, eng)) 823 | 824 | #wait on the queue until everything has been processed 825 | queue.join_with_timeout(timeout) 826 | 827 | #run the last func 828 | args[-1](obj, eng) 829 | _synchronize.__name__ = 'SYNCHRONIZE' 830 | return _synchronize 831 | 832 | 833 | Configuration (i.e. what would admins write): 834 | 835 | .. code-block:: python 836 | 837 | from workflow.patterns import SYNCHRONIZE 838 | from my_module_x import task_a,task_b,task_c,task_d 839 | 840 | [ 841 | SYNCHRONIZE(task_b,task_c,task_d, task_a) 842 | ] 843 | 844 | .. .. automodule:: workflow 845 | .. :members: 846 | 847 | GenericWorkflowEngine API 848 | ========================= 849 | 850 | This documentation is automatically generated from Workflow's source code. 851 | 852 | .. autoclass:: workflow.engine.GenericWorkflowEngine 853 | :members: 854 | 855 | .. autoclass:: workflow.engine.MachineState 856 | :members: 857 | 858 | .. autoclass:: workflow.engine.Callbacks 859 | :members: 860 | 861 | .. autoclass:: workflow.engine._Signal 862 | :members: 863 | 864 | DbWorkflowEngine API 865 | ==================== 866 | 867 | .. autoclass:: workflow.engine_db.DbWorkflowEngine 868 | :members: 869 | 870 | .. autoclass:: workflow.engine_db.WorkflowStatus 871 | :members: 872 | 873 | .. autoclass:: workflow.engine_db.ObjectStatus 874 | :members: 875 | 876 | .. include:: ../CONTRIBUTING.rst 877 | 878 | 879 | Release notes 880 | --------------- 881 | Here you can find the `release notes`_. 882 | 883 | 884 | Changelog 885 | ------------ 886 | Here you can find the `full changelog for this version`_. 887 | 888 | 889 | License 890 | ======= 891 | 892 | .. include:: ../LICENSE 893 | 894 | 895 | .. _release notes: _static/RELEASE_NOTES.txt 896 | .. _full changelog for this version: _static/CHANGELOG.txt 897 | -------------------------------------------------------------------------------- /docs/requirements.txt: -------------------------------------------------------------------------------- 1 | -e .[docs,tests] 2 | -------------------------------------------------------------------------------- /pytest.ini: -------------------------------------------------------------------------------- 1 | [pytest] 2 | addopts = --pep8 --ignore=docs --cov=workflow --cov-report=term-missing 3 | pep8ignore = 4 | test_*.py E501 E731 E402 E241 E222 5 | -------------------------------------------------------------------------------- /run-tests.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # 3 | # This file is part of Workflow. 4 | # Copyright (C) 2014 CERN. 5 | # 6 | # Workflow is free software; you can redistribute it and/or modify it 7 | # under the terms of the Revised BSD License; see LICENSE file for 8 | # more details. 9 | 10 | sphinx-build -qnNW docs docs/_build/html 11 | python setup.py test 12 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # This file is part of Workflow. 4 | # Copyright (C) 2016 CERN. 5 | # 6 | # Workflow is free software; you can redistribute it and/or modify it 7 | # under the terms of the Revised BSD License; see LICENSE file for 8 | # more details. 9 | 10 | [aliases] 11 | test=pytest 12 | 13 | [build_sphinx] 14 | source-dir = docs/ 15 | build-dir = docs/_build 16 | all_files = 1 17 | 18 | [bdist_wheel] 19 | universal = 1 20 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # This file is part of Workflow. 4 | # Copyright (C) 2011, 2014, 2015, 2016, 2018 CERN. 5 | # 6 | # Workflow is free software; you can redistribute it and/or modify it 7 | # under the terms of the Revised BSD License; see LICENSE file for 8 | # more details. 9 | 10 | """Simple workflows for Python""" 11 | 12 | import os 13 | import platform 14 | 15 | from setuptools import find_packages, setup 16 | 17 | readme = open('README.rst').read() 18 | 19 | tests_require = [ 20 | 'coverage>=4.0', 21 | 'mock>=1.0.0', 22 | 'isort>=4.2.2', 23 | 'pytest-cache>=1.0', 24 | 'pytest-cov>=1.8.0', 25 | 'pytest-pep8>=1.0.6', 26 | 'pytest>=2.8.0', 27 | ] 28 | 29 | extras_require = { 30 | 'docs': [ 31 | 'Sphinx>=1.4.2', 32 | ], 33 | 'tests': tests_require, 34 | } 35 | 36 | extras_require['all'] = [] 37 | for reqs in extras_require.values(): 38 | extras_require['all'].extend(reqs) 39 | 40 | install_requires = [ 41 | 'autosemver~=0.5', 42 | 'configobj>4.7.0', 43 | 'blinker>=1.3', 44 | 'six', 45 | ] 46 | 47 | setup_requires = [ 48 | 'autosemver~=0.5', 49 | 'pytest-runner>=2.6.2', 50 | ] 51 | 52 | if platform.python_version_tuple() < ('3', '4'): 53 | install_requires.append('enum34>=1.0.4') 54 | 55 | packages = find_packages(exclude=['docs', 'tests']) 56 | 57 | URL = 'https://github.com/inveniosoftware/workflow' 58 | 59 | setup( 60 | name='workflow', 61 | description=__doc__, 62 | packages=packages, 63 | scripts=['bin/run_workflow.py'], 64 | author='Invenio Collaboration', 65 | author_email='info@inveniosoftware.org', 66 | url=URL, 67 | keywords=['workflows', 'finite state machine', 'task execution'], 68 | zip_safe=False, 69 | include_package_data=True, 70 | platforms='any', 71 | extras_require=extras_require, 72 | install_requires=install_requires, 73 | setup_requires=setup_requires, 74 | tests_require=tests_require, 75 | autosemver={ 76 | 'bugtracker_url': URL + '/issues', 77 | }, 78 | classifiers=[ 79 | 'Programming Language :: Python', 80 | 'Programming Language :: Python :: 2', 81 | 'Programming Language :: Python :: 2.7', 82 | 'Programming Language :: Python :: 3', 83 | 'Programming Language :: Python :: 3.6', 84 | 'Development Status :: 5 - Production/Stable', 85 | 'Environment :: Other Environment', 86 | 'Intended Audience :: Developers', 87 | 'License :: OSI Approved :: BSD License', 88 | 'Operating System :: OS Independent', 89 | 'Topic :: Software Development :: Libraries :: Python Modules', 90 | 'Topic :: Software Development :: Libraries :: Application Frameworks', 91 | 'Topic :: Utilities', 92 | ], 93 | ) 94 | -------------------------------------------------------------------------------- /tests/global.ini: -------------------------------------------------------------------------------- 1 | STRING = string 2 | ARRAY = one, two, three 3 | OVERRIDEN = global 4 | VARIABLE = %(OVERRIDEN)s 5 | -------------------------------------------------------------------------------- /tests/local.ini: -------------------------------------------------------------------------------- 1 | 2 | string = %(VARIABLE)s/local 3 | 4 | OVERRIDEN = local 5 | 6 | 7 | -------------------------------------------------------------------------------- /tests/local2.ini: -------------------------------------------------------------------------------- 1 | string = second 2 | 3 | 4 | -------------------------------------------------------------------------------- /tests/test_config.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # This file is part of Workflow. 4 | # Copyright (C) 2014, 2015 CERN. 5 | # 6 | # Workflow is free software; you can redistribute it and/or modify it 7 | # under the terms of the Revised BSD License; see LICENSE file for 8 | # more details. 9 | 10 | import unittest 11 | import os 12 | import logging 13 | from six import StringIO 14 | 15 | from workflow.config import config_reader 16 | 17 | 18 | class TestConfig(object): 19 | 20 | """Tests of the WE interface""" 21 | 22 | def setup_method(self, method): 23 | config_reader.setBasedir(os.path.dirname(__file__)) 24 | self._old_handlers = logging.root.handlers[:] 25 | 26 | def teardown_method(self, method): 27 | logging.root.handlers = self._old_handlers[:] 28 | 29 | def test_basics(self): 30 | config_reader.init(os.path.join(os.path.dirname(__file__), 31 | 'local.ini')) 32 | assert config_reader.STRING == 'string' 33 | assert config_reader.ARRAY == ['one', 'two', 'three'] 34 | assert config_reader.OVERRIDEN == 'local' 35 | assert config_reader.string == 'global/local' 36 | 37 | config_reader.init(os.path.join(os.path.dirname(__file__), 38 | 'local2.ini')) 39 | assert config_reader.STRING == 'string' 40 | assert config_reader.ARRAY == ['one', 'two', 'three'] 41 | assert config_reader.OVERRIDEN == 'global' 42 | assert config_reader.string == 'second' 43 | 44 | def test_logger(self): 45 | """The WF logger should not affect other loggers.""" 46 | from workflow import engine 47 | 48 | logging.root.handlers = [] 49 | engine.LOG.handlers = [] 50 | 51 | other_logger = logging.getLogger('other') 52 | wf_logger = engine.get_logger('workflow.test') 53 | 54 | test_io = StringIO() 55 | root_io = StringIO() 56 | other_io = StringIO() 57 | 58 | logging.root.addHandler(logging.StreamHandler(root_io)) 59 | other_logger.addHandler(logging.StreamHandler(other_io)) 60 | wf_logger.addHandler(logging.StreamHandler(test_io)) 61 | 62 | # set the root level to WARNING; wf should honour parent level 63 | logging.root.setLevel(logging.WARNING) 64 | 65 | logging.warn('root warn') 66 | other_logger.warn('other warn') 67 | wf_logger.warn('wf warn') 68 | 69 | logging.info('root info') 70 | other_logger.info('other info') 71 | wf_logger.info('wf info') 72 | 73 | assert root_io.getvalue() == "root warn\nother warn\n" # Root logger should have two msgs 74 | assert other_io.getvalue() == "other warn\n" # Other logger should have one msg 75 | assert test_io.getvalue() == "wf warn\n" # Wf logger should have one msg 76 | 77 | root_io.seek(0) 78 | other_io.seek(0) 79 | test_io.seek(0) 80 | 81 | # now set too to DEBUG and wf to INFO 82 | logging.root.setLevel(logging.DEBUG) 83 | engine.reset_all_loggers(logging.WARNING) 84 | 85 | logging.warn('root warn') 86 | other_logger.warn('other warn') 87 | wf_logger.warn('wf warn') 88 | 89 | logging.info('root info') 90 | other_logger.info('other info') 91 | wf_logger.info('wf info') 92 | 93 | assert root_io.getvalue() == "root warn\nother warn\n" "root info\nother info\n" # Root logger should have four msgs 94 | assert other_io.getvalue() == "other warn\nother info\n" # Other logger should have two msg 95 | assert test_io.getvalue() == "wf warn\n" # Wf logger should have one msg 96 | -------------------------------------------------------------------------------- /tests/test_engine.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # This file is part of Workflow. 4 | # Copyright (C) 2014, 2015, 2016 CERN. 5 | # 6 | # Workflow is free software; you can redistribute it and/or modify it 7 | # under the terms of the Revised BSD License; see LICENSE file for 8 | # more details. 9 | 10 | import os 11 | import sys 12 | import mock 13 | 14 | import pytest 15 | 16 | import re 17 | 18 | from workflow.patterns.controlflow import IF_ELSE 19 | from workflow.engine import GenericWorkflowEngine, HaltProcessing 20 | from workflow.errors import WorkflowError 21 | 22 | 23 | p = os.path.abspath(os.path.dirname(__file__) + '/../') 24 | if p not in sys.path: 25 | sys.path.append(p) 26 | 27 | 28 | def m(key=None): 29 | def _m(token, inst): 30 | current_sem = token.getFeature('sem', '') 31 | new_feature = (current_sem + ' ' + key).strip() 32 | token.setFeatureKw(sem=new_feature) 33 | _m.__name__ = 'string appender' 34 | return _m 35 | 36 | 37 | def if_str_token_jump(value='', step=0): 38 | def x(token, inst): 39 | if step >= 0: 40 | feature_str = 'token_back' 41 | else: 42 | feature_str = 'token_forward' 43 | if str(token) == value and not token.getFeature(feature_str): 44 | token.setFeature(feature_str, 1) 45 | inst.jump_token(step) 46 | return lambda token, inst: x(token, inst) 47 | 48 | 49 | def jump_call(step=0): 50 | if step < 0: 51 | def x(token, inst): 52 | if not token.getFeature('back'): 53 | token.setFeature('back', 1) 54 | inst.jump_call(step) 55 | return lambda token, inst: x(token, inst) 56 | return lambda token, inst: inst.jump_call(step) 57 | 58 | 59 | def break_loop(): 60 | return lambda token, inst: inst.break_current_loop() 61 | 62 | 63 | def workflow_error(): 64 | def _error(token, inst): 65 | raise WorkflowError("oh no!") 66 | return _error 67 | 68 | 69 | def stop_processing(): 70 | return lambda token, inst: inst.stop() 71 | 72 | 73 | def halt_processing(): 74 | return lambda token, inst: inst.halt() 75 | 76 | 77 | def next_token(): 78 | return lambda token, inst: inst.continue_next_token() 79 | 80 | 81 | def get_first(doc): 82 | return doc[0].getFeature('sem') 83 | 84 | 85 | def get_xth(doc, xth): 86 | return doc[xth].getFeature('sem') 87 | 88 | 89 | def stop_if_token_equals(value=None): 90 | def x(token, inst): 91 | if str(token) == value: 92 | inst.stopProcessing() 93 | return lambda token, inst: x(token, inst) 94 | 95 | 96 | class FakeToken(object): 97 | 98 | def __init__(self, data, **attributes): 99 | self.data = data 100 | self.pos = None # set TokenCollection on obj return 101 | # here link to TokenCollection (when returning) 102 | self.backreference = None 103 | self.__prev = 0 104 | self.__next = 0 105 | self.__attributes = {} 106 | for attr_name, attr_value in attributes.items(): 107 | self.setFeature(attr_name, attr_value) 108 | 109 | def __str__(self): 110 | return str(self.data) 111 | 112 | def __repr__(self): 113 | return 'Token(%s, **%s)' % (repr(self.data), repr(self.__attributes)) 114 | 115 | def getFeature(self, key, default=None): 116 | try: 117 | return self.__attributes[key] 118 | except KeyError: 119 | return default 120 | 121 | def setFeature(self, key, value): 122 | self.__attributes[key] = value 123 | 124 | def setFeatureKw(self, **kwargs): 125 | for key, value in kwargs.items(): 126 | self.setFeature(key, value) 127 | 128 | 129 | class TestWorkflowEngine(object): 130 | 131 | """Tests using FakeTokens in place of strings""" 132 | 133 | def setup_method(self, method): 134 | self.key = '*' 135 | self.wfe = GenericWorkflowEngine() 136 | self.data = ['one', 'two', 'three', 'four', 'five'] 137 | self.tokens = [FakeToken(x, type='*') for x in self.data] 138 | 139 | def teardown_method(self, method): 140 | pass 141 | 142 | @pytest.mark.parametrize("_,tokens,exception,exception_msg", ( 143 | ("int", 49, WorkflowError, "not an iterable"), 144 | ("str", "hello", WorkflowError, "not an iterable"), 145 | ("object", object, WorkflowError, "not an iterable"), 146 | )) 147 | def test_objects_are_of_bad_type(self, _, tokens, exception, exception_msg): 148 | with pytest.raises(exception) as exc_info: 149 | self.wfe.process(tokens) 150 | assert exception_msg in exc_info.value.args[0] 151 | 152 | def test_empty_object_list_logs_warning(self): 153 | assert hasattr(self.wfe, 'log') 154 | self.wfe.log = mock.Mock() 155 | self.wfe.callbacks.replace([lambda o, e: None]) 156 | self.wfe.process([]) 157 | self.wfe.log.warning.assert_called_once_with('List of objects is empty. Running workflow ' 158 | 'on empty set has no effect.') 159 | 160 | def test_current_taskname_resolution(self): 161 | workflow = [m('test')] 162 | self.wfe.callbacks.replace(workflow, self.key) 163 | self.wfe.process(self.tokens) 164 | assert self.wfe.current_taskname == 'string appender' 165 | 166 | workflow = [lambda obj, eng: 1] 167 | self.wfe.callbacks.replace(workflow, self.key) 168 | self.wfe.process(self.tokens) 169 | assert self.wfe.current_taskname == '' 170 | 171 | workflow = [ 172 | IF_ELSE( 173 | lambda obj, eng: True, 174 | [lambda obj, eng: 1], 175 | [lambda obj, eng: 2], 176 | ) 177 | ] 178 | self.wfe.callbacks.replace(workflow, self.key) 179 | # This test will break if someone changes IF_ELSE. TODO: Mock 180 | # Note: Python3 has much stronger introspection, thus the `.*`. 181 | assert re.match(r'\[, ' 182 | r'\[ at 0x[0-f]+>\], ' 183 | r', ' 184 | r'\[ at 0x[0-f]+>\]\]', 185 | self.wfe.current_taskname) 186 | 187 | def test_current_object_returns_correct_object(self): 188 | self.wfe.callbacks.replace([halt_processing()]) 189 | 190 | assert self.wfe.current_object is None 191 | with pytest.raises(HaltProcessing): 192 | self.wfe.process(self.tokens) 193 | assert self.wfe.current_object is self.tokens[0] 194 | with pytest.raises(HaltProcessing): 195 | self.wfe.restart('current', 'next') 196 | assert self.wfe.current_object is self.tokens[1] 197 | 198 | @pytest.mark.parametrize("_,callbacks,expected_result", ( 199 | ( 200 | 'skips_forward_with_acceptable_increment', 201 | [ 202 | m('mouse'), 203 | [m('dog'), jump_call(2), m('cat'), m('puppy'), m('python')], 204 | m('horse'), 205 | ], 206 | 'mouse dog puppy python horse' 207 | ), 208 | 209 | ( 210 | 'skips_forward_with_increment_that_is_too_large', 211 | [ 212 | m('mouse'), 213 | [m('dog'), jump_call(50), m('cat'), m('puppy'), m('python')], 214 | m('horse'), 215 | ], 216 | 'mouse dog horse' 217 | ), 218 | 219 | ( 220 | 'jumps_forward_outside_of_nest', 221 | [ 222 | jump_call(3), 223 | m('mouse'), 224 | [m('dog'), m('cat'), m('puppy'), m('python')], 225 | m('horse'), 226 | ], 227 | 'horse' 228 | ), 229 | 230 | ( 231 | 'skips_backwards_with_acceptable_decrement', 232 | [ 233 | m('mouse'), 234 | [m('dog'), jump_call(-1), m('cat'), m('puppy')], 235 | m('horse'), 236 | ], 237 | 'mouse dog dog cat puppy horse' 238 | ), 239 | 240 | ( 241 | 'skips_backwards_with_decrement_that_is_too_large', 242 | [ 243 | m('mouse'), 244 | [m('dog'), m('cat'), jump_call(-50), m('puppy'), m('python')], 245 | m('horse'), 246 | ], 247 | 'mouse dog cat dog cat puppy python horse' 248 | ), 249 | 250 | ( 251 | 'skips_backwards_outside_of_nest', 252 | [ 253 | m('mouse'), 254 | [m('dog'), m('cat'), m('puppy'), m('python')], 255 | m('horse'), 256 | jump_call(-2) 257 | ], 258 | 'mouse dog cat puppy python horse dog cat puppy python horse' 259 | ) 260 | )) 261 | def test_jump_call(self, _, callbacks, expected_result): 262 | self.wfe.callbacks.add_many(callbacks, self.key) 263 | self.wfe.process(self.tokens) 264 | t = get_first(self.tokens) 265 | assert t == expected_result 266 | 267 | # --------- complicated loop ----------- 268 | 269 | @pytest.mark.parametrize("_,workflow,expected_result", ( 270 | ( 271 | 'simple', 272 | [ 273 | m('mouse'), 274 | [ 275 | m('dog'), 276 | [m('cat'), m('puppy')], 277 | [m('python'), [m('wasp'), m('leon')]], 278 | m('horse'), 279 | ] 280 | ], 281 | 'mouse dog cat puppy python wasp leon horse' 282 | ), 283 | 284 | ( 285 | 'with_nested_jumps', 286 | [ 287 | jump_call(2), 288 | m('mouse'), 289 | [ 290 | m('dog'), 291 | [m('cat'), m('puppy')], 292 | [m('python'), jump_call(-2), [m('wasp'), m('leon')]], 293 | m('horse'), 294 | ] 295 | ], 296 | 'dog cat puppy python python wasp leon horse' 297 | ) 298 | )) 299 | def test_multi_nested_workflows(self, _, workflow, expected_result): 300 | self.wfe.callbacks.add_many(workflow, self.key) 301 | self.wfe.process(self.tokens) 302 | t = get_first(self.tokens) 303 | assert t == expected_result 304 | 305 | @pytest.mark.parametrize("_,workflow,expected_result", ( 306 | ( 307 | 'simple', 308 | [ 309 | m('mouse'), 310 | [ 311 | m('dog'), 312 | [m('cat'), m('puppy')], 313 | [m('python'), break_loop(), [m('wasp'), m('leon')]], 314 | m('horse'), 315 | ] 316 | ], 317 | 'mouse dog cat puppy python horse' 318 | ), 319 | 320 | ( 321 | 'break_loop_outside_of_nest', 322 | [ 323 | break_loop(), 324 | m('mouse'), 325 | [ 326 | m('dog'), 327 | [m('cat'), m('puppy')], 328 | [m('python'), [m('wasp'), m('leon')]], 329 | m('horse'), 330 | ] 331 | ], 332 | None 333 | ) 334 | )) 335 | def test_break_from_this_loop(self, _, workflow, expected_result): 336 | self.wfe.callbacks.add_many(workflow, self.key) 337 | self.wfe.process(self.tokens) 338 | t = get_first(self.tokens) 339 | assert t == expected_result 340 | 341 | # ----------- StopProcessing -------------------------- 342 | 343 | def test_engine_immediatelly_stops(self): 344 | self.wfe.callbacks.add_many([ 345 | stop_processing(), 346 | m('mouse'), 347 | [ 348 | m('dog'), 349 | [m('cat'), m('puppy')], 350 | [ 351 | m('python'), 352 | [m('wasp'), m('leon')] 353 | ], 354 | m('horse'), 355 | ] 356 | ], self.key) 357 | self.wfe.process(self.tokens) 358 | t = get_first(self.tokens) 359 | assert get_xth(self.tokens, 0) is None 360 | assert get_xth(self.tokens, 1) is None 361 | assert get_xth(self.tokens, 2) is None 362 | 363 | def test_engine_stops_half_way_through(self): 364 | self.wfe.callbacks.add_many([ 365 | m('mouse'), 366 | [ 367 | m('dog'), 368 | [m('cat'), m('puppy')], 369 | [m('python'), stop_if_token_equals('four'), [m('wasp'), m('leon')]], 370 | m('horse'), 371 | ] 372 | ], self.key) 373 | self.wfe.process(self.tokens) 374 | full_result = 'mouse dog cat puppy python wasp leon horse' 375 | result_until_stop = 'mouse dog cat puppy python' 376 | assert get_xth(self.tokens, 0) == full_result # 'one' 377 | assert get_xth(self.tokens, 1) == full_result # 'two' 378 | assert get_xth(self.tokens, 2) == full_result # 'three' 379 | assert get_xth(self.tokens, 3) == result_until_stop # 'four' 380 | assert get_xth(self.tokens, 4) is None # 'five', engine stopped 381 | 382 | # ---------- jump_token ------------- 383 | 384 | def test_engine_moves_to_next_token(self): 385 | self.wfe.callbacks.add_many( 386 | [ 387 | m('mouse'), 388 | [ 389 | m('dog'), 390 | [m('cat'), m('puppy')], 391 | [m('python'), next_token(), [m('wasp'), m('leon')]], 392 | m('horse'), 393 | ] 394 | ], self.key) 395 | self.wfe.process(self.tokens) 396 | result_until_next_token = 'mouse dog cat puppy python' 397 | for i in range(5): 398 | assert get_xth(self.tokens, i) == result_until_next_token 399 | 400 | def test_workflow_09a(self): 401 | self.wfe.callbacks.add_many([ 402 | m('mouse'), 403 | [ 404 | m('dog'), if_str_token_jump('four', -2), 405 | [m('cat'), m('puppy')], 406 | m('horse'), 407 | ] 408 | ], self.key) 409 | self.wfe.process(self.tokens) 410 | t = get_first(self.tokens) 411 | r1 = 'mouse dog cat puppy horse' # one, five 412 | r2 = 'mouse dog cat puppy horse mouse dog cat puppy horse' # two, three 413 | r3 = 'mouse dog mouse dog cat puppy horse' # four 414 | assert get_xth(self.tokens, 0) == r1 415 | assert get_xth(self.tokens, 1) == r2 416 | assert get_xth(self.tokens, 2) == r2 417 | assert get_xth(self.tokens, 3) == r3 418 | assert get_xth(self.tokens, 4) == r1 419 | 420 | def test_workflow_09b(self): 421 | self.wfe.callbacks.add_many([ 422 | m('mouse'), 423 | [ 424 | m('dog'), 425 | if_str_token_jump('two', 2), 426 | [m('cat'), m('puppy')], 427 | m('horse'), 428 | ] 429 | ], self.key) 430 | self.wfe.process(self.tokens) 431 | t = get_first(self.tokens) 432 | r1 = 'mouse dog cat puppy horse' # one, four, five 433 | r2 = 'mouse dog' # two 434 | r3 = None # three 435 | assert get_xth(self.tokens, 0) == r1 436 | assert get_xth(self.tokens, 1) == r2 437 | assert get_xth(self.tokens, 2) == r3 438 | assert get_xth(self.tokens, 3) == r1 439 | assert get_xth(self.tokens, 4) == r1 440 | 441 | # ----------------- HaltProcessing -------------------- 442 | 443 | def test_50_halt_processing_mid_workflow(self): 444 | other_wfe = GenericWorkflowEngine() 445 | other_wfe.callbacks.add_many([ 446 | m('mouse'), 447 | [ 448 | m('dog'), 449 | [m('cat'), m('puppy')], 450 | [m('python'), halt_processing()], 451 | m('horse'), 452 | ] 453 | ], self.key) 454 | with pytest.raises(HaltProcessing): 455 | other_wfe.process(self.tokens) 456 | 457 | t = get_first(self.tokens) 458 | assert get_xth(self.tokens, 0) == 'mouse dog cat puppy python' 459 | assert get_xth(self.tokens, 1) is None 460 | assert get_xth(self.tokens, 2) is None 461 | 462 | compl = 'mouse dog cat puppy python horse' 463 | compl1 = 'mouse dog cat puppy python' 464 | 465 | @pytest.mark.parametrize("obj,task,results", ( 466 | ('prev', 'prev', (compl + " python", compl1, None)), 467 | ('prev', 'current', (compl, compl1, None)), # current task is to halt 468 | ('prev', 'next', (compl + " horse", compl1 + " " + compl1, None)), 469 | ('prev', 'first', (compl + " " + compl1, compl1, None)), 470 | 471 | ('current', 'prev', (compl, compl1 + " python", None)), 472 | ('current', 'current', (compl, compl1, None)), 473 | ('current', 'next', (compl, compl1 + " horse", compl1)), 474 | ('current', 'first', (compl, compl1 + " " + compl1, None)), 475 | 476 | ('next', 'prev', (compl, compl1, "python")), 477 | ('next', 'current', (compl, compl1, None)), 478 | ('next', 'next', (compl, compl1, "horse")), 479 | ('next', 'first', (compl, compl1, compl1)), 480 | 481 | ('first', 'prev', (compl + " python", compl1, None)), 482 | ('first', 'current', (compl, compl1, None)), # current task is to halt 483 | ('first', 'next', (compl + " horse", compl1 + " " + compl1, None)), 484 | ('first', 'first', (compl + " " + compl1, compl1, None)), 485 | )) 486 | def test_51_workflow_restart_after_halt(self, obj, task, results): 487 | self.wfe.callbacks.add_many([ 488 | m('mouse'), 489 | [ 490 | m('dog'), 491 | [m('cat'), m('puppy')], 492 | [m('python'), halt_processing()], 493 | m('horse'), 494 | ] 495 | ], self.key) 496 | with pytest.raises(HaltProcessing): 497 | self.wfe.process(self.tokens) 498 | 499 | assert get_xth(self.tokens, 0) == 'mouse dog cat puppy python' 500 | assert get_xth(self.tokens, 1) is None 501 | assert get_xth(self.tokens, 2) is None 502 | 503 | # this should pick up from the point where we stopped 504 | with pytest.raises(HaltProcessing): 505 | self.wfe.restart('current', 'next') 506 | 507 | assert get_xth(self.tokens, 0) == 'mouse dog cat puppy python horse' 508 | assert get_xth(self.tokens, 1) == 'mouse dog cat puppy python' 509 | assert get_xth(self.tokens, 2) is None 510 | 511 | with pytest.raises(HaltProcessing): 512 | self.wfe.restart(obj, task) 513 | 514 | assert (get_xth(self.tokens, 0), 515 | get_xth(self.tokens, 1), 516 | get_xth(self.tokens, 2)) == results 517 | 518 | def test_restart_accepts_new_objects(self): 519 | workflow = [m('test')] 520 | self.wfe.callbacks.replace(workflow, self.key) 521 | self.wfe.process(self.tokens) 522 | 523 | new_data = ['a', 'b', 'c', 'd', 'e'] 524 | new_tokens = [FakeToken(x, type='*') for x in new_data] 525 | 526 | self.wfe.restart('first', 'first', objects=new_tokens) 527 | 528 | assert self.wfe.objects == new_tokens 529 | 530 | def test_has_completed(self): 531 | self.wfe.callbacks.replace([ 532 | m('mouse'), 533 | halt_processing(), 534 | m('horse'), 535 | ]) 536 | assert self.wfe.has_completed is False 537 | with pytest.raises(HaltProcessing): 538 | self.wfe.process([self.tokens[0]]) 539 | assert self.wfe.has_completed is False 540 | self.wfe.restart('current', 'next') 541 | assert self.wfe.has_completed is True 542 | 543 | def test_nested_workflow_halt(self): 544 | other_wfe = GenericWorkflowEngine() 545 | wfe = self.wfe 546 | 547 | other_wfe.callbacks.add_many([ 548 | m('mouse'), 549 | [ 550 | m('dog'), 551 | [m('cat'), m('puppy')], 552 | [m('python'), halt_processing()], 553 | m('horse'), 554 | ] 555 | ], self.key) 556 | 557 | wfe.callbacks.add_many([ 558 | m('mouse'), 559 | [ 560 | m('dog'), 561 | [m('cat'), m('puppy')], 562 | [m('python'), lambda o, e: other_wfe.process(self.tokens)], 563 | m('horse'), 564 | ] 565 | ], self.key) 566 | with pytest.raises(HaltProcessing): 567 | wfe.process(self.tokens) 568 | 569 | t = get_first(self.tokens) 570 | assert get_xth(self.tokens, 0) == 'mouse dog cat puppy python mouse dog cat puppy python' 571 | assert get_xth(self.tokens, 1) is None 572 | assert get_xth(self.tokens, 2) is None 573 | 574 | @pytest.mark.parametrize("callbacks,kwargs,result,exception", ( 575 | ( 576 | [ 577 | m('mouse'), 578 | [ 579 | m('dog'), 580 | [m('cat'), m('puppy')], 581 | [m('python'), workflow_error()], 582 | m('horse'), 583 | ] 584 | ], 585 | {}, 586 | 'mouse dog cat puppy python', 587 | WorkflowError, 588 | ), 589 | ( 590 | [ 591 | m('mouse'), 592 | [ 593 | m('dog'), 594 | [m('cat'), m('puppy')], 595 | [m('python'), workflow_error()], 596 | m('horse'), 597 | ] 598 | ], 599 | { 600 | 'stop_on_error': False, 601 | }, 602 | 'mouse dog cat puppy python', 603 | None, 604 | ), 605 | ( 606 | [ 607 | m('mouse'), 608 | [ 609 | m('dog'), 610 | [m('cat'), m('puppy')], 611 | [m('python'), halt_processing()], 612 | m('horse'), 613 | ] 614 | ], 615 | { 616 | 'stop_on_halt': False, 617 | }, 618 | 'mouse dog cat puppy python', 619 | None, 620 | ), 621 | )) 622 | def test_process_smash_through(self, callbacks, kwargs, result, exception): 623 | self.wfe.callbacks.add_many(callbacks, self.key) 624 | 625 | if exception: 626 | with pytest.raises(exception): 627 | self.wfe.process(self.tokens, **kwargs) 628 | else: 629 | self.wfe.process(self.tokens, **kwargs) 630 | for idx, dummy in enumerate(self.tokens): 631 | assert get_xth(self.tokens, idx) == result 632 | -------------------------------------------------------------------------------- /tests/test_engine_db.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # This file is part of Workflow. 4 | # Copyright (C) 2015, 2016 CERN. 5 | # 6 | # Workflow is free software; you can redistribute it and/or modify it 7 | # under the terms of the Revised BSD License; see LICENSE file for 8 | # more details. 9 | 10 | import os 11 | import sys 12 | 13 | from collections import Iterable 14 | 15 | import mock 16 | import pytest 17 | 18 | from workflow.engine import HaltProcessing, TransitionActions 19 | from workflow.engine_db import ( 20 | DbWorkflowEngine, 21 | ObjectStatus, 22 | WorkflowStatus, 23 | DbProcessingFactory, 24 | ) 25 | from workflow.utils import classproperty 26 | 27 | 28 | p = os.path.abspath(os.path.dirname(__file__) + '/../') 29 | if p not in sys.path: 30 | sys.path.append(p) 31 | 32 | 33 | class DummyDbObj(object): 34 | def __init__(self): 35 | self._status = None 36 | 37 | def save(self, status): 38 | pass 39 | 40 | @property 41 | def uuid(self): 42 | pass 43 | 44 | @property 45 | def name(self): 46 | pass 47 | 48 | @property 49 | def status(self): 50 | return self._status 51 | 52 | @property 53 | def objects(self): 54 | pass 55 | 56 | 57 | def m(key=None): 58 | def _m(token, inst): 59 | token.data(key) 60 | _m.__name__ = 'string appender' 61 | return _m 62 | 63 | 64 | class FakeToken(object): 65 | 66 | def __init__(self, data, **attributes): 67 | self.data = data 68 | self._status = None 69 | self._id_workflow = None # TODO: Remove this 70 | self._callback_pos = None 71 | 72 | def save(self, status=None, callback_pos=None, id_workflow=None): 73 | self._status = status 74 | self._callback_pos = callback_pos 75 | self._id_workflow = id_workflow 76 | 77 | @classproperty 78 | def known_statuses(cls): 79 | return ObjectStatus 80 | 81 | 82 | class TestObjectStatus(object): 83 | 84 | @pytest.mark.parametrize("status, name", ( 85 | (ObjectStatus.INITIAL, "New"), 86 | (ObjectStatus.COMPLETED, "Done"), 87 | (ObjectStatus.HALTED, "Need action"), 88 | (ObjectStatus.RUNNING, "In process"), 89 | (ObjectStatus.ERROR, "Error"), 90 | )) 91 | def test_object_status_name_returns_correct_name(self, status, name): 92 | assert status.label == name 93 | 94 | 95 | class TestWorkflowEngineDb(object): 96 | 97 | def setup_method(self, method): 98 | self.dummy_db_obj = mock.Mock(spec=DummyDbObj()) 99 | self.dummy_db_obj.save(WorkflowStatus.NEW) 100 | self.wfe = DbWorkflowEngine(self.dummy_db_obj) 101 | self.data = ['one', 'two', 'three', 'four', 'five'] 102 | self.tokens = [mock.Mock(spec=FakeToken(x)) for x in self.data] 103 | 104 | def teardown_method(self, method): 105 | pass 106 | 107 | @mock.patch.object(TransitionActions, 'HaltProcessing') 108 | def test_halt_processing_calls_parent(self, mock_HaltProcessing): 109 | self.wfe.callbacks.add_many([ 110 | m('mouse'), 111 | lambda obj, eng: eng.halt() 112 | ]) 113 | self.wfe.process(self.tokens) 114 | 115 | assert mock_HaltProcessing.call_count == len(self.data) 116 | for args_list in mock_HaltProcessing.call_args_list: 117 | args_list = args_list[0] 118 | assert isinstance(args_list[0], FakeToken) 119 | assert isinstance(args_list[1], DbWorkflowEngine) 120 | assert isinstance(args_list[2], Iterable) 121 | assert isinstance(args_list[3][1], HaltProcessing) # exc_info 122 | 123 | def test_halt_processing_saves_eng_and_obj(self): 124 | self.wfe.callbacks.add_many([ 125 | lambda obj, eng: eng.halt('please wait') 126 | ]) 127 | with pytest.raises(HaltProcessing): 128 | self.wfe.process(self.tokens) 129 | 130 | token = self.tokens[0] 131 | 132 | assert token.save.call_count == 2 133 | assert token.save.call_args_list[0][1]['status'] == token.known_statuses.RUNNING 134 | assert token.save.call_args_list[1][1]['status'] == token.known_statuses.HALTED 135 | 136 | def test_halt_processing_saves_correct_statuses(self): 137 | self.wfe.callbacks.add_many([ 138 | lambda obj, eng: eng.halt('please wait') 139 | ]) 140 | with pytest.raises(HaltProcessing): 141 | self.wfe.process(self.tokens) 142 | 143 | # Token saved 144 | token = self.tokens[0] 145 | assert token.save.call_count == 2 146 | assert token.save.call_args_list[0][1]['status'] == token.known_statuses.RUNNING 147 | assert token.save.call_args_list[1][1]['status'] == token.known_statuses.HALTED 148 | 149 | # Engine saved 150 | assert self.dummy_db_obj.save.call_count == 3 151 | assert self.dummy_db_obj.save.call_args_list[0][0] == (WorkflowStatus.NEW, ) 152 | assert self.dummy_db_obj.save.call_args_list[1][0] == (WorkflowStatus.RUNNING, ) 153 | assert self.dummy_db_obj.save.call_args_list[2][0] == (WorkflowStatus.HALTED, ) 154 | 155 | # Sorry, no parametrization here because mocks won't do. 156 | def test_before_object_save_object(self): 157 | DbProcessingFactory.before_object(self.wfe, self.tokens, self.tokens[0]) 158 | assert self.tokens[0].save.call_count == 1 159 | assert self.tokens[0].save.call_args_list[0][1]['status'] == self.tokens[0].known_statuses.RUNNING 160 | 161 | # Sorry, no parametrization here because mocks won't do. 162 | def test_after_object_save_object(self): 163 | DbProcessingFactory.after_object(self.wfe, self.tokens, self.tokens[0]) 164 | assert self.tokens[0].save.call_count == 1 165 | assert self.tokens[0].save.call_args_list[0][1]['status'] == self.tokens[0].known_statuses.COMPLETED 166 | 167 | @pytest.mark.parametrize("method, status, has_completed", ( 168 | (DbProcessingFactory.before_processing, WorkflowStatus.RUNNING, False), 169 | (DbProcessingFactory.after_processing, WorkflowStatus.HALTED, False), 170 | (DbProcessingFactory.after_processing, WorkflowStatus.COMPLETED, True), 171 | )) 172 | def test_after_processing_save_status(self, method, status, has_completed): 173 | self.wfe.__class__.has_completed = mock.PropertyMock(return_value=has_completed) 174 | with mock.patch.object(self.wfe, 'save'): 175 | method(self.wfe, self.tokens) 176 | assert self.wfe.save.call_count == 1 177 | assert self.wfe.save.call_args_list[0][0] == (status, ) 178 | 179 | def test_before_processing_save_status(self): 180 | with mock.patch.object(self.wfe, 'save'): 181 | self.wfe.processing_factory.before_processing(self.wfe, []) 182 | assert self.wfe.save.call_count == 1 183 | assert self.wfe.save.call_args_list[0][0] == (WorkflowStatus.RUNNING,) 184 | -------------------------------------------------------------------------------- /tests/test_engine_interface.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # This file is part of Workflow. 4 | # Copyright (C) 2014, 2015 CERN. 5 | # 6 | # Workflow is free software; you can redistribute it and/or modify it 7 | # under the terms of the Revised BSD License; see LICENSE file for 8 | # more details. 9 | 10 | import os 11 | import sys 12 | from copy import deepcopy 13 | 14 | import mock 15 | import pytest 16 | from six import iteritems 17 | 18 | from workflow.engine import MachineState, Callbacks, GenericWorkflowEngine 19 | 20 | 21 | p = os.path.abspath(os.path.dirname(__file__) + '/../') 22 | if p not in sys.path: 23 | sys.path.append(p) 24 | 25 | 26 | def obj_append(key): 27 | def _m(obj, eng): 28 | obj.append(key) 29 | return _m 30 | 31 | 32 | def stop_if_obj_str_eq(value): 33 | def x(obj, eng): 34 | if str(obj) == value: 35 | eng.stopProcessing() 36 | return lambda obj, eng: x(obj, eng) 37 | 38 | 39 | def jump_call(step=0): 40 | return lambda obj, eng: eng.jump_call(step) 41 | 42 | 43 | class TestSignals(object): 44 | 45 | @pytest.mark.parametrize("signal_name", ( 46 | 'workflow_started', 47 | 'workflow_halted', 48 | 'workflow_finished', 49 | )) 50 | @pytest.mark.skipif(sys.version_info > (3, ), 51 | reason="create_autospec broken in py3") 52 | def test_signals_are_emitted(self, signal_name): 53 | from workflow.engine import Signal 54 | from workflow import signals 55 | 56 | # Create engine 57 | eng = mock.create_autospec(GenericWorkflowEngine) 58 | 59 | # Call signal 60 | with mock.patch.object(signals, signal_name, autospec=True): 61 | getattr(Signal, signal_name)(eng) 62 | getattr(signals, signal_name).send.assert_called_once_with() 63 | 64 | def test_log_warning_if_signals_lib_is_missing(self): 65 | from workflow.engine import Signal 66 | 67 | orig_import = __import__ 68 | 69 | def import_mock(name, *args): 70 | if name == 'workflow.signals': 71 | raise ImportError 72 | return orig_import(name, *args) 73 | 74 | # Patch the engine so that we can inspect calls 75 | with mock.patch('workflow.engine.GenericWorkflowEngine') as patched_GWE: 76 | eng = patched_GWE.return_value 77 | # Patch __import__ so that importing workflow.signals raises ImportError 78 | if sys.version_info < (3, ): 79 | builtins_module = '__builtin__' 80 | else: 81 | builtins_module = 'builtins' 82 | with mock.patch(builtins_module + '.__import__', side_effect=import_mock): 83 | Signal.workflow_started(eng) 84 | eng.log.warning.assert_called_once_with("Could not import signals lib; " 85 | "ignoring all future signal calls.") 86 | 87 | 88 | def TestMachineState(object): 89 | 90 | def test_machine_state_does_not_allow_token_pos_below_minus_one(self): 91 | ms = MachineState() 92 | ms.token_pos = 1 93 | ms.token_pos = 0 94 | ms.token_pos = -1 95 | with pytest.raises(AttributeError): 96 | ms.token_pos = -2 97 | 98 | @pytest.mark.parametrize("params, token_pos, callback_pos", ( 99 | (tuple(), -1, [0]), 100 | ((5, [1, 2]), 5, [1, 2]), 101 | )) 102 | def test_machine_state_reads_defaults(self, params, token_pos, callback_pos): 103 | "Test initialization of machine state with and without args.""" 104 | ms = MachineState(*params) 105 | assert ms.token_pos == token_pos 106 | assert ms.callback_pos == callback_pos 107 | 108 | lmb = [ 109 | lambda a: a, 110 | lambda b: b + 1, 111 | lambda c: c + 2, 112 | lambda d: d + 3 113 | ] 114 | 115 | 116 | class TestCallbacks(object): 117 | 118 | @pytest.mark.parametrize("key,ret,exception", ( 119 | ('*', [], KeyError), 120 | (None, {}, None), 121 | )) 122 | def test_callbacks_return_correct_when_empty(self, key, ret, exception): 123 | cbs = Callbacks() 124 | if exception: 125 | with pytest.raises(exception) as exc_info: 126 | cbs.get(key) 127 | assert 'No workflow is registered for the key: ' + key in exc_info.value.args[0] 128 | else: 129 | assert cbs.get(key) == ret 130 | 131 | @pytest.fixture() 132 | def cbs(self): 133 | return Callbacks() 134 | 135 | @pytest.mark.parametrize("in_dict,ret", ( 136 | ( 137 | {'a': [lmb[0], lmb[1]], 'b': [lmb[2], lmb[3]]}, 138 | {'a': [lmb[0], lmb[1]], 'b': [lmb[2], lmb[3]]}, 139 | ), 140 | ( 141 | {'a': [lmb[0], (lmb[1], lmb[2])]}, 142 | {'a': [lmb[0], lmb[1], lmb[2]]}, 143 | ), 144 | ( 145 | {'a': [lmb[0], ((lmb[1],), lmb[2])]}, 146 | {'a': [lmb[0], lmb[1], lmb[2]]}, 147 | ), 148 | )) 149 | def test_callbacks_get_return_correct_after_add_many(self, cbs, in_dict, ret): 150 | # Run `add_many` 151 | for key, val in iteritems(in_dict): 152 | cbs.add_many(val, key) 153 | # Existing keys 154 | for key, val in iteritems(ret): 155 | assert cbs.get(key) == val 156 | 157 | def test_callbacks_replace_from_used(self, cbs): 158 | cbs.add_many(lmb, '*') 159 | lmb_rev = lmb[::-1] 160 | cbs.replace(lmb_rev, '*') 161 | 162 | assert cbs.get('*') == lmb_rev 163 | 164 | def test_callbacks_clear_maintains_exception(self, cbs): 165 | cbs.add_many(lmb, 'some-key') 166 | cbs.clear() 167 | cbs.add_many(lmb, 'some-key') 168 | with pytest.raises(KeyError) as exc_info: 169 | cbs.get('missing') 170 | assert 'No workflow is registered for the key: ' + 'missing' in exc_info.value.args[0] 171 | 172 | 173 | class TestGenericWorkflowEngine(object): 174 | 175 | """Tests of the WE interface""" 176 | 177 | def setup_method(self, method): 178 | # Don't turn this into some generator. One needs to be able to see what 179 | # the input is. 180 | self.d0 = [['one'], ['two'], ['three'], ['four'], ['five']] 181 | self.d1 = [['one'], ['two'], ['three'], ['four'], ['five']] 182 | self.d2 = [['one'], ['two'], ['three'], ['four'], ['five']] 183 | 184 | def teardown_method(self, method): 185 | pass 186 | 187 | def test_init(self): 188 | 189 | # init with empty to full parameters 190 | we1 = GenericWorkflowEngine() 191 | 192 | callbacks = [ 193 | obj_append('mouse'), 194 | [obj_append('dog'), jump_call(1), obj_append('cat'), obj_append('puppy')], 195 | obj_append('horse'), 196 | ] 197 | 198 | we1.addManyCallbacks('*', deepcopy(callbacks)) 199 | 200 | we1.process(self.d1) 201 | 202 | def test_configure(self): 203 | 204 | callbacks_list = [ 205 | obj_append('mouse'), 206 | [obj_append('dog'), jump_call(1), obj_append('cat'), obj_append('puppy')], 207 | obj_append('horse'), 208 | ] 209 | 210 | we = GenericWorkflowEngine() 211 | we.addManyCallbacks('*', callbacks_list) 212 | 213 | # process using defaults 214 | we.process(self.d1) 215 | r = 'one mouse dog cat puppy horse'.split() 216 | 217 | we = GenericWorkflowEngine() 218 | we.addManyCallbacks('*', callbacks_list) 219 | we.process(self.d2) 220 | 221 | assert self.d1[0] == r 222 | assert self.d2[0] == r 223 | assert self.d1 == self.d2 224 | 225 | # ------------ tests configuring the we -------------------- 226 | def test_workflow01(self): 227 | 228 | class GenericWEWithXChooser(GenericWorkflowEngine): 229 | def callback_chooser(self, obj): 230 | return self.callbacks.get('x') 231 | 232 | we0 = GenericWorkflowEngine() 233 | we1 = GenericWorkflowEngine() 234 | we2 = GenericWEWithXChooser() 235 | 236 | we0.addManyCallbacks('*', [ 237 | obj_append('mouse'), 238 | [obj_append('dog'), jump_call(1), obj_append('cat'), obj_append('puppy')], 239 | obj_append('horse'), 240 | ]) 241 | we1.setWorkflow([ 242 | obj_append('mouse'), 243 | [obj_append('dog'), jump_call(1), obj_append('cat'), obj_append('puppy')], 244 | obj_append('horse'), 245 | ]) 246 | we2.addManyCallbacks('x', [ 247 | obj_append('mouse'), 248 | [obj_append('dog'), jump_call(1), obj_append('cat'), obj_append('puppy')], 249 | obj_append('horse'), 250 | ]) 251 | 252 | we0.process(self.d0) 253 | we1.process(self.d1) 254 | we2.process(self.d2) 255 | 256 | assert self.d0 == self.d1 257 | assert self.d0 == self.d2 258 | -------------------------------------------------------------------------------- /tests/test_patterns.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # This file is part of Workflow. 4 | # Copyright (C) 2014, 2015 CERN. 5 | # 6 | # Workflow is free software; you can redistribute it and/or modify it 7 | # under the terms of the Revised BSD License; see LICENSE file for 8 | # more details. 9 | 10 | import sys 11 | import os 12 | import time 13 | import random 14 | 15 | p = os.path.abspath(os.path.dirname(__file__) + '/../') 16 | if p not in sys.path: 17 | sys.path.append(p) 18 | 19 | from workflow.engine import GenericWorkflowEngine 20 | import workflow.patterns.controlflow as cf 21 | import workflow.patterns.utils as ut 22 | 23 | 24 | def i(key): 25 | def _i(obj, eng): 26 | obj.insert(0, key) 27 | return _i 28 | 29 | 30 | def a(key): 31 | def _a(obj, eng): 32 | obj.append(key) 33 | return _a 34 | 35 | 36 | def e(key, val): 37 | def _e(obj, eng): 38 | eng.store[key] = val 39 | return _e 40 | 41 | 42 | def printer(val): 43 | def _printer(obj, eng): 44 | lock = eng.store['lock'] 45 | w = float(random.randint(1, 5)) / 100 46 | for _ in range(5): 47 | try: 48 | lock.acquire() 49 | obj.append(val) 50 | finally: 51 | lock.release() 52 | time.sleep(w) 53 | return _printer 54 | 55 | 56 | class TestGenericWorkflowEngine(object): 57 | 58 | """Tests of the WE interface""" 59 | 60 | def setup_method(self, method): 61 | self.key = '*' 62 | 63 | def teardown_method(self, method): 64 | pass 65 | 66 | def getDoc(self, val=None): 67 | if val: 68 | return [[x] for x in val.split()] 69 | return [[x] for x in u"one two three four five".split()] 70 | 71 | def addTestCallbacks(self, no, eng): 72 | if type == 1: 73 | eng.addManyCallbacks() 74 | 75 | # --------- initialization --------------- 76 | 77 | def test_IF_ELSE01(self): 78 | we = GenericWorkflowEngine() 79 | doc = self.getDoc() 80 | 81 | we.setWorkflow([i('add'), 82 | cf.IF_ELSE(lambda o, e: o[1] == 'three', 83 | [a('3'), a('33')], 84 | [a('other'), [a('nested'), a('branch')]]), 85 | a('end')]) 86 | we.process(doc) 87 | 88 | r = [' '.join(doc[x]) for x in range(len(doc))] 89 | 90 | assert r[0] == 'add one other nested branch end' 91 | assert r[1] == 'add two other nested branch end' 92 | assert r[2] == 'add three 3 33 end' 93 | assert r[3] == 'add four other nested branch end' 94 | assert r[4] == 'add five other nested branch end' 95 | 96 | def test_IF_ELSE02(self): 97 | we = GenericWorkflowEngine() 98 | doc = self.getDoc() 99 | 100 | we.setWorkflow([i('add'), 101 | cf.IF_ELSE(lambda o, e: o[1] == 'three', 102 | a('3'), 103 | a('other')) 104 | ]) 105 | we.process(doc) 106 | 107 | r = [' '.join(doc[x]) for x in range(len(doc))] 108 | 109 | assert r[0] == 'add one other' 110 | assert r[1] == 'add two other' 111 | assert r[2] == 'add three 3' 112 | assert r[3] == 'add four other' 113 | assert r[4] == 'add five other' 114 | 115 | def test_IF_ELSE03(self): 116 | we = GenericWorkflowEngine() 117 | doc = self.getDoc() 118 | 119 | doc[3].append('4') 120 | 121 | def test(v): 122 | return lambda o, e: v in o 123 | 124 | we.setWorkflow([i('add'), 125 | cf.IF_ELSE( 126 | test('three'), 127 | [a('xxx'), cf.IF_ELSE(test('xxx'), 128 | [a('6'), cf.IF_ELSE( 129 | test('6'), 130 | a('six'), 131 | (a('only-3s'), a('error')))], 132 | a('ok'))], 133 | [cf.IF_ELSE( 134 | test('4'), 135 | cf.IF_ELSE(test('four'), 136 | [a('44'), [[[a('forty')]]]], 137 | a('error')), 138 | a('not-four'))]), 139 | a('end'), 140 | cf.IF_ELSE(test('error'), 141 | a('gosh!'), 142 | a('OK')) 143 | ]) 144 | we.process(doc) 145 | 146 | r = [' '.join(doc[x]) for x in range(len(doc))] 147 | 148 | assert r[0] == 'add one not-four end OK' 149 | assert r[1] == 'add two not-four end OK' 150 | assert r[2] == 'add three xxx 6 six end OK' 151 | assert r[3] == 'add four 4 44 forty end OK' 152 | assert r[4] == 'add five not-four end OK' 153 | 154 | # -------------- parallel split ------------------ 155 | 156 | def test_PARALLEL_SPLIT01(self): 157 | we = GenericWorkflowEngine() 158 | doc = self.getDoc() 159 | 160 | we.setWorkflow([i('start'), 161 | cf.PARALLEL_SPLIT( 162 | printer('p1'), 163 | printer('p2'), 164 | printer('p3'), 165 | printer('p4'), 166 | printer('p5')), 167 | lambda o, e: time.sleep(.1), 168 | a('end') 169 | ]) 170 | we.process(doc) 171 | r = [' '.join(doc[x]) for x in range(len(doc))] 172 | 173 | assert doc[0][0] == 'start' 174 | assert doc[0][1] == 'one' 175 | assert doc[1][0] == 'start' 176 | assert doc[1][1] == 'two' 177 | 178 | # end must have been inserted while printers were running 179 | # mixed together with them 180 | all_pos = set() 181 | for x in range(len(doc)): 182 | pos = doc[x].index('end') 183 | assert pos > 2 184 | assert pos < len(doc[x]) 185 | all_pos.add(pos) 186 | 187 | # --------------- nested parallel splits -------------------- 188 | def test_PARALLEL_SPLIT02(self): 189 | """TODO: this test is failing, but that is because sometimes it does 190 | not take into accounts threads being executed in random mannger""" 191 | 192 | we = GenericWorkflowEngine() 193 | doc = self.getDoc()[0:1] 194 | 195 | we.setWorkflow([i('start'), 196 | cf.PARALLEL_SPLIT( 197 | [ 198 | cf.PARALLEL_SPLIT( 199 | printer('p0'), 200 | printer('p0a'), 201 | cf.PARALLEL_SPLIT( 202 | printer('p0b'), 203 | printer('p0c') 204 | ), 205 | ), 206 | printer('xx') 207 | ], 208 | [ 209 | a('AAA'), 210 | printer('p2b') 211 | ], 212 | printer('p3'), 213 | [ 214 | a('p4a'), 215 | printer('p4b'), 216 | printer('p4c') 217 | ], 218 | [printer('p5'), cf.PARALLEL_SPLIT( 219 | printer('p6'), 220 | printer('p7'), 221 | [printer('p8a'), printer('p8b')], 222 | )]), 223 | a('end') 224 | ]) 225 | we.process(doc) 226 | 227 | # give threads time to finish 228 | time.sleep(2) 229 | 230 | assert doc[0][0] == 'start' 231 | assert doc[0][1] == 'one' 232 | 233 | # at least the fist object should have them all 234 | # print doc[0] 235 | for x in ['p0', 'p0a', 'p0b', 'p0c', 'xx', 'AAA', 'p2b', 'p3', 'p4a', 236 | 'p4b', 'p4c', 'p5', 'p6', 'p8a', 'p8b']: 237 | doc[0].index(x) # will fail if not present 238 | 239 | # ------------ parallel split that does nasty things -------------- 240 | def test_PARALLEL_SPLIT03(self): 241 | we = GenericWorkflowEngine() 242 | doc = self.getDoc() 243 | 244 | we.setWorkflow([i('start'), 245 | cf.PARALLEL_SPLIT( 246 | [cf.IF(lambda obj, eng: 'jump-verified' in obj, a('error')), 247 | cf.PARALLEL_SPLIT( 248 | [cf.IF(lambda obj, eng: 'nasty-jump' in obj, 249 | [a('jump-ok'), 250 | lambda obj, eng: ('nasty-jump' in obj and 251 | obj.append('jump-verified'))]), 252 | cf.PARALLEL_SPLIT( 253 | a('ok-1'), 254 | a('ok-2'), 255 | cf.IF(lambda obj, eng: 'ok-3' not in obj, 256 | lambda obj, eng: (obj.append('ok-3') and 257 | eng.break_current_loop())), 258 | a('ok-4')), 259 | 260 | a('xx'), 261 | lambda obj, eng: ('jump-verified' in obj and 262 | eng.break_current_loop()), 263 | a('nasty-jump'), 264 | cf.TASK_JUMP_IF( 265 | lambda obj, eng: 'jump-verified' not in obj, -100)]), 266 | ], 267 | [a('AAA'), a('p2b')]), 268 | a('end') 269 | ]) 270 | we.process(doc) 271 | # give threads time to finish 272 | time.sleep(.5) 273 | 274 | d = doc[0] 275 | 276 | # at least the fist object should have them all 277 | # print doc[0] 278 | for x in ['nasty-jump', 'jump-verified', 'ok-3']: 279 | d.index(x) # will fail if not present 280 | 281 | assert d.count('ok-1') > 1 282 | 283 | # --------------- choice pattern -------------------- 284 | 285 | def test_CHOICE01(self): 286 | we = GenericWorkflowEngine() 287 | doc = self.getDoc()[0:1] 288 | 289 | def arbiter(obj, eng): 290 | return obj[-1] 291 | 292 | we.setWorkflow([i('start'), 293 | cf.CHOICE(arbiter, 294 | end=(lambda obj, eng: obj.append('error')), 295 | bam=(lambda obj, eng: obj.append('bom')), 296 | bim=(lambda obj, eng: obj.append('bam')), 297 | bom=(lambda obj, eng: obj.append('bum')), 298 | one=(lambda obj, eng: obj.append('bim')), 299 | bum=cf.STOP(), 300 | ), 301 | cf.TASK_JUMP_BWD(-1)]) 302 | we.process(doc) 303 | 304 | d = ' '.join(doc[0]) 305 | 306 | assert 'bim bam bom bum' in d 307 | assert 'error' not in d 308 | assert len(doc[0]) == 6 309 | 310 | def test_CHOICE02(self): 311 | we = GenericWorkflowEngine() 312 | doc = self.getDoc()[0:1] 313 | 314 | def arbiter(obj, eng): 315 | return obj[-1] 316 | 317 | we.setWorkflow([i('start'), 318 | cf.CHOICE(arbiter, 319 | ('bom', lambda obj, eng: obj.append('bum')), 320 | ('one', lambda obj, eng: obj.append('bim')), 321 | ('bum', cf.STOP()), 322 | ('end', lambda obj, 323 | eng: obj.append('error')), 324 | ('bam', lambda obj, eng: obj.append('bom')), 325 | ('bim', lambda obj, eng: obj.append('bam'))), 326 | cf.TASK_JUMP_BWD(-1)]) 327 | we.process(doc) 328 | 329 | d = ' '.join(doc[0]) 330 | 331 | assert 'bim bam bom bum' in d 332 | assert 'error' not in d 333 | assert len(doc[0]) == 6 334 | 335 | def test_CHOICE03(self): 336 | we = GenericWorkflowEngine() 337 | doc = self.getDoc()[0:1] 338 | 339 | def arbiter(obj, eng): 340 | return obj[-1] 341 | 342 | we.setWorkflow([i('start'), 343 | cf.CHOICE(arbiter, 344 | ('bam', lambda obj, eng: obj.append('bom')), 345 | ('end', lambda obj, 346 | eng: obj.append('error')), 347 | ('bim', lambda obj, eng: obj.append('bam')), 348 | bom=(lambda obj, eng: obj.append('bum')), 349 | one=(lambda obj, eng: obj.append('bim')), 350 | bum=cf.STOP(), 351 | ), 352 | cf.TASK_JUMP_BWD(-1)]) 353 | we.process(doc) 354 | 355 | d = ' '.join(doc[0]) 356 | 357 | assert 'bim bam bom bum' in d 358 | assert 'error' not in d 359 | assert len(doc[0]) == 6 360 | 361 | # ------------------- testing simple merge ----------------------- 362 | def test_SIMPLE_MERGE03(self): 363 | we = GenericWorkflowEngine() 364 | doc = self.getDoc()[0:1] 365 | 366 | we.setWorkflow([i('start'), 367 | cf.SIMPLE_MERGE( 368 | lambda obj, eng: obj.append('bom'), 369 | lambda obj, eng: obj.append('error'), 370 | lambda obj, eng: obj.append('bam'), 371 | lambda obj, eng: obj.append('bum'), 372 | lambda obj, eng: obj.append('end'), 373 | ), 374 | ]) 375 | we.process(doc) 376 | 377 | d = ' '.join(doc[0]) 378 | 379 | assert 'start' in d 380 | assert 'bom' in d 381 | assert 'error' not in d 382 | assert 'end' in d 383 | 384 | # ------------------- testing RUN_WF ----------------------------- 385 | def test_RUN_WF01(self): 386 | """Test wfe is reinit=False, eng must remember previous invocations""" 387 | we = GenericWorkflowEngine() 388 | doc = self.getDoc()[0:1] 389 | 390 | we.setWorkflow( 391 | [ 392 | i('start'), 393 | ut.RUN_WF( 394 | [ 395 | lambda obj, eng: obj.append('bom'), 396 | lambda obj, eng: obj.append('bam'), 397 | lambda obj, eng: obj.append('bum'), 398 | lambda obj, eng: obj.append('end'), 399 | lambda obj, eng: obj.append( 400 | eng.store.setdefault('eng-end', '')), 401 | e('eng-end', 'eng-end') 402 | ], 403 | data_connector=lambda obj, eng: [obj], 404 | outkey='#wfe', 405 | ), 406 | ] 407 | ) 408 | we.process(doc) 409 | 410 | d = ' '.join(doc[0]) 411 | 412 | assert 'start' in d 413 | assert 'bom' in d 414 | assert 'bam' in d 415 | assert 'bum' in d 416 | assert 'end' in d 417 | assert 'eng-end' not in d 418 | 419 | # run the same thing again 420 | we.process(doc) 421 | 422 | d = ' '.join(doc[0]) 423 | assert 'start' in d 424 | assert d.count('bom') == 2 425 | assert d.count('bam') == 2 426 | assert d.count('bum') == 2 427 | assert 'end' in d 428 | assert 'eng-end' in d # now it must be present 429 | 430 | def test_RUN_WF02(self): 431 | """Test wfe is reinit=True - eng must not remember""" 432 | we = GenericWorkflowEngine() 433 | doc = self.getDoc()[0:1] 434 | 435 | we.callbacks.replace( 436 | [ 437 | i('start'), 438 | ut.RUN_WF( 439 | [ 440 | lambda obj, eng: obj.append('bom'), 441 | lambda obj, eng: obj.append('bam'), 442 | lambda obj, eng: obj.append('bum'), 443 | lambda obj, eng: obj.append('end'), 444 | lambda obj, eng: obj.append( 445 | eng.store.setdefault('eng-end', '')), 446 | e('eng-end', 'eng-end') 447 | ], 448 | data_connector=lambda obj, eng: [obj], 449 | outkey='#wfe', 450 | reinit=True 451 | ), 452 | ] 453 | ) 454 | we.process(doc) 455 | 456 | d = ' '.join(doc[0]) 457 | 458 | assert 'start' in d 459 | assert 'bom' in d 460 | assert 'bam' in d 461 | assert 'bum' in d 462 | assert 'end' in d 463 | assert 'eng-end' not in d 464 | 465 | # run the same thing again 466 | we.process(doc) 467 | 468 | d = ' '.join(doc[0]) 469 | assert 'start' in d 470 | assert d.count('bom') == 2 471 | assert d.count('bam') == 2 472 | assert d.count('bum') == 2 473 | assert 'end' in d 474 | assert 'eng-end' not in d # it must not be present if reinit=True 475 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py26, py27, py33, py34 3 | 4 | [testenv] 5 | deps = pytest 6 | pytest-cov 7 | pytest-pep8 8 | commands = {envpython} setup.py test 9 | -------------------------------------------------------------------------------- /workflow/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # This file is part of Workflow. 4 | # Copyright (C) 2011, 2014 CERN. 5 | # 6 | # Workflow is free software; you can redistribute it and/or modify it 7 | # under the terms of the Revised BSD License; see LICENSE file for 8 | # more details. 9 | 10 | """Workflow engine is a Finite State Machine with memory.""" 11 | 12 | from .version import __version__ 13 | -------------------------------------------------------------------------------- /workflow/config.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # This file is part of Workflow. 4 | # Copyright (C) 2011, 2014 CERN. 5 | # 6 | # Workflow is free software; you can redistribute it and/or modify it 7 | # under the terms of the Revised BSD License; see LICENSE file for 8 | # more details. 9 | 10 | """ 11 | Provide a class for reading global configuration options. 12 | 13 | The reader in itself is using configobj to access the ini files. The reader 14 | should be initialized (from the project root) with the path of the folder where 15 | configuration files live. 16 | """ 17 | 18 | import inspect 19 | import os 20 | import sys 21 | import traceback 22 | 23 | from configobj import Section, OPTION_DEFAULTS, ConfigObjError, ConfigObj 24 | 25 | 26 | class CustomConfigObj(ConfigObj): 27 | 28 | """Add support for key lookup in parent configuration. 29 | 30 | This is very small change into the default ``ConfigObj`` class 31 | - the only difference is in the ``parent_config`` parameter, if passed 32 | we will add it to the new instance and interpolation will use the 33 | values of the ``parent_config`` for lookup. 34 | """ 35 | 36 | def __init__(self, infile=None, options=None, configspec=None, 37 | encoding=None, interpolation=True, raise_errors=False, 38 | list_values=True, create_empty=False, file_error=False, 39 | stringify=True, indent_type=None, default_encoding=None, 40 | unrepr=False, write_empty_values=False, _inspec=False, 41 | parent_config=None): 42 | """Parse a config file or create a config file object.""" 43 | self._inspec = _inspec 44 | # init the superclass 45 | # this is the only change - we pass the parent configobj if 46 | # available, to have lookup use its values 47 | Section.__init__(self, parent_config or self, 0, self) 48 | 49 | infile = infile or [] 50 | if options is not None: 51 | import warnings 52 | warnings.warn('Passing in an options dictionary to ConfigObj() ', 53 | 'is deprecated. Use **options instead.', 54 | DeprecationWarning, stacklevel=2) 55 | 56 | _options = {'configspec': configspec, 57 | 'encoding': encoding, 'interpolation': interpolation, 58 | 'raise_errors': raise_errors, 'list_values': list_values, 59 | 'create_empty': create_empty, 'file_error': file_error, 60 | 'stringify': stringify, 'indent_type': indent_type, 61 | 'default_encoding': default_encoding, 'unrepr': unrepr, 62 | 'write_empty_values': write_empty_values} 63 | 64 | options = dict(options or {}) 65 | options.update(_options) 66 | 67 | # XXXX this ignores an explicit list_values = True in combination 68 | # with _inspec. The user should *never* do that anyway, but still... 69 | if _inspec: 70 | options['list_values'] = False 71 | 72 | defaults = OPTION_DEFAULTS.copy() 73 | # TODO: check the values too. 74 | for entry in options: 75 | if entry not in defaults: 76 | raise TypeError('Unrecognised option "%s".' % entry) 77 | 78 | # Add any explicit options to the defaults 79 | defaults.update(options) 80 | self._initialise(defaults) 81 | configspec = defaults['configspec'] 82 | self._original_configspec = configspec 83 | self._load(infile, configspec) 84 | 85 | 86 | class ConfigReader(object): 87 | 88 | """Facilitate easy reading/access to the INI style configuration. 89 | 90 | Modules/workflows should not import this, but config instance 91 | 92 | .. code-block:: python 93 | 94 | from workflow import config 95 | 96 | During instantion, reader loads the global config file - usually from 97 | ``./cfg/global.ini`` The values will be accessible as attributes, eg: 98 | ``reader.BASEDIR reader.sectionX.VAL`` 99 | 100 | When workflow/module is accessing an attribute, the reader will also load 101 | special configuration (workflow-specific configuration) which has the same 102 | name as a workflow/module. 103 | 104 | Example: workflow 'load_seman_components.py' 105 | 106 | .. code-block:: python 107 | 108 | from merkur.config import reader 109 | reader.LOCAL_VALUE 110 | 111 | # at this moment, reader will check if exists 112 | # %basedir/etc/load_seman_components.ini if yes, the reader will load 113 | # the configuration and store it inside ._local if no LOCAL_VALUE 114 | # exists, error will be raised if in the meantime, some other module 115 | # imported reader and tries to access an attribute, the reader will 116 | # recognize the caller is different, will update the local config and 117 | # will server workflow-specifi configuration automatically 118 | 119 | You can pass a list of basedir folders - in that case, only the last one 120 | will be used for lookup of local configurations, but the global values will 121 | be inherited from all global.ini files found in the basedir folders. 122 | """ 123 | 124 | def __init__(self, basedir=os.path.abspath(os.path.dirname(__file__)), 125 | caching=True): 126 | """Initialize configuration reader.""" 127 | object.__init__(self) 128 | self._local = {} 129 | self._global = {} 130 | self._on_demand = {} 131 | self._recent_caller = '' 132 | self._caching = caching 133 | self._main_config = None 134 | self._basedir = [] 135 | 136 | self.setBasedir(basedir) 137 | 138 | # load configurations 139 | if isinstance(basedir, list) or isinstance(basedir, tuple): 140 | files = [] 141 | for d in self._basedir: 142 | if os.path.exists(d): 143 | files.append( 144 | os.path.abspath(os.path.join(d, 'global.ini'))) 145 | self.update(files) 146 | else: 147 | self.update() 148 | 149 | def __getattr__(self, key): 150 | """Return configuration value. 151 | 152 | 1. First lookup in the local values; 153 | 2. then in the global values. 154 | """ 155 | # first find out who is trying to access us 156 | frame = inspect.currentframe().f_back 157 | 158 | # TODO - make it try the hierarchy first? 159 | if frame: 160 | cfile = self._getCallerPath(frame) 161 | if cfile: 162 | caller = self._getCallerName(cfile) 163 | if caller != self._recent_caller: 164 | # TODO: make it optional, allow for read-once-updates 165 | self.update_local(caller) # update config 166 | 167 | if key in self._local: 168 | return self._local[key] 169 | elif key in self._global: 170 | return self._global[key] # raise error ok 171 | else: 172 | global_cfg_path = self._main_config and os.path.abspath( 173 | self._main_config.filename) or 'None' 174 | local_cfg_path = self._findConfigPath(self._recent_caller) 175 | raise AttributeError( 176 | 'Attribute "%s" not defined\n' 177 | 'global_config: %s\nlocal_config: %s' % ( 178 | key, global_cfg_path, local_cfg_path)) 179 | 180 | def _getCallerId(self, frame): 181 | if frame: 182 | cfile = self._getCallerPath(frame) 183 | if cfile: 184 | caller = self._getCallerName(cfile) 185 | return caller 186 | 187 | def getBaseDir(self): 188 | """Get basedir path.""" 189 | return self._basedir 190 | 191 | def setBasedir(self, basedir): 192 | """Set a new basedir path. 193 | 194 | This is a root of the configuration directives from which other paths 195 | are resolved. 196 | """ 197 | if not (isinstance(basedir, list) or isinstance(basedir, tuple)): 198 | basedir = [basedir] 199 | new_base = [] 200 | for b in basedir: 201 | b = os.path.abspath(b) 202 | if b[0] != '\\': 203 | b = b.replace('\\', '/') 204 | b = b[0].lower() + b[1:] 205 | if b not in new_base: 206 | new_base.append(b) 207 | self._basedir = new_base 208 | self.update() 209 | 210 | def update(self, files=None, replace_keys={}): 211 | """Update values reading them from the main configuration file(s). 212 | 213 | :param files: list of configuration files 214 | (if empty, default file is read) 215 | :param replace_keys: dictionary of values that you want to replace 216 | this allows you to change config at runtime, but IT IS NOT 217 | RECOMMENDED to change anything else than global values (and you can 218 | change only top level values). If you don't know what you are 219 | doing, do not replace any keys! 220 | """ 221 | if files is None: 222 | files = self._makeAllConfigPaths('global') 223 | 224 | updated = 0 225 | for file in files: 226 | if os.path.exists(file): 227 | # if we have more files, we will wrap/inherit them into one 228 | # object this object should not be probably usef for writing 229 | config = self._main_config = CustomConfigObj( 230 | file, encoding='UTF8', parent_config=self._main_config 231 | ) 232 | if replace_keys: 233 | for k, v in replace_keys.items(): 234 | if k in config: 235 | config[k] = v 236 | self._update(self._global, config) 237 | updated += 1 238 | return updated 239 | 240 | def init(self, filename): 241 | """Initialize configuration file.""" 242 | if not os.path.exists(filename): 243 | filename = self._findConfigPath(filename) 244 | caller = self._getCallerId(inspect.currentframe().f_back) 245 | if not(self.update_local(caller, filename)): 246 | raise Exception('Config file: %s does not exist' % filename) 247 | 248 | def update_local(self, name, file=None): 249 | """Update the local, workflow-specific cache. 250 | 251 | :param name: name of the calling module (without suffix) 252 | :param file: file to load config from (if empty, default ini file 253 | will be sought) 254 | """ 255 | self._recent_caller = name 256 | self._local = {} 257 | if file is None: 258 | file = self._findConfigPath(name) 259 | 260 | if file and os.path.exists(file): 261 | config = CustomConfigObj(file, 262 | encoding='UTF8', 263 | parent_config=self._main_config) 264 | 265 | self._update(self._local, config) 266 | return True 267 | 268 | def load(self, cfgfile, force_reload=False, failonerror=True, 269 | replace_keys={}): 270 | """Load configuration file on demand. 271 | 272 | :param cfgfile: path to the file, the path may be relative, in 273 | that case we will try to guess it using set basedir. Or it 274 | can be absolute 275 | :param force_reload: returns cached configuration or reloads 276 | it again from file if force_reload=True 277 | :param failonerror: bool, raise Exception when config file 278 | is not found/loaded 279 | :return: config object or None 280 | 281 | example: 282 | c = config.load('some-file.txt') 283 | c.some.key 284 | """ 285 | realpath = None 286 | if os.path.exists(cfgfile): 287 | realpath = cfgfile 288 | else: 289 | new_p = self._findConfigPath(cfgfile) 290 | if new_p: 291 | realpath = new_p 292 | else: 293 | new_p = self._findConfigPath(cfgfile.rsplit('.', 1)[0]) 294 | if new_p: 295 | realpath = new_p 296 | 297 | if not realpath: 298 | if failonerror: 299 | raise Exception('Cannot find: %s' % cfgfile) 300 | else: 301 | sys.stderr.write('Cannot find: %s' % cfgfile) 302 | return 303 | 304 | if realpath in self._on_demand and not force_reload: 305 | return ConfigWrapper(realpath, self._on_demand[realpath]) 306 | 307 | try: 308 | config = CustomConfigObj(realpath, 309 | encoding='UTF8', 310 | parent_config=self._main_config) 311 | if replace_keys: 312 | for k, v in replace_keys.items(): 313 | if k in config: 314 | config[k] = v 315 | except ConfigObjError as msg: 316 | if failonerror: 317 | raise ConfigObjError(msg) 318 | else: 319 | self.traceback.print_exc() 320 | return 321 | 322 | self._on_demand[realpath] = {} 323 | self._update(self._on_demand[realpath], config) 324 | return ConfigWrapper(realpath, self._on_demand[realpath]) 325 | 326 | def get(self, key, failonerror=True): 327 | """Get value from the key identified by string, eg. `index.dir`.""" 328 | parts = key.split('.') 329 | pointer = self 330 | try: 331 | for p in parts: 332 | pointer = getattr(pointer, p) 333 | return pointer 334 | except (KeyError, AttributeError): 335 | global_cfg_path = self._main_config and os.path.abspath( 336 | self._main_config.filename) or 'None' 337 | local_cfg_path = self._findConfigPath(self._recent_caller) 338 | m = ('Attribute "%s" not defined\nglobal_config: %s\n' 339 | 'local_config: %s' % (key, global_cfg_path, local_cfg_path)) 340 | if failonerror: 341 | raise AttributeError(m) 342 | else: 343 | sys.stderr.write(m) 344 | 345 | def _getCallerPath(self, frame): 346 | cfile = os.path.abspath(inspect.getfile(frame)).replace('\\', '/') 347 | f = __file__.replace('\\', '/') 348 | if f != cfile: 349 | return cfile 350 | 351 | def _getCallerName(self, path): 352 | cfile = os.path.split(path)[1] 353 | return cfile.rsplit('.', 1)[0] 354 | 355 | def getCallersConfig(self, failonerror=True): 356 | """Get the value from the calling workflow configuration. 357 | 358 | This is useful if we want to access configuration of the object 359 | that included us. 360 | 361 | :param key: name of the key to access, it is a string in a dot notation 362 | """ 363 | # first find out who is trying to access us 364 | caller = '' 365 | frame = inspect.currentframe().f_back 366 | frame = inspect.currentframe().f_back 367 | if frame: 368 | frame = frame.f_back 369 | if frame: 370 | caller = self._getCallerName(self._getCallerPath(frame)) 371 | path = self._findConfigPath(caller) 372 | if path: 373 | config = self.load(path) 374 | if config: 375 | return config 376 | if failonerror: 377 | raise Exception('Error, cannot find the caller') 378 | 379 | def _findConfigPath(self, name): 380 | """Find the most specific config path.""" 381 | for path in reversed(self._makeAllConfigPaths(name)): 382 | if os.path.exists(path): 383 | return path 384 | 385 | def _makeAllConfigPaths(self, name): 386 | f = [] 387 | for d in self._basedir: 388 | path = '%s/%s.ini' % (d, name) 389 | f.append(path.replace('\\', '/')) 390 | return f 391 | 392 | def _update(self, pointer, config): 393 | for key, cfg_val in config.items(): 394 | if isinstance(cfg_val, Section): 395 | o = cfgval() 396 | for k, v in cfg_val.items(): 397 | if isinstance(v, Section): 398 | o2 = cfgval() 399 | o.__setattr__(k, o2) 400 | self._update(o2, v) 401 | else: 402 | o.__setattr__(k, v) 403 | pointer[key] = o 404 | else: 405 | pointer[key] = cfg_val 406 | 407 | def __str__(self): 408 | """Return textual representation of the current config. 409 | 410 | It can be used for special purposes (i.e. to save values 411 | somewhere and reload them -- however, they will be a simple 412 | dictionaries of textual values; without special powers. This 413 | class also does not provide ways to load such dumped values, 414 | we would be circumventing configobj and that is no good. 415 | """ 416 | return ("{{'global_config' : {0._global}, " 417 | "'local_config' : {0._local}, " 418 | "'on_demand_config': {0._on_demand}, " 419 | "'recent_caller' : '{0._recent_caller}'}}".format(self)) 420 | 421 | 422 | class cfgval(dict): 423 | 424 | """Wrapper for configuration value.""" 425 | 426 | __getattr__ = dict.__getitem__ 427 | __setattr__ = dict.__setitem__ 428 | 429 | def __repr__(self): 430 | """Return representation of :class:`cfgval` instance.""" 431 | # return '{%s}' % ',\n'.join(map(lambda o: "'.%s': %s" % (o[0], 432 | # repr(o[1])), self.__dict__.items())) 433 | return '%s\n%s' % ('#cfgwrapper', repr(self)) 434 | 435 | 436 | class ConfigWrapper(object): 437 | 438 | """Configuration wrapper.""" 439 | 440 | def __init__(self, realpath, config): 441 | """Set instance `realpath` and `config` values.""" 442 | self.__dict__['_config'] = config 443 | self.__dict__['_realpath'] = realpath 444 | 445 | def __getattr__(self, key): 446 | """Return value from instance `config` value.""" 447 | return self._config[key] 448 | 449 | def __setattr__(self, key, value): 450 | """Store value in instance `config` dictionary.""" 451 | self._config.__setitem__(key, value) 452 | 453 | def get(self, key): 454 | """Allow recursive dotted key access to configuration.""" 455 | parts = key.split('.') 456 | pointer = self 457 | for p in parts: 458 | pointer = getattr(pointer, p) 459 | return pointer 460 | 461 | def __str__(self): 462 | """Return string representation with `config` and `realpath`.""" 463 | return "%s #config from: %s" % (self._config, self._realpath) 464 | 465 | 466 | # The config instance is a configuration reader 467 | # The configuration can sit in different places 468 | 469 | __cfgdir = None 470 | # set by environmental variable 471 | if 'WORKFLOWCFG' in os.environ: 472 | __cfgdir = os.environ['WORKFLOWCFG'] 473 | if os.pathsep in __cfgdir: 474 | __cfgdir = __cfgdir.split(os.pathsep) 475 | else: 476 | __cfgdir = [os.path.abspath(os.path.dirname(__file__)), 477 | os.path.abspath(os.path.dirname(__file__) + '/cfg')] 478 | 479 | 480 | # This instance has access to all global/local config options 481 | config_reader = ConfigReader(basedir=__cfgdir) 482 | -------------------------------------------------------------------------------- /workflow/deprecation.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # This file is part of Invenio. 3 | # Copyright (C) 2013, 2014, 2015, 2016 CERN. 4 | # 5 | # Invenio is free software; you can redistribute it and/or 6 | # modify it under the terms of the GNU General Public License as 7 | # published by the Free Software Foundation; either version 2 of the 8 | # License, or (at your option) any later version. 9 | # 10 | # Invenio is distributed in the hope that it will be useful, but 11 | # WITHOUT ANY WARRANTY; without even the implied warranty of 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 13 | # General Public License for more details. 14 | # 15 | # You should have received a copy of the GNU General Public License 16 | # along with Invenio; if not, write to the Free Software Foundation, Inc., 17 | # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. 18 | 19 | """Deprecation warning related helpers.""" 20 | import warnings 21 | from functools import wraps 22 | 23 | 24 | # Improved version of http://code.activestate.com/recipes/391367-deprecated/ 25 | def deprecated(message, category=DeprecationWarning): 26 | def wrap(func): 27 | """Decorator which can be used to mark functions as deprecated. 28 | 29 | :param message: text to include in the warning 30 | :param category: warning category exception class 31 | """ 32 | @wraps(func) 33 | def new_func(*args, **kwargs): 34 | warnings.warn(message, category, stacklevel=3) 35 | return func(*args, **kwargs) 36 | return new_func 37 | return wrap 38 | -------------------------------------------------------------------------------- /workflow/engine_db.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # This file is part of Workflow. 4 | # Copyright (C) 2011, 2012, 2014, 2015, 2016 CERN. 5 | # 6 | # Workflow is free software; you can redistribute it and/or modify it 7 | # under the terms of the Revised BSD License; see LICENSE file for 8 | # more details. 9 | 10 | """The workflow engine extension of GenericWorkflowEngine.""" 11 | 12 | from __future__ import absolute_import 13 | 14 | import traceback 15 | 16 | from enum import Enum 17 | 18 | from six import reraise 19 | 20 | from .engine import ( 21 | GenericWorkflowEngine, 22 | TransitionActions, 23 | ProcessingFactory, 24 | ) 25 | from .errors import WorkflowError 26 | from .utils import classproperty 27 | 28 | 29 | class EnumLabel(Enum): 30 | def __init__(self, label): 31 | self.label = self.labels[label] 32 | 33 | @classproperty 34 | def labels(cls): 35 | raise NotImplementedError 36 | 37 | 38 | class WorkflowStatus(EnumLabel): 39 | """Define the known workflow statuses. """ 40 | 41 | NEW = 0 42 | RUNNING = 1 43 | HALTED = 2 44 | ERROR = 3 45 | COMPLETED = 4 46 | 47 | @classproperty 48 | def labels(cls): 49 | return { 50 | 0: "New", 51 | 1: "Running", 52 | 2: "Halted", 53 | 3: "Error", 54 | 4: "Completed", 55 | } 56 | 57 | 58 | class ObjectStatus(EnumLabel): 59 | """Specify the known object statuses.""" 60 | 61 | INITIAL = 0 62 | COMPLETED = 1 63 | HALTED = 2 64 | RUNNING = 3 65 | ERROR = 4 66 | 67 | @classproperty 68 | def labels(cls): 69 | return { 70 | 0: "New", 71 | 1: "Done", 72 | 2: "Need action", 73 | 3: "In process", 74 | 4: "Error", 75 | } 76 | 77 | 78 | class DbWorkflowEngine(GenericWorkflowEngine): 79 | """GenericWorkflowEngine with DB persistence. 80 | 81 | Adds a SQLAlchemy database model to save workflow states and 82 | workflow data. 83 | 84 | Overrides key functions in GenericWorkflowEngine to implement 85 | logging and certain workarounds for storing data before/after 86 | task calls (This part will be revisited in the future). 87 | """ 88 | 89 | def __init__(self, db_obj, **kwargs): 90 | """Instantiate a new BibWorkflowEngine object. 91 | 92 | :param db_obj: the workflow engine 93 | :type db_obj: Workflow 94 | 95 | This object is needed to run a workflow and control the workflow, 96 | like at which step of the workflow execution is currently at, as well 97 | as control object manipulation inside the workflow. 98 | 99 | You can pass several parameters to personalize your engine, 100 | but most of the time you will not need to create this object yourself 101 | as the :py:mod:`.api` is there to do it for you. 102 | 103 | :param db_obj: instance of a Workflow object. 104 | :type db_obj: Workflow 105 | """ 106 | self.db_obj = db_obj 107 | super(DbWorkflowEngine, self).__init__() 108 | 109 | @classproperty 110 | def processing_factory(cls): 111 | """Provide a proccessing factory.""" 112 | return DbProcessingFactory 113 | 114 | @classproperty 115 | def known_statuses(cls): 116 | return WorkflowStatus 117 | 118 | @property 119 | def name(self): 120 | """Return the name.""" 121 | return self.db_obj.name 122 | 123 | @property 124 | def status(self): 125 | """Return the status.""" 126 | return self.db_obj.status 127 | 128 | @property 129 | def uuid(self): 130 | """Return the status.""" 131 | return self.db_obj.uuid 132 | 133 | @property 134 | def database_objects(self): 135 | """Return the objects associated with this workflow.""" 136 | return self.db_obj.objects 137 | 138 | @property 139 | def final_objects(self): 140 | """Return the objects associated with this workflow.""" 141 | return [obj for obj in self.database_objects 142 | if obj.status in [obj.known_statuses.COMPLETED]] 143 | 144 | @property 145 | def halted_objects(self): 146 | """Return the objects associated with this workflow.""" 147 | return [obj for obj in self.database_objects 148 | if obj.status in [obj.known_statuses.HALTED]] 149 | 150 | @property 151 | def running_objects(self): 152 | """Return the objects associated with this workflow.""" 153 | return [obj for obj in self.database_objects 154 | if obj.status in [obj.known_statuses.RUNNING]] 155 | 156 | def __repr__(self): 157 | """Allow to represent the DbWorkflowEngine.""" 158 | return "" % (self.name,) 159 | 160 | def __str__(self, log=False): 161 | """Allow to print the DbWorkflowEngine.""" 162 | return """------------------------------- 163 | DbWorkflowEngine 164 | ------------------------------- 165 | %s 166 | ------------------------------- 167 | """ % (self.db_obj.__str__(),) 168 | 169 | def save(self, status=None): 170 | """Save the workflow instance to database.""" 171 | # This workflow continues a previous execution. 172 | self.db_obj.save(status) 173 | 174 | 175 | class DbTransitionAction(TransitionActions): 176 | """Transition actions on engine exceptions for persistence object. 177 | 178 | ..note:: 179 | Typical actions to take here is store the new state of the object and 180 | save it, save the engine, log a message and finally call `super`. 181 | """ 182 | @staticmethod 183 | def HaltProcessing(obj, eng, callbacks, exc_info): 184 | """Action to take when HaltProcessing is raised.""" 185 | e = exc_info[1] 186 | obj.save(status=obj.known_statuses.HALTED, 187 | task_counter=eng.state.callback_pos, 188 | id_workflow=eng.uuid) 189 | eng.save(status=WorkflowStatus.HALTED) 190 | message = "Workflow '%s' halted at task %s with message: %s" % \ 191 | (eng.name, eng.current_taskname or "Unknown", e.message) 192 | eng.log.warning(message) 193 | super(DbTransitionAction, DbTransitionAction).HaltProcessing( 194 | obj, eng, callbacks, exc_info 195 | ) 196 | 197 | @staticmethod 198 | def Exception(obj, eng, callbacks, exc_info): 199 | """Action to take when an otherwise unhandled exception is raised.""" 200 | exception_repr = ''.join(traceback.format_exception(*exc_info)) 201 | msg = "Error:\n%s" % (exception_repr) 202 | eng.log.error(msg) 203 | if obj: 204 | # Sets an error message as a tuple (title, details) 205 | obj.set_error_message(exception_repr) 206 | obj.save(status=obj.known_statuses.ERROR, 207 | callback_pos=eng.state.callback_pos, 208 | id_workflow=eng.uuid) 209 | eng.save(WorkflowStatus.ERROR) 210 | try: 211 | super(DbTransitionAction, DbTransitionAction).Exception( 212 | obj, eng, callbacks, exc_info 213 | ) 214 | except Exception: 215 | # We expect this to reraise 216 | pass 217 | # Change the type of the Exception to WorkflowError, but use its tb 218 | reraise(WorkflowError( 219 | message=exception_repr, id_workflow=eng.uuid, 220 | id_object=eng.state.token_pos), None, exc_info[2] 221 | ) 222 | 223 | 224 | class DbProcessingFactory(ProcessingFactory): 225 | """Processing factory for persistence requirements.""" 226 | 227 | @classproperty 228 | def transition_exception_mapper(cls): 229 | """Define our for handling transition exceptions.""" 230 | return DbTransitionAction 231 | 232 | @staticmethod 233 | def before_object(eng, objects, obj): 234 | """Action to take before the proccessing of an object begins.""" 235 | obj.save(status=obj.known_statuses.RUNNING, 236 | id_workflow=eng.db_obj.uuid) 237 | super(DbProcessingFactory, DbProcessingFactory).before_object( 238 | eng, objects, obj 239 | ) 240 | 241 | @staticmethod 242 | def after_object(eng, objects, obj): 243 | """Action to take once the proccessing of an object completes.""" 244 | # We save each object once it is fully run through 245 | obj.save(status=obj.known_statuses.COMPLETED, 246 | id_workflow=eng.db_obj.uuid) 247 | super(DbProcessingFactory, DbProcessingFactory).after_object( 248 | eng, objects, obj 249 | ) 250 | 251 | @staticmethod 252 | def before_processing(eng, objects): 253 | """Executed before processing the workflow.""" 254 | eng.save(WorkflowStatus.RUNNING) 255 | super(DbProcessingFactory, DbProcessingFactory).before_processing( 256 | eng, objects 257 | ) 258 | 259 | @staticmethod 260 | def after_processing(eng, objects): 261 | """Action after process to update status.""" 262 | if eng.has_completed: 263 | eng.save(WorkflowStatus.COMPLETED) 264 | else: 265 | eng.save(WorkflowStatus.HALTED) 266 | -------------------------------------------------------------------------------- /workflow/errors.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # This file is part of Workflow. 4 | # Copyright (C) 2011, 2012, 2014, 2015, 2016 CERN. 5 | # 6 | # Workflow is free software; you can redistribute it and/or modify it 7 | # under the terms of the Revised BSD License; see LICENSE file for 8 | # more details. 9 | 10 | 11 | """Contains standard error messages for workflows module.""" 12 | 13 | import sys 14 | from functools import wraps, partial 15 | from types import MethodType 16 | 17 | 18 | def with_str(args): 19 | """Attach bound method `__str__` to a class instance. 20 | 21 | :type args: list or tuple 22 | :param args: 23 | args[0], is taken from `self` and printed 24 | args[1], if exists, is iterated over and each element is looked in 25 | `self` and printed in a `key=value` format. 26 | 27 | Usage example: 28 | ..code-block:: python 29 | @with_str(('message', ('foo', 'bar'))) 30 | class MyException(Exception): 31 | def __init__(self, message, a, b): 32 | self.message = message 33 | self.foo = a 34 | self.bar = b 35 | 36 | >> print(MyException('hello', 1, 2)) 37 | >> MyException(hello, foo=1, bar=2), 38 | """ 39 | @wraps(with_str) 40 | def wrapper(Klass): 41 | def __str__(args, self): 42 | """String representation.""" 43 | real_args = getattr(self, args[0]) 44 | try: 45 | real_kwargs = {} 46 | for key in args[1]: 47 | real_kwargs[key] = getattr(self, key) 48 | except IndexError: 49 | real_kwargs = {} 50 | return "{class_name}({real_args}, {real_kwargs})".format( 51 | class_name=Klass.__name__, 52 | real_args=real_args, 53 | real_kwargs=', '.join(('{k}={v}'.format(k=key, v=val) 54 | for key, val in real_kwargs.items()))) 55 | if sys.version_info >= (3, ): 56 | Klass.__str__ = MethodType(partial(__str__, args), Klass) 57 | else: 58 | Klass.__str__ = MethodType(partial(__str__, args), None, Klass) 59 | return Klass 60 | return wrapper 61 | 62 | 63 | class WorkflowTransition(Exception): 64 | """Base class for workflow exceptions.""" 65 | 66 | 67 | class StopProcessing(WorkflowTransition): 68 | """Stop current workflow.""" 69 | 70 | 71 | @with_str(('message', ('action', 'payload'))) 72 | class HaltProcessing(WorkflowTransition): # Used to be WorkflowHalt 73 | """Halt the workflow (can be used for nested workflow engines). 74 | 75 | Also contains the widget and other information to be displayed. 76 | """ 77 | def __init__(self, message="", action=None, payload=None): 78 | """Instanciate a HaltProcessing object.""" 79 | super(HaltProcessing, self).__init__() 80 | self.message = message 81 | self.action = action 82 | self.payload = payload 83 | 84 | 85 | class ContinueNextToken(WorkflowTransition): 86 | """Jump up to next token (it can be called many levels deep).""" 87 | 88 | 89 | class JumpToken(WorkflowTransition): 90 | """Jump N steps in the given direction.""" 91 | 92 | 93 | class JumpTokenForward(WorkflowTransition): 94 | """Jump N steps forwards.""" 95 | 96 | 97 | class JumpTokenBack(WorkflowTransition): 98 | """Jump N steps back.""" 99 | 100 | 101 | class JumpCall(WorkflowTransition): 102 | """In one loop ``[call, call...]``, jump `x` steps.""" 103 | 104 | 105 | # Deprecated 106 | class JumpCallForward(WorkflowTransition): 107 | """In one loop ``[call, call...]``, jump `x` steps forward.""" 108 | 109 | 110 | # Deprecated 111 | class JumpCallBack(WorkflowTransition): 112 | """In one loop ``[call, call...]``, jump `x` steps forward.""" 113 | 114 | 115 | class BreakFromThisLoop(WorkflowTransition): 116 | """Break from this loop, but do not stop processing.""" 117 | 118 | 119 | @with_str(('message', ('id_workflow', 'id_object', 'payload'))) 120 | class WorkflowError(Exception): 121 | """Raised when workflow experiences an error.""" 122 | 123 | def __init__(self, message, id_workflow=None, 124 | id_object=None, payload=None): 125 | """Instanciate a WorkflowError object.""" 126 | self.message = message 127 | self.id_workflow = id_workflow 128 | self.id_object = id_object 129 | self.payload = payload 130 | # Needed for passing an exception through message queue 131 | super(WorkflowError, self).__init__(message) 132 | 133 | 134 | @with_str(('message', ('workflow_name', 'payload'))) 135 | class WorkflowDefinitionError(Exception): 136 | """Raised when workflow definition is missing.""" 137 | 138 | def __init__(self, message, workflow_name, payload=None): 139 | """Instanciate a WorkflowDefinitionError object.""" 140 | self.message = message 141 | self.workflow_name = workflow_name 142 | self.payload = payload 143 | super(WorkflowDefinitionError, self).__init__(message, workflow_name, 144 | payload) 145 | 146 | 147 | @with_str(('message', ('obj_status', 'id_object'))) 148 | class WorkflowObjectStatusError(Exception): 149 | """Raised when workflow object has an unknown or missing version.""" 150 | 151 | def __init__(self, message, id_object, obj_status): 152 | """Instanciate a WorkflowObjectStatusError object.""" 153 | self.message = message 154 | self.obj_status = obj_status 155 | self.id_object = id_object 156 | 157 | 158 | class WorkflowAPIError(Exception): 159 | """Raised when there is a problem with parameters at the API level.""" 160 | 161 | 162 | class SkipToken(WorkflowTransition): 163 | """Used by workflow engine to skip the current process of an object.""" 164 | 165 | 166 | class AbortProcessing(WorkflowTransition): 167 | """Used by workflow engine to abort the engine execution.""" 168 | -------------------------------------------------------------------------------- /workflow/patterns/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # This file is part of Workflow. 4 | # Copyright (C) 2011, 2014 CERN. 5 | # 6 | # Workflow is free software; you can redistribute it and/or modify it 7 | # under the terms of the Revised BSD License; see LICENSE file for 8 | # more details. 9 | 10 | """Workflow patterns definitions.""" 11 | 12 | # basic flow commands 13 | from .controlflow import (STOP, BREAK, OBJ_JUMP_BWD, OBJ_JUMP_FWD, OBJ_NEXT, 14 | TASK_JUMP_BWD, TASK_JUMP_FWD, TASK_JUMP_IF) 15 | 16 | 17 | # conditions 18 | from .controlflow import IF, IF_NOT, IF_ELSE, WHILE 19 | 20 | # basic patterns 21 | from .controlflow import PARALLEL_SPLIT, SYNCHRONIZE, SIMPLE_MERGE, CHOICE 22 | 23 | 24 | # helper functions 25 | from .utils import (EMPTY_CALL, ENG_GET, ENG_SET, OBJ_SET, OBJ_GET, ERROR, TRY, 26 | RUN_WF, CALLFUNC, DEBUG_CYCLE, PROFILE) 27 | -------------------------------------------------------------------------------- /workflow/patterns/controlflow.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # This file is part of Workflow. 4 | # Copyright (C) 2011, 2014, 2015, 2016, 2017 CERN. 5 | # 6 | # Workflow is free software; you can redistribute it and/or modify it 7 | # under the terms of the Revised BSD License; see LICENSE file for 8 | # more details. 9 | 10 | """Basic control flow patterns. 11 | 12 | See http://www.yawlfoundation.org/pages/resources/patterns.html#basic 13 | """ 14 | 15 | import threading 16 | import time 17 | import collections 18 | from functools import wraps, partial 19 | 20 | from six.moves import _thread as thread, queue 21 | from six import string_types 22 | 23 | from .utils import with_nice_docs 24 | from ..engine import Callbacks 25 | 26 | 27 | MAX_TIMEOUT = 30000 28 | 29 | 30 | @with_nice_docs 31 | def TASK_JUMP_BWD(step=-1): 32 | """Jump to the previous task - eng.jump_call. 33 | 34 | Example: A, B, TASK_JUMP_FWD(-2), C, D, ... 35 | will produce: A, B, A, B, A, B, ... (recursion!) 36 | :param step: int, must not be positive number 37 | """ 38 | def _move_back(obj, eng): 39 | eng.jump_call(step) 40 | _move_back.__name__ = 'TASK_JUMP_BWD' 41 | return _move_back 42 | 43 | 44 | @with_nice_docs 45 | def TASK_JUMP_FWD(step=1): 46 | """Jump to the next task - eng.jump_call() 47 | example: A, B, TASK_JUMP_FWD(2), C, D, ... 48 | will produce: A, B, D 49 | :param step: int 50 | """ 51 | def _x(obj, eng): 52 | eng.jump_call(step) 53 | _x.__name__ = 'TASK_JUMP_FWD' 54 | return _x 55 | 56 | 57 | @with_nice_docs 58 | def TASK_JUMP_IF(cond, step): 59 | """Jump in the specified direction if the condition 60 | evaluates to True, the difference from other IF conditions 61 | is that this one does not insert the code inside a [] block 62 | :param cond: function 63 | :param step: int, negative jumps back, positive forward 64 | """ 65 | def jump(obj, eng): 66 | return cond(obj, eng) and eng.jump_call(step) 67 | 68 | return jump 69 | 70 | 71 | @with_nice_docs 72 | def BREAK(): 73 | """Stop execution of the current block while keeping workflow running. 74 | 75 | Usage: ``eng.break_current_loop()``. 76 | """ 77 | def x(obj, eng): 78 | eng.break_current_loop() 79 | x.__name__ = 'BREAK' 80 | return x 81 | 82 | 83 | @with_nice_docs 84 | def STOP(): 85 | """Unconditional stop of the workflow execution.""" 86 | def x(obj, eng): 87 | eng.stopProcessing() 88 | x.__name__ = 'STOP' 89 | return x 90 | 91 | 92 | @with_nice_docs 93 | def HALT(): 94 | """Unconditional stop of the workflow execution.""" 95 | def x(obj, eng): 96 | eng.haltProcessing() 97 | x.__name__ = 'HALT' 98 | return x 99 | 100 | 101 | @with_nice_docs 102 | def OBJ_NEXT(): 103 | """Stop the workflow execution for the current object and start 104 | the same worfklow for the next object - eng.break_current_loop().""" 105 | def x(obj, eng): 106 | eng.break_current_loop() 107 | x.__name__ = 'OBJ_NEXT' 108 | return x 109 | 110 | 111 | @with_nice_docs 112 | def OBJ_JUMP_FWD(step=1): 113 | """Stop the workflow execution, jumps to xth consecutive object 114 | and starts executing the workflow on it - eng.jumpTokenForward() 115 | :param step: int, relative jump from the current obj, must not be 116 | negative number 117 | """ 118 | def x(obj, eng): 119 | eng.jumpTokenForward(step) 120 | x.__name__ = 'OBJ_JUMP_FWD' 121 | return x 122 | 123 | 124 | @with_nice_docs 125 | def OBJ_JUMP_BWD(step=-1): 126 | """Stop the workflow execution, jumps to xth antecedent object 127 | and starts executing the workflow on it - eng.jumpTokenForward() 128 | :param step: int, relative jump from the current obj, must not be 129 | negative number 130 | """ 131 | def _x(obj, eng): 132 | eng.jumpTokenBackward(step) 133 | _x.__name__ = 'OBJ_JUMP_BWD' 134 | return _x 135 | 136 | # ------------------------- some conditions --------------------------------- # 137 | 138 | 139 | @with_nice_docs 140 | def IF(cond, branch): 141 | """Implement condition, if cond evaluates to True branch is executed. 142 | 143 | :param cond: callable, function that decides 144 | :param branch: block of functions to run 145 | 146 | @attention: the branch is inserted inside [] block, therefore jumping is 147 | limited only inside the branch 148 | """ 149 | def _x(obj, eng): 150 | return cond(obj, eng) and eng.jump_call(1) \ 151 | or eng.break_current_loop() 152 | _x.__name__ = 'IF' 153 | return [_x, branch] 154 | 155 | 156 | @with_nice_docs 157 | def IF_NOT(cond, branch): 158 | """Implements condition, if cond evaluates to False 159 | branch is executed 160 | :param cond: callable, function that decides 161 | :param branch: block of functions to run 162 | 163 | @attention: the branch is inserted inside [] block, therefore jumping is 164 | limited only inside the branch 165 | """ 166 | def _x(obj, eng): 167 | if cond(obj, eng): 168 | eng.break_current_loop() 169 | return 1 170 | _x.__name__ = 'IF_NOT' 171 | return [_x, branch] 172 | 173 | 174 | @with_nice_docs 175 | def IF_ELSE(cond, branch1, branch2): 176 | """Implements condition, if cond evaluates to True 177 | branch1 is executed, otherwise branch2 178 | :param cond: callable, function that decides 179 | :param branch1: block of functions to run [if=true] 180 | :param branch2: block of functions to run [else] 181 | 182 | @attention: the branch is inserted inside [] block, therefore jumping is 183 | limited only inside the branch 184 | """ 185 | if branch1 is None or branch2 is None: 186 | raise Exception("Neither of the branches can be None/empty") 187 | 188 | def _x(obj, eng): 189 | return cond(obj, eng) and eng.jump_call(1) \ 190 | or eng.jump_call(3) 191 | _x.__name__ = 'IF_ELSE' 192 | return [_x, branch1, BREAK(), branch2] 193 | 194 | 195 | @with_nice_docs 196 | def WHILE(cond, branch): 197 | """Keeps executing branch as long as the condition cond is True 198 | :param cond: callable, function that decides 199 | :param branch: block of functions to run [if=true] 200 | """ 201 | # quite often i passed a function, which results in errors 202 | if callable(branch): 203 | branch = (branch,) 204 | # we don't know what is hiding inside branch 205 | branch = tuple(Callbacks.cleanup_callables(branch)) 206 | 207 | def _x(obj, eng): 208 | if not cond(obj, eng): 209 | eng.break_current_loop() 210 | _x.__name__ = 'WHILE' 211 | return [_x, branch, TASK_JUMP_BWD(-(len(branch) + 1))] 212 | 213 | 214 | @with_nice_docs 215 | def CMP(a, b, op): 216 | """Task that can be used in if or something else to compare two values. 217 | 218 | :param a: left-hand-side value 219 | :param b: right-hand-side value 220 | :param op: Operator can be : 221 | eq , gt , gte , lt , lte 222 | == , > , >= , < , <= 223 | :return: bool: result of the test 224 | """ 225 | @wraps(CMP) 226 | def _CMP(obj, eng): 227 | a_ = a 228 | b_ = b 229 | while callable(a_): 230 | a_ = a_(obj, eng) 231 | while callable(b_): 232 | b_ = b_(obj, eng) 233 | return { 234 | "eq": lambda a_, b_: a_ == b_, 235 | "gt": lambda a_, b_: a_ > b_, 236 | "gte": lambda a_, b_: a_ >= b_, 237 | "lt": lambda a_, b_: a_ < b_, 238 | "lte": lambda a_, b_: a_ <= b_, 239 | 240 | "==": lambda a_, b_: a_ == b_, 241 | ">": lambda a_, b_: a_ > b_, 242 | ">=": lambda a_, b_: a_ >= b_, 243 | "<": lambda a_, b_: a_ < b_, 244 | "<=": lambda a_, b_: a_ <= b_, 245 | "in": lambda a_, b_: b_ in a_, 246 | }[op](a_, b_) 247 | _CMP.hide = True 248 | return _CMP 249 | 250 | 251 | def _setter(key, obj, eng, step, val): 252 | eng.extra_data[key] = val 253 | 254 | 255 | @with_nice_docs 256 | def FOR(get_list_function, setter, branch, cache_data=False, order="ASC"): 257 | """For loop that stores the current item. 258 | :param get_list_function: function returning the list on which we should 259 | iterate. 260 | :param branch: block of functions to run 261 | :param savename: name of variable to save the current loop state in the 262 | extra_data in case you want to reuse the value somewhere in a task. 263 | :param cache_data: can be True or False in case of True, the list will be 264 | cached in memory instead of being recomputed everytime. In case of caching 265 | the list is no more dynamic. 266 | :param order: because we should iterate over a list you can choose in which 267 | order you want to iterate over your list from start to end(ASC) or from end 268 | to start (DSC). 269 | :param setter: function to call in order to save the current item of the 270 | list that is being iterated over. 271 | expected to take arguments (obj, eng, val) 272 | :param getter: function to call in order to retrieve the current item of 273 | the list that is being iterated over. expected to take arguments(obj, eng) 274 | """ 275 | # be sane 276 | assert order in ('ASC', 'DSC') 277 | # sanitize string better 278 | if isinstance(setter, string_types): 279 | setter = partial(_setter, setter) 280 | # quite often i passed a function, which results in errors 281 | if callable(branch): 282 | branch = (branch,) 283 | # we don't know what is hiding inside branch 284 | branch = tuple(Callbacks.cleanup_callables(branch)) 285 | 286 | def _for(obj, eng): 287 | step = str(eng.getCurrTaskId()) # eg '[1]' 288 | if "_Iterators" not in eng.extra_data: 289 | eng.extra_data["_Iterators"] = {} 290 | 291 | def get_list(): 292 | try: 293 | return eng.extra_data["_Iterators"][step]["cache"] 294 | except KeyError: 295 | if callable(get_list_function): 296 | return get_list() 297 | elif isinstance(get_list_function, collections.Iterable): 298 | return list(get_list_function) 299 | else: 300 | raise TypeError("get_list_function is not a callable nor a" 301 | " iterable") 302 | 303 | my_list_to_process = get_list() 304 | 305 | # First time we are in this step 306 | if step not in eng.extra_data["_Iterators"]: 307 | eng.extra_data["_Iterators"][step] = {} 308 | # Cache list 309 | if cache_data: 310 | eng.extra_data["_Iterators"][step]["cache"] = get_list() 311 | # Initialize step value 312 | eng.extra_data["_Iterators"][step]["value"] = { 313 | "ASC": 0, 314 | "DSC": len(my_list_to_process) - 1}[order] 315 | # Store previous data 316 | if 'current_data' in eng.extra_data["_Iterators"][step]: 317 | eng.extra_data["_Iterators"][step]["previous_data"] = \ 318 | eng.extra_data["_Iterators"][step]["current_data"] 319 | 320 | # Increment or decrement step value 321 | step_value = eng.extra_data["_Iterators"][step]["value"] 322 | currently_within_list_bounds = \ 323 | (order == "ASC" and step_value < len(my_list_to_process)) or \ 324 | (order == "DSC" and step_value > -1) 325 | if currently_within_list_bounds: 326 | # Store current data for ourselves 327 | eng.extra_data["_Iterators"][step]["current_data"] = \ 328 | my_list_to_process[step_value] 329 | # Store for the user 330 | if setter: 331 | setter(obj, eng, step, my_list_to_process[step_value]) 332 | if order == 'ASC': 333 | eng.extra_data["_Iterators"][step]["value"] += 1 334 | elif order == 'DSC': 335 | eng.extra_data["_Iterators"][step]["value"] -= 1 336 | else: 337 | setter(obj, eng, step, 338 | eng.extra_data["_Iterators"][step]["previous_data"]) 339 | del eng.extra_data["_Iterators"][step] 340 | eng.break_current_loop() 341 | 342 | _for.__name__ = 'FOR' 343 | return [_for, branch, TASK_JUMP_BWD(-(len(branch) + 1))] 344 | 345 | 346 | @with_nice_docs 347 | def PARALLEL_SPLIT(*args): 348 | """Start task in parallel. 349 | 350 | @attention: tasks A,B,C,D... are not addressable, you can't 351 | you can't use jumping to them (they are invisible to 352 | the workflow engine). Though you can jump inside the 353 | branches 354 | @attention: tasks B,C,D... will be running on their own 355 | once you have started them, and we are not waiting for 356 | them to finish. Workflow will continue executing other 357 | tasks while B,C,D... might be still running. 358 | @attention: a new engine is spawned for each branch or code, 359 | all operations works as expected, but mind that the branches 360 | know about themselves, they don't see other tasks outside. 361 | They are passed the object, but not the old workflow 362 | engine object 363 | @postcondition: eng object will contain lock (to be used 364 | by threads) 365 | """ 366 | def _parallel_split(obj, eng, calls): 367 | lock = thread.allocate_lock() 368 | eng.store['lock'] = lock 369 | for func in calls: 370 | new_eng = eng.duplicate() 371 | new_eng.setWorkflow( 372 | [lambda o, e: e.store.update({'lock': lock}), func] 373 | ) 374 | thread.start_new_thread(new_eng.process, ([obj], )) 375 | # new_eng.process([obj]) 376 | return lambda o, e: _parallel_split(o, e, args) 377 | 378 | 379 | @with_nice_docs 380 | def SYNCHRONIZE(*args, **kwargs): 381 | """ 382 | After the execution of task B, task C, and task D, task E can be executed. 383 | :param *args: args can be a mix of callables and list of callables 384 | the simplest situation comes when you pass a list of callables 385 | they will be simply executed in parallel. 386 | But if you pass a list of callables (branch of callables) 387 | which is potentionally a new workflow, we will first create a 388 | workflow engine with the workflows, and execute the branch in it 389 | @attention: you should never jump out of the synchronized branches 390 | """ 391 | timeout = MAX_TIMEOUT 392 | if 'timeout' in kwargs: 393 | timeout = kwargs['timeout'] 394 | 395 | if len(args) < 2: 396 | raise Exception('You must pass at least two callables') 397 | 398 | def _synchronize(obj, eng): 399 | queue = MyTimeoutQueue() 400 | # spawn a pool of threads, and pass them queue instance 401 | for i in range(len(args) - 1): 402 | t = MySpecialThread(queue) 403 | t.setDaemon(True) 404 | t.start() 405 | 406 | for func in args[0:-1]: 407 | if isinstance(func, list) or isinstance(func, tuple): 408 | new_eng = eng.duplicate() 409 | new_eng.setWorkflow(func) 410 | queue.put(lambda: new_eng.process([obj])) 411 | else: 412 | queue.put(lambda: func(obj, eng)) 413 | 414 | # wait on the queue until everything has been processed 415 | queue.join_with_timeout(timeout) 416 | 417 | # run the last func 418 | args[-1](obj, eng) 419 | _synchronize.__name__ = 'SYNCHRONIZE' 420 | return _synchronize 421 | 422 | 423 | @with_nice_docs 424 | def CHOICE(arbiter, *predicates, **kwpredicates): 425 | """ 426 | A choice is made to execute either task B, task C or task D 427 | after execution of task A. 428 | :param arbiter: a function which returns some value (the value 429 | must be inside the predicates dictionary) 430 | :param predicates: list of callables, the first item must be the 431 | value returned by the arbiter, example: 432 | ('submit', task_a), 433 | ('upload' : task_a, [task_b, task_c]...) 434 | :param **kwpredicates: you can supply predicates also as a 435 | keywords, example 436 | CHOICE(arbiter, one=lambda...., two=[lambda o,e:...., ...]) 437 | @postcondition: all tasks are 'jumpable' 438 | 439 | """ 440 | workflow = [] 441 | mapping = {} 442 | for branch in predicates: 443 | workflow.append(branch[1:]) 444 | mapping[branch[0]] = len(workflow) 445 | workflow.append(BREAK()) 446 | 447 | for k, v in kwpredicates.items(): 448 | workflow.append(v) 449 | mapping[k] = len(workflow) 450 | workflow.append(BREAK()) 451 | 452 | def _exclusive_choice(obj, eng): 453 | val = arbiter(obj, eng) 454 | i = mapping[val] # die on error 455 | eng.jump_call(i) 456 | c = _exclusive_choice 457 | c.__name__ = arbiter.__name__ 458 | workflow.insert(0, c) 459 | return workflow 460 | 461 | 462 | @with_nice_docs 463 | def SIMPLE_MERGE(*args): 464 | """ 465 | Task E will be started when any one of the tasks B, C or D completes. 466 | This pattern though makes a context assumption: there is no 467 | parallelism preceding task E. 468 | """ 469 | 470 | if len(args) < 2: 471 | raise Exception("You must suply at least 2 callables") 472 | 473 | final_task = args[-1] 474 | workflow = [] 475 | total = ((len(args) - 1) * 2) + 1 476 | for branch in args[0:-1]: 477 | total -= 2 478 | workflow.append(branch) 479 | workflow.append(TASK_JUMP_FWD(total)) 480 | 481 | workflow.append(final_task) 482 | return workflow 483 | 484 | 485 | class MyTimeoutQueue(queue.Queue): 486 | 487 | def join_with_timeout(self, timeout): 488 | self.all_tasks_done.acquire() 489 | try: 490 | endtime = time.time() + timeout 491 | while self.unfinished_tasks: 492 | remaining = endtime - time.time() 493 | if remaining <= 0.0: 494 | raise threading.ThreadError('NotFinished') 495 | time.sleep(.05) 496 | self.all_tasks_done.wait(remaining) 497 | finally: 498 | self.all_tasks_done.release() 499 | 500 | 501 | class MySpecialThread(threading.Thread): 502 | 503 | def __init__(self, itemq, *args, **kwargs): 504 | threading.Thread.__init__(self, *args, **kwargs) 505 | self.itemq = itemq 506 | 507 | def run(self): 508 | call = self.itemq.get() 509 | call() 510 | -------------------------------------------------------------------------------- /workflow/patterns/utils.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # This file is part of Workflow. 4 | # Copyright (C) 2011, 2014, 2015 CERN. 5 | # 6 | # Workflow is free software; you can redistribute it and/or modify it 7 | # under the terms of the Revised BSD License; see LICENSE file for 8 | # more details. 9 | 10 | from __future__ import print_function 11 | 12 | import sys 13 | 14 | import collections 15 | import inspect 16 | import pstats 17 | import six 18 | import timeit 19 | import traceback 20 | from functools import wraps 21 | 22 | from workflow.errors import WorkflowTransition 23 | 24 | 25 | try: 26 | import cProfile 27 | except ImportError: 28 | import profile as cProfile 29 | 30 | 31 | def with_nice_docs(func): 32 | """Add nice documentation to the function returned by another function. 33 | 34 | Adds the extra parameter ``comment``, that might be used to override the 35 | automatically generated docs. This is specially useful for all the control 36 | flow functions defined here. 37 | 38 | Args: 39 | func(callable): function to decorate, that must return a function. 40 | comment(string): override for the automatically generated docs. 41 | 42 | Returns: 43 | callable: the function that ``func`` would return with the extra nice 44 | docstring. 45 | """ 46 | def _comment_from_params(*args, **kwargs): 47 | args_doc = ( 48 | 'args(' + ', '.join( 49 | str(arg) for arg in args 50 | ) + ')' 51 | ) 52 | kwargs_doc = ( 53 | 'kwargs(' + ', '.join( 54 | '%s<%s>' % (key, type(value)) for key, value in kwargs.items() 55 | ) + ')' 56 | ) 57 | return func.__name__ + ': ' + args_doc + '; ' + kwargs_doc + '.' 58 | 59 | @wraps(func) 60 | def _decorated_func(*args, **kwargs): 61 | comment = kwargs.pop('comment', _comment_from_params(*args, **kwargs)) 62 | inner_func = func(*args, **kwargs) 63 | if callable(inner_func): 64 | inner_func.__doc__ = comment 65 | elif isinstance(inner_func, collections.Iterable): 66 | inner_func[0].__doc__ = comment 67 | 68 | return inner_func 69 | 70 | return _decorated_func 71 | 72 | 73 | @with_nice_docs 74 | def RUN_WF(workflow, 75 | engine=None, 76 | data_connector=None, 77 | pass_eng=[], 78 | pass_always=None, 79 | outkey='RUN_WF', 80 | reinit=False): 81 | """Task for running other workflow - i.e. new workflow engine will 82 | be created and the workflow run. The workflow engine is garbage 83 | collected together with the function. Therefore you can run the 84 | function many times and it will reuse the already-loaded WE. 85 | 86 | :param workflow: normal workflow tasks definition 87 | :param engine: class of the engine to create WE, if None, the new 88 | WFE instance will be of the same class as the calling WFE. 89 | Attention, changes in the classes of the WFE instances may have 90 | many consequences, so be careful. For example, if you use 91 | serialiazable WFE instance, but you create another instance of WFE 92 | which is not serializable, then you will be in problems. 93 | --- 94 | :param data_connector: callback which will prepare data and pass 95 | the corrent objects into the workflow engine (from the calling 96 | engine into the called WE), if not present, the current obj is 97 | passed (possibly wrapped into a list) 98 | :param pass_eng: list of keys corresponding to the values, that should 99 | be passed from the calling engine to the called engine. This is 100 | called only once, during initialization. 101 | :param outkey: if outkey is present, the initialized new 102 | workflow engine is stored into the calling workflow engine 103 | so that you can get access to it from outside. This instance 104 | will be available at runtime 105 | :param reinit: if True, wfe will be re-instantiated always 106 | for every invocation of the function 107 | """ 108 | @wraps(RUN_WF) 109 | def x(obj, eng=None): 110 | 111 | if not outkey and reinit: 112 | raise AssertionError("Cannot use `reinit` without `outkey`.") 113 | 114 | if engine: # user supplied class 115 | engine_cls = engine 116 | else: 117 | engine_cls = eng.__class__ 118 | 119 | new_eng = engine_cls() 120 | 121 | if not reinit: 122 | try: 123 | new_eng = eng.extra_data[outkey] 124 | except KeyError: 125 | # This is the first time we are starting this engine, so 126 | # failing to reinit is normal. 127 | pass 128 | 129 | new_eng.callbacks.replace(workflow) 130 | 131 | if outkey: 132 | eng.extra_data.setdefault(outkey, new_eng) 133 | 134 | # pass data from the old wf engine to the new one 135 | to_remove = [] 136 | for k in pass_eng: 137 | new_eng.store[k] = eng.extra_data[k] 138 | if not pass_always and not reinit: 139 | to_remove.append(k) 140 | if to_remove: 141 | for k in to_remove: 142 | pass_eng.remove(k) 143 | 144 | if data_connector: 145 | data = data_connector(obj, eng) 146 | new_eng.process(data) 147 | else: 148 | new_eng.process(obj) 149 | x.__name__ = 'RUN_WF' 150 | return x 151 | 152 | # ------------------------- useful structures ------------------------------- # 153 | 154 | 155 | def EMPTY_CALL(obj, eng): 156 | """Empty call that does nothing""" 157 | pass 158 | 159 | 160 | @with_nice_docs 161 | def ENG_GET(something): 162 | """this is the same as lambda obj, eng: eng.extra_data.get(something) 163 | :param something: str, key of the object to retrieve 164 | :return: value of the key from eng object 165 | """ 166 | @wraps(ENG_GET) 167 | def x(obj, eng): 168 | return eng.extra_data.setdefault(something, None) 169 | return x 170 | 171 | 172 | @with_nice_docs 173 | def ENG_SET(key, value): 174 | """this is the same as lambda obj, eng: eng.extra_data.update({'key': value}) 175 | :param key: str, key of the object to retrieve 176 | :param value: anything 177 | @attention: this call is executed when the workflow is created 178 | therefore, the key and value must exist at the time 179 | (obj and eng don't exist yet) 180 | """ 181 | @wraps(ENG_SET) 182 | def _eng_set(obj, eng): 183 | eng.extra_data[key] = value 184 | return _eng_set 185 | 186 | 187 | @with_nice_docs 188 | def OBJ_GET(something, cond='all'): 189 | """this is the same as lambda obj, eng: something in obj and obj[something] 190 | :param something: str, key of the object to retrieve or list of strings 191 | :param cond: how to evaluate several keys, all|any|many 192 | :return: value of the key from obj object, if you are looking at several 193 | keys, then a list is returned. Watch for empty and None returns! 194 | 195 | """ 196 | @wraps(OBJ_GET) 197 | def x(obj, eng): 198 | if isinstance(something, six.string_types): 199 | return something in obj and obj[something] 200 | else: 201 | if cond.lower() == 'any': 202 | for o in something: 203 | if o in obj and obj[o]: 204 | return obj[o] 205 | elif cond.lower() == 'many': 206 | r = {} 207 | for o in something: 208 | if o in obj and obj[o]: 209 | r[o] = obj[o] 210 | return r 211 | else: 212 | r = {} 213 | for o in something: 214 | if o in obj and obj[o]: 215 | r[o] = obj[o] 216 | else: 217 | return False 218 | return r 219 | 220 | x.__name__ = 'OBJ_GET' 221 | return x 222 | 223 | 224 | @with_nice_docs 225 | def OBJ_SET(key, value): 226 | """this is the same as lambda obj, eng: obj.__setitem__(key, value) 227 | :param key: str, key of the object to retrieve 228 | :param value: anything 229 | @attention: this call is executed when the workflow is created 230 | therefore, the key and value must exist at the time 231 | (obj and eng don't exist yet) 232 | """ 233 | @wraps(OBJ_SET) 234 | def x(obj, eng): 235 | obj[key] = value 236 | x.__name__ = 'OBJ_SET' 237 | return x 238 | 239 | # ----------------------- error handlling ------------------------------- 240 | 241 | 242 | @with_nice_docs 243 | def ERROR(msg='Error in the workflow'): 244 | """Throws uncatchable error stopping execution and printing the message""" 245 | caller = inspect.getmodule(inspect.currentframe().f_back) 246 | if caller: 247 | caller = caller.__file__ 248 | else: 249 | caller = '' 250 | 251 | @wraps(ERROR) 252 | def x(obj, eng): 253 | raise Exception('in %s : %s' % (caller, msg)) 254 | x.__name__ = 'ERROR' 255 | return x 256 | 257 | 258 | @with_nice_docs 259 | def TRY(onecall, retry=1, onfailure=Exception, verbose=True): 260 | """Wrap the call in try...except statement and eventually 261 | retries when failure happens 262 | :param attempts: how many times to retry 263 | :param onfailure: exception to raise or callable to call on failure, 264 | if callable, then it will receive standard obj, eng arguments 265 | """ 266 | 267 | if not callable(onecall): 268 | raise Exception('You can wrap only one callable with TRY') 269 | 270 | @wraps(TRY) 271 | def x(obj, eng): 272 | tries = 1 + retry 273 | i = 0 274 | while i < tries: 275 | try: 276 | onecall(obj, eng) 277 | break # success 278 | except WorkflowTransition: 279 | raise # just let it propagate 280 | except: 281 | if verbose: 282 | eng.log.error('Error reported from the call') 283 | traceback.print_exc() 284 | i += 1 285 | if i >= tries: 286 | if isinstance(onfailure, Exception): 287 | raise onfailure 288 | elif callable(onfailure): 289 | onfailure(obj, eng) 290 | else: 291 | raise Exception( 292 | 'Error after attempting to run: %s' % onecall) 293 | 294 | x.__name__ = 'TRY' 295 | return x 296 | 297 | 298 | @with_nice_docs 299 | def PROFILE(call, output=None, 300 | stats=['time', 'calls', 'cumulative', 'pcalls']): 301 | """Run the call(s) inside profiler 302 | :param call: either function or list of functions 303 | - if it is a single callable, it will be executed 304 | - if it is a list of callables, a new workflow engine (a duplicate) 305 | will be created, the workflow will be set with the calls, and 306 | calls executed; thus by providing list of functions, you are 307 | actually profiling also the workflow engine! 308 | :param output: where to save the stats, if empty, it will be printed 309 | to stdout 310 | :param stats: list of statistical outputs, 311 | default is: time, calls, cumulative, pcalls 312 | @see pstats module for explanation 313 | """ 314 | @wraps(PROFILE) 315 | def x(obj, eng): 316 | if isinstance(call, list) or isinstance(call, tuple): 317 | new_eng = eng.duplicate() 318 | new_eng.setWorkflow(call) 319 | 320 | def profileit(): 321 | return new_eng.process([obj]) 322 | else: 323 | def profileit(): 324 | return call(obj, eng) 325 | 326 | if output: 327 | cProfile.runctx('profileit()', globals(), locals(), output) 328 | else: 329 | cProfile.runctx('profileit()', globals(), locals()) 330 | 331 | if output and stats: 332 | for st in stats: 333 | fname = '%s.stats-%s' % (output, st) 334 | fo = open(fname, 'w') 335 | 336 | p = pstats.Stats(output, stream=fo) 337 | p.strip_dirs() 338 | p.sort_stats(st) 339 | p.print_stats() 340 | 341 | fo.close() 342 | x.__name__ = 'PROFILE' 343 | return x 344 | 345 | 346 | @with_nice_docs 347 | def DEBUG_CYCLE(stmt, setup=None, 348 | onerror=None, 349 | debug_stopper=None, 350 | **kwargs): 351 | """Workflow task DEBUG_CYCLE used to repeatedly execute 352 | certain call - you can effectively reload modules and 353 | hotplug the new code, debug at runtime. The call is 354 | taking advantage of the internal python timeit module. 355 | The parameters are the same as for timeit module - i.e. 356 | 357 | :param stmt: string to evaluate (i.e. "print sys") 358 | :param setup: initialization (i.e. "import sys") 359 | 360 | The debug_stopper is a callback that receives (eng, obj) 361 | *after* execution of the main call. If debug_stopper 362 | returns True, it means 'stop', don't continue. 363 | 364 | :param onerror: string (like the setup) which will 365 | be appended to setup in case of error. I.e. if execution 366 | failed, you can reload the module and try again. This 367 | gets fired only after an exception! 368 | 369 | You can also pass any number of arguments as keywords, 370 | they will be available to your function at the runtime. 371 | 372 | Here is example of testing engine task: 373 | 374 | >>> from merkur.box.code import debug_cycle 375 | >>> def debug_stopper(obj, eng): 376 | ... if obj: 377 | ... return True 378 | ... 379 | >>> def engtask(config, something): 380 | ... def x(obj, eng): 381 | ... print(config) 382 | ... print(something) 383 | ... return x 384 | ... 385 | >>> config = {'some': 'value'} 386 | >>> debug_cycle = testpass.debug_cycle 387 | >>> c = DEBUG_CYCLE("engtask(config, something)(obj,eng)", 388 | ... "from __main__ import engtask", 389 | ... config=config, 390 | ... something='hi!', 391 | ... ) 392 | >>> c('eng', 'obj') 393 | {'some': 'value'} 394 | hi! 395 | >>> 396 | 397 | You can of course test any other python calls, not only 398 | engine tasks with this function. If you want to reload 399 | code, use the setup argument: 400 | 401 | c = DEBUG_CYCLE("mm.engtask(config, something)(obj,eng)", 402 | ... "import mm;reload(mm)", 403 | ... config=config) 404 | 405 | """ 406 | 407 | if not callable(debug_stopper): 408 | def debug_stopper(obj, eng): 409 | return False 410 | debug_stopper = debug_stopper 411 | to_pass = {} 412 | if kwargs: 413 | to_pass.update(kwargs) 414 | 415 | @wraps(DEBUG_CYCLE) 416 | def x(obj, eng): 417 | 418 | storage = [0, debug_stopper, True] # counter, callback, flag 419 | 420 | def _timer(): 421 | if storage[0] == 0: 422 | storage[0] = 1 423 | return timeit.default_timer() 424 | else: 425 | storage[0] = 0 426 | try: 427 | if storage[1](obj, eng): 428 | storage[2] = False 429 | except: 430 | traceback.print_exc() 431 | storage[2] = False 432 | return timeit.default_timer() 433 | 434 | class Timer(timeit.Timer): 435 | 436 | def timeit(self): 437 | # i am taking advantage of the timeit template 438 | # and passing in the self object inside the array 439 | timing = self.inner([self], self.timer) 440 | return timing 441 | 442 | error_caught = False 443 | while storage[2]: 444 | try: 445 | # include passed in arguments and also obj, eng 446 | _setup = ';'.join( 447 | ['%s=_it[0].topass[\'%s\']' % (k, k) 448 | for k, v in to_pass.items()]) 449 | _setup += '\nobj=_it[0].obj\neng=_it[0].eng' 450 | _setup += '\n%s' % setup 451 | if error_caught and onerror: 452 | _setup += '\n%s' % onerror 453 | try: 454 | t = Timer(stmt, _setup, _timer) 455 | except: 456 | traceback.print_exc() 457 | break 458 | 459 | # set reference to the passed in values 460 | t.topass = to_pass 461 | t.obj = obj 462 | t.eng = eng 463 | 464 | # print(t.src) 465 | print('Execution time: %.3s' % (t.timeit())) 466 | except: 467 | traceback.print_exc() 468 | lasterr = traceback.format_exc().splitlines() 469 | if '' in lasterr[-2]: 470 | sys.stderr.write( 471 | 'Error most likely in compilation, printing the ' 472 | 'source code:\n%s%s\n%s\n' % ( 473 | '=' * 60, t.src, '=' * 60)) 474 | break 475 | 476 | x.__name__ = 'DEBUG_CYCLE' 477 | return x 478 | 479 | 480 | @with_nice_docs 481 | def CALLFUNC(func, outkey=None, debug=False, stopper=None, 482 | args=[], oeargs=[], ekeys={}, okeys={}, **kwargs): 483 | """Workflow task CALLFUNC 484 | This wf task allows you to call any function 485 | :param func: identification of the function, it can be either 486 | string (fully qualified function name) or the callable 487 | itself 488 | :param outkey: results of the call will be stored inside 489 | eng.extra_data[outkey] if outkey != None 490 | :param debug: boolean, if True, we will run the call in a 491 | loop, reloading the module after each error 492 | :param stopper: a callable which will receive obj, eng 493 | after each run. If the callable returns True, we will 494 | stop running the func (only applicable when debug=True) 495 | :param args: params passed on to the function 496 | :param ekeys: dictionary of key:value pairs, we will take 497 | 'value' from the engine, and pass it to the function under 498 | the 'key' name. 499 | :param okeys: the same as ekeys, only that values are taken 500 | from the obj 501 | :param oeargs: definition of arguments that should be put 502 | inside the *args; you can use syntactic sugar to instruct 503 | system where to take the value, for example Eseman - will 504 | take eng.extra_data['seman'] -- 'O' [capital letter Oooo] means 505 | take the value from obj 506 | :param **kwargs: other kwargs passed on to the function 507 | :return: nothing, value is stored inside obj[outkey] 508 | """ 509 | mod, new_func = _get_mod_func(func) 510 | args = list(args) 511 | 512 | @wraps(CALLFUNC) 513 | def x(obj, eng): 514 | try: 515 | for key in oeargs: 516 | first_key, rest_key = key[0], key[1:] 517 | if first_key == 'O': 518 | args.append(obj[rest_key]) 519 | elif first_key == 'E' and rest_key in eng.extra_data: 520 | args.append(eng.extra_data.setdefault(rest_key)) 521 | else: 522 | if key in obj: 523 | args.append(obj[key]) 524 | elif key in eng.extra_data: 525 | args.append(eng.extra_data.setdefault(key)) 526 | else: 527 | raise Exception( 528 | "%s is not inside obj nor eng, try specifying " 529 | "Okey or Ekey" % key) 530 | except Exception as msg: 531 | eng.log.error(traceback.format_exc()) 532 | eng.log.error( 533 | 'Check your "oeargs" configuration. ' 534 | 'Key "%s" not available' % key) 535 | sys.exit(1) 536 | 537 | for k, v in ekeys.items(): 538 | kwargs[k] = eng.extra_data[v] 539 | for k, v in okeys.items(): 540 | kwargs[k] = obj[v] 541 | 542 | if debug: 543 | universal_repeater(mod, new_func, stopper, *args, **kwargs) 544 | else: 545 | if outkey: 546 | obj[outkey] = new_func(*args, **kwargs) 547 | else: 548 | new_func(*args, **kwargs) 549 | x.__name__ = 'CALLFUNC' 550 | return x 551 | 552 | # ----------------- not wf tasks ----------------------------- 553 | 554 | 555 | def _get_mod_func(func): 556 | """for a given callable finds its module - imports it 557 | and returns module, call -- module can be reloaded""" 558 | # find out module of this call 559 | def get_mod(modid, __name__): 560 | mod = __import__(modid) 561 | components = modid.split('.') 562 | for comp in components[1:]: 563 | mod = getattr(mod, comp) 564 | return getattr(mod, __name__), mod 565 | 566 | if callable(func): 567 | m = func.__module__ 568 | n = func.__name__ 569 | new_func, mod = get_mod(m, n) 570 | else: 571 | m, n = str(func).rsplit('.', 1) 572 | new_func, mod = get_mod(m, n) 573 | return mod, new_func 574 | 575 | 576 | def debug_simple(func, *args, **kwargs): 577 | """Run func with *args, **kwargs and reloads it 578 | after each failure - this is not a wfe task""" 579 | mod, new_func = _get_mod_func(func) 580 | universal_repeater(mod, new_func) 581 | 582 | 583 | def universal_repeater(mod, call, stopper=None, *args, **kwargs): 584 | """Universal while loop.""" 585 | fname = call.__name__ 586 | while True: 587 | if callable(stopper) and stopper(*args, **kwargs): 588 | break 589 | try: 590 | call(*args, **kwargs) 591 | except: 592 | traceback.print_exc() 593 | reload(mod) 594 | call = getattr(mod, fname) 595 | -------------------------------------------------------------------------------- /workflow/signals.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # This file is part of Workflow. 4 | # Copyright (C) 2011, 2012, 2014, 2015 CERN. 5 | # 6 | # Workflow is free software; you can redistribute it and/or modify it 7 | # under the terms of the Revised BSD License; see LICENSE file for 8 | # more details. 9 | 10 | 11 | """Contain signals emitted from workflows module. 12 | 13 | Import with care. It is not neccessary that `blinker` is available.""" 14 | 15 | from blinker import Namespace 16 | _signals = Namespace() 17 | 18 | 19 | workflow_halted = _signals.signal('workflow_halted') 20 | """ 21 | This signal is sent when a workflow engine's halt function is called. 22 | Sender is the bibworkflow object that was running before the workflow 23 | was halted. 24 | """ 25 | 26 | workflow_started = _signals.signal('workflow_started') 27 | """ 28 | This signal is sent when a workflow is started. 29 | Sender is the workflow engine object running the workflow. 30 | """ 31 | 32 | workflow_finished = _signals.signal('workflow_finished') 33 | """ 34 | This signal is sent when a workflow is finished. 35 | Sender is the workflow engine object running the workflow. 36 | """ 37 | 38 | workflow_error = _signals.signal('workflow_error') 39 | """ 40 | This signal is sent when a workflow object gets an error. 41 | Sender is the workflow engine object that was running before the workflow 42 | got the error. 43 | """ 44 | -------------------------------------------------------------------------------- /workflow/utils.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # This file is part of Workflow. 4 | # Copyright (C) 2011, 2012, 2014, 2015, 2016 CERN. 5 | # 6 | # Workflow is free software; you can redistribute it and/or modify it 7 | # under the terms of the Revised BSD License; see LICENSE file for 8 | # more details. 9 | 10 | 11 | # https://mail.python.org/pipermail/python-ideas/2011-January/008958.html 12 | class staticproperty(object): 13 | """Property decorator for static methods.""" 14 | 15 | def __init__(self, function): 16 | self._function = function 17 | 18 | def __get__(self, instance, owner): 19 | return self._function() 20 | 21 | 22 | class classproperty(object): 23 | """Property decorator for class methods.""" 24 | 25 | def __init__(self, function): 26 | self._function = function 27 | 28 | def __get__(self, instance, owner): 29 | return self._function(owner) 30 | -------------------------------------------------------------------------------- /workflow/version.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # This file is part of Workflow. 4 | # Copyright (C) 2014, 2016 CERN. 5 | # 6 | # Workflow is free software; you can redistribute it and/or modify it 7 | # under the terms of the Revised BSD License; see LICENSE file for 8 | # more details. 9 | 10 | """ 11 | Version information for workflow package. 12 | 13 | This file is imported by ``workflow.__init__``, and parsed by 14 | ``setup.py`` as well as ``docs/conf.py``. 15 | """ 16 | import autosemver 17 | 18 | 19 | __version__ = autosemver.packaging.get_current_version( 20 | project_name='workflow' 21 | ) 22 | --------------------------------------------------------------------------------