├── .gitignore ├── README.md ├── docs ├── Makefile ├── conf.py ├── index.rst ├── intro.rst ├── modules.rst ├── whizbang.http.orm.rst ├── whizbang.http.rst ├── whizbang.http.views.rst ├── whizbang.kvs.rst ├── whizbang.queue.config.rst ├── whizbang.queue.rst ├── whizbang.queue.util.rst └── whizbang.rst ├── examples └── twooter │ ├── db.sqlite3 │ ├── models.py │ ├── requirements.txt │ ├── resources │ └── twoot.js │ ├── runserver.py │ └── sockets.py ├── images ├── home.png ├── resource.png └── resources.png ├── omega ├── __init__.py ├── http │ ├── __init__.py │ ├── cache.py │ ├── core.py │ ├── orm │ │ ├── __init__.py │ │ └── model.py │ ├── utils.py │ └── views │ │ ├── __init__.py │ │ ├── generated.py │ │ ├── nosql.py │ │ ├── orm.py │ │ ├── static.py │ │ ├── template.py │ │ └── templates │ │ ├── base.html │ │ ├── index.html │ │ ├── resource.html │ │ └── resources.html ├── kvs │ ├── __init__.py │ ├── client.py │ ├── server.py │ ├── test │ │ └── app.py │ └── tests │ │ └── test_kvs.py ├── log │ └── README.md ├── queue │ ├── README.md │ ├── __init__.py │ ├── config │ │ ├── __init__.py │ │ └── settings.py │ ├── message.py │ ├── monitor.py │ ├── queue.py │ ├── serialization.py │ ├── test │ │ ├── app.py │ │ └── config │ │ │ ├── __init__.py │ │ │ └── settings.py │ ├── util │ │ ├── __init__.py │ │ └── xrange_helper.py │ └── worker.py ├── search │ └── README.md ├── settings │ └── README.md └── stat │ └── README.md ├── requirements.txt └── tests ├── config ├── __init__.py └── settings.py └── test_queue.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | .*sw[op] 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Omega! 2 | 3 | **Omega - The web *platform* that "has an app for that"** 4 | 5 | ## What is Omega? 6 | 7 | Omega is an attempt to bring back innovation to Python web frameworks. Its 8 | goal is to be more than a web framework; Omega aims to be a platform 9 | on which *any* type of web application can be built, batteries included. 10 | That means Omega ships with support for creating ORM-backed CRUD 11 | applications, NoSQL REST APIs, real-time applications using Websockets, and 12 | simple, mostly static page applications. 13 | 14 | To this end, Omega will include the tools/libraries listed below. Of course, 15 | it's still in its infancy, so many of the items below are vaporware. 16 | 17 | ### `search` 18 | 19 | Support for full-text search in web applications. *Coming Soon*. 20 | 21 | ### `kvs` 22 | 23 | A pure-Python NoSQL database usable as a backing store for a web application. 24 | *In progress*. 25 | 26 | ### `queue` 27 | 28 | A distributed, asychronous task queue for out-of-band/background task execution in web 29 | applications. Includes support for `cron`-like job scheduling. *In progress.* 30 | 31 | ### `log/stat` 32 | 33 | A centralized logging/metrics server with fully browsable logs and metric 34 | reporting. Includes monitoring for metrics like process uptime, request speed, 35 | number of exceptions, etc. *Coming Soon* 36 | 37 | ### `settings` 38 | 39 | A centralized live-settings server capable of providing service discovery 40 | capabilities as well as a management frontent for traditional database-backed 41 | application settings. *Coming Soon* 42 | 43 | ### `http` 44 | 45 | A micro web framework with macro capabilities. *In progress.* 46 | 47 | Here are a few examples of the vastly different types of applications you can already 48 | build in an instant: 49 | 50 | ##### ORM-backed CRUD application 51 | 52 | `runserver.py` 53 | 54 | ```python 55 | 56 | from omega.http.core import create_app 57 | from omega.http.orm import create_engine 58 | from werkzeug import run_simple 59 | from models import Twoot, User 60 | 61 | if __name__ == '__main__': 62 | app = create_app(__name__) 63 | app.engine(create_engine('sqlite+pysqlite:///db.sqlite3')) 64 | app.orm_resource(Twoot) 65 | app.orm_resource(User) 66 | app.auto_generate_home() 67 | run_simple('127.0.0.1', 5000, app, use_debugger=True, use_reloader=True) 68 | 69 | ``` 70 | 71 | `models.py` 72 | 73 | ```python 74 | 75 | import datetime 76 | 77 | from omega.http.orm import Model, Column, String, DateTime, Integer, relationship, ForeignKey 78 | 79 | class User(Model): 80 | __tablename__ = 'user' 81 | 82 | id = Column(Integer, primary_key=True) 83 | user_name = Column(String) 84 | joined_at = Column(DateTime, default=datetime.datetime.now()) 85 | 86 | def __str__(self): 87 | return self.user_name 88 | 89 | class Twoot(Model): 90 | __tablename__ = 'twoot' 91 | 92 | id = Column(Integer, primary_key=True) 93 | content = Column(String) 94 | posted_at = Column(DateTime, default=datetime.datetime.now()) 95 | user_id = Column(Integer, ForeignKey('user.id')) 96 | user = relationship(User) 97 | 98 | ``` 99 | 100 | This gives you the following, for free: 101 | 102 | ###### Homepage 103 | 104 | ![Home page shot](/images/home.png) 105 | 106 | ###### List of objects 107 | 108 | ![Resources shot](/images/resources.png) 109 | 110 | By adding a `sockets.py` file with the following contents (and creating a new 111 | route function using the `route` decorator): 112 | 113 | ```python 114 | from socketio.namespace import BaseNamespace 115 | 116 | 117 | class ChatNamespace(BaseNamespace): 118 | sockets = {} 119 | 120 | def on_chat(self, msg): 121 | self.emit('chat', msg) 122 | 123 | def recv_connect(self): 124 | self.sockets[id(self)] = self 125 | 126 | def disconnect(self, *args, **kwargs): 127 | if id(self) in self.sockets: 128 | del self.sockets[id(self)] 129 | super(ChatNamespace, self).disconnect(*args, **kwargs) 130 | 131 | @classmethod 132 | def broadcast(self, event, message): 133 | for ws in self.sockets.values(): 134 | ws.emit(event, message) 135 | ``` 136 | 137 | (in `runserver.py`): 138 | 139 | ```python 140 | @app.route('/chat', methods=['POST']) 141 | def chat(request): 142 | """Route chat posts to the *chat* handler function. Broadcast the message 143 | to all users.""" 144 | message = '{}: {}'.format(request.form['user'], request.form['message']) 145 | if message: 146 | ChatNamespace.broadcast('message', message) 147 | return Response() 148 | ``` 149 | 150 | You now have free real-time chat that degrades gracefully via socket.io. Connect 151 | two browsers to the root page and type a message in one browser. It will pop up 152 | in the chat area of the other browser. 153 | -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- 1 | # Makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = sphinx-build 7 | PAPER = 8 | BUILDDIR = _build 9 | 10 | # User-friendly check for sphinx-build 11 | ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) 12 | $(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) 13 | endif 14 | 15 | # Internal variables. 16 | PAPEROPT_a4 = -D latex_paper_size=a4 17 | PAPEROPT_letter = -D latex_paper_size=letter 18 | ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 19 | # the i18n builder cannot share the environment and doctrees with the others 20 | I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 21 | 22 | .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext 23 | 24 | help: 25 | @echo "Please use \`make ' where is one of" 26 | @echo " html to make standalone HTML files" 27 | @echo " dirhtml to make HTML files named index.html in directories" 28 | @echo " singlehtml to make a single large HTML file" 29 | @echo " pickle to make pickle files" 30 | @echo " json to make JSON files" 31 | @echo " htmlhelp to make HTML files and a HTML help project" 32 | @echo " qthelp to make HTML files and a qthelp project" 33 | @echo " devhelp to make HTML files and a Devhelp project" 34 | @echo " epub to make an epub" 35 | @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" 36 | @echo " latexpdf to make LaTeX files and run them through pdflatex" 37 | @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" 38 | @echo " text to make text files" 39 | @echo " man to make manual pages" 40 | @echo " texinfo to make Texinfo files" 41 | @echo " info to make Texinfo files and run them through makeinfo" 42 | @echo " gettext to make PO message catalogs" 43 | @echo " changes to make an overview of all changed/added/deprecated items" 44 | @echo " xml to make Docutils-native XML files" 45 | @echo " pseudoxml to make pseudoxml-XML files for display purposes" 46 | @echo " linkcheck to check all external links for integrity" 47 | @echo " doctest to run all doctests embedded in the documentation (if enabled)" 48 | 49 | clean: 50 | rm -rf $(BUILDDIR)/* 51 | 52 | html: 53 | $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html 54 | @echo 55 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." 56 | 57 | dirhtml: 58 | $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml 59 | @echo 60 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." 61 | 62 | singlehtml: 63 | $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml 64 | @echo 65 | @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." 66 | 67 | pickle: 68 | $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle 69 | @echo 70 | @echo "Build finished; now you can process the pickle files." 71 | 72 | json: 73 | $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json 74 | @echo 75 | @echo "Build finished; now you can process the JSON files." 76 | 77 | htmlhelp: 78 | $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp 79 | @echo 80 | @echo "Build finished; now you can run HTML Help Workshop with the" \ 81 | ".hhp project file in $(BUILDDIR)/htmlhelp." 82 | 83 | qthelp: 84 | $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp 85 | @echo 86 | @echo "Build finished; now you can run "qcollectiongenerator" with the" \ 87 | ".qhcp project file in $(BUILDDIR)/qthelp, like this:" 88 | @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Whizbang.qhcp" 89 | @echo "To view the help file:" 90 | @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Whizbang.qhc" 91 | 92 | devhelp: 93 | $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp 94 | @echo 95 | @echo "Build finished." 96 | @echo "To view the help file:" 97 | @echo "# mkdir -p $$HOME/.local/share/devhelp/Whizbang" 98 | @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Whizbang" 99 | @echo "# devhelp" 100 | 101 | epub: 102 | $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub 103 | @echo 104 | @echo "Build finished. The epub file is in $(BUILDDIR)/epub." 105 | 106 | latex: 107 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 108 | @echo 109 | @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." 110 | @echo "Run \`make' in that directory to run these through (pdf)latex" \ 111 | "(use \`make latexpdf' here to do that automatically)." 112 | 113 | latexpdf: 114 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 115 | @echo "Running LaTeX files through pdflatex..." 116 | $(MAKE) -C $(BUILDDIR)/latex all-pdf 117 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 118 | 119 | latexpdfja: 120 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 121 | @echo "Running LaTeX files through platex and dvipdfmx..." 122 | $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja 123 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 124 | 125 | text: 126 | $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text 127 | @echo 128 | @echo "Build finished. The text files are in $(BUILDDIR)/text." 129 | 130 | man: 131 | $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man 132 | @echo 133 | @echo "Build finished. The manual pages are in $(BUILDDIR)/man." 134 | 135 | texinfo: 136 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 137 | @echo 138 | @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." 139 | @echo "Run \`make' in that directory to run these through makeinfo" \ 140 | "(use \`make info' here to do that automatically)." 141 | 142 | info: 143 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 144 | @echo "Running Texinfo files through makeinfo..." 145 | make -C $(BUILDDIR)/texinfo info 146 | @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." 147 | 148 | gettext: 149 | $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale 150 | @echo 151 | @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." 152 | 153 | changes: 154 | $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes 155 | @echo 156 | @echo "The overview file is in $(BUILDDIR)/changes." 157 | 158 | linkcheck: 159 | $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck 160 | @echo 161 | @echo "Link check complete; look for any errors in the above output " \ 162 | "or in $(BUILDDIR)/linkcheck/output.txt." 163 | 164 | doctest: 165 | $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest 166 | @echo "Testing of doctests in the sources finished, look at the " \ 167 | "results in $(BUILDDIR)/doctest/output.txt." 168 | 169 | xml: 170 | $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml 171 | @echo 172 | @echo "Build finished. The XML files are in $(BUILDDIR)/xml." 173 | 174 | pseudoxml: 175 | $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml 176 | @echo 177 | @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." 178 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # Omega documentation build configuration file, created by 4 | # sphinx-quickstart on Sun Jun 15 21:44:48 2014. 5 | # 6 | # This file is execfile()d with the current directory set to its 7 | # containing dir. 8 | # 9 | # Note that not all possible configuration values are present in this 10 | # autogenerated file. 11 | # 12 | # All configuration values have a default; values that are commented out 13 | # serve to show the default. 14 | 15 | import sys 16 | import os 17 | import sphinx_rtd_theme 18 | 19 | 20 | # ... 21 | 22 | # Activate the theme. 23 | # If extensions (or modules to document with autodoc) are in another directory, 24 | # add these directories to sys.path here. If the directory is relative to the 25 | # documentation root, use os.path.abspath to make it absolute, like shown here. 26 | #sys.path.insert(0, os.path.abspath('.')) 27 | 28 | # -- General configuration ------------------------------------------------ 29 | 30 | # If your documentation needs a minimal Sphinx version, state it here. 31 | #needs_sphinx = '1.0' 32 | 33 | # Add any Sphinx extension module names here, as strings. They can be 34 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom 35 | # ones. 36 | extensions = ['sphinx.ext.autodoc'] 37 | 38 | # Add any paths that contain templates here, relative to this directory. 39 | templates_path = ['_templates'] 40 | 41 | # The suffix of source filenames. 42 | source_suffix = '.rst' 43 | 44 | # The encoding of source files. 45 | #source_encoding = 'utf-8-sig' 46 | 47 | # The master toctree document. 48 | master_doc = 'index' 49 | 50 | # General information about the project. 51 | project = u'Omega' 52 | copyright = u'2014, Jeff Knupp' 53 | 54 | # The version info for the project you're documenting, acts as replacement for 55 | # |version| and |release|, also used in various other places throughout the 56 | # built documents. 57 | # 58 | # The short X.Y version. 59 | version = '0.0.1' 60 | # The full version, including alpha/beta/rc tags. 61 | release = '0.0.1' 62 | 63 | # The language for content autogenerated by Sphinx. Refer to documentation 64 | # for a list of supported languages. 65 | #language = None 66 | 67 | # There are two options for replacing |today|: either, you set today to some 68 | # non-false value, then it is used: 69 | #today = '' 70 | # Else, today_fmt is used as the format for a strftime call. 71 | #today_fmt = '%B %d, %Y' 72 | 73 | # List of patterns, relative to source directory, that match files and 74 | # directories to ignore when looking for source files. 75 | exclude_patterns = ['_build'] 76 | 77 | # The reST default role (used for this markup: `text`) to use for all 78 | # documents. 79 | #default_role = None 80 | 81 | # If true, '()' will be appended to :func: etc. cross-reference text. 82 | #add_function_parentheses = True 83 | 84 | # If true, the current module name will be prepended to all description 85 | # unit titles (such as .. function::). 86 | #add_module_names = True 87 | 88 | # If true, sectionauthor and moduleauthor directives will be shown in the 89 | # output. They are ignored by default. 90 | #show_authors = False 91 | 92 | # The name of the Pygments (syntax highlighting) style to use. 93 | pygments_style = 'sphinx' 94 | 95 | # A list of ignored prefixes for module index sorting. 96 | #modindex_common_prefix = [] 97 | 98 | # If true, keep warnings as "system message" paragraphs in the built documents. 99 | #keep_warnings = False 100 | 101 | 102 | # -- Options for HTML output ---------------------------------------------- 103 | 104 | # The theme to use for HTML and HTML Help pages. See the documentation for 105 | # a list of builtin themes. 106 | html_theme = "sphinx_rtd_theme" 107 | 108 | # Theme options are theme-specific and customize the look and feel of a theme 109 | # further. For a list of options available for each theme, see the 110 | # documentation. 111 | #html_theme_options = {} 112 | 113 | # Add any paths that contain custom themes here, relative to this directory. 114 | html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] 115 | 116 | # The name for this set of Sphinx documents. If None, it defaults to 117 | # " v documentation". 118 | #html_title = None 119 | 120 | # A shorter title for the navigation bar. Default is the same as html_title. 121 | #html_short_title = None 122 | 123 | # The name of an image file (relative to this directory) to place at the top 124 | # of the sidebar. 125 | #html_logo = None 126 | 127 | # The name of an image file (within the static path) to use as favicon of the 128 | # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 129 | # pixels large. 130 | #html_favicon = None 131 | 132 | # Add any paths that contain custom static files (such as style sheets) here, 133 | # relative to this directory. They are copied after the builtin static files, 134 | # so a file named "default.css" will overwrite the builtin "default.css". 135 | html_static_path = ['_static'] 136 | 137 | # Add any extra paths that contain custom files (such as robots.txt or 138 | # .htaccess) here, relative to this directory. These files are copied 139 | # directly to the root of the documentation. 140 | #html_extra_path = [] 141 | 142 | # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, 143 | # using the given strftime format. 144 | #html_last_updated_fmt = '%b %d, %Y' 145 | 146 | # If true, SmartyPants will be used to convert quotes and dashes to 147 | # typographically correct entities. 148 | #html_use_smartypants = True 149 | 150 | # Custom sidebar templates, maps document names to template names. 151 | #html_sidebars = {} 152 | 153 | # Additional templates that should be rendered to pages, maps page names to 154 | # template names. 155 | #html_additional_pages = {} 156 | 157 | # If false, no module index is generated. 158 | #html_domain_indices = True 159 | 160 | # If false, no index is generated. 161 | #html_use_index = True 162 | 163 | # If true, the index is split into individual pages for each letter. 164 | #html_split_index = False 165 | 166 | # If true, links to the reST sources are added to the pages. 167 | #html_show_sourcelink = True 168 | 169 | # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. 170 | #html_show_sphinx = True 171 | 172 | # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. 173 | #html_show_copyright = True 174 | 175 | # If true, an OpenSearch description file will be output, and all pages will 176 | # contain a tag referring to it. The value of this option must be the 177 | # base URL from which the finished HTML is served. 178 | #html_use_opensearch = '' 179 | 180 | # This is the file name suffix for HTML files (e.g. ".xhtml"). 181 | #html_file_suffix = None 182 | 183 | # Output file base name for HTML help builder. 184 | htmlhelp_basename = 'Omegadoc' 185 | 186 | 187 | # -- Options for LaTeX output --------------------------------------------- 188 | 189 | latex_elements = { 190 | # The paper size ('letterpaper' or 'a4paper'). 191 | #'papersize': 'letterpaper', 192 | 193 | # The font size ('10pt', '11pt' or '12pt'). 194 | #'pointsize': '10pt', 195 | 196 | # Additional stuff for the LaTeX preamble. 197 | #'preamble': '', 198 | } 199 | 200 | # Grouping the document tree into LaTeX files. List of tuples 201 | # (source start file, target name, title, 202 | # author, documentclass [howto, manual, or own class]). 203 | latex_documents = [ 204 | ('index', 'Omega.tex', u'Omega Documentation', 205 | u'Jeff Knupp', 'manual'), 206 | ] 207 | 208 | # The name of an image file (relative to this directory) to place at the top of 209 | # the title page. 210 | #latex_logo = None 211 | 212 | # For "manual" documents, if this is true, then toplevel headings are parts, 213 | # not chapters. 214 | #latex_use_parts = False 215 | 216 | # If true, show page references after internal links. 217 | #latex_show_pagerefs = False 218 | 219 | # If true, show URL addresses after external links. 220 | #latex_show_urls = False 221 | 222 | # Documents to append as an appendix to all manuals. 223 | #latex_appendices = [] 224 | 225 | # If false, no module index is generated. 226 | #latex_domain_indices = True 227 | 228 | 229 | # -- Options for manual page output --------------------------------------- 230 | 231 | # One entry per manual page. List of tuples 232 | # (source start file, name, description, authors, manual section). 233 | man_pages = [ 234 | ('index', 'omega', u'Omega Documentation', 235 | [u'Jeff Knupp'], 1) 236 | ] 237 | 238 | # If true, show URL addresses after external links. 239 | #man_show_urls = False 240 | 241 | 242 | # -- Options for Texinfo output ------------------------------------------- 243 | 244 | # Grouping the document tree into Texinfo files. List of tuples 245 | # (source start file, target name, title, author, 246 | # dir menu entry, description, category) 247 | texinfo_documents = [ 248 | ('index', 'Omega', u'Omega Documentation', 249 | u'Jeff Knupp', 'Omega', 'One line description of project.', 250 | 'Miscellaneous'), 251 | ] 252 | 253 | # Documents to append as an appendix to all manuals. 254 | #texinfo_appendices = [] 255 | 256 | # If false, no module index is generated. 257 | #texinfo_domain_indices = True 258 | 259 | # How to display URL addresses: 'footnote', 'no', or 'inline'. 260 | #texinfo_show_urls = 'footnote' 261 | 262 | # If true, do not generate a @detailmenu in the "Top" node's menu. 263 | #texinfo_no_detailmenu = False 264 | -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | .. Whizbang documentation master file, created by 2 | sphinx-quickstart on Sun Jun 15 21:44:48 2014. 3 | You can adapt this file completely to your liking, but it should at least 4 | contain the root `toctree` directive. 5 | 6 | Welcome to Whizbang's documentation! 7 | ==================================== 8 | 9 | Contents: 10 | 11 | .. toctree:: 12 | :maxdepth: 2 13 | 14 | intro 15 | quickstart 16 | 17 | Indices and tables 18 | ================== 19 | 20 | * :ref:`genindex` 21 | * :ref:`modindex` 22 | * :ref:`search` 23 | 24 | -------------------------------------------------------------------------------- /docs/intro.rst: -------------------------------------------------------------------------------- 1 | Introduction to Whizbang 2 | ======================== 3 | 4 | Whizbang is a web application suite that gives you the tools required to build 5 | vastly different types of modern web applications. Here are a few of the 6 | types of applications Whizbang makes it easy to create: 7 | 8 | * Static sites or dynamic sites that don't require storage 9 | * NoSQL-based REST APIs 10 | * CRUD sites using an ORM to interface with a traditional relational database 11 | * Real-time sites using websockets 12 | 13 | In order to allow you to build such different types of sites, whizbang offers 14 | functionality generally found in some webframeworks plus a number of tools found 15 | in almost none. For example, Whizbang provides: 16 | 17 | * A pure-Python Key/Value store using ZeroMQ for transport 18 | * A logging/statistics service that centralizes collection and display of 19 | metrics 20 | * A search module that allows for efficient, indexed searching in pure Python 21 | * A distributed task queue with *cron*-like functionality as well as the ability 22 | to asynchronously complete tasks launched over the network (again, using 23 | ZeroMQ for transport). 24 | -------------------------------------------------------------------------------- /docs/modules.rst: -------------------------------------------------------------------------------- 1 | whizbang 2 | ======== 3 | 4 | .. toctree:: 5 | :maxdepth: 4 6 | 7 | whizbang 8 | -------------------------------------------------------------------------------- /docs/whizbang.http.orm.rst: -------------------------------------------------------------------------------- 1 | whizbang.http.orm package 2 | ========================= 3 | 4 | Submodules 5 | ---------- 6 | 7 | whizbang.http.orm.model module 8 | ------------------------------ 9 | 10 | .. automodule:: whizbang.http.orm.model 11 | :members: 12 | :undoc-members: 13 | :show-inheritance: 14 | 15 | 16 | Module contents 17 | --------------- 18 | 19 | .. automodule:: whizbang.http.orm 20 | :members: 21 | :undoc-members: 22 | :show-inheritance: 23 | -------------------------------------------------------------------------------- /docs/whizbang.http.rst: -------------------------------------------------------------------------------- 1 | whizbang.http package 2 | ===================== 3 | 4 | Subpackages 5 | ----------- 6 | 7 | .. toctree:: 8 | 9 | whizbang.http.orm 10 | whizbang.http.views 11 | 12 | Submodules 13 | ---------- 14 | 15 | whizbang.http.cache module 16 | -------------------------- 17 | 18 | .. automodule:: whizbang.http.cache 19 | :members: 20 | :undoc-members: 21 | :show-inheritance: 22 | 23 | whizbang.http.core module 24 | ------------------------- 25 | 26 | .. automodule:: whizbang.http.core 27 | :members: 28 | :undoc-members: 29 | :show-inheritance: 30 | 31 | whizbang.http.interactions module 32 | --------------------------------- 33 | 34 | .. automodule:: whizbang.http.interactions 35 | :members: 36 | :undoc-members: 37 | :show-inheritance: 38 | 39 | whizbang.http.resource module 40 | ----------------------------- 41 | 42 | .. automodule:: whizbang.http.resource 43 | :members: 44 | :undoc-members: 45 | :show-inheritance: 46 | 47 | whizbang.http.utils module 48 | -------------------------- 49 | 50 | .. automodule:: whizbang.http.utils 51 | :members: 52 | :undoc-members: 53 | :show-inheritance: 54 | 55 | 56 | Module contents 57 | --------------- 58 | 59 | .. automodule:: whizbang.http 60 | :members: 61 | :undoc-members: 62 | :show-inheritance: 63 | -------------------------------------------------------------------------------- /docs/whizbang.http.views.rst: -------------------------------------------------------------------------------- 1 | whizbang.http.views package 2 | =========================== 3 | 4 | Submodules 5 | ---------- 6 | 7 | whizbang.http.views.generated module 8 | ------------------------------------ 9 | 10 | .. automodule:: whizbang.http.views.generated 11 | :members: 12 | :undoc-members: 13 | :show-inheritance: 14 | 15 | whizbang.http.views.nosql module 16 | -------------------------------- 17 | 18 | .. automodule:: whizbang.http.views.nosql 19 | :members: 20 | :undoc-members: 21 | :show-inheritance: 22 | 23 | whizbang.http.views.orm module 24 | ------------------------------ 25 | 26 | .. automodule:: whizbang.http.views.orm 27 | :members: 28 | :undoc-members: 29 | :show-inheritance: 30 | 31 | whizbang.http.views.static module 32 | --------------------------------- 33 | 34 | .. automodule:: whizbang.http.views.static 35 | :members: 36 | :undoc-members: 37 | :show-inheritance: 38 | 39 | whizbang.http.views.template module 40 | ----------------------------------- 41 | 42 | .. automodule:: whizbang.http.views.template 43 | :members: 44 | :undoc-members: 45 | :show-inheritance: 46 | 47 | 48 | Module contents 49 | --------------- 50 | 51 | .. automodule:: whizbang.http.views 52 | :members: 53 | :undoc-members: 54 | :show-inheritance: 55 | -------------------------------------------------------------------------------- /docs/whizbang.kvs.rst: -------------------------------------------------------------------------------- 1 | whizbang.kvs package 2 | ==================== 3 | 4 | Submodules 5 | ---------- 6 | 7 | whizbang.kvs.client module 8 | -------------------------- 9 | 10 | .. automodule:: whizbang.kvs.client 11 | :members: 12 | :undoc-members: 13 | :show-inheritance: 14 | 15 | whizbang.kvs.message module 16 | --------------------------- 17 | 18 | .. automodule:: whizbang.kvs.message 19 | :members: 20 | :undoc-members: 21 | :show-inheritance: 22 | 23 | whizbang.kvs.server module 24 | -------------------------- 25 | 26 | .. automodule:: whizbang.kvs.server 27 | :members: 28 | :undoc-members: 29 | :show-inheritance: 30 | 31 | 32 | Module contents 33 | --------------- 34 | 35 | .. automodule:: whizbang.kvs 36 | :members: 37 | :undoc-members: 38 | :show-inheritance: 39 | -------------------------------------------------------------------------------- /docs/whizbang.queue.config.rst: -------------------------------------------------------------------------------- 1 | whizbang.queue.config package 2 | ============================= 3 | 4 | Submodules 5 | ---------- 6 | 7 | whizbang.queue.config.settings module 8 | ------------------------------------- 9 | 10 | .. automodule:: whizbang.queue.config.settings 11 | :members: 12 | :undoc-members: 13 | :show-inheritance: 14 | 15 | 16 | Module contents 17 | --------------- 18 | 19 | .. automodule:: whizbang.queue.config 20 | :members: 21 | :undoc-members: 22 | :show-inheritance: 23 | -------------------------------------------------------------------------------- /docs/whizbang.queue.rst: -------------------------------------------------------------------------------- 1 | whizbang.queue package 2 | ====================== 3 | 4 | Subpackages 5 | ----------- 6 | 7 | .. toctree:: 8 | 9 | whizbang.queue.config 10 | whizbang.queue.util 11 | 12 | Submodules 13 | ---------- 14 | 15 | whizbang.queue.message module 16 | ----------------------------- 17 | 18 | .. automodule:: whizbang.queue.message 19 | :members: 20 | :undoc-members: 21 | :show-inheritance: 22 | 23 | whizbang.queue.monitor module 24 | ----------------------------- 25 | 26 | .. automodule:: whizbang.queue.monitor 27 | :members: 28 | :undoc-members: 29 | :show-inheritance: 30 | 31 | whizbang.queue.queue module 32 | --------------------------- 33 | 34 | .. automodule:: whizbang.queue.queue 35 | :members: 36 | :undoc-members: 37 | :show-inheritance: 38 | 39 | whizbang.queue.serialization module 40 | ----------------------------------- 41 | 42 | .. automodule:: whizbang.queue.serialization 43 | :members: 44 | :undoc-members: 45 | :show-inheritance: 46 | 47 | whizbang.queue.worker module 48 | ---------------------------- 49 | 50 | .. automodule:: whizbang.queue.worker 51 | :members: 52 | :undoc-members: 53 | :show-inheritance: 54 | 55 | 56 | Module contents 57 | --------------- 58 | 59 | .. automodule:: whizbang.queue 60 | :members: 61 | :undoc-members: 62 | :show-inheritance: 63 | -------------------------------------------------------------------------------- /docs/whizbang.queue.util.rst: -------------------------------------------------------------------------------- 1 | whizbang.queue.util package 2 | =========================== 3 | 4 | Submodules 5 | ---------- 6 | 7 | whizbang.queue.util.xrange_helper module 8 | ---------------------------------------- 9 | 10 | .. automodule:: whizbang.queue.util.xrange_helper 11 | :members: 12 | :undoc-members: 13 | :show-inheritance: 14 | 15 | 16 | Module contents 17 | --------------- 18 | 19 | .. automodule:: whizbang.queue.util 20 | :members: 21 | :undoc-members: 22 | :show-inheritance: 23 | -------------------------------------------------------------------------------- /docs/whizbang.rst: -------------------------------------------------------------------------------- 1 | whizbang package 2 | ================ 3 | 4 | Subpackages 5 | ----------- 6 | 7 | .. toctree:: 8 | 9 | whizbang.http 10 | whizbang.kvs 11 | whizbang.queue 12 | 13 | Module contents 14 | --------------- 15 | 16 | .. automodule:: whizbang 17 | :members: 18 | :undoc-members: 19 | :show-inheritance: 20 | -------------------------------------------------------------------------------- /examples/twooter/db.sqlite3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jeffknupp/omega/922ceb42011ae22ae0798fd333f2286634451ca4/examples/twooter/db.sqlite3 -------------------------------------------------------------------------------- /examples/twooter/models.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | 3 | from omega.http.orm import ( 4 | Model, 5 | Column, 6 | String, 7 | DateTime, 8 | Integer, 9 | relationship, 10 | ForeignKey, 11 | ) 12 | 13 | 14 | class User(Model): 15 | """A Twooter User""" 16 | __tablename__ = 'user' 17 | 18 | id = Column(Integer, primary_key=True) 19 | user_name = Column(String) 20 | joined_at = Column(DateTime, default=datetime.datetime.now()) 21 | 22 | 23 | class Twoot(Model): 24 | """A Twoot message""" 25 | __tablename__ = 'twoot' 26 | 27 | id = Column(Integer, primary_key=True) 28 | content = Column(String) 29 | posted_at = Column(DateTime, default=datetime.datetime.now()) 30 | user_id = Column(Integer, ForeignKey('user.id')) 31 | user = relationship(User) 32 | -------------------------------------------------------------------------------- /examples/twooter/requirements.txt: -------------------------------------------------------------------------------- 1 | Flask==0.10.1 2 | Flask-SQLAlchemy==1.0 3 | Flask-WTF==0.9.5 4 | Jinja2==2.7.3 5 | MarkupSafe==0.23 6 | SQLAlchemy==0.9.4 7 | WTForms==1.0.5 8 | Werkzeug==0.9.6 9 | gevent==1.0.1 10 | gevent-socketio==0.3.6 11 | gevent-websocket==0.9.3 12 | greenlet==0.4.2 13 | itsdangerous==0.24 14 | psycopg2==2.5.3 15 | pyzmq==14.3.1 16 | wsgiref==0.1.2 17 | -------------------------------------------------------------------------------- /examples/twooter/resources/twoot.js: -------------------------------------------------------------------------------- 1 | { 2 | "user": "*user", 3 | "content": "This is awesome", 4 | "posted_at": "2014-01-03 01:02:03" 5 | } 6 | -------------------------------------------------------------------------------- /examples/twooter/runserver.py: -------------------------------------------------------------------------------- 1 | """Code for the Twooter application.""" 2 | from werkzeug import Response 3 | from omega.http.core import create_app 4 | from sqlalchemy import create_engine 5 | from gevent import monkey 6 | monkey.patch_all() 7 | from models import Twoot, User 8 | from sockets import ChatNamespace 9 | 10 | 11 | app = create_app(__name__) 12 | 13 | 14 | @app.route('/chat', methods=['POST']) 15 | def chat(request): 16 | """Route chat posts to the *chat* handler function. Broadcast the message 17 | to all users.""" 18 | message = '{}: {}'.format(request.form['user'], request.form['message']) 19 | if message: 20 | ChatNamespace.broadcast('message', message) 21 | return Response() 22 | 23 | 24 | if __name__ == '__main__': 25 | # Set the SQLAlchemy database engine 26 | app.engine(create_engine( 27 | 'postgresql+psycopg2://jknupp@localhost/omega')) 28 | # Auto-generate an ORM resource view from the given models 29 | app.orm_resource(Twoot) 30 | app.orm_resource(User) 31 | # Create a socketio endpoint for chatting 32 | app.namespace('/chats', ChatNamespace) 33 | # Auto generate a home page for the project 34 | app.auto_generate_home() 35 | app.run(debug=True) 36 | -------------------------------------------------------------------------------- /examples/twooter/sockets.py: -------------------------------------------------------------------------------- 1 | """SocketIO implementation for Chatting. Requires very small amount of code.""" 2 | 3 | from socketio.namespace import BaseNamespace 4 | 5 | 6 | class ChatNamespace(BaseNamespace): 7 | """A namespace to handle chatting.""" 8 | sockets = {} 9 | 10 | def on_chat(self, msg): 11 | """Act on a *chat* event.""" 12 | ChatNamespace.emit('chat', msg) 13 | 14 | def recv_connect(self): 15 | """Add the connected socket to the given socket.""" 16 | print "Got a socket connection" 17 | self.sockets[id(self)] = self 18 | 19 | def disconnect(self, *args, **kwargs): 20 | """Remove the given socket from the list.""" 21 | print "Got a socket disconnection" 22 | if id(self) in self.sockets: 23 | del self.sockets[id(self)] 24 | super(ChatNamespace, self).disconnect(*args, **kwargs) 25 | 26 | @classmethod 27 | def broadcast(cls, event, message): 28 | """Broadcast the message to all connected sockets.""" 29 | for write_socket in cls.sockets.values(): 30 | write_socket.emit(event, message) 31 | -------------------------------------------------------------------------------- /images/home.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jeffknupp/omega/922ceb42011ae22ae0798fd333f2286634451ca4/images/home.png -------------------------------------------------------------------------------- /images/resource.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jeffknupp/omega/922ceb42011ae22ae0798fd333f2286634451ca4/images/resource.png -------------------------------------------------------------------------------- /images/resources.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jeffknupp/omega/922ceb42011ae22ae0798fd333f2286634451ca4/images/resources.png -------------------------------------------------------------------------------- /omega/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jeffknupp/omega/922ceb42011ae22ae0798fd333f2286634451ca4/omega/__init__.py -------------------------------------------------------------------------------- /omega/http/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jeffknupp/omega/922ceb42011ae22ae0798fd333f2286634451ca4/omega/http/__init__.py -------------------------------------------------------------------------------- /omega/http/cache.py: -------------------------------------------------------------------------------- 1 | class DummyCache(dict): 2 | pass 3 | -------------------------------------------------------------------------------- /omega/http/core.py: -------------------------------------------------------------------------------- 1 | """Omega's core HTTP module. 2 | 3 | Application objects are created by instantiating the 4 | :class:`omega.http.core.WebApplication` class. 5 | """ 6 | from sqlalchemy.orm import sessionmaker 7 | 8 | from werkzeug.wrappers import Request, Response 9 | from werkzeug.routing import Map, Rule 10 | from werkzeug import run_simple 11 | 12 | from omega.http.views.template import TemplateView 13 | from omega.http.views.static import StaticFileView 14 | from omega.http.views.nosql import NoSQLResourceView 15 | from omega.http.views.orm import ORMResourceView 16 | from omega.http.views.generated import GeneratedIndexView 17 | from omega.http.orm import Model 18 | from werkzeug.wsgi import SharedDataMiddleware 19 | 20 | from socketio.server import SocketIOServer 21 | from socketio import socketio_manage 22 | 23 | 24 | class WebApplication(object): 25 | """A WSGI application object responsible for handling requests. 26 | 27 | :param name: The name of the application. 28 | :param url_map: A map of :class:`werzeug.routing.Rule` objects to endpoint 29 | names. 30 | :param debug: True if the application is in debug mode. 31 | """ 32 | 33 | def __init__(self, name): 34 | """Return a new application object with built-in static file routing 35 | already setup. 36 | 37 | TODO: Make static directory configurable. 38 | 39 | :param str name: The name of the application. 40 | """ 41 | self.name = name 42 | self.url_map = Map([Rule('/static/', endpoint='static')]) 43 | self._routes = {'static': StaticFileView()} 44 | self._engine = None 45 | self.debug = None 46 | self._orm_resources = [] 47 | self._namespaces = {} 48 | 49 | def page(self, endpoint, template_name): 50 | """Register a template to be served for the specified *endpoint*. 51 | 52 | :param str endpoint: The endpoint (i.e. '/home') this template should be served at. 53 | :param str template_name: The name of the template to be rendered. 54 | """ 55 | name = template_name.split('.')[0] 56 | self.url_map.add(Rule(endpoint, endpoint=name)) 57 | self._routes[name] = TemplateView(template_name) 58 | 59 | def nosql_resource(self, resource): 60 | """Register a NoSQL resource.""" 61 | self.url_map.add(Rule('/' + resource.name, endpoint=resource.name)) 62 | self.url_map.add(Rule('/' + resource.name + '/', endpoint=resource.name)) 63 | self._routes[resource.name] = NoSQLResourceView(resource) 64 | 65 | def orm_resource(self, cls, resource_view_class=ORMResourceView): 66 | """Register an ORM resource at an endpoint defined by the class's 67 | *__endpoint__* attribute. 68 | 69 | A reasonably complete set of RESTful endpoints are created/supported, 70 | with support for all HTTP methods as well as form-based manipulation of 71 | resources. 72 | 73 | :param cls: The ORM class to register. 74 | :type cls: :class:`omega.http.orm.model.Resource` 75 | :param resource_view_class: The class to generate the resource's views 76 | from. This should be a subclass of 77 | :class:`omega.http.views.orm.ORMResourceView` 78 | """ 79 | name = cls.endpoint()[1:] 80 | self._orm_resources.append(name) 81 | orm_view = resource_view_class(name, cls, self.Session) 82 | self.url_map.add(Rule('/' + name, endpoint='get_{}s'.format(name), methods=['GET'])) 83 | self.url_map.add(Rule('/' + name, endpoint='post_{}'.format(name), methods=['POST'])) 84 | self.url_map.add(Rule('/' + name + '/', endpoint='get_{}'.format(name), methods=['GET'])) 85 | self.url_map.add(Rule('/' + name + '/', endpoint='put_{}'.format(name), methods=['PUT'])) 86 | self.url_map.add(Rule('/' + name + '/', endpoint='patch_{}'.format(name), methods=['PATCH'])) 87 | self.url_map.add(Rule('/' + name + '/', endpoint='delete_{}'.format(name), methods=['DELETE'])) 88 | self.url_map.add(Rule('/' + name + '//delete', endpoint='delete_{}'.format(name), methods=['POST'])) 89 | self.url_map.add(Rule('/' + name + '//edit', endpoint='post_{}'.format(name), methods=['POST'])) 90 | self._routes['get_{}s'.format(name)] = orm_view.handle_get_collection 91 | self._routes['get_{}'.format(name)] = orm_view.handle_get 92 | self._routes['put_{}'.format(name)] = orm_view.handle_put 93 | self._routes['patch_{}'.format(name)] = orm_view.handle_patch 94 | self._routes['delete_{}'.format(name)] = orm_view.handle_delete 95 | self._routes['post_{}'.format(name)] = orm_view.handle_post 96 | 97 | def namespace(self, endpoint, namespace_class): 98 | """Register a socketio namespace class at the given *endpoint*. 99 | 100 | :param str endpoint: The endpoint (i.e. '/chat') to register. 101 | :param namespace_class: SocketIO Namespace to register at the 102 | given endpoint. 103 | :type namespace_class: :class:`socketio.namespace.BaseNamespace` 104 | """ 105 | self._namespaces[endpoint] = namespace_class 106 | 107 | def auto_generate_home(self): 108 | """Auto-generate an index page that lists all the ORM resources 109 | registered.""" 110 | self.url_map.add(Rule('/', endpoint='home')) 111 | self._routes['home'] = GeneratedIndexView(self._orm_resources) 112 | 113 | def route(self, rule, **kwargs): 114 | """Register *function* to handle requests to *endpoint*. 115 | 116 | :param str endpoint: The endpoint (i.e. `/foo/bar`) to register. 117 | :param function: The view function to handle requests. 118 | """ 119 | def decorator(f): 120 | methods = kwargs.pop('methods', None) 121 | self.url_map.add(Rule( 122 | rule, 123 | endpoint=rule, 124 | methods=methods, 125 | *kwargs)) 126 | self._routes[rule] = f 127 | return f 128 | return decorator 129 | 130 | def dispatch_request(self, urls, request): 131 | """Dispatch the incoming *request* to the given view function or class. 132 | 133 | If the "request" is a socket.io message, route it to the registered 134 | namespace. 135 | 136 | :param urls: The URLs parsed by binding the URL map to the environment. 137 | :param request: The current HTTP request. 138 | :type request :class:`werkzeug.Request`: 139 | """ 140 | if request.path.startswith('/socket.io'): 141 | try: 142 | socketio_manage(request.environ, self._namespaces, request) 143 | except: 144 | print("Exception while handling socketio connection") 145 | return Response() 146 | else: 147 | response = urls.dispatch( 148 | lambda e, v: self._routes[e]( 149 | request, **v)) 150 | if isinstance(response, (unicode, str)): 151 | headers = {'Content-type': 'text/html'} 152 | response = Response(response, headers=headers) 153 | if not response: 154 | headers = {'Content-type': 'text/html'} 155 | response = Response('404 Not Found', headers=headers) 156 | response.status_code = 404 157 | return response 158 | 159 | def wsgi_app(self, environ, start_response): 160 | """Return a response to the request described by *environ*. 161 | 162 | Invokes the application object as a WSGI application and returns an 163 | HTTP Response. 164 | 165 | :param environ: The current environment variables. 166 | :param start_response: A callable to provide the response. 167 | """ 168 | request = Request(environ) 169 | urls = self.url_map.bind_to_environ(environ) 170 | response = self.dispatch_request(urls, request) 171 | return response(environ, start_response) 172 | 173 | def engine(self, engine): 174 | """Set the application's database engine to be *engine*. 175 | 176 | :param engine: The SQLAlchemy engine object to bind. 177 | """ 178 | self._engine = engine 179 | self.Session = sessionmaker(bind=engine) 180 | Model.metadata.create_all(self._engine) 181 | 182 | def run(self, host='127.0.0.1', port=5000, debug=None): 183 | """Run the application at the given host and port. 184 | 185 | :param str host: The hostname to run the application on. 186 | :param int port: The port to run the application on. 187 | """ 188 | 189 | if debug is not None: 190 | self.debug = debug 191 | SocketIOServer( 192 | ('0.0.0.0', 5000), 193 | SharedDataMiddleware(self, {}), 194 | resource="socket.io", 195 | policy_server=False).serve_forever() 196 | 197 | def __call__(self, environ, start_response): 198 | return self.wsgi_app(environ, start_response) 199 | 200 | def create_app(name): 201 | return WebApplication(name) 202 | -------------------------------------------------------------------------------- /omega/http/orm/__init__.py: -------------------------------------------------------------------------------- 1 | from sqlalchemy import Column, Integer, DateTime, Float, String, ForeignKey, create_engine 2 | from sqlalchemy.orm import (relationship, backref) 3 | from omega.http.orm.model import Model 4 | 5 | __all__ = [ 6 | 'Model', 'Column', 'Integer', 'DateTime', 'Double', 7 | 'String', 'relationship', 'ForeignKey'] 8 | -------------------------------------------------------------------------------- /omega/http/orm/model.py: -------------------------------------------------------------------------------- 1 | """Omega's ORM module.""" 2 | from sqlalchemy.ext.declarative import declarative_base 3 | 4 | 5 | class Resource(object): 6 | __endpoint__ = None 7 | 8 | def to_json(self): 9 | values = {} 10 | for column in self.__table__.columns: 11 | values[column] = getattr(self, column) 12 | return values 13 | 14 | def __iter__(self): 15 | for column in self.__table__.columns: 16 | yield (column.name, getattr(self, column.name)) 17 | 18 | @classmethod 19 | def endpoint(cls): 20 | return cls.__endpoint__ or '/' + cls.__tablename__ 21 | 22 | def url(self): 23 | url = self.endpoint() + '/' 24 | url += str(getattr(self, self.__table__.primary_key.columns.values()[0].name)) 25 | return url 26 | 27 | Model = declarative_base(cls=(Resource,)) 28 | -------------------------------------------------------------------------------- /omega/http/utils.py: -------------------------------------------------------------------------------- 1 | """Utilities for the omega http module.""" 2 | 3 | from werkzeug import Response 4 | 5 | 6 | def mimetype_for_path(path): 7 | """Return the Internet Type for the given path. 8 | 9 | :param str path: Path to detect type for 10 | 11 | """ 12 | 13 | if '.css' in path: 14 | return 'text/css' 15 | elif '.js' in path: 16 | return 'script/js' 17 | 18 | 19 | def make_response(content): 20 | """Return a Response object suitable for return. 21 | 22 | Note: This currently always returns HTML responses. For a JSON response, 23 | override the Content-type header to be 'application/json'. 24 | 25 | :param content: Content to be returned by the response 26 | 27 | """ 28 | return Response(content, headers={'Content-type': 'text/html'}) 29 | -------------------------------------------------------------------------------- /omega/http/views/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jeffknupp/omega/922ceb42011ae22ae0798fd333f2286634451ca4/omega/http/views/__init__.py -------------------------------------------------------------------------------- /omega/http/views/generated.py: -------------------------------------------------------------------------------- 1 | """Module responsible for generating an "index" view (root page, or home page, 2 | view) from available routes.""" 3 | from jinja2 import Environment, PackageLoader 4 | from omega.http.utils import make_response 5 | 6 | 7 | class GeneratedIndexView(object): 8 | """A view that renders the index.html template with all the registered 9 | routes as context data.""" 10 | def __init__(self, routes): 11 | self.routes = routes 12 | self.env = Environment(loader=PackageLoader('omega.http.views')) 13 | self.template = self.env.get_template('index.html') 14 | 15 | def __call__(self, request): 16 | return make_response(self.template.render(routes=self.routes)) 17 | -------------------------------------------------------------------------------- /omega/http/views/nosql.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | import json 3 | import re 4 | import uuid 5 | 6 | from werkzeug import Response 7 | 8 | from omega.kvs.server import KVStore 9 | from omega.http.utils import make_response 10 | 11 | DATE = re.compile("\d+-\d+-\d+ \d+:\d+:\d+") 12 | 13 | 14 | class NoSQLResourceView(object): 15 | """View class for NoSQLResourceView-based resource routing.""" 16 | def __init__(self, resource, store=KVStore()): 17 | self.name = resource.name 18 | self._resource = resource 19 | self._cache = store 20 | self._resources = set() 21 | 22 | def __call__(self, request, primary_key=None): 23 | if request.method == 'POST': 24 | return self.handle_post(request) 25 | elif request.method == 'PUT': 26 | return self.handle_put(request, primary_key) 27 | elif request.method == 'PATCH': 28 | return self.handle_patch(request, primary_key) 29 | elif request.method == 'DELETE': 30 | return self.handle_delete(request, primary_key) 31 | elif request.method == 'HEAD': 32 | return self.handle_head(request) 33 | elif request.method == 'OPTIONS': 34 | return self.handle_options(request) 35 | 36 | elif request.method == 'GET': 37 | if not primary_key: 38 | response = Response( 39 | self.all_resources(), 40 | headers={'Content-type': 'application/json'}) 41 | else: 42 | response = Response( 43 | json.dumps(self._cache.get(primary_key)), 44 | headers={'Content-type': 'application/json'}) 45 | return response 46 | 47 | def handle_post(self, request): 48 | """Return a :class:`werkzeug.Response` object after handling the POST 49 | call.""" 50 | try: 51 | resource = json.loads(request.data) 52 | except ValueError: 53 | return bad_request('No JSON object detected') 54 | error = self.validate_data(resource) 55 | if error: 56 | return bad_request(error) 57 | resource_id = str(uuid.uuid4()) 58 | resource['_id'] = resource_id 59 | self._resources.add(resource_id) 60 | self._cache.put(resource_id, resource) 61 | return Response( 62 | json.dumps(to_json(resource)), 63 | headers={'Content-type': 'application/json'}) 64 | 65 | def handle_put(self, request, primary_key): 66 | """Return a :class:`werkzeug.Response` object after handling the PUT 67 | call.""" 68 | resource = self._cache.get(primary_key) 69 | resource.update(json.loads(request.data)) 70 | error = self.validate_data(resource) 71 | if error: 72 | return bad_request(error) 73 | self._cache.put(primary_key, resource) 74 | return Response( 75 | json.dumps(to_json(resource)), 76 | headers={'Content-type': 'application/json'}) 77 | 78 | def handle_patch(self, request, primary_key): 79 | """Return a :class:`werkzeug.Response` object after handling the PUT 80 | call.""" 81 | resource = self._cache.get(primary_key) 82 | resource.update(json.loads(request.data)) 83 | self._cache.put(primary_key, resource) 84 | return Response( 85 | json.dumps(to_json(resource)), 86 | headers={'Content-type': 'application/json'}) 87 | 88 | def handle_delete(self, request, primary_key): 89 | """Return a :class:`werkzeug.Response` object after handling 90 | the DELETE call.""" 91 | self._cache.delete(primary_key) 92 | return Response( 93 | ('204 No Content'), headers={'Content-type': 'application/json'}) 94 | 95 | def validate_against_definition(self, fields): 96 | """Return an error message if there are extra or missing fields on the 97 | message.""" 98 | defined_fields = set(self._resource.fields.keys()) 99 | if fields - defined_fields: 100 | return 'Unknown fields [{}]'.format( 101 | ', '.join([f for f in fields - defined_fields])) 102 | if defined_fields - fields: 103 | return 'Missing fields [{}]'.format( 104 | ', '.join([f for f in defined_fields - fields])) 105 | 106 | def validate_data(self, resource): 107 | """Return an error message if the given resource's data is 108 | not valid based on our definition.""" 109 | fields = set(resource.keys()) 110 | error = self.validate_against_definition(fields) 111 | if error: 112 | return error 113 | for field, value in resource.items(): 114 | if field not in self._fields.keys: 115 | return 'Unknown field [{}]'.format(field) 116 | if isinstance(value, (str, unicode)): 117 | match = re.match(DATE, value) 118 | if match and match.group(0): 119 | resource[field] = datetime.datetime.strptime( 120 | value, '%Y-%m-%d %H:%M:%S') 121 | value = resource[field] 122 | if not isinstance(value, self._definition[field]['type']): 123 | return 'Invalid type for field [{}]. Expected [{}]' \ 124 | 'but got [{}]'.format( 125 | field, 126 | self._definition[field]['type'], 127 | type(value)) 128 | 129 | def all_resources(self): 130 | """Return all resources as a list of JSON objects.""" 131 | return json.dumps( 132 | [to_json(self._cache.get(r)) for r in self._resources]) 133 | 134 | 135 | def to_json(resource): 136 | """Return the JSON representation of the resource.""" 137 | values = resource 138 | for field, value in resource.items(): 139 | if isinstance(value, datetime.datetime): 140 | values[field] = str(value) 141 | return values 142 | 143 | 144 | def bad_request(message): 145 | response = make_response(json.dumps({'status': 'ERR', 'message': message})) 146 | response.status_code = 400 147 | response.headers = {'Content-type': 'application/json'} 148 | return response 149 | -------------------------------------------------------------------------------- /omega/http/views/orm.py: -------------------------------------------------------------------------------- 1 | """Module containing code to automatically generate RESTful API from an ORM 2 | model.""" 3 | from jinja2 import Environment, PackageLoader 4 | from werkzeug import redirect 5 | from wtforms.ext.sqlalchemy.orm import model_form 6 | 7 | from omega.http.utils import make_response 8 | 9 | 10 | class ORMResourceView(object): 11 | """Generates a REST API from an ORM model.""" 12 | 13 | def __init__(self, name, cls, session): 14 | self.name = name 15 | self.cls = cls 16 | self._session = session() 17 | self.form = model_form(self.cls, self._session) 18 | self.env = Environment(loader=PackageLoader('omega.http.views')) 19 | 20 | def handle_get(self, request, primary_key): 21 | """Return a response for a get request for a specific resource. 22 | 23 | :param request: The incoming Request object 24 | :param primary_key: The primary_key of the ORM model to retrieve 25 | 26 | """ 27 | 28 | resource = self._session.query(self.cls).get(primary_key) 29 | if not resource: 30 | return None 31 | self.template = self.env.get_template('resource.html') 32 | form = self.form(request.form, resource) 33 | return make_response(self.template.render( 34 | resource=resource, name=self.name, form=form)) 35 | 36 | def handle_get_collection(self, request): 37 | """Return a response for a get request on the entire collection. 38 | 39 | :param request: The incoming Request object. 40 | 41 | """ 42 | 43 | resources = self._session.query(self.cls).all() 44 | self.template = self.env.get_template('resources.html') 45 | 46 | return make_response( 47 | self.template.render( 48 | resources=resources, form=self.form(), name=self.cls.__name__)) 49 | 50 | def handle_post(self, request, primary_key=None): 51 | """Return a :class:`werkzeug.Response` object after handling the POST 52 | call. 53 | 54 | :param request: The incoming Request object. 55 | :param primary_key: The primary_key of the ORM model to retrieve 56 | 57 | """ 58 | 59 | if primary_key: 60 | print 'pirmary_key' 61 | resource = self._session.query(self.cls).get(primary_key) 62 | form = self.form(request.form, obj=resource) 63 | if form.validate(): 64 | form.populate_obj(resource) 65 | resource = self._session.merge(resource) 66 | self._session.commit() 67 | else: 68 | print 'cls' 69 | resource = self.cls() 70 | form = self.form(request.form, obj=resource) 71 | form.populate_obj(resource) 72 | self._session.add(resource) 73 | self._session.commit() 74 | return redirect(resource.url()) 75 | 76 | def handle_put(self, request, primary_key): 77 | """Return a :class:`werkzeug.Response` object after handling the PUT 78 | call. 79 | 80 | :param request: The incoming Request object. 81 | :param primary_key: The primary_key of the ORM model to retrieve 82 | 83 | """ 84 | 85 | resource = self._session.query(self.cls).get(primary_key) 86 | if resource: 87 | for field, value in request.form.items(): 88 | setattr(resource, field, value) 89 | else: 90 | resource = self.cls(request.form) 91 | error = self.validate_data(resource) 92 | if error: 93 | return None 94 | self._session.add(resource) 95 | self._session.commit() 96 | 97 | def handle_patch(self, request, primary_key): 98 | """Return a :class:`werkzeug.Response` object after handling the PUT 99 | call. 100 | 101 | :param request: The incoming Request object. 102 | :param primary_key: The primary_key of the ORM model to retrieve 103 | 104 | """ 105 | 106 | resource = self._session.query(self.cls).get(primary_key) 107 | if resource: 108 | for field, value in request.form.items(): 109 | setattr(resource, field, value) 110 | else: 111 | resource = self.cls(request.form) 112 | error = self.validate_data(resource) 113 | if error: 114 | return None 115 | self._session.add(resource) 116 | self._session.commit() 117 | template = self.env.get_template('resource.html') 118 | 119 | return make_response(template.render(resource=resource)) 120 | 121 | def handle_delete(self, request, primary_key): 122 | """Return a :class:`werkzeug.Response` object after handling 123 | the DELETE call or a POST to /delete. 124 | 125 | :param request: The incoming Request object. 126 | :param primary_key: The primary_key of the ORM model to retrieve 127 | 128 | """ 129 | 130 | resource = self._session.query(self.cls).get(primary_key) 131 | self._session.delete(resource) 132 | self._session.commit() 133 | return redirect('/' + self.name) 134 | 135 | def validate_data(self, resource): 136 | """Return an error message if the given resource's data is 137 | not valid based on our definition. 138 | 139 | :param resource: The resource to be validated 140 | 141 | """ 142 | for name, value in self.cls.__table__.columns.items(): 143 | if getattr(resource, name) != value.type: 144 | return False 145 | return True 146 | -------------------------------------------------------------------------------- /omega/http/views/static.py: -------------------------------------------------------------------------------- 1 | """Module containing code to serve static files.""" 2 | import os 3 | from omega.http.utils import make_response, mimetype_for_path 4 | 5 | 6 | class StaticFileView(object): 7 | """View class for static file routing.""" 8 | def __call__(self, *args, **kwargs): 9 | file_path = kwargs['file_path'] 10 | with open(os.path.join('static', file_path), 'r') as file_handle: 11 | response = make_response(file_handle.read()) 12 | response.headers['Content-type'] = mimetype_for_path(file_path) 13 | return response 14 | -------------------------------------------------------------------------------- /omega/http/views/template.py: -------------------------------------------------------------------------------- 1 | """Module for creating views that simply render a template.""" 2 | 3 | from jinja2 import Environment, FileSystemLoader 4 | from omega.http.utils import make_response 5 | 6 | 7 | class TemplateView(object): 8 | """View class for direct-to-template routing.""" 9 | def __init__(self, template_name): 10 | self.env = Environment(loader=FileSystemLoader('templates')) 11 | self.template = self.env.get_template(template_name) 12 | 13 | def __call__(self, request): 14 | return make_response(self.template.render()) 15 | -------------------------------------------------------------------------------- /omega/http/views/templates/base.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | Omega! 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 65 | 91 | 92 | 96 | {% block head %} 97 | {% endblock head %} 98 | 99 | 100 |
101 |

OMEGA!

102 |
103 |
104 |
105 | {% block breadcrumb %} 106 |

Home

107 | {% endblock breadcrumb %} 108 |
109 | {% if messages %} 110 |
111 | Error: 112 |
    113 | {% for message in messages %} 114 |
  • {{message}}
    • 115 | {% endfor %} 116 |
117 |
118 | {% endif %} 119 | 120 | 121 | {% block content %} 122 | {% endblock %} 123 | {% block extra_body %} 124 | {% endblock extra_body %} 125 |
126 |
127 |
128 |

Live User Chat

129 |
130 | 131 |
132 |
133 |
134 |
135 | 136 |
137 |
138 | 139 |
140 | 141 |
142 |
143 |
144 | 145 | 146 | -------------------------------------------------------------------------------- /omega/http/views/templates/index.html: -------------------------------------------------------------------------------- 1 | {% extends "base.html" %} 2 | 3 | {% block content %} 4 |

Available Routes

5 |
    6 | {% for route in routes %} 7 |
    8 |
  • 9 |

    /{{ route }}

    10 |
    • 11 |
    12 | {% endfor %} 13 |
14 | {% endblock content %} 15 | -------------------------------------------------------------------------------- /omega/http/views/templates/resource.html: -------------------------------------------------------------------------------- 1 | {% extends "base.html" %} 2 | 3 | {% block breadcrumb %} 4 |

Home / {{ name }}

5 | {% endblock breadcrumb %} 6 | {% block content %} 7 | 8 |

{{ name|capitalize }}

9 |

list all {{ name }}s

10 |
11 |
12 | {% for field in form %} 13 |
14 | {{ field.label }} 15 | {{ field }} 16 |
17 | {% endfor %} 18 |
19 | 20 |
21 |
22 | 23 |
24 | 25 |
26 | {% endblock %} 27 | -------------------------------------------------------------------------------- /omega/http/views/templates/resources.html: -------------------------------------------------------------------------------- 1 | {% extends "base.html" %} 2 | 3 | {% block breadcrumb %} 4 |

Home / {{ name }}

5 | {% endblock breadcrumb %} 6 | 7 | {% block content %} 8 | 9 |

All {{ name }}s

10 |
11 |

Create New {{ name }}

12 |
13 | {% for field in form %} 14 |
15 | {{ field.label }} 16 | {{ field }} 17 |
18 | {% endfor %} 19 | 20 |
21 |
22 | 23 | 24 | 25 | {% for column, value in resources[0] %} 26 | 27 | {% endfor %} 28 | 29 | 30 | {% for resource in resources %} 31 | 32 | {% for column, value in resource %} 33 | 34 | {% endfor %} 35 | 36 | {% endfor %} 37 | 38 |
Go{{ column }}
{{ value }}
39 | 40 | {% endblock content %} 41 | -------------------------------------------------------------------------------- /omega/kvs/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jeffknupp/omega/922ceb42011ae22ae0798fd333f2286634451ca4/omega/kvs/__init__.py -------------------------------------------------------------------------------- /omega/kvs/client.py: -------------------------------------------------------------------------------- 1 | import zmq 2 | 3 | 4 | class Client(object): 5 | 6 | def __init__(self, port): 7 | context = zmq.Context() 8 | self._socket = context.socket(zmq.REQ) 9 | self._socket.connect('tcp://localhost:{}'.format(port)) 10 | 11 | def get(self, key): 12 | message = {'command': 'GET', 'value': key} 13 | self._socket.send_json(message) 14 | response = self._socket.recv_json() 15 | return response 16 | 17 | def put(self, key, value): 18 | message = {'command': 'PUT', 'key': key, 'value': value} 19 | self._socket.send_json(message) 20 | response = self._socket.recv_json() 21 | return response 22 | 23 | 24 | if __name__ == '__main__': 25 | c = Client(9090) 26 | print c.put('foo', 'bar') 27 | print c.get('foo') 28 | -------------------------------------------------------------------------------- /omega/kvs/server.py: -------------------------------------------------------------------------------- 1 | import zmq 2 | 3 | 4 | class KVStore(object): 5 | def __init__(self): 6 | self._store = {} 7 | 8 | def put(self, key, value): 9 | self._store[key] = value 10 | 11 | def get(self, key): 12 | return self._store.get(key, None) 13 | 14 | def delete(self, key): 15 | del self._store[key] 16 | 17 | 18 | class Server(object): 19 | 20 | SUCCESS, FAILURE = range(2) 21 | 22 | def __init__(self, port): 23 | context = zmq.Context() 24 | self.socket = context.socket(zmq.REP) 25 | self.socket.bind('tcp://*:{}'.format(port)) 26 | self._store = KVStore() 27 | self._terminate = False 28 | 29 | def stop(self): 30 | self._terminate = True 31 | 32 | def start(self): 33 | poll = zmq.Poller() 34 | poll.register(self.socket, zmq.POLLIN) 35 | while not self._terminate: 36 | events = poll.poll(1000) 37 | if events: 38 | message = self.socket.recv_json() 39 | if message['command'] == 'GET': 40 | self.socket.send_json({ 41 | 'status': self.SUCCESS, 42 | 'message': self._store.get(message['value'])}) 43 | elif message['command'] == 'PUT': 44 | key, value = message['key'], message['value'] 45 | self._store.put(key, value) 46 | self.socket.send_json({'status': self.SUCCESS, 47 | 'message': 'Key set'}) 48 | else: 49 | self.socket.send_json({ 50 | 'status': self.FAILURE, 51 | 'message': 'Unknown command [{}]'.format(message['command']) 52 | }) 53 | 54 | if __name__ == '__main__': 55 | s = Server(9090) 56 | s.start() 57 | -------------------------------------------------------------------------------- /omega/kvs/test/app.py: -------------------------------------------------------------------------------- 1 | import threading 2 | 3 | from omega.kvs.server import Server 4 | from omega.kvs.client import Client 5 | 6 | def main(): 7 | server = Server(9090) 8 | server_thread = threading.Thread(target=server.start) 9 | server_thread.start() 10 | 11 | client = Client(9090) 12 | client.put('foo', 'bar') 13 | client.get('foo') 14 | client.get('baz') 15 | server.stop() 16 | server_thread.join() 17 | if __name__ == '__main__': 18 | main() 19 | -------------------------------------------------------------------------------- /omega/kvs/tests/test_kvs.py: -------------------------------------------------------------------------------- 1 | """Tests for the KVStore.""" 2 | 3 | 4 | import threading 5 | 6 | import pytest 7 | 8 | from omega.kvs.server import KVStore, Server 9 | from omega.kvs.client import Client 10 | from omega.kvs.message import CommandMessage, StatusMessage 11 | 12 | @pytest.yield_fixture 13 | def server(): 14 | server = Server(9090) 15 | server_thread = threading.Thread(target=server.start) 16 | server_thread.start() 17 | yield 18 | server.stop() 19 | server_thread.join() 20 | 21 | @pytest.fixture 22 | def client(): 23 | client = Client(9090) 24 | return client 25 | 26 | def test_set_from_client(server, client): 27 | response = client.put('foo', 'bar') 28 | assert response == StatusMessage('OK') 29 | -------------------------------------------------------------------------------- /omega/log/README.md: -------------------------------------------------------------------------------- 1 | Placeholder for `log` application 2 | -------------------------------------------------------------------------------- /omega/queue/README.md: -------------------------------------------------------------------------------- 1 | Placeholder for 'queue' application 2 | -------------------------------------------------------------------------------- /omega/queue/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jeffknupp/omega/922ceb42011ae22ae0798fd333f2286634451ca4/omega/queue/__init__.py -------------------------------------------------------------------------------- /omega/queue/config/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jeffknupp/omega/922ceb42011ae22ae0798fd333f2286634451ca4/omega/queue/config/__init__.py -------------------------------------------------------------------------------- /omega/queue/config/settings.py: -------------------------------------------------------------------------------- 1 | """Configuration file for brokest.""" 2 | CONFIG = { 3 | 'front-end-host': '127.0.0.1', 4 | 'front-end-port': 7070, 5 | 'back-end-host': '127.0.0.1', 6 | 'back-end-port': 7080, 7 | 'monitor-host': '127.0.0.1', 8 | 'monitor-port': 7090, 9 | } 10 | -------------------------------------------------------------------------------- /omega/queue/message.py: -------------------------------------------------------------------------------- 1 | class Message(object): 2 | """Brokest message structure.""" 3 | 4 | def __init__(self, runnable_string, args, kwargs): 5 | self.runnable_string = runnable_string 6 | self.args = args 7 | self.kwargs = kwargs 8 | -------------------------------------------------------------------------------- /omega/queue/monitor.py: -------------------------------------------------------------------------------- 1 | """Monitor client for brokest broker.""" 2 | 3 | import sys 4 | import zmq 5 | 6 | class Monitor(object): 7 | def __init__(self, num_messages=0): 8 | self._num_messages = num_messages 9 | self.socket = zmq.Context().socket(zmq.SUB) 10 | self.socket.connect('tcp://127.0.0.1:9090') 11 | self.socket.setsockopt(zmq.SUBSCRIBE, '') 12 | self._terminate = False 13 | self._seen_messages = 0 14 | 15 | def terminate(self): 16 | self._terminate = True 17 | 18 | def start(self): 19 | while not self._terminate: 20 | if self._num_messages and self._num_messages == self._seen_messages: 21 | break 22 | message = self.socket.recv() 23 | 24 | if __name__ == '__main__': 25 | m = Monitor() 26 | m.start() 27 | -------------------------------------------------------------------------------- /omega/queue/queue.py: -------------------------------------------------------------------------------- 1 | """Brokest client library.""" 2 | from __future__ import absolute_import 3 | import logging 4 | 5 | import zmq 6 | from omega.queue import serialization 7 | 8 | from omega.queue.message import Message 9 | from config.settings import CONFIG 10 | 11 | LOGGER = logging.getLogger(__name__) 12 | logging.basicConfig(level=logging.DEBUG) 13 | 14 | def async(f): 15 | def wrapped(*args, **kwargs): 16 | return f(*args, **kwargs) 17 | wrapped.delay = delay(f) 18 | return wrapped 19 | 20 | class delay(object): 21 | def __init__(self, f): 22 | self._func = f 23 | self.socket = zmq.Context().socket(zmq.REQ) 24 | self.socket.connect('tcp://{}:{}'.format( 25 | CONFIG['front-end-host'], 26 | CONFIG['front-end-port'])) 27 | 28 | def __call__(self, *args, **kwargs): 29 | want_results = kwargs.pop('want_results') 30 | self.queue(self._func, *args, **kwargs) 31 | if not want_results: 32 | self.socket.recv_pyobj() 33 | else: 34 | return Result(self.socket.recv_pyobj) 35 | 36 | def queue(self, runnable, *args, **kwargs): 37 | """Return the result of running the task *runnable* with the given 38 | arguments.""" 39 | message = Message( 40 | serialization.dumps(runnable), 41 | args, 42 | kwargs) 43 | LOGGER.info('Sending [{}] with args[{}] and kwargs[{}] to {}:{}'.format( 44 | runnable, 45 | message.args, 46 | message.kwargs, 47 | CONFIG['front-end-host'], 48 | CONFIG['front-end-port'])) 49 | self.socket.send_pyobj(message) 50 | 51 | 52 | class Result(object): 53 | def __init__(self, runnable): 54 | self._runnable = runnable 55 | 56 | def get(self): 57 | return self._runnable() 58 | 59 | class Broker(object): 60 | def __init__(self): 61 | self._device = zmq.devices.MonitoredQueue(zmq.ROUTER, zmq.DEALER, zmq.PUB) 62 | self._device.bind_in('tcp://{}:{}'.format( 63 | CONFIG['front-end-host'], 64 | CONFIG['front-end-port'])) 65 | self._device.bind_out('tcp://{}:{}'.format( 66 | CONFIG['back-end-host'], 67 | CONFIG['back-end-port'])) 68 | self._device.bind_mon('tcp://{}:{}'.format( 69 | CONFIG['monitor-host'], 70 | CONFIG['monitor-port'])) 71 | 72 | def start(self): 73 | self._device.start() 74 | -------------------------------------------------------------------------------- /omega/queue/serialization.py: -------------------------------------------------------------------------------- 1 | """ 2 | This class is defined to override standard pickle functionality 3 | 4 | The goals of it follow: 5 | -Serialize lambdas and nested functions to compiled byte code 6 | -Deal with main module correctly 7 | -Deal with other non-serializable objects 8 | 9 | It does not include an unpickler, as standard python unpickling suffices 10 | 11 | Copyright (c) 2009-2012 `PiCloud, Inc. `_. All rights reserved. 12 | 13 | email: contact@picloud.com 14 | 15 | The cloud package is free software; you can redistribute it and/or 16 | modify it under the terms of the GNU Lesser General Public 17 | License as published by the Free Software Foundation; either 18 | version 2.1 of the License, or (at your option) any later version. 19 | 20 | This package is distributed in the hope that it will be useful, 21 | but WITHOUT ANY WARRANTY; without even the implied warranty of 22 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 23 | Lesser General Public License for more details. 24 | 25 | You should have received a copy of the GNU Lesser General Public 26 | License along with this package; if not, see 27 | http://www.gnu.org/licenses/lgpl-2.1.html 28 | """ 29 | 30 | 31 | import operator 32 | import os 33 | import pickle 34 | import struct 35 | import sys 36 | import types 37 | from functools import partial 38 | import itertools 39 | from copy_reg import _extension_registry, _inverted_registry, _extension_cache 40 | import new 41 | import dis 42 | import traceback 43 | 44 | #relevant opcodes 45 | STORE_GLOBAL = chr(dis.opname.index('STORE_GLOBAL')) 46 | DELETE_GLOBAL = chr(dis.opname.index('DELETE_GLOBAL')) 47 | LOAD_GLOBAL = chr(dis.opname.index('LOAD_GLOBAL')) 48 | GLOBAL_OPS = [STORE_GLOBAL, DELETE_GLOBAL, LOAD_GLOBAL] 49 | 50 | HAVE_ARGUMENT = chr(dis.HAVE_ARGUMENT) 51 | EXTENDED_ARG = chr(dis.EXTENDED_ARG) 52 | 53 | import logging 54 | cloudLog = logging.getLogger("Cloud.Transport") 55 | 56 | try: 57 | import ctypes 58 | except (MemoryError, ImportError): 59 | logging.warning('Exception raised on importing ctypes. Likely python bug.. some functionality will be disabled', exc_info = True) 60 | ctypes = None 61 | PyObject_HEAD = None 62 | else: 63 | 64 | # for reading internal structures 65 | PyObject_HEAD = [ 66 | ('ob_refcnt', ctypes.c_size_t), 67 | ('ob_type', ctypes.c_void_p), 68 | ] 69 | 70 | 71 | try: 72 | from cStringIO import StringIO 73 | except ImportError: 74 | from StringIO import StringIO 75 | from util import islambda 76 | from util import xrange_helper 77 | 78 | #debug variables intended for developer use: 79 | printSerialization = False 80 | printMemoization = False 81 | 82 | useForcedImports = True # Should I use forced imports for tracking? 83 | 84 | def error_msg(msg, loglevel=logging.WARN, exc_info = 0): 85 | """Print an error message to pilog if running on cloud; otherwise send to stderr""" 86 | from .. import _getcloud 87 | roc = _getcloud().running_on_cloud() 88 | if roc: 89 | pilog_module = __import__('pimployee.log', fromlist=['log']) 90 | pilog_module.pilogger.log(loglevel, msg, exc_info = exc_info) 91 | else: 92 | print >> sys.stderr, msg 93 | if exc_info: 94 | ei = sys.exc_info() 95 | traceback.print_exception(ei[0], ei[1], ei[2], None, sys.stderr) 96 | 97 | class CloudPickler(pickle.Pickler): 98 | 99 | dispatch = pickle.Pickler.dispatch.copy() 100 | savedForceImports = False 101 | savedDjangoEnv = False #hack tro transport django environment 102 | os_env_vars = [] # OS environment variables to copy over 103 | 104 | def __init__(self, file, protocol=None, min_size_to_save= 0): 105 | pickle.Pickler.__init__(self,file,protocol) 106 | self.modules = set() #set of modules needed to depickle 107 | self.globals_ref = {} # map ids to dictionary. used to ensure that functions can share global env 108 | 109 | def dump(self, obj): 110 | # note: not thread safe 111 | # minimal side-effects, so not fixing 112 | recurse_limit = 3000 113 | base_recurse = sys.getrecursionlimit() 114 | if base_recurse < recurse_limit: 115 | sys.setrecursionlimit(recurse_limit) 116 | self.inject_addons() 117 | try: 118 | return pickle.Pickler.dump(self, obj) 119 | except RuntimeError, e: 120 | if 'recursion' in e.args[0]: 121 | msg = """Could not pickle object as excessively deep recursion required. 122 | Try _fast_serialization=2 or contact PiCloud support""" 123 | raise pickle.PicklingError(msg) 124 | raise 125 | finally: 126 | new_recurse = sys.getrecursionlimit() 127 | if new_recurse == recurse_limit: 128 | sys.setrecursionlimit(base_recurse) 129 | 130 | def save_buffer(self, obj): 131 | """Fallback to save_string""" 132 | pickle.Pickler.save_string(self,str(obj)) 133 | dispatch[buffer] = save_buffer 134 | 135 | #block broken objects 136 | def save_unsupported(self, obj, pack=None): 137 | raise pickle.PicklingError("Cannot pickle objects of type %s" % type(obj)) 138 | dispatch[types.GeneratorType] = save_unsupported 139 | 140 | #python2.6+ supports slice pickling. some py2.5 extensions might as well. We just test it 141 | try: 142 | slice(0,1).__reduce__() 143 | except TypeError: #can't pickle - 144 | dispatch[slice] = save_unsupported 145 | 146 | #itertools objects do not pickle! 147 | for v in itertools.__dict__.values(): 148 | if type(v) is type: 149 | dispatch[v] = save_unsupported 150 | 151 | 152 | def save_dict(self, obj): 153 | """hack fix 154 | If the dict is a global, deal with it in a special way 155 | """ 156 | #print 'saving', obj 157 | if obj is __builtins__: 158 | self.save_reduce(_get_module_builtins, (), obj=obj) 159 | else: 160 | pickle.Pickler.save_dict(self, obj) 161 | dispatch[pickle.DictionaryType] = save_dict 162 | 163 | 164 | def save_module(self, obj, pack=struct.pack): 165 | """ 166 | Save a module as an import 167 | """ 168 | #print 'try save import', obj.__name__ 169 | self.modules.add(obj) 170 | self.save_reduce(subimport,(obj.__name__,), obj=obj) 171 | dispatch[types.ModuleType] = save_module #new type 172 | 173 | def save_codeobject(self, obj, pack=struct.pack): 174 | """ 175 | Save a code object 176 | """ 177 | #print 'try to save codeobj: ', obj 178 | args = ( 179 | obj.co_argcount, obj.co_nlocals, obj.co_stacksize, obj.co_flags, obj.co_code, 180 | obj.co_consts, obj.co_names, obj.co_varnames, obj.co_filename, obj.co_name, 181 | obj.co_firstlineno, obj.co_lnotab, obj.co_freevars, obj.co_cellvars 182 | ) 183 | self.save_reduce(types.CodeType, args, obj=obj) 184 | dispatch[types.CodeType] = save_codeobject #new type 185 | 186 | def save_function(self, obj, name=None, pack=struct.pack): 187 | """ Registered with the dispatch to handle all function types. 188 | 189 | Determines what kind of function obj is (e.g. lambda, defined at 190 | interactive prompt, etc) and handles the pickling appropriately. 191 | """ 192 | write = self.write 193 | 194 | name = obj.__name__ 195 | modname = pickle.whichmodule(obj, name) 196 | #print 'which gives %s %s %s' % (modname, obj, name) 197 | try: 198 | themodule = sys.modules[modname] 199 | except KeyError: # eval'd items such as namedtuple give invalid items for their function __module__ 200 | modname = '__main__' 201 | 202 | if modname == '__main__': 203 | themodule = None 204 | 205 | if themodule: 206 | self.modules.add(themodule) 207 | 208 | if self.os_env_vars: 209 | 210 | # recursion hackery 211 | os_env_vars = self.os_env_vars 212 | self.os_env_vars = [] 213 | old_saved_django = self.savedDjangoEnv 214 | self.savedDjangoEnv = True 215 | 216 | write(pickle.MARK) 217 | 218 | if isinstance(os_env_vars, dict): 219 | os_mapping = os_env_vars 220 | else: 221 | os_mapping = {} 222 | for env_var in os_env_vars: 223 | if env_var in os.environ: 224 | os_mapping[env_var] = os.environ[env_var] 225 | self.save_reduce(env_vars_load, (os_mapping,)) # note: nothing sensible to memoize 226 | write(pickle.POP_MARK) 227 | 228 | self.savedDjangoEnv = old_saved_django 229 | 230 | if modname == 'cloud.shell': 231 | # don't save django environment if we are using shell.exec 232 | self.savedDjangoEnv = True 233 | 234 | elif not self.savedDjangoEnv: 235 | # hack for django - if we detect the settings module, we transport it 236 | # unfortunately this module is dynamically, not statically, resolved, so 237 | # dependency analysis never detects it 238 | 239 | django_settings = os.environ.get('DJANGO_SETTINGS_MODULE', '') 240 | if django_settings: 241 | django_mod = sys.modules.get(django_settings) 242 | if django_mod: 243 | cloudLog.debug('Transporting django settings %s during save of %s', django_mod, name) 244 | self.savedDjangoEnv = True 245 | self.modules.add(django_mod) 246 | write(pickle.MARK) 247 | self.save_reduce(django_settings_load, (django_mod.__name__,), obj=django_mod) 248 | write(pickle.POP_MARK) 249 | 250 | 251 | # if func is lambda, def'ed at prompt, is in main, or is nested, then 252 | # we'll pickle the actual function object rather than simply saving a 253 | # reference (as is done in default pickler), via save_function_tuple. 254 | if islambda(obj) or obj.func_code.co_filename == '' or themodule == None: 255 | #Force server to import modules that have been imported in main 256 | modList = None 257 | if themodule == None and not self.savedForceImports: 258 | mainmod = sys.modules['__main__'] 259 | if useForcedImports and hasattr(mainmod,'___pyc_forcedImports__'): 260 | modList = list(mainmod.___pyc_forcedImports__) 261 | self.savedForceImports = True 262 | self.save_function_tuple(obj, modList) 263 | return 264 | else: # func is nested 265 | klass = getattr(themodule, name, None) 266 | if klass is None or klass is not obj: 267 | self.save_function_tuple(obj, [themodule]) 268 | return 269 | 270 | if obj.__dict__: 271 | # essentially save_reduce, but workaround needed to avoid recursion 272 | self.save(_restore_attr) 273 | write(pickle.MARK + pickle.GLOBAL + modname + '\n' + name + '\n') 274 | self.memoize(obj) 275 | self.save(obj.__dict__) 276 | write(pickle.TUPLE + pickle.REDUCE) 277 | else: 278 | write(pickle.GLOBAL + modname + '\n' + name + '\n') 279 | self.memoize(obj) 280 | dispatch[types.FunctionType] = save_function 281 | 282 | def save_function_tuple(self, func, forced_imports): 283 | """ Pickles an actual func object. 284 | 285 | A func comprises: code, globals, defaults, closure, and dict. We 286 | extract and save these, injecting reducing functions at certain points 287 | to recreate the func object. Keep in mind that some of these pieces 288 | can contain a ref to the func itself. Thus, a naive save on these 289 | pieces could trigger an infinite loop of save's. To get around that, 290 | we first create a skeleton func object using just the code (this is 291 | safe, since this won't contain a ref to the func), and memoize it as 292 | soon as it's created. The other stuff can then be filled in later. 293 | """ 294 | save = self.save 295 | write = self.write 296 | 297 | # save the modules (if any) 298 | if forced_imports: 299 | write(pickle.MARK) 300 | save(_modules_to_main) 301 | #print 'forced imports are', forced_imports 302 | 303 | # do not save our own references 304 | forced_names = [m.__name__ for m in forced_imports if not m.__name__.startswith('cloud')] 305 | save((forced_names,)) 306 | 307 | #save((forced_imports,)) 308 | write(pickle.REDUCE) 309 | write(pickle.POP_MARK) 310 | 311 | code, f_globals, defaults, closure, dct, base_globals = self.extract_func_data(func) 312 | 313 | save(_fill_function) # skeleton function updater 314 | write(pickle.MARK) # beginning of tuple that _fill_function expects 315 | 316 | # create a skeleton function object and memoize it 317 | save(_make_skel_func) 318 | save((code, len(closure), base_globals)) 319 | write(pickle.REDUCE) 320 | self.memoize(func) 321 | 322 | # save the rest of the func data needed by _fill_function 323 | save(f_globals) 324 | save(defaults) 325 | save(closure) 326 | save(dct) 327 | write(pickle.TUPLE) 328 | write(pickle.REDUCE) # applies _fill_function on the tuple 329 | 330 | @staticmethod 331 | def extract_code_globals(co): 332 | """ 333 | Find all globals names read or written to by codeblock co 334 | """ 335 | code = co.co_code 336 | names = co.co_names 337 | out_names = set() 338 | 339 | n = len(code) 340 | i = 0 341 | extended_arg = 0 342 | while i < n: 343 | op = code[i] 344 | 345 | i = i+1 346 | if op >= HAVE_ARGUMENT: 347 | oparg = ord(code[i]) + ord(code[i+1])*256 + extended_arg 348 | extended_arg = 0 349 | i = i+2 350 | if op == EXTENDED_ARG: 351 | extended_arg = oparg*65536L 352 | if op in GLOBAL_OPS: 353 | out_names.add(names[oparg]) 354 | #print 'extracted', out_names, ' from ', names 355 | return out_names 356 | 357 | def extract_func_data(self, func): 358 | """ 359 | Turn the function into a tuple of data necessary to recreate it: 360 | code, globals, defaults, closure, dict 361 | """ 362 | code = func.func_code 363 | 364 | # extract all global ref's 365 | func_global_refs = CloudPickler.extract_code_globals(code) 366 | if code.co_consts: # see if nested function have any global refs 367 | for const in code.co_consts: 368 | if type(const) is types.CodeType and const.co_names: 369 | func_global_refs = func_global_refs.union( CloudPickler.extract_code_globals(const)) 370 | # process all variables referenced by global environment 371 | f_globals = {} 372 | for var in func_global_refs: 373 | #Some names, such as class functions are not global - we don't need them 374 | if func.func_globals.has_key(var): 375 | f_globals[var] = func.func_globals[var] 376 | 377 | # defaults requires no processing 378 | defaults = func.func_defaults 379 | 380 | def get_contents(cell): 381 | try: 382 | return cell.cell_contents 383 | except ValueError, e: #cell is empty error on not yet assigned 384 | raise pickle.PicklingError('Function to be pickled has free variables that are referenced before assignment in enclosing scope') 385 | 386 | 387 | # process closure 388 | if func.func_closure: 389 | closure = map(get_contents, func.func_closure) 390 | else: 391 | closure = [] 392 | 393 | # save the dict 394 | dct = func.func_dict 395 | 396 | if printSerialization: 397 | outvars = ['code: ' + str(code) ] 398 | outvars.append('globals: ' + str(f_globals)) 399 | outvars.append('defaults: ' + str(defaults)) 400 | outvars.append('closure: ' + str(closure)) 401 | print 'function ', func, 'is extracted to: ', ', '.join(outvars) 402 | 403 | base_globals = self.globals_ref.get(id(func.func_globals), {}) 404 | self.globals_ref[id(func.func_globals)] = base_globals 405 | 406 | return (code, f_globals, defaults, closure, dct, base_globals) 407 | 408 | def save_class_obj(self, cobj, name=None, pack=struct.pack): 409 | """Save a class object that a reference cannot be provided to""" 410 | 411 | #note: Third party types might crash this - add better checks! 412 | d = dict(cobj.__dict__) #copy dict proxy to a dict 413 | constructor_dct = {'__doc__' : d.pop('__doc__')} 414 | if not isinstance(d.get('__dict__', None), property): # don't extract dict that are properties 415 | d.pop('__dict__',None) 416 | elif '__dict__' in d: 417 | constructor_dct['__dict__'] = d.pop('__dict__') 418 | #pass 419 | 420 | d.pop('__weakref__',None) # never needed in serialization 421 | 422 | # hack as __new__ is stored differently in the __dict__ 423 | new_override = d.get('__new__', None) 424 | if new_override: 425 | constructor_dct['__new__'] = cobj.__new__ 426 | d.pop('__new__', None) 427 | 428 | # beginning of class logic 429 | 430 | self.save(_fill_class) 431 | 432 | self.write(pickle.MARK) # beginning of tuple that _fill_class expects 433 | 434 | self.save(type(cobj)) 435 | self.save((cobj.__name__,cobj.__bases__,constructor_dct)) # inject rest of dictionary later 436 | self.write(pickle.REDUCE) 437 | self.memoize(cobj) 438 | 439 | #print 'save %s' % d['__dict__'] 440 | # save dct needed 441 | self.save(d) 442 | self.write(pickle.TUPLE) 443 | self.write(pickle.REDUCE) # apply _fill_class on tuple 444 | 445 | #self.save_reduce(type(cobj), (cobj.__name__, cobj.__bases__, d), obj = cobj) 446 | 447 | def save_global(self, obj, name=None, pack=struct.pack): 448 | write = self.write 449 | memo = self.memo 450 | 451 | if name is None: 452 | name = obj.__name__ 453 | 454 | modname = getattr(obj, "__module__", None) 455 | if modname is None: 456 | modname = pickle.whichmodule(obj, name) 457 | 458 | try: 459 | __import__(modname) 460 | themodule = sys.modules[modname] 461 | except (ImportError, KeyError, AttributeError): #should never occur 462 | raise pickle.PicklingError( 463 | "Can't pickle %r: Module %s cannot be found" % 464 | (obj, modname)) 465 | 466 | if modname == '__main__': 467 | themodule = None 468 | 469 | if themodule: 470 | self.modules.add(themodule) 471 | 472 | sendRef = True 473 | typ = type(obj) 474 | #print 'saving', obj, typ 475 | try: 476 | try: #Deal with case when getattribute fails with exceptions 477 | klass = getattr(themodule, name) 478 | except (AttributeError): 479 | if modname == '__builtin__': #new.* are misrepeported 480 | modname = 'new' 481 | __import__(modname) 482 | themodule = sys.modules[modname] 483 | try: 484 | klass = getattr(themodule, name) 485 | except AttributeError, a: 486 | #print themodule, name, obj, type(obj) 487 | raise pickle.PicklingError("Can't pickle builtin %s" % obj) 488 | else: 489 | raise 490 | 491 | except (ImportError, KeyError, AttributeError): 492 | if typ == types.TypeType or typ == types.ClassType: 493 | sendRef = False 494 | else: #we can't deal with this 495 | raise 496 | else: 497 | if klass is not obj and (typ == types.TypeType or typ == types.ClassType): 498 | sendRef = False 499 | if not sendRef: 500 | self.save_class_obj(obj, name, pack) 501 | return 502 | 503 | if self.proto >= 2: 504 | code = _extension_registry.get((modname, name)) 505 | if code: 506 | assert code > 0 507 | if code <= 0xff: 508 | write(pickle.EXT1 + chr(code)) 509 | elif code <= 0xffff: 510 | write("%c%c%c" % (pickle.EXT2, code&0xff, code>>8)) 511 | else: 512 | write(pickle.EXT4 + pack("= 2 and getattr(func, "__name__", "") == "__newobj__": 623 | #Added fix to allow transient 624 | cls = args[0] 625 | if not hasattr(cls, "__new__"): 626 | raise pickle.PicklingError( 627 | "args[0] from __newobj__ args has no __new__") 628 | if obj is not None and cls is not obj.__class__: 629 | raise pickle.PicklingError( 630 | "args[0] from __newobj__ args has the wrong class") 631 | args = args[1:] 632 | save(cls) 633 | 634 | #Don't pickle transient entries 635 | if hasattr(obj, '__transient__'): 636 | transient = obj.__transient__ 637 | state = state.copy() 638 | 639 | for k in list(state.keys()): 640 | if k in transient: 641 | del state[k] 642 | 643 | save(args) 644 | write(pickle.NEWOBJ) 645 | else: 646 | save(func) 647 | save(args) 648 | write(pickle.REDUCE) 649 | 650 | if obj is not None: 651 | self.memoize(obj) 652 | 653 | # More new special cases (that work with older protocols as 654 | # well): when __reduce__ returns a tuple with 4 or 5 items, 655 | # the 4th and 5th item should be iterators that provide list 656 | # items and dict items (as (key, value) tuples), or None. 657 | 658 | if listitems is not None: 659 | self._batch_appends(listitems) 660 | 661 | if dictitems is not None: 662 | self._batch_setitems(dictitems) 663 | 664 | if state is not None: 665 | #print 'obj %s has state %s' % (obj, state) 666 | save(state) 667 | write(pickle.BUILD) 668 | 669 | 670 | def save_xrange(self, obj): 671 | """Save an xrange object in python 2.5 672 | Python 2.6 supports this natively 673 | """ 674 | range_params = xrange_helper.xrange_params(obj) 675 | self.save_reduce(_build_xrange,range_params) 676 | 677 | #python2.6+ supports xrange pickling. some py2.5 extensions might as well. We just test it 678 | try: 679 | xrange(0).__reduce__() 680 | except TypeError: #can't pickle -- use PiCloud pickler 681 | dispatch[xrange] = save_xrange 682 | 683 | def save_partial(self, obj): 684 | """Partial objects do not serialize correctly in python2.x -- this fixes the bugs""" 685 | self.save_reduce(_genpartial, (obj.func, obj.args, obj.keywords)) 686 | 687 | if sys.version_info < (2,7): #2.7 supports partial pickling 688 | dispatch[partial] = save_partial 689 | 690 | 691 | def save_file(self, obj): 692 | """Save a file""" 693 | import StringIO as pystringIO #we can't use cStringIO as it lacks the name attribute 694 | from ..transport.adapter import SerializingAdapter 695 | 696 | if not hasattr(obj, 'name') or not hasattr(obj, 'mode'): 697 | raise pickle.PicklingError("Cannot pickle files that do not map to an actual file") 698 | if obj.name == '': 699 | return self.save_reduce(getattr, (sys,'stdout'), obj=obj) 700 | if obj.name == '': 701 | return self.save_reduce(getattr, (sys,'stderr'), obj=obj) 702 | if obj.name == '': 703 | raise pickle.PicklingError("Cannot pickle standard input") 704 | if hasattr(obj, 'isatty') and obj.isatty(): 705 | raise pickle.PicklingError("Cannot pickle files that map to tty objects") 706 | if 'r' not in obj.mode: 707 | raise pickle.PicklingError("Cannot pickle files that are not opened for reading") 708 | name = obj.name 709 | try: 710 | fsize = os.stat(name).st_size 711 | except OSError: 712 | raise pickle.PicklingError("Cannot pickle file %s as it cannot be stat" % name) 713 | 714 | if obj.closed: 715 | #create an empty closed string io 716 | retval = pystringIO.StringIO("") 717 | retval.close() 718 | elif not fsize: #empty file 719 | retval = pystringIO.StringIO("") 720 | try: 721 | tmpfile = file(name) 722 | tst = tmpfile.read(1) 723 | except IOError: 724 | raise pickle.PicklingError("Cannot pickle file %s as it cannot be read" % name) 725 | tmpfile.close() 726 | if tst != '': 727 | raise pickle.PicklingError("Cannot pickle file %s as it does not appear to map to a physical, real file" % name) 728 | elif fsize > SerializingAdapter.max_transmit_data: 729 | raise pickle.PicklingError("Cannot pickle file %s as it exceeds cloudconf.py's max_transmit_data of %d" % 730 | (name,SerializingAdapter.max_transmit_data)) 731 | else: 732 | try: 733 | tmpfile = file(name) 734 | contents = tmpfile.read(SerializingAdapter.max_transmit_data) 735 | tmpfile.close() 736 | except IOError: 737 | raise pickle.PicklingError("Cannot pickle file %s as it cannot be read" % name) 738 | retval = pystringIO.StringIO(contents) 739 | curloc = obj.tell() 740 | retval.seek(curloc) 741 | 742 | retval.name = name 743 | self.save(retval) #save stringIO 744 | self.memoize(obj) 745 | 746 | dispatch[file] = save_file 747 | """Special functions for Add-on libraries""" 748 | 749 | def inject_numpy(self): 750 | numpy = sys.modules.get('numpy') 751 | if not numpy or not hasattr(numpy, 'ufunc'): 752 | return 753 | self.dispatch[numpy.ufunc] = self.__class__.save_ufunc 754 | 755 | numpy_tst_mods = ['numpy', 'scipy.special'] 756 | def save_ufunc(self, obj): 757 | """Hack function for saving numpy ufunc objects""" 758 | name = obj.__name__ 759 | for tst_mod_name in self.numpy_tst_mods: 760 | tst_mod = sys.modules.get(tst_mod_name, None) 761 | if tst_mod: 762 | if name in tst_mod.__dict__: 763 | self.save_reduce(_getobject, (tst_mod_name, name)) 764 | return 765 | raise pickle.PicklingError('cannot save %s. Cannot resolve what module it is defined in' % str(obj)) 766 | 767 | def inject_timeseries(self): 768 | """Handle bugs with pickling scikits timeseries""" 769 | tseries = sys.modules.get('scikits.timeseries.tseries') 770 | if not tseries or not hasattr(tseries, 'Timeseries'): 771 | return 772 | self.dispatch[tseries.Timeseries] = self.__class__.save_timeseries 773 | 774 | def save_timeseries(self, obj): 775 | import scikits.timeseries.tseries as ts 776 | 777 | func, reduce_args, state = obj.__reduce__() 778 | if func != ts._tsreconstruct: 779 | raise pickle.PicklingError('timeseries using unexpected reconstruction function %s' % str(func)) 780 | state = (1, 781 | obj.shape, 782 | obj.dtype, 783 | obj.flags.fnc, 784 | obj._data.tostring(), 785 | ts.getmaskarray(obj).tostring(), 786 | obj._fill_value, 787 | obj._dates.shape, 788 | obj._dates.__array__().tostring(), 789 | obj._dates.dtype, #added -- preserve type 790 | obj.freq, 791 | obj._optinfo, 792 | ) 793 | return self.save_reduce(_genTimeSeries, (reduce_args, state)) 794 | 795 | def inject_email(self): 796 | """Block email LazyImporters from being saved""" 797 | email = sys.modules.get('email') 798 | if not email: 799 | return 800 | self.dispatch[email.LazyImporter] = self.__class__.save_unsupported 801 | 802 | def inject_addons(self): 803 | """Plug in system. Register additional pickling functions if modules already loaded""" 804 | self.inject_numpy() 805 | self.inject_timeseries() 806 | self.inject_email() 807 | 808 | """Python Imaging Library""" 809 | def save_image(self, obj): 810 | if not obj.im and obj.fp and 'r' in obj.fp.mode and obj.fp.name \ 811 | and not obj.fp.closed and (not hasattr(obj, 'isatty') or not obj.isatty()): 812 | #if image not loaded yet -- lazy load 813 | self.save_reduce(_lazyloadImage,(obj.fp,), obj=obj) 814 | else: 815 | #image is loaded - just transmit it over 816 | self.save_reduce(_generateImage, (obj.size, obj.mode, obj.tostring()), obj=obj) 817 | 818 | """ 819 | def memoize(self, obj): 820 | pickle.Pickler.memoize(self, obj) 821 | if printMemoization: 822 | print 'memoizing ' + str(obj) 823 | """ 824 | 825 | 826 | 827 | # Shorthands for legacy support 828 | 829 | def dump(obj, file, protocol=2): 830 | CloudPickler(file, protocol).dump(obj) 831 | 832 | def dumps(obj, protocol=2): 833 | file = StringIO() 834 | 835 | cp = CloudPickler(file,protocol) 836 | cp.dump(obj) 837 | 838 | #print 'cloud dumped', str(obj), str(cp.modules) 839 | 840 | return file.getvalue() 841 | 842 | def loads(s): 843 | return pickle.loads(s) 844 | 845 | #hack for __import__ not working as desired 846 | def subimport(name): 847 | __import__(name) 848 | return sys.modules[name] 849 | 850 | #hack to load django settings: 851 | def django_settings_load(name): 852 | modified_env = False 853 | 854 | if 'DJANGO_SETTINGS_MODULE' not in os.environ: 855 | os.environ['DJANGO_SETTINGS_MODULE'] = name # must set name first due to circular deps 856 | modified_env = True 857 | try: 858 | module = subimport(name) 859 | except Exception, i: 860 | error_msg('Could not import django settings %s:' % (name), exc_info = True) 861 | if modified_env: 862 | del os.environ['DJANGO_SETTINGS_MODULE'] 863 | else: 864 | #add project directory to sys,path: 865 | if hasattr(module,'__file__'): 866 | dirname = os.path.split(module.__file__)[0] + '/' 867 | sys.path.append(dirname) 868 | 869 | error_msg('Loaded Django settings', logging.DEBUG) 870 | 871 | def env_vars_load(env_dct): 872 | #print 'loading environment vars %s' % env_dct 873 | for var, value in env_dct.items(): 874 | os.environ[var] = value 875 | error_msg('Loaded Environment variables %s' % env_dct.keys(), logging.DEBUG) 876 | 877 | # restores function attributes 878 | def _restore_attr(obj, attr): 879 | for key, val in attr.items(): 880 | setattr(obj, key, val) 881 | return obj 882 | 883 | def _get_module_builtins(): 884 | return pickle.__builtins__ 885 | 886 | def _modules_to_main(modList): 887 | """Force every module in modList to be placed into main""" 888 | if not modList: 889 | return 890 | 891 | main = sys.modules['__main__'] 892 | 893 | seen_errors = set() # avoid excessive exceptions; cache top-level module failing and reason 894 | for modname in modList: 895 | if type(modname) is str: 896 | try: 897 | mod = __import__(modname) 898 | except Exception, i: #catch all... 899 | key = modname.split('.',1)[0], str(i) 900 | if key not in seen_errors: 901 | seen_errors.add(key) 902 | error_msg('could not import %s\n. Your function may unexpectedly error due to this import failing; \ 903 | Likely due to version mismatch or non-transferred python extension. Specific error was:\n' % modname, exc_info = True) 904 | else: 905 | setattr(main,mod.__name__, mod) 906 | else: 907 | #REVERSE COMPATIBILITY FOR CLOUD CLIENT 1.5 (WITH EPD) 908 | #In old version actual module was sent 909 | setattr(main,modname.__name__, modname) 910 | 911 | #object generators: 912 | def _build_xrange(start, step, len): 913 | """Built xrange explicitly""" 914 | return xrange(start, start + step*len, step) 915 | 916 | def _genpartial(func, args, kwds): 917 | if not args: 918 | args = () 919 | if not kwds: 920 | kwds = {} 921 | return partial(func, *args, **kwds) 922 | 923 | 924 | def _fill_function(func, globals, defaults, closure, dct): 925 | """ Fills in the rest of function data into the skeleton function object 926 | that were created via _make_skel_func(). 927 | """ 928 | func.func_globals.update(globals) 929 | func.func_defaults = defaults 930 | func.func_dict = dct 931 | 932 | if len(closure) != len(func.func_closure): 933 | raise pickle.UnpicklingError("closure lengths don't match up") 934 | for i in range(len(closure)): 935 | _change_cell_value(func.func_closure[i], closure[i]) 936 | 937 | return func 938 | 939 | def _fill_class(cls, dct): 940 | """Fill in a class dictionary 941 | This hack is necessary because functions in class dictionary may reference class 942 | """ 943 | for key, val in dct.items(): 944 | setattr(cls, key, val) 945 | 946 | return cls 947 | 948 | def _make_skel_func(code, num_closures, base_globals = None): 949 | """ Creates a skeleton function object that contains just the provided 950 | code and the correct number of cells in func_closure. All other 951 | func attributes (e.g. func_globals) are empty. 952 | """ 953 | #build closure (cells): 954 | if not ctypes: 955 | raise Exception('ctypes failed to import; cannot build function') 956 | 957 | cellnew = ctypes.pythonapi.PyCell_New 958 | cellnew.restype = ctypes.py_object 959 | cellnew.argtypes = (ctypes.py_object,) 960 | dummy_closure = tuple(map(lambda i: cellnew(None), range(num_closures))) 961 | 962 | if base_globals is None: 963 | base_globals = {} 964 | base_globals['__builtins__'] = __builtins__ 965 | 966 | return types.FunctionType(code, base_globals, 967 | None, None, dummy_closure) 968 | 969 | # this piece of opaque code is needed below to modify 'cell' contents 970 | cell_changer_code = new.code( 971 | 1, 1, 2, 0, 972 | ''.join([ 973 | chr(dis.opmap['LOAD_FAST']), '\x00\x00', 974 | chr(dis.opmap['DUP_TOP']), 975 | chr(dis.opmap['STORE_DEREF']), '\x00\x00', 976 | chr(dis.opmap['RETURN_VALUE']) 977 | ]), 978 | (), (), ('newval',), '', 'cell_changer', 1, '', ('c',), () 979 | ) 980 | 981 | def _change_cell_value(cell, newval): 982 | """ Changes the contents of 'cell' object to newval """ 983 | return new.function(cell_changer_code, {}, None, (), (cell,))(newval) 984 | 985 | """Constructors for 3rd party libraries 986 | Note: These can never be renamed due to client compatibility issues""" 987 | 988 | def _getobject(modname, attribute): 989 | mod = __import__(modname) 990 | return mod.__dict__[attribute] 991 | 992 | def _generateImage(size, mode, str_rep): 993 | """Generate image from string representation""" 994 | import Image 995 | i = Image.new(mode, size) 996 | i.fromstring(str_rep) 997 | return i 998 | 999 | def _lazyloadImage(fp): 1000 | import Image 1001 | fp.seek(0) #works in almost any case 1002 | return Image.open(fp) 1003 | 1004 | """Timeseries""" 1005 | def _genTimeSeries(reduce_args, state): 1006 | import scikits.timeseries.tseries as ts 1007 | from numpy import ndarray 1008 | from numpy.ma import MaskedArray 1009 | 1010 | 1011 | time_series = ts._tsreconstruct(*reduce_args) 1012 | 1013 | #from setstate modified 1014 | (ver, shp, typ, isf, raw, msk, flv, dsh, dtm, dtyp, frq, infodict) = state 1015 | #print 'regenerating %s' % dtyp 1016 | 1017 | MaskedArray.__setstate__(time_series, (ver, shp, typ, isf, raw, msk, flv)) 1018 | _dates = time_series._dates 1019 | #_dates.__setstate__((ver, dsh, typ, isf, dtm, frq)) #use remote typ 1020 | ndarray.__setstate__(_dates,(dsh,dtyp, isf, dtm)) 1021 | _dates.freq = frq 1022 | _dates._cachedinfo.update(dict(full=None, hasdups=None, steps=None, 1023 | toobj=None, toord=None, tostr=None)) 1024 | # Update the _optinfo dictionary 1025 | time_series._optinfo.update(infodict) 1026 | return time_series 1027 | 1028 | -------------------------------------------------------------------------------- /omega/queue/test/app.py: -------------------------------------------------------------------------------- 1 | from queue import async 2 | 3 | 4 | @async 5 | def add(a, b): 6 | return a + b 7 | 8 | 9 | def main(): 10 | result = add.delay(3, 4, want_results=True) 11 | print result.get() 12 | print add(3, 4) 13 | 14 | if __name__ == '__main__': 15 | main() 16 | -------------------------------------------------------------------------------- /omega/queue/test/config/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jeffknupp/omega/922ceb42011ae22ae0798fd333f2286634451ca4/omega/queue/test/config/__init__.py -------------------------------------------------------------------------------- /omega/queue/test/config/settings.py: -------------------------------------------------------------------------------- 1 | """Configuration file for brokest.""" 2 | CONFIG = { 3 | 'front-end-host': '127.0.0.1', 4 | 'front-end-port': 7070, 5 | 'back-end-host': '127.0.0.1', 6 | 'back-end-port': 7080, 7 | 'monitor-host': '127.0.0.1', 8 | 'monitor-port': 7090, 9 | } 10 | -------------------------------------------------------------------------------- /omega/queue/util/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | Various utility functions 3 | 4 | Copyright (c) 2009 `PiCloud, Inc. `_. All rights reserved. 5 | 6 | email: contact@picloud.com 7 | 8 | The cloud package is free software; you can redistribute it and/or 9 | modify it under the terms of the GNU Lesser General Public 10 | License as published by the Free Software Foundation; either 11 | version 2.1 of the License, or (at your option) any later version. 12 | 13 | This package is distributed in the hope that it will be useful, 14 | but WITHOUT ANY WARRANTY; without even the implied warranty of 15 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 16 | Lesser General Public License for more details. 17 | 18 | You should have received a copy of the GNU Lesser General Public 19 | License along with this package; if not, see 20 | http://www.gnu.org/licenses/lgpl-2.1.html 21 | """ 22 | 23 | import sys 24 | import types 25 | import cPickle 26 | import inspect 27 | import datetime 28 | import os 29 | 30 | from functools import partial 31 | from warnings import warn 32 | 33 | def islambda(func): 34 | return getattr(func,'func_name') == '' 35 | 36 | def funcname(func): 37 | """Return name of a callable (function, class, partial, etc.)""" 38 | module = "" 39 | if hasattr(func,'__module__'): 40 | module = (func.__module__ if func.__module__ else '__main__') 41 | """Return a human readable name associated with a function""" 42 | if inspect.ismethod(func): 43 | nme = '.'.join([module,func.im_class.__name__,func.__name__]) 44 | elif inspect.isfunction(func): 45 | nme = '.'.join([module,func.__name__]) 46 | elif inspect.isbuiltin(func): 47 | return '.'.join([module,func.__name__]) 48 | elif isinstance(func,partial): 49 | return 'partial_of_' + funcname(func.func) 50 | elif inspect.isclass(func): 51 | nme = '.'.join([module,func.__name__]) 52 | if hasattr(func, '__init__') and inspect.ismethod(func.__init__): 53 | func = func.__init__ 54 | else: 55 | return nme #can't extract more info for classes 56 | 57 | else: 58 | nme = 'type %s' % type(func) 59 | if hasattr(func, '__name__'): 60 | nme = '%s of %s' % (func.__name__, type(func)) 61 | return nme 62 | nme += ' at ' + ':'.join([func.func_code.co_filename,str(func.func_code.co_firstlineno)]) 63 | return nme 64 | 65 | def min_args(func): 66 | """Return minimum (required) number args this function has""" 67 | if inspect.isfunction(func): 68 | op_args = len(func.func_defaults) if func.func_defaults else 0 69 | return func.func_code.co_argcount - op_args 70 | elif inspect.ismethod(func): 71 | return min_args(func.im_func) - 1 72 | elif inspect.isclass(func): 73 | if hasattr(func, '__init__'): #check class constructor 74 | return min_args(func.__init__) 75 | else: 76 | return 0 77 | 78 | raise TypeError('cannot deal with type: %s' % type(func)) 79 | 80 | def max_args(func): 81 | """Return maximum (required + default) number of arguments callable can take""" 82 | if inspect.isfunction(func): 83 | return func.func_code.co_argcount 84 | elif inspect.ismethod(func): 85 | return max_args(func.im_func) - 1 86 | elif inspect.isclass(func) and hasattr(func, '__init__'): #check class constructor 87 | if hasattr(func, '__init__'): #check class constructor 88 | return max_args(func.__init__) 89 | else: 90 | return 0 91 | raise TypeError('cannot deal with type: %s' % type(func)) 92 | 93 | 94 | def getargspec(func): 95 | """Returns an argspec or None if it can't be resolved 96 | Our argspec is similar to inspect's except the name & if it is a method is appended as the first argument 97 | Returns (name, is_method, args, *args, **kwargs, defaults) 98 | """ 99 | 100 | try: 101 | argspec = inspect.getargspec(func) 102 | except TypeError: 103 | return None 104 | 105 | out_list = [func.__name__, int(inspect.ismethod(func))] 106 | out_list.extend(argspec) 107 | return out_list 108 | 109 | def validate_func_arguments(func, test_args, test_kwargs): 110 | """First pass validation to see if args/kwargs are compatible with the argspec 111 | Probably doesn't catch everything that will error 112 | Known to miss: 113 | Validate that anonymous tuple params receive tuples 114 | 115 | This is only valid for python 2.x 116 | 117 | Returns true if validation passed; false if validation not supported 118 | Exception raised if validation fails 119 | """ 120 | try: 121 | argspec = inspect.getargspec(func) 122 | except TypeError: #we can't check non-functions 123 | return False 124 | 125 | return validate_func_arguments_from_spec( (func.__name__, int(inspect.ismethod(func))) + argspec, 126 | test_args, 127 | test_kwargs.keys()) 128 | 129 | 130 | def validate_func_arguments_from_spec(argspec, test_args, test_kwargs_keys): 131 | 132 | name, is_method, args, varargs, varkw, defaults = argspec 133 | 134 | if defaults == None: 135 | defaults = [] 136 | else: 137 | defaults = list(defaults) 138 | 139 | if is_method: #ignore self/cls 140 | args = args[1:] 141 | 142 | 143 | name += '()' #conform to python error reporting 144 | test_args_len = len(test_args) 145 | 146 | #kwd exist? 147 | if not varkw: 148 | for kw in test_kwargs_keys: 149 | if kw not in args: 150 | raise TypeError("%s got an unexpected keyword argument '%s'" % (name, kw)) 151 | 152 | #kwd not already bound by passed arg? 153 | kwd_bound = args[test_args_len:] #These must all be default or bound to kwds 154 | if not varkw: 155 | for kw in test_kwargs_keys: 156 | if kw not in kwd_bound: 157 | raise TypeError("%s got multiple values for keyword argument '%s'" % (name, kw)) 158 | 159 | #verify argument count 160 | firstdefault = len(args) - len(defaults) 161 | nondefargs = args[:firstdefault] 162 | 163 | defaults_injected = 0 164 | for kw in test_kwargs_keys: 165 | if kw in nondefargs: 166 | defaults.append(None) #pretend another default is there for counting 167 | defaults_injected += 1 168 | 169 | min = len(args) - len(defaults) 170 | max = len(args) 171 | 172 | #correct for default injection 173 | min+=defaults_injected 174 | max+=defaults_injected 175 | test_args_len += defaults_injected 176 | 177 | if varargs: 178 | max = sys.maxint 179 | if min < 0: 180 | min = 0 181 | 182 | if test_args_len < min or max < test_args_len: 183 | err_msg = '%s takes %s arguments (%d given)' 184 | 185 | if min == max: 186 | arg_c_msg = 'exactly %s' % min 187 | elif test_args_len < min: 188 | arg_c_msg = 'at least %s' % min 189 | else: 190 | arg_c_msg = 'at most %s' % max 191 | 192 | raise TypeError(err_msg % (name, arg_c_msg, test_args_len)) 193 | 194 | return True 195 | 196 | def fix_time_element(dct, key): 197 | """Fix time elements in dictionaries coming off the wire""" 198 | item = dct.get(key) 199 | if item == 'None': #returned by web instead of a NoneType None 200 | item = None 201 | dct[key] = item 202 | if item: 203 | dct[key] = datetime.datetime.strptime(item,'%Y-%m-%d %H:%M:%S') 204 | return dct 205 | 206 | def fix_sudo_path(path): 207 | """Correct permissions on path if using sudo from another user and keeping old users home directory""" 208 | 209 | if os.name != 'posix': 210 | return 211 | 212 | sudo_uid = os.environ.get('SUDO_UID') 213 | sudo_user = os.environ.get('SUDO_USER') 214 | 215 | if sudo_uid != None and sudo_user: 216 | sudo_uid = int(sudo_uid) 217 | home = os.environ.get('HOME') 218 | sudo_user_home = os.path.expanduser('~' + sudo_user) 219 | 220 | # important: Only make modifications if user's home was not changed with sudo (e.g. sudo -H) 221 | if home == sudo_user_home: 222 | sudo_gid = os.environ.get('SUDO_GID') 223 | sudo_gid = int(sudo_gid) if sudo_gid else -1 224 | try: 225 | os.chown(path, sudo_uid, sudo_gid) 226 | except Exception, e: 227 | warn('PiCloud cannot fix SUDO Paths. Error is %s:%s' % (type(e), str(e))) 228 | 229 | 230 | """Ordered Dictionary""" 231 | import UserDict 232 | class OrderedDict(UserDict.DictMixin): 233 | 234 | def __init__(self, it = None): 235 | self._keys = [] 236 | self._data = {} 237 | if it: 238 | for k,v in it: 239 | self.__setitem__(k,v) 240 | 241 | 242 | def __setitem__(self, key, value): 243 | if key not in self._data: 244 | self._keys.append(key) 245 | self._data[key] = value 246 | 247 | def insertAt(self, loc, key, value): 248 | if key in self._data: 249 | del self._data[self._data.index(key)] 250 | self._keys.insert(loc, key) 251 | self._data[key] = value 252 | 253 | def __getitem__(self, key): 254 | return self._data[key] 255 | 256 | 257 | def __delitem__(self, key): 258 | del self._data[key] 259 | self._keys.remove(key) 260 | 261 | 262 | def keys(self): 263 | return list(self._keys) 264 | 265 | def copy(self): 266 | copyDict = OrderedDict() 267 | copyDict._data = self._data.copy() 268 | copyDict._keys = self._keys[:] 269 | return copyDict 270 | 271 | """Python 2.5 support""" 272 | from itertools import izip, chain, repeat 273 | if sys.version_info[:2] < (2,6): 274 | def izip_longest(*args): 275 | def sentinel(counter = ([None]*(len(args)-1)).pop): 276 | yield counter() # yields the fillvalue, or raises IndexError 277 | fillers = repeat(None) 278 | iters = [chain(it, sentinel(), fillers) for it in args] 279 | try: 280 | for tup in izip(*iters): 281 | yield tup 282 | except IndexError: 283 | pass 284 | 285 | if __name__ == '__main__': 286 | """Validate the validate_func_arguments function""" 287 | def foo0(): 288 | pass 289 | 290 | def foo1(a): 291 | pass 292 | 293 | def foo2(a, b=2): 294 | pass 295 | 296 | def foo21(a, b): 297 | pass 298 | 299 | def foo3(a, (x,y), b): 300 | """lovely anonymous function""" 301 | pass 302 | 303 | def consist(func, *args, **kwargs): 304 | typerror = None 305 | try: 306 | func(*args, **kwargs) 307 | except TypeError, e: 308 | typerror = e 309 | 310 | print '%s %s %s' % (func, args, kwargs) 311 | try: 312 | validate_func_arguments(func, args, kwargs) 313 | except TypeError, e: 314 | if not typerror: 315 | print 'unexpected typerror! %s' % str(e) 316 | raise 317 | else: 318 | print '%s == %s' % (typerror, str(e)) 319 | else: 320 | if typerror: 321 | print 'missed error! %s' % typerror 322 | raise 323 | else: 324 | print 'no error!' 325 | 326 | consist(foo0) 327 | consist(foo0, 2) 328 | consist(foo0, k=2) 329 | consist(foo0, 3, k=4) 330 | 331 | consist(foo1) 332 | consist(foo1, b=2) 333 | consist(foo1, a=2) 334 | consist(foo1, 2) 335 | consist(foo1, 3) 336 | consist(foo1, 3, a=2) 337 | consist(foo1, 3, b=2) 338 | 339 | consist(foo2) 340 | consist(foo2, b=2) 341 | consist(foo2, b=2, c=3) 342 | consist(foo2, a=2) 343 | consist(foo2, a=2, b=2) 344 | consist(foo2, a=2, b=2, c=3) 345 | consist(foo2, 2, a=10) 346 | consist(foo2, 3) 347 | consist(foo2, 3, 4) 348 | consist(foo2, 3, 4, 7) 349 | consist(foo2, 3, b=2) 350 | consist(foo2, 3, a=10, b=2) 351 | consist(foo2, 3, b=2, c=2) 352 | consist(foo2, 3, a=10, b=2, c=4) 353 | 354 | consist(foo21, 3, 4) 355 | consist(foo21, 3, b=4) 356 | consist(foo21, a=3, b=4) 357 | consist(foo21, b=4) 358 | consist(foo21, a=4) 359 | consist(foo21) 360 | consist(foo21, 4, 3, 5) 361 | 362 | consist(foo3, 2, (4,3), 9) 363 | consist(foo3, 2, (4,3), b=9) 364 | consist(foo3, 2, (4,3), a=9) 365 | consist(foo3, 2, (4,3), a=9, b=9) 366 | consist(foo3, 2, a=9, b=9) 367 | consist(foo3, 2, (4,3)) 368 | 369 | #we can't catch below.. 370 | #consist(foo3, 2, 10, 12) 371 | 372 | -------------------------------------------------------------------------------- /omega/queue/util/xrange_helper.py: -------------------------------------------------------------------------------- 1 | """ 2 | XRange serialization routines 3 | 4 | Note: This code is in desperate need of cleanup. But options are limited. 5 | Some bad API choices were made 2 years ago, and cannot be reverted 6 | 7 | The code deals with several types. 8 | The PiecewiseXrange is a list subclass that has xranges internally 9 | Example: 10 | [1,3,xrange(5,9), 20] 11 | This defines the list [1,3,5,6,7,8,20] more compactly 12 | 13 | This also deals with JSONable PiecewiseXrange. These encodings can then be encoded by JSON. They look like: 14 | [1,3,['xrange', 5, 9, 1], 20] 15 | 16 | Note: 17 | Single integers are also considered PiecewiseXrange (e.g. 9 is a PiecewiseXrange) 18 | Single xranges are also PiecewiseXrange. (e.g. xrange(10) is a PiecewiseXrange) 19 | 20 | 21 | Copyright (c) 2009 `PiCloud, Inc. `_. All rights reserved. 22 | 23 | email: contact@picloud.com 24 | 25 | The cloud package is free software; you can redistribute it and/or 26 | modify it under the terms of the GNU Lesser General Public 27 | License as published by the Free Software Foundation; either 28 | version 2.1 of the License, or (at your option) any later version. 29 | 30 | This package is distributed in the hope that it will be useful, 31 | but WITHOUT ANY WARRANTY; without even the implied warranty of 32 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 33 | Lesser General Public License for more details. 34 | 35 | You should have received a copy of the GNU Lesser General Public 36 | License along with this package; if not, see 37 | http://www.gnu.org/licenses/lgpl-2.1.html 38 | """ 39 | 40 | 41 | def xrange_params(xrangeobj): 42 | """Returns a 3 element tuple describing the xrange start, step, and len respectively 43 | 44 | Note: Only guarentees that elements of xrange are the same. parameters may be different. 45 | e.g. xrange(1,1) is interpretted as xrange(0,0); both behave the same though w/ iteration 46 | """ 47 | 48 | xrange_len = len(xrangeobj) 49 | if not xrange_len: #empty 50 | return (0,1,0) 51 | start = xrangeobj[0] 52 | if xrange_len == 1: #one element 53 | return start, 1, 1 54 | return (start, xrangeobj[1] - xrangeobj[0], xrange_len) 55 | 56 | 57 | """Encoding (generally for placing within a JSON object):""" 58 | 59 | def encode_maybe_xrange(obj, allowLists = True): 60 | """If obj is a picewiseXrange, make it a JSONable PiecewiseXrange 61 | Else return object 62 | If allowLists is False only an xrange can be encoded 63 | If true, PiecewiseXranges can be encoded 64 | """ 65 | if isinstance(obj, xrange): 66 | return ['xrange'] + list(xrange_params(obj)) 67 | if allowLists and isinstance(obj, list): 68 | return [encode_maybe_xrange(elm, allowLists = False) for elm in obj] 69 | return obj 70 | 71 | def decode_maybe_xrange(obj, allowLists = True): 72 | """Decode an object that may be a JSONable piecewise xrange, into 73 | a regular PiecewiseXrange 74 | Else return object""" 75 | if isinstance(obj, list): 76 | if len(obj) == 4 and obj[0] == 'xrange': #an xrange object 77 | return xrange(obj[1], obj[1] + obj[2]*obj[3], obj[2]) 78 | elif allowLists: #decode internal xranges 79 | return [decode_maybe_xrange(elm, allowLists = False) for elm in obj] 80 | return obj 81 | 82 | def decode_xrange_list(obj): 83 | """First call decode_maybe_xrange(obj) 84 | Then flatten the list, so no xranges are contained""" 85 | 86 | obj = decode_maybe_xrange(obj, allowLists = False) #outer layer might be an xrange 87 | if isinstance(obj, xrange): 88 | return list(obj) #convert to list 89 | outlst = [] 90 | for elem in obj: 91 | if isinstance(elem, list) and len(elem) == 4 and elem[0] == 'xrange': 92 | outlst.extend(decode_maybe_xrange(elem, allowLists = False)) 93 | else: 94 | outlst.append(elem) 95 | return outlst 96 | 97 | """Support for piece-wise xranges - effectively a list of xranges""" 98 | class PiecewiseXrange(list): 99 | """A PiecewiseXrange is best explained with an example: 100 | 101 | [1,3,xrange(5,9), 20] is a PiecewiseXrange that defines 102 | 103 | [1,3,5,6,7,8,20] more compactly 104 | 105 | This class exists to provide the my_iter method which iterates the elements defined. 106 | in example, myiter (in xrangeMode) would return: 107 | 1,3,5,6,7,8,20 108 | 109 | However, __iter__ returns: 110 | 1,3,xrange(5,9), 20 111 | """ 112 | 113 | def __init__(self): 114 | self._xrangeMode = False 115 | 116 | def to_xrange_mode(self): 117 | """Turn on xrange iterator""" 118 | self._xrangeMode = True 119 | 120 | def my_iter(self): 121 | """Ideally, this would overload __iter__, but that presents massive pickling issues""" 122 | if self._xrangeMode: 123 | return xrange_iter(self) 124 | else: 125 | return self.__iter__() 126 | 127 | def xrange_iter(iterable): #use a generator to iterate 128 | for elm in iterable.__iter__(): 129 | if isinstance(elm, xrange): 130 | for subelm in elm: 131 | yield subelm 132 | else: 133 | yield elm 134 | 135 | def maybe_xrange_iter(lst, coerce = True): 136 | """if lst is a PiecewiseXrange or coerce, iterate as though lst is a piecewise xrange 137 | otherwise, normal lst iteration 138 | """ 139 | if isinstance(lst, PiecewiseXrange): 140 | return lst.my_iter() 141 | elif coerce: 142 | return xrange_iter(lst) 143 | else: 144 | return lst.__iter__() 145 | 146 | 147 | def filter_xrange_list(func, xrange_list): 148 | """ Input is a PiecewiseXrange 149 | 150 | Returns a reasonably compact PiecewiseXrange consisting of all elements in the input where 151 | where func(elem) evaluates to True 152 | 153 | Behavior notes: 154 | If an xrange reduces to one element, it is replaced by an integer 155 | e.g. xrange(n,n+1) = n 156 | """ 157 | 158 | if not hasattr(xrange_list,'__iter__') or isinstance(xrange_list, xrange): 159 | xrange_list = [xrange_list] 160 | 161 | single_range = 2 #if > 0, then outList is just a single xrange 162 | no_xrange_output = True #outList has no xranges inserted 163 | 164 | outList = PiecewiseXrange() 165 | for elm in xrange_list: 166 | if isinstance(elm, (int, long)): #elm is 167 | if func(elm): 168 | outList.append(elm) 169 | single_range = 0 #individual elements present - so not single xrange 170 | elif isinstance(elm, xrange): 171 | step = xrange_params(elm)[1] 172 | basenum = None 173 | for num in elm: #iterate through xrange 174 | if func(num): 175 | if basenum == None: 176 | basenum = num 177 | else: 178 | if basenum != None: #push back xrange 179 | if num - step == basenum: #only one element: push an integer 180 | outList.append(basenum) 181 | single_range = 0 182 | else: 183 | outList.append(xrange(basenum, num, step)) 184 | single_range-=1 185 | no_xrange_output = False 186 | basenum = None 187 | if basenum != None: #cleanup 188 | num+=step 189 | if num - step == basenum: #only one element: push an integer 190 | outList.append(basenum) 191 | single_range = 0 192 | else: 193 | outList.append(xrange(basenum, num, step)) 194 | single_range-=1 195 | no_xrange_output = False 196 | else: 197 | raise TypeError('%s (type %s) is not of type int, long or xrange' % (elm, type(elm))) 198 | 199 | if outList: 200 | if not no_xrange_output: 201 | if single_range > 0: #only one xrange appended - just return it 202 | return outList[0] 203 | else: 204 | outList.to_xrange_mode() 205 | return outList 206 | 207 | def iterate_xrange_limit(pwx, limit): 208 | """ 209 | Split PiecewiseXrange *pwx* into individual pieceWiseXranges containing no more than limit elements 210 | Iterator returns individual partitions 211 | """ 212 | if not pwx: #empty list/non case 213 | return 214 | 215 | if isinstance(pwx, xrange): 216 | if len(pwx) <= limit: 217 | yield pwx 218 | return 219 | #needs to parition 220 | pwx = [pwx] 221 | 222 | #main algorithm 223 | outlist = [] 224 | cnt = 0 #number of items appended 225 | for xr in pwx: 226 | if isinstance(xr, (int, long)): 227 | outlist.append(xr) 228 | cnt+=1 229 | if cnt >= limit: 230 | yield outlist 231 | outlist = [] 232 | cnt =0 233 | elif isinstance(xr, xrange): 234 | while True: 235 | if len(xr) + cnt <= limit: 236 | outlist.append(xr) 237 | cnt+=len(xr) 238 | break 239 | else: #break apart xrange 240 | allowed = limit - cnt 241 | start, step, xl_len = xrange_params(xr) 242 | breakpoint = start+ step*allowed 243 | to_app = xrange(start, breakpoint, step) 244 | outlist.append(to_app) 245 | yield outlist 246 | outlist = [] 247 | cnt=0 248 | xr = xrange(breakpoint,breakpoint+(xl_len-allowed)*step, step) 249 | if len(outlist) >= limit: 250 | yield outlist 251 | outlist = [] 252 | 253 | else: 254 | raise TypeError('%s (type %s) is not of type int, long or xrange' % (xr, type(xr))) 255 | if outlist: 256 | yield outlist 257 | 258 | """ 259 | Quick unit test 260 | (TODO: Move to unit tests) 261 | """ 262 | 263 | if __name__ == '__main__': 264 | 265 | for m in [xrange(0,10,2), xrange(0,1), xrange(0), xrange(0,-2), xrange(0,6), xrange(0,7,3)]: 266 | me = encode_maybe_xrange(m) 267 | n = decode_maybe_xrange(me) 268 | print m, me, n 269 | 270 | #split 271 | print list(iterate_xrange_limit(m,1000)) 272 | print list(iterate_xrange_limit(m,3)) 273 | print [list(x[0]) if type(x) is list else x for x in iterate_xrange_limit(m,3)] 274 | 275 | print 'filters!' 276 | xrl = filter_xrange_list(lambda x: x%10, xrange(10,100)) 277 | print xrl 278 | 279 | print 'it limits!' 280 | lmt = list(iterate_xrange_limit(xrl,15)) 281 | print 'chunks of 15', lmt 282 | 283 | lmt_read = [[list(x) for x in y] for y in lmt] 284 | lmt2 = [reduce(lambda x, y: x+y, x) for x in lmt_read] 285 | print 'still 15', lmt2 286 | 287 | lenz = [len(x) for x in lmt2] 288 | print lenz 289 | 290 | print list(iterate_xrange_limit(xrange(25), 13)) 291 | it = iterate_xrange_limit(xrange(25), 13) 292 | print 'iterate!' 293 | try: 294 | while True: 295 | v = it.next() 296 | print type(v), v 297 | except StopIteration: 298 | pass 299 | 300 | -------------------------------------------------------------------------------- /omega/queue/worker.py: -------------------------------------------------------------------------------- 1 | """Broker-less distributed task queue.""" 2 | from __future__ import absolute_import 3 | import pickle 4 | import logging 5 | import multiprocessing 6 | 7 | import zmq 8 | 9 | from config.settings import CONFIG 10 | 11 | LOGGER = logging.getLogger(__name__) 12 | logging.basicConfig(level=logging.DEBUG) 13 | 14 | class Worker(object): 15 | """A remote task executor.""" 16 | 17 | def __init__(self, host='127.0.0.1', port=7080, worker_id=0): 18 | """Initialize worker.""" 19 | LOGGER.info('Starting worker [{}]'.format(worker_id)) 20 | self.host = host 21 | self.port = port 22 | self._id = worker_id 23 | self._context = None 24 | self._socket = None 25 | self._terminate = False 26 | 27 | def terminate(self): 28 | """Stop listening for tasks.""" 29 | self._terminate = True 30 | 31 | def start(self): 32 | """Start listening for tasks.""" 33 | self._context = zmq.Context() 34 | self._socket = self._context.socket(zmq.REP) 35 | self._socket.connect(CONFIG['bind_back']) 36 | while not self._terminate: 37 | message = self._socket.recv_pyobj() 38 | runnable = pickle.loads(message.runnable_string) 39 | args = message.args 40 | kwargs = message.kwargs 41 | response = self._do_work(runnable, args, kwargs) 42 | self._socket.send_pyobj(response) 43 | 44 | def _do_work(self, task, args, kwargs): 45 | """Return the result of executing the given task.""" 46 | LOGGER.info('[{}] running [{}] with args [{}] and kwargs [{}]'.format(self._id, 47 | task, args, kwargs)) 48 | return task(*args, **kwargs) 49 | 50 | if __name__ == '__main__': 51 | w = Worker() 52 | w.start() 53 | -------------------------------------------------------------------------------- /omega/search/README.md: -------------------------------------------------------------------------------- 1 | Placeholder for `search` application 2 | -------------------------------------------------------------------------------- /omega/settings/README.md: -------------------------------------------------------------------------------- 1 | Placeholder for `settings` application 2 | -------------------------------------------------------------------------------- /omega/stat/README.md: -------------------------------------------------------------------------------- 1 | Placeholder for `stat` application 2 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | Jinja2==2.7.3 2 | MarkupSafe==0.23 3 | SQLAlchemy==0.9.4 4 | WTForms==2.0.1 5 | Werkzeug==0.9.6 6 | cloud==2.8.5 7 | gevent==1.0.1 8 | gevent-socketio==0.3.6 9 | gevent-websocket==0.9.3 10 | greenlet==0.4.2 11 | psycopg2==2.5.3 12 | py==1.4.20 13 | pytest==2.5.2 14 | pyzmq==14.3.1 15 | wsgiref==0.1.2 16 | -------------------------------------------------------------------------------- /tests/config/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jeffknupp/omega/922ceb42011ae22ae0798fd333f2286634451ca4/tests/config/__init__.py -------------------------------------------------------------------------------- /tests/config/settings.py: -------------------------------------------------------------------------------- 1 | """Configuration file for brokest.""" 2 | CONFIG = { 3 | 'bind_front': 'inproc://name', 4 | 'bind_back': 'inproc://name', 5 | 'front-end-host': '127.0.0.1', 6 | 'front-end-port': 7070, 7 | 'back-end-host': '127.0.0.1', 8 | 'back-end-port': 7080, 9 | 'monitor-host': '127.0.0.1', 10 | 'monitor-port': 7090, 11 | } 12 | -------------------------------------------------------------------------------- /tests/test_queue.py: -------------------------------------------------------------------------------- 1 | from whizbang.queue.queue import async 2 | 3 | @async 4 | def add(a, b): 5 | return a + b 6 | 7 | def test_direct_call(): 8 | """Can we still call the function directly (no async functionality)?""" 9 | assert add(3, 4) == 7 10 | --------------------------------------------------------------------------------