├── .gitignore ├── .travis.yml ├── LICENSE ├── MANIFEST.in ├── README.rst ├── bookstore ├── __init__.py ├── cloudfiles.py └── swift.py ├── requirements.txt ├── setup.cfg ├── setup.py ├── tasks.py └── tests └── test_bookstore.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.py[cod] 2 | 3 | # C extensions 4 | *.so 5 | 6 | # Packages 7 | *.egg 8 | *.egg-info 9 | dist 10 | build 11 | eggs 12 | parts 13 | bin 14 | var 15 | sdist 16 | develop-eggs 17 | .installed.cfg 18 | lib 19 | lib64 20 | 21 | # Installer logs 22 | pip-log.txt 23 | 24 | # Unit test / coverage reports 25 | .coverage 26 | .tox 27 | nosetests.xml 28 | 29 | # Translations 30 | *.mo 31 | 32 | # Mr Developer 33 | .mr.developer.cfg 34 | .project 35 | .pydevproject 36 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | 3 | python: 4 | - 2.7 5 | 6 | before_install: 7 | - sudo apt-get install -qq libzmq3-dev 8 | - if [[ $TRAVIS_PYTHON_VERSION == 'pypy' ]]; then pip install -q --use-mirrors cffi; fi 9 | - if [[ $TRAVIS_PYTHON_VERSION != 'pypy' ]]; then pip install -q --use-mirrors cython; fi 10 | - pip install -U setuptools 11 | - pip install invoke==0.4.0 pytest==2.3.5 12 | 13 | install: 14 | - pip install . 15 | 16 | script: invoke test 17 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, and 10 | distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by the copyright 13 | owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all other entities 16 | that control, are controlled by, or are under common control with that entity. 17 | For the purposes of this definition, "control" means (i) the power, direct or 18 | indirect, to cause the direction or management of such entity, whether by 19 | contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the 20 | outstanding shares, or (iii) beneficial ownership of such entity. 21 | 22 | "You" (or "Your") shall mean an individual or Legal Entity exercising 23 | permissions granted by this License. 24 | 25 | "Source" form shall mean the preferred form for making modifications, including 26 | but not limited to software source code, documentation source, and configuration 27 | files. 28 | 29 | "Object" form shall mean any form resulting from mechanical transformation or 30 | translation of a Source form, including but not limited to compiled object code, 31 | generated documentation, and conversions to other media types. 32 | 33 | "Work" shall mean the work of authorship, whether in Source or Object form, made 34 | available under the License, as indicated by a copyright notice that is included 35 | in or attached to the work (an example is provided in the Appendix below). 36 | 37 | "Derivative Works" shall mean any work, whether in Source or Object form, that 38 | is based on (or derived from) the Work and for which the editorial revisions, 39 | annotations, elaborations, or other modifications represent, as a whole, an 40 | original work of authorship. For the purposes of this License, Derivative Works 41 | shall not include works that remain separable from, or merely link (or bind by 42 | name) to the interfaces of, the Work and Derivative Works thereof. 43 | 44 | "Contribution" shall mean any work of authorship, including the original version 45 | of the Work and any modifications or additions to that Work or Derivative Works 46 | thereof, that is intentionally submitted to Licensor for inclusion in the Work 47 | by the copyright owner or by an individual or Legal Entity authorized to submit 48 | on behalf of the copyright owner. For the purposes of this definition, 49 | "submitted" means any form of electronic, verbal, or written communication sent 50 | to the Licensor or its representatives, including but not limited to 51 | communication on electronic mailing lists, source code control systems, and 52 | issue tracking systems that are managed by, or on behalf of, the Licensor for 53 | the purpose of discussing and improving the Work, but excluding communication 54 | that is conspicuously marked or otherwise designated in writing by the copyright 55 | owner as "Not a Contribution." 56 | 57 | "Contributor" shall mean Licensor and any individual or Legal Entity on behalf 58 | of whom a Contribution has been received by Licensor and subsequently 59 | incorporated within the Work. 60 | 61 | 2. Grant of Copyright License. 62 | 63 | Subject to the terms and conditions of this License, each Contributor hereby 64 | grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, 65 | irrevocable copyright license to reproduce, prepare Derivative Works of, 66 | publicly display, publicly perform, sublicense, and distribute the Work and such 67 | Derivative Works in Source or Object form. 68 | 69 | 3. Grant of Patent License. 70 | 71 | Subject to the terms and conditions of this License, each Contributor hereby 72 | grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, 73 | irrevocable (except as stated in this section) patent license to make, have 74 | made, use, offer to sell, sell, import, and otherwise transfer the Work, where 75 | such license applies only to those patent claims licensable by such Contributor 76 | that are necessarily infringed by their Contribution(s) alone or by combination 77 | of their Contribution(s) with the Work to which such Contribution(s) was 78 | submitted. If You institute patent litigation against any entity (including a 79 | cross-claim or counterclaim in a lawsuit) alleging that the Work or a 80 | Contribution incorporated within the Work constitutes direct or contributory 81 | patent infringement, then any patent licenses granted to You under this License 82 | for that Work shall terminate as of the date such litigation is filed. 83 | 84 | 4. Redistribution. 85 | 86 | You may reproduce and distribute copies of the Work or Derivative Works thereof 87 | in any medium, with or without modifications, and in Source or Object form, 88 | provided that You meet the following conditions: 89 | 90 | You must give any other recipients of the Work or Derivative Works a copy of 91 | this License; and 92 | You must cause any modified files to carry prominent notices stating that You 93 | changed the files; and 94 | You must retain, in the Source form of any Derivative Works that You distribute, 95 | all copyright, patent, trademark, and attribution notices from the Source form 96 | of the Work, excluding those notices that do not pertain to any part of the 97 | Derivative Works; and 98 | If the Work includes a "NOTICE" text file as part of its distribution, then any 99 | Derivative Works that You distribute must include a readable copy of the 100 | attribution notices contained within such NOTICE file, excluding those notices 101 | that do not pertain to any part of the Derivative Works, in at least one of the 102 | following places: within a NOTICE text file distributed as part of the 103 | Derivative Works; within the Source form or documentation, if provided along 104 | with the Derivative Works; or, within a display generated by the Derivative 105 | Works, if and wherever such third-party notices normally appear. The contents of 106 | the NOTICE file are for informational purposes only and do not modify the 107 | License. You may add Your own attribution notices within Derivative Works that 108 | You distribute, alongside or as an addendum to the NOTICE text from the Work, 109 | provided that such additional attribution notices cannot be construed as 110 | modifying the License. 111 | You may add Your own copyright statement to Your modifications and may provide 112 | additional or different license terms and conditions for use, reproduction, or 113 | distribution of Your modifications, or for any such Derivative Works as a whole, 114 | provided Your use, reproduction, and distribution of the Work otherwise complies 115 | with the conditions stated in this License. 116 | 117 | 5. Submission of Contributions. 118 | 119 | Unless You explicitly state otherwise, any Contribution intentionally submitted 120 | for inclusion in the Work by You to the Licensor shall be under the terms and 121 | conditions of this License, without any additional terms or conditions. 122 | Notwithstanding the above, nothing herein shall supersede or modify the terms of 123 | any separate license agreement you may have executed with Licensor regarding 124 | such Contributions. 125 | 126 | 6. Trademarks. 127 | 128 | This License does not grant permission to use the trade names, trademarks, 129 | service marks, or product names of the Licensor, except as required for 130 | reasonable and customary use in describing the origin of the Work and 131 | reproducing the content of the NOTICE file. 132 | 133 | 7. Disclaimer of Warranty. 134 | 135 | Unless required by applicable law or agreed to in writing, Licensor provides the 136 | Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, 137 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, 138 | including, without limitation, any warranties or conditions of TITLE, 139 | NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are 140 | solely responsible for determining the appropriateness of using or 141 | redistributing the Work and assume any risks associated with Your exercise of 142 | permissions under this License. 143 | 144 | 8. Limitation of Liability. 145 | 146 | In no event and under no legal theory, whether in tort (including negligence), 147 | contract, or otherwise, unless required by applicable law (such as deliberate 148 | and grossly negligent acts) or agreed to in writing, shall any Contributor be 149 | liable to You for damages, including any direct, indirect, special, incidental, 150 | or consequential damages of any character arising as a result of this License or 151 | out of the use or inability to use the Work (including but not limited to 152 | damages for loss of goodwill, work stoppage, computer failure or malfunction, or 153 | any and all other commercial damages or losses), even if such Contributor has 154 | been advised of the possibility of such damages. 155 | 156 | 9. Accepting Warranty or Additional Liability. 157 | 158 | While redistributing the Work or Derivative Works thereof, You may choose to 159 | offer, and charge a fee for, acceptance of support, warranty, indemnity, or 160 | other liability obligations and/or rights consistent with this License. However, 161 | in accepting such obligations, You may act only on Your own behalf and on Your 162 | sole responsibility, not on behalf of any other Contributor, and only if You 163 | agree to indemnify, defend, and hold each Contributor harmless for any liability 164 | incurred by, or claims asserted against, such Contributor by reason of your 165 | accepting any such warranty or additional liability. 166 | 167 | END OF TERMS AND CONDITIONS 168 | 169 | APPENDIX: How to apply the Apache License to your work 170 | 171 | To apply the Apache License to your work, attach the following boilerplate 172 | notice, with the fields enclosed by brackets "[]" replaced with your own 173 | identifying information. (Don't include the brackets!) The text should be 174 | enclosed in the appropriate comment syntax for the file format. We also 175 | recommend that a file or class name and description of purpose be included on 176 | the same "printed page" as the copyright notice for easier identification within 177 | third-party archives. 178 | 179 | Copyright [yyyy] [name of copyright owner] 180 | 181 | Licensed under the Apache License, Version 2.0 (the "License"); 182 | you may not use this file except in compliance with the License. 183 | You may obtain a copy of the License at 184 | 185 | http://www.apache.org/licenses/LICENSE-2.0 186 | 187 | Unless required by applicable law or agreed to in writing, software 188 | distributed under the License is distributed on an "AS IS" BASIS, 189 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 190 | See the License for the specific language governing permissions and 191 | limitations under the License. 192 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include README.rst LICENSE requirements.txt 2 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | Bookstore 2 | ========= 3 | 4 | .. image:: https://badge.fury.io/py/bookstore.png 5 | :target: http://badge.fury.io/py/bookstore 6 | 7 | .. image:: https://travis-ci.org/rgbkrk/bookstore.png?branch=master 8 | :target: https://travis-ci.org/rgbkrk/bookstore 9 | 10 | ⚠️ **This is the 1.x series of `bookstore`. The bookstore 2.x series lives on as a spiritual successor relying on S3, in https://github.com/nteract/bookstore** ⚠️ 11 | 12 | Stores IPython notebooks automagically onto OpenStack clouds through Swift. 13 | 14 | *Add your provider with a pull request!* 15 | 16 | **Note: Bookstore requires IPython 1.0+** 17 | 18 | Bookstore currently has generic support for OpenStack Swift and simplified 19 | authentication for Rackspace's CloudFiles. Bookstore also handles IPython notebook's 20 | autosave/checkpoint feature and as of the latest release supports multiple checkpoints: 21 | 22 | .. image:: https://pbs.twimg.com/media/BVD3olXCMAA2rzb.png 23 | :alt: Multiple checkpoints 24 | 25 | Once installed and configured (added to an ipython profile), just launch 26 | IPython notebook like normal: 27 | 28 | .. code-block:: bash 29 | 30 | $ ipython notebook 31 | 2013-08-01 13:44:19.199 [NotebookApp] Using existing profile dir: u'/Users/rgbkrk/.ipython/profile_default' 32 | 2013-08-01 13:44:25.384 [NotebookApp] Using MathJax from CDN: http://cdn.mathjax.org/mathjax/latest/MathJax.js 33 | 2013-08-01 13:44:25.400 [NotebookApp] Serving rgbkrk's notebooks on Rackspace CloudFiles from container: notebooks 34 | 2013-08-01 13:44:25.400 [NotebookApp] The IPython Notebook is running at: http://127.0.0.1:9999/ 35 | 2013-08-01 13:44:25.400 [NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). 36 | 37 | Installation 38 | ------------ 39 | 40 | Simply: 41 | 42 | .. code-block:: bash 43 | 44 | $ pip install bookstore 45 | 46 | Alternatively, you can always pull from the master branch if you're the adventurous type: 47 | 48 | .. code-block:: bash 49 | 50 | $ pip install git+https://github.com/rgbkrk/bookstore.git 51 | 52 | Installation isn't the end though. You need to configure your account details 53 | as well as where you'll be storing the notebooks. 54 | 55 | Configuration 56 | ------------- 57 | 58 | Bookstore has to be added to an IPython profile and configured to work with 59 | your OpenStack provider. 60 | 61 | If you want to keep it simple, just add your configuration to the default configuration located at: 62 | 63 | .. code-block:: bash 64 | 65 | ~/.ipython/profile_default/ipython_notebook_config.py 66 | 67 | Alternatively, you can create a brand new notebook profile for bookstore: 68 | 69 | .. code-block:: bash 70 | 71 | $ ipython profile create swiftstore 72 | [ProfileCreate] Generating default config file: u'/Users/theuser/.ipython/profile_swiftstore/ipython_config.py' 73 | [ProfileCreate] Generating default config file: u'/Users/theuser/.ipython/profile_swiftstore/ipython_notebook_config.py' 74 | 75 | When launching, just set the custom profile you want to use 76 | 77 | .. code-block:: bash 78 | 79 | $ ipython notebook --profile=swiftstore 80 | 81 | Each provider has their own setup for authentication. 82 | 83 | On OpenStack Swift using Keystone Authentication 84 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 85 | 86 | OpenStack (generic, non provider specific) has quite a few details you'll need 87 | to configure, namely account name, account key, auth endpoint, and region. 88 | You'll possibly need a tenant id and a tenant name. 89 | 90 | Add this to your ipython notebook profile *ipython_notebook_config.py*, making 91 | sure it comes after the config declaration ``c = get_config()``. 92 | 93 | .. code-block:: python 94 | 95 | # Setup IPython Notebook to write notebooks to a Swift Cluster 96 | # that uses Keystone for authentication 97 | c.NotebookApp.notebook_manager_class = 'bookstore.swift.KeystoneNotebookManager' 98 | 99 | # Account details for OpenStack 100 | c.KeystoneNotebookManager.account_name = USER_NAME 101 | c.KeystoneNotebookManager.account_key = API_KEY 102 | c.KeystoneNotebookManager.auth_endpoint = u'127.0.0.1:8021' 103 | c.KeystoneNotebookManager.tenant_id = TENANT_ID 104 | c.KeystoneNotebookManager.tenant_name = TENANT_NAME 105 | c.KeystoneNotebookManager.region = 'RegionOne' 106 | 107 | # Container on OpenStack Swift 108 | c.KeystoneNotebookManager.container_name = u'notebooks' 109 | 110 | On Rackspace's CloudFiles 111 | ~~~~~~~~~~~~~~~~~~~~~~~~~ 112 | 113 | The Rackspace CloudFileNotebookManager simply needs your ``USER_NAME`` and ``API_KEY``. You can also configure the region to store your notebooks (e.g. ``'SYD'``, ``'ORD'``, ``'DFW'``, ``'LON'``). Note: If you're using Rackspace UK, set your region to ``'LON'``. 114 | 115 | Add this to your ipython notebook profile *ipython_notebook_config.py*, making 116 | sure it comes after the config declaration ``c = get_config()``. 117 | 118 | .. code-block:: python 119 | 120 | # Setup IPython Notebook to write notebooks to CloudFiles 121 | c.NotebookApp.notebook_manager_class = 'bookstore.cloudfiles.CloudFilesNotebookManager' 122 | 123 | # Set your user name and API Key 124 | c.CloudFilesNotebookManager.account_name = USER_NAME 125 | c.CloudFilesNotebookManager.account_key = API_KEY 126 | 127 | # Container on CloudFiles 128 | c.CloudFilesNotebookManager.container_name = u'notebooks' 129 | 130 | Contributing 131 | ------------ 132 | 133 | Send a pull request on `GitHub `_. It's 134 | that simple. More than happy to respond to issues on GitHub as well. 135 | 136 | -------------------------------------------------------------------------------- /bookstore/__init__.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # -*- coding: utf-8 -*- 4 | '''Bookstore 5 | 6 | Stores IPython notebooks automagically onto OpenStack clouds through Swift. 7 | ''' 8 | 9 | __title__ = 'bookstore' 10 | __version__ = '1.0.0' 11 | __build__ = 0x010000 12 | __author__ = 'Kyle Kelley' 13 | __license__ = 'Apache 2.0' 14 | __copyright__ = 'Copyright 2013 Kyle Kelley' 15 | 16 | from . import swift 17 | from . import cloudfiles 18 | -------------------------------------------------------------------------------- /bookstore/cloudfiles.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # -*- coding: utf-8 -*- 4 | 5 | """A notebook manager that uses Rackspace CloudFiles. 6 | 7 | Requires IPython 1.0.0+ 8 | 9 | Add this to your ipython notebook profile (`ipython_notebook_config.py`): 10 | 11 | c.NotebookApp.notebook_manager_class = 'bookstore.cloudfiles.CloudFilesNotebookManager' 12 | c.CloudFilesNotebookManager.account_name = USER_NAME 13 | c.CloudFilesNotebookManager.account_key = API_KEY 14 | c.CloudFilesNotebookManager.container_name = u'notebooks' 15 | 16 | You'll need to replace `USER_NAME` and `API_KEY` with your actual username and 17 | api key of course. You can get the API key from the cloud control panel after 18 | logging in. 19 | 20 | It's easy to set up a notebook profile if you don't have one: 21 | 22 | $ ipython profile create bookstore 23 | [ProfileCreate] Generating default config file: u'/Users/theuser/.ipython/profile_bookstore/ipython_config.py' 24 | [ProfileCreate] Generating default config file: u'/Users/theuser/.ipython/profile_bookstore/ipython_notebook_config.py' 25 | 26 | You can also use your default config, located at 27 | 28 | ~/.ipython/profile_default/ipython_notebook_config.py 29 | 30 | """ 31 | 32 | import pyrax 33 | from IPython.utils.traitlets import Unicode 34 | from .swift import SwiftNotebookManager 35 | from tornado import web 36 | 37 | 38 | class CloudFilesNotebookManager(SwiftNotebookManager): 39 | """Manages IPython notebooks on Rackspace's Cloud. 40 | 41 | Rackspace is a known entity (configured OpenStack), so the setup is 42 | simplified. 43 | """ 44 | 45 | account_name = Unicode('', config=True, help='Rackspace username') 46 | account_key = Unicode('', config=True, help='Rackspace API Key') 47 | region = Unicode('DFW', config=True, help='Region') 48 | identity_type = "rackspace" 49 | 50 | def __init__(self, **kwargs): 51 | """Sets up the NotebookManager using the credentials supplied from the 52 | IPython configuration. 53 | """ 54 | super(CloudFilesNotebookManager, self).__init__(**kwargs) 55 | pyrax.set_setting("identity_type", self.identity_type) 56 | pyrax.set_setting("region", self.region) 57 | 58 | pyrax.set_credentials(username=self.account_name, 59 | api_key=self.account_key) 60 | self.cf = pyrax.cloudfiles 61 | 62 | self.container = self.cf.create_container(self.container_name) 63 | 64 | def info_string(self): 65 | """Returns a status string about the Rackspace CloudFiles Notebook 66 | Manager 67 | """ 68 | info = ("Serving {}'s notebooks on Rackspace CloudFiles from " 69 | "container {} in the {} region.") 70 | return info.format(self.account_name, self.container_name, self.region) 71 | -------------------------------------------------------------------------------- /bookstore/swift.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # -*- coding: utf-8 -*- 4 | 5 | """ 6 | A notebook manager that uses OpenStack Swift object storage. 7 | 8 | Requires IPython 1.0.0+ 9 | 10 | Add this to your ipython notebook profile (`ipython_notebook_config.py`), 11 | filling in details for your OpenStack implementation. 12 | 13 | c.NotebookApp.notebook_manager_class = 'bookstore.swift.KeystoneNotebookManager' 14 | c.KeystoneNotebookManager.account_name = USER_NAME 15 | c.KeystoneNotebookManager.account_key = API_KEY 16 | c.KeystoneNotebookManager.container_name = 'notebooks' 17 | c.KeystoneNotebookManager.auth_endpoint = '127.0.0.1:8021' 18 | c.KeystoneNotebookManager.tenant_id = TENANT_ID 19 | c.KeystoneNotebookManager.tenant_name = TENANT_NAME 20 | c.KeystoneNotebookManager.region = 'RegionOne' 21 | 22 | It's easy to set up a notebook profile if you don't have one: 23 | 24 | $ ipython profile create swiftstore 25 | [ProfileCreate] Generating default config file: '/Users/theuser/.ipython/profile_swiftstore/ipython_config.py' 26 | [ProfileCreate] Generating default config file: '/Users/theuser/.ipython/profile_swiftstore/ipython_notebook_config.py' 27 | 28 | You can also use your default config, located at 29 | 30 | ~/.ipython/profile_default/ipython_notebook_config.py 31 | 32 | Notebooks are stored by uuid and checkpoints are stored relative to this uuid 33 | 34 | {notebook_id}/checkpoints/{checkpoint_id} 35 | 36 | """ 37 | 38 | from datetime import datetime 39 | 40 | import pyrax 41 | from pyrax.exceptions import NoSuchContainer 42 | 43 | from tornado import web 44 | 45 | from IPython.html.services.notebooks.nbmanager import NotebookManager 46 | 47 | from IPython.nbformat import current 48 | from IPython.utils.traitlets import Unicode 49 | from IPython.utils.tz import utcnow, tzUTC 50 | 51 | import uuid 52 | 53 | from bookstore import __version__ 54 | 55 | METADATA_NBNAME = 'x-object-meta-nbname' 56 | METADATA_CHK_ID = 'x-object-meta-checkpoint-id' 57 | METADATA_LAST_MODIFIED = 'x-object-meta-nb-last-modified' 58 | METADATA_NB_ID = 'x-object-meta-notebook-id' 59 | 60 | DATE_FORMAT = "%X-%x" 61 | 62 | NB_DNEXIST_ERR = 'Notebook does not exist: {}' 63 | NB_SAVE_UNK_ERR = 'Unexpected error while saving notebook: {}' 64 | NB_DEL_UNK_ERR = 'Unexpected error while deleting notebook: {}' 65 | CHK_SAVE_UNK_ERR = 'Unexpected error while saving checkpoint: {}' 66 | 67 | 68 | class SwiftNotebookManager(NotebookManager): 69 | """This is a base class to be subclassed by OpenStack providers. The swift 70 | object storage should work across implementations, only authentication 71 | should differ. 72 | """ 73 | 74 | user_agent = "bookstore v{version}".format(version=__version__) 75 | container_name = Unicode('notebooks', config=True, 76 | help='Container name for notebooks.') 77 | 78 | def __init__(self, **kwargs): 79 | super(SwiftNotebookManager, self).__init__(**kwargs) 80 | pyrax.set_setting("custom_user_agent", self.user_agent) 81 | 82 | def load_notebook_names(self): 83 | """On startup load the notebook ids and names from OpenStack Swift. 84 | 85 | The object names are the notebook ids and the notebook names are stored 86 | as object metadata. 87 | """ 88 | # Cached version of the mapping of notebook IDs to notebook names 89 | self.mapping = {} 90 | 91 | # Grab only top level notebooks 92 | objects = self.container.get_objects(delimiter='/') 93 | 94 | for obj in objects: 95 | nb_id = obj.name 96 | metadata = obj.get_metadata() 97 | 98 | if(METADATA_NBNAME in metadata): 99 | name = metadata[METADATA_NBNAME] 100 | self.mapping[nb_id] = name 101 | 102 | def list_notebooks(self): 103 | """List all notebooks in the container. 104 | 105 | This version uses `self.mapping` as the authoritative notebook list. 106 | """ 107 | data = [dict(notebook_id=nb_id, name=name) 108 | for nb_id, name in list(self.mapping.items())] 109 | data = sorted(data, key=lambda item: item['name']) 110 | return data 111 | 112 | def read_notebook_object(self, notebook_id): 113 | """Get the object representation of a notebook by notebook_id.""" 114 | if not self.notebook_exists(notebook_id): 115 | raise web.HTTPError(404, NB_DNEXIST_ERR.format(notebook_id)) 116 | try: 117 | obj = self.container.get_object(notebook_id) 118 | 119 | # Read in the entire notebook file into s 120 | s = obj.get() 121 | except: 122 | raise web.HTTPError(500, 'Notebook cannot be read.') 123 | try: 124 | nb = current.reads(s, 'json') 125 | except: 126 | raise web.HTTPError(500, 'Unreadable JSON notebook.') 127 | 128 | last_modified = utcnow() 129 | return last_modified, nb 130 | 131 | def write_notebook_object(self, nb, notebook_id=None): 132 | """Save an existing notebook object by notebook_id.""" 133 | 134 | try: 135 | new_name = nb.metadata.name 136 | except AttributeError: 137 | raise web.HTTPError(400, 'Missing notebook name') 138 | 139 | if notebook_id is None: 140 | notebook_id = self.new_notebook_id(new_name) 141 | 142 | if notebook_id not in self.mapping: 143 | raise web.HTTPError(404, NB_DNEXIST_ERR.format(notebook_id)) 144 | 145 | try: 146 | data = current.writes(nb, 'json') 147 | except Exception as e: 148 | raise web.HTTPError(400, NB_SAVE_UNK_ERR.format(e)) 149 | 150 | metadata = {METADATA_NBNAME: new_name} 151 | try: 152 | obj = self.container.store_object(notebook_id, data) 153 | obj.set_metadata(metadata) 154 | except Exception as e: 155 | raise web.HTTPError(400, NB_SAVE_UNK_ERR.format(e)) 156 | 157 | self.mapping[notebook_id] = new_name 158 | return notebook_id 159 | 160 | def delete_notebook(self, notebook_id): 161 | """Delete notebook by notebook_id. 162 | 163 | Also deletes checkpoints for the notebook. 164 | """ 165 | if not self.notebook_exists(notebook_id): 166 | raise web.HTTPError(404, NB_DNEXIST_ERR.format(notebook_id)) 167 | try: 168 | for obj in self.container.get_objects(prefix=notebook_id): 169 | obj.delete() 170 | except Exception as e: 171 | raise web.HTTPError(400, NB_DEL_UNK_ERR.format(e)) 172 | else: 173 | self.delete_notebook_id(notebook_id) 174 | 175 | def get_checkpoint_path(self, notebook_id, checkpoint_id): 176 | """Returns the canonical checkpoint path based on the notebook_id and 177 | checkpoint_id 178 | """ 179 | checkpoint_path_format = "{}/checkpoints/{}" 180 | return checkpoint_path_format.format(notebook_id, checkpoint_id) 181 | 182 | def new_checkpoint_id(self): 183 | """Generate a new checkpoint_id and store its mapping.""" 184 | return unicode(uuid.uuid4()) 185 | 186 | # Required Checkpoint methods 187 | 188 | def create_checkpoint(self, notebook_id): 189 | """Create a checkpoint of the current state of a notebook 190 | 191 | Returns a dictionary with a checkpoint_id and the timestamp from the 192 | last modification 193 | """ 194 | 195 | self.log.info("Creating checkpoint for notebook {}".format( 196 | notebook_id)) 197 | 198 | # We pull the next available checkpoint id (1UP) 199 | checkpoints = self.container.get_objects(prefix=(notebook_id + "/")) 200 | 201 | checkpoint_id = self.new_checkpoint_id() 202 | 203 | checkpoint_path = self.get_checkpoint_path(notebook_id, checkpoint_id) 204 | 205 | last_modified = utcnow() 206 | 207 | metadata = { 208 | METADATA_CHK_ID: checkpoint_id, 209 | METADATA_LAST_MODIFIED: last_modified.strftime(DATE_FORMAT), 210 | METADATA_NB_ID: notebook_id 211 | } 212 | try: 213 | self.log.info("Copying notebook {} to {}".format( 214 | notebook_id, checkpoint_path)) 215 | self.cf.copy_object(container=self.container_name, 216 | obj=notebook_id, 217 | new_container=self.container_name, 218 | new_obj_name=checkpoint_path) 219 | 220 | obj = self.container.get_object(checkpoint_path) 221 | obj.set_metadata(metadata) 222 | 223 | except Exception as e: 224 | raise web.HTTPError(400, CHK_SAVE_UNK_ERR.format(e)) 225 | 226 | info = dict(checkpoint_id=checkpoint_id, 227 | last_modified=last_modified) 228 | 229 | return info 230 | 231 | def list_checkpoints(self, notebook_id): 232 | """Return a list of checkpoints for a given notebook""" 233 | # Going to have to re-think this later. This is just something to try 234 | # out for the moment 235 | self.log.info("Listing checkpoints for notebook {}".format( 236 | notebook_id)) 237 | try: 238 | objects = self.container.get_objects(prefix=(notebook_id + "/")) 239 | 240 | self.log.debug("Checkpoints = {}".format(objects)) 241 | 242 | checkpoints = [] 243 | for obj in objects: 244 | try: 245 | metadata = obj.get_metadata() 246 | self.log.debug("Object: {}".format(obj.name)) 247 | self.log.debug("Metadata: {}".format(metadata)) 248 | 249 | last_modified = datetime.strptime( 250 | metadata[METADATA_LAST_MODIFIED], 251 | DATE_FORMAT) 252 | last_modified = last_modified.replace(tzinfo=tzUTC()) 253 | info = dict( 254 | checkpoint_id=metadata[METADATA_CHK_ID], 255 | last_modified=last_modified, 256 | ) 257 | checkpoints.append(info) 258 | 259 | except Exception as e: 260 | self.log.error("Unable to pull metadata") 261 | self.log.error("Exception: {}".format(e)) 262 | 263 | except Exception as e: 264 | raise web.HTTPError(400, "Unexpected error while listing" + 265 | "checkpoints") 266 | 267 | checkpoints = sorted(checkpoints, key=lambda item: item['last_modified']) 268 | 269 | self.log.debug("Checkpoints to list: {}".format(checkpoints)) 270 | 271 | return checkpoints 272 | 273 | def restore_checkpoint(self, notebook_id, checkpoint_id): 274 | """Restore a notebook from one of its checkpoints. 275 | 276 | Actually overwrites the existing notebook 277 | """ 278 | 279 | self.log.info("Restoring checkpoint {} for notebook {}".format( 280 | checkpoint_id, notebook_id)) 281 | 282 | if not self.notebook_exists(notebook_id): 283 | raise web.HTTPError(404, NB_DNEXIST_ERR.format(notebook_id)) 284 | 285 | checkpoint_path = self.get_checkpoint_path(notebook_id, checkpoint_id) 286 | 287 | try: 288 | self.cf.copy_object(container=self.container_name, 289 | obj=checkpoint_path, 290 | new_container=self.container_name, 291 | new_obj_name=notebook_id) 292 | except: 293 | raise web.HTTPError(500, 'Checkpoint could not be restored.') 294 | 295 | def delete_checkpoint(self, notebook_id, checkpoint_id): 296 | """Delete a checkpoint for a notebook""" 297 | 298 | self.log.info("Deleting checkpoint {} for notebook {}".format( 299 | checkpoint_id, notebook_id)) 300 | 301 | if not self.notebook_exists(notebook_id): 302 | raise web.HTTPError(404, NB_DNEXIST_ERR.format(notebook_id)) 303 | 304 | checkpoint_path = self.get_checkpoint_path(notebook_id, checkpoint_id) 305 | 306 | try: 307 | self.container.delete_object(checkpoint_path) 308 | except Exception as e: 309 | nb_delete_err_msg = 'Unexpected error while deleting notebook: {}' 310 | raise web.HTTPError(400, nb_delete_err_msg.format(e)) 311 | 312 | def info_string(self): 313 | info = ("Serving {}'s notebooks from OpenStack Swift " 314 | "storage container: {}") 315 | return info.format(self.account_name, self.container_name) 316 | 317 | 318 | class KeystoneNotebookManager(SwiftNotebookManager): 319 | """Manages IPython notebooks on OpenStack Swift, using Keystone 320 | authentication. 321 | 322 | Extend this class with the defaults for your OpenStack provider to make 323 | configuration for clients easier. 324 | """ 325 | account_name = Unicode('', config=True, help='OpenStack account name.') 326 | account_key = Unicode('', config=True, help='OpenStack account key.') 327 | auth_endpoint = Unicode('', config=True, help='Authentication endpoint.') 328 | 329 | region = Unicode('RegionOne', config=True, 330 | help='Region (e.g. RegionOne, ORD, LON)') 331 | 332 | tenant_id = Unicode('', config=True, 333 | help='The tenant ID used for authentication') 334 | tenant_name = Unicode('', config=True, 335 | help='The tenant name used for authentication') 336 | 337 | identity_type = 'keystone' 338 | 339 | def __init__(self, **kwargs): 340 | super(SwiftNotebookManager, self).__init__(**kwargs) 341 | pyrax.set_setting("identity_type", self.identity_type) 342 | pyrax.set_setting("auth_endpoint", self.auth_endpoint) 343 | pyrax.set_setting("region", self.region) 344 | pyrax.set_setting("tenant_id", self.tenant_id) 345 | pyrax.set_setting("tenant_name", self.tenant_name) 346 | 347 | # Set creds and authenticate 348 | pyrax.set_credentials(username=self.account_name, 349 | api_key=self.account_key) 350 | 351 | self.cf = pyrax.cloudfiles 352 | 353 | try: 354 | self.container = self.cf.get_container(self.container_name) 355 | except NoSuchContainer: 356 | self.container = self.cf.create_container(self.container_name) 357 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | pyrax>=1.4.10 2 | ipython[notebook]>=1.1.0 3 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [wheel] 2 | universal = 1 3 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # -*- coding: utf-8 -*- 4 | 5 | import os 6 | import sys 7 | 8 | import re 9 | 10 | try: 11 | from setuptools import setup 12 | except ImportError: 13 | from distutils.core import setup 14 | 15 | # Backwards compatibility for Python 2.x 16 | try: 17 | from itertools import ifilter 18 | filter = ifilter 19 | except ImportError: 20 | pass 21 | 22 | 23 | def get_version(): 24 | ''' 25 | Version slurping without importing bookstore, since dependencies may not be 26 | met until setup is run. 27 | ''' 28 | version_regex = re.compile(r"__version__\s+=\s+" 29 | r"['\"](\d+.\d+.\d+\w*)['\"]$") 30 | versions = filter(version_regex.match, open("bookstore/__init__.py")) 31 | 32 | try: 33 | version = next(versions) 34 | except StopIteration: 35 | raise Exception("Bookstore version not set") 36 | 37 | return version_regex.match(version).group(1) 38 | 39 | version = get_version() 40 | 41 | # Utility for publishing the bookstore, courtesy kennethreitz/requests 42 | if sys.argv[-1] == 'publish': 43 | print("Publishing bookstore {version}".format(version=version)) 44 | os.system('python setup.py sdist upload') 45 | sys.exit() 46 | 47 | packages = ['bookstore'] 48 | requires = [] 49 | 50 | with open('requirements.txt') as reqs: 51 | requires = reqs.read().splitlines() 52 | 53 | setup(name='bookstore', 54 | version=version, 55 | description='IPython notebook storage on OpenStack Swift + Rackspace.', 56 | long_description=open('README.rst').read(), 57 | author='Kyle Kelley', 58 | author_email='rgbkrk@gmail.com', 59 | url='http://github.com/rgbkrk/bookstore', 60 | packages=packages, 61 | package_data={'': ['LICENSE']}, 62 | include_package_data=False, 63 | install_requires=requires, 64 | license=open('LICENSE').read(), 65 | zip_safe=False, 66 | classifiers=( 67 | 'Development Status :: 5 - Production/Stable', 68 | 'Intended Audience :: Developers', 69 | 'Intended Audience :: Science/Research', 70 | 'Framework :: IPython', 71 | 'Environment :: OpenStack', 72 | 'License :: OSI Approved :: Apache Software License', 73 | 'Natural Language :: English', 74 | 'Programming Language :: Python', 75 | 'Programming Language :: Python :: 2.6', 76 | 'Programming Language :: Python :: 2.7', 77 | 'Topic :: System :: Distributed Computing', 78 | ), 79 | ) 80 | -------------------------------------------------------------------------------- /tasks.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | import bookstore 5 | from invoke import run, task 6 | 7 | 8 | @task 9 | def test(): 10 | run('py.test', pty=True) 11 | -------------------------------------------------------------------------------- /tests/test_bookstore.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | """ 5 | Tests for bookstore 6 | """ 7 | 8 | import unittest 9 | import doctest 10 | 11 | import bookstore 12 | 13 | 14 | class BookstoreTestCase(unittest.TestCase): 15 | def setUp(self): 16 | pass 17 | 18 | def tearDown(self): 19 | pass 20 | 21 | def test_entry_points(self): 22 | bookstore.swift 23 | bookstore.cloudfiles 24 | 25 | if __name__ == "__main__": 26 | unittest.main() 27 | --------------------------------------------------------------------------------