├── .coveragerc ├── .gitignore ├── .travis.yml ├── AUTHORS ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── Makefile ├── README.rst ├── docs ├── Makefile ├── backends.rst ├── changelog.rst ├── conf.py ├── fields.rst ├── index.rst ├── make.bat ├── requirements.txt ├── signals.rst └── tasks.rst ├── queued_storage ├── __init__.py ├── backends.py ├── conf.py ├── fields.py ├── models.py ├── signals.py ├── tasks.py └── utils.py ├── setup.cfg ├── setup.py ├── tests ├── __init__.py ├── celeryconfig.py ├── models.py ├── requirements.txt ├── settings.py ├── tasks.py └── test_storages.py └── tox.ini /.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | source = queued_storage 3 | branch = 1 4 | 5 | [report] 6 | omit = *tests* 7 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.egg-info 2 | *.egg 3 | .coverage 4 | *.pyc 5 | .project 6 | .pydevproject 7 | test.db 8 | build/ 9 | .tox/ 10 | htmlcov/ 11 | *.pyc 12 | .coverage 13 | .python-version 14 | .idea 15 | dist/ 16 | .cache 17 | .eggs 18 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | sudo: false 2 | language: python 3 | cache: 4 | directories: 5 | - $HOME/.cache/pip 6 | matrix: 7 | include: 8 | - python: "2.7" 9 | env: TOXENV=py27-django-17 10 | - python: "2.7" 11 | env: TOXENV=py27-django-18 12 | - python: "2.7" 13 | env: TOXENV=py27-django-19 14 | - python: "2.7" 15 | env: TOXENV=py27-django-110 16 | - python: "2.7" 17 | env: TOXENV=py27-django-111 18 | 19 | - python: "3.4" 20 | env: TOXENV=py34-django-17 21 | - python: "3.4" 22 | env: TOXENV=py34-django-18 23 | - python: "3.4" 24 | env: TOXENV=py34-django-19 25 | - python: "3.4" 26 | env: TOXENV=py34-django-110 27 | - python: "3.4" 28 | env: TOXENV=py34-django-111 29 | 30 | - python: "3.5" 31 | env: TOXENV=py35-django-18 32 | - python: "3.5" 33 | env: TOXENV=py35-django-19 34 | - python: "3.5" 35 | env: TOXENV=py35-django-110 36 | - python: "3.5" 37 | env: TOXENV=py35-django-111 38 | - python: "3.5" 39 | env: TOXENV=py35-django-dev 40 | 41 | - python: "3.6" 42 | env: TOXENV=py36-django-dev 43 | 44 | allow_failures: 45 | - env: TOXENV=py35-django-dev 46 | - env: TOXENV=py36-django-dev 47 | 48 | install: pip install tox tox-travis coveralls wheel 49 | script: tox 50 | after_success: coveralls 51 | deploy: 52 | provider: pypi 53 | user: jazzband 54 | distributions: "sdist bdist_wheel" 55 | on: 56 | tags: true 57 | repo: jazzband/django-queued-storage 58 | condition: "$TOXENV = py27-django-110" 59 | password: 60 | secure: scGiNi9NXOdVaWzly7Tc65SBbC7iUVkcaO9ibBFfJq2QB0QM7es9QeXhpBz/biHbgCh40zuP8o2XmmsENxyaebb712k657YFS6fAcNmau7txCMOxDceQNoN/2OhSmGpYsegl8EdwBd4HsoB0fpLuq4bUg6OUkVkqR4pe4afq4O0= 61 | -------------------------------------------------------------------------------- /AUTHORS: -------------------------------------------------------------------------------- 1 | Maintainers: 2 | Jannis Leidel 3 | Ashley Camba Garrido 4 | 5 | Contributors: 6 | 7 | Alen Mujezinovic 8 | Allisson Azevedo 9 | Ashley Camba 10 | Celia Oakley 11 | Éric Araujo 12 | Hank Sims 13 | Josh VanderLinden 14 | Malcolm Tredinnick 15 | Sean Brant 16 | Shabda Raaj 17 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Code of Conduct 2 | 3 | As contributors and maintainers of the Jazzband projects, and in the interest of 4 | fostering an open and welcoming community, we pledge to respect all people who 5 | contribute through reporting issues, posting feature requests, updating documentation, 6 | submitting pull requests or patches, and other activities. 7 | 8 | We are committed to making participation in the Jazzband a harassment-free experience 9 | for everyone, regardless of the level of experience, gender, gender identity and 10 | expression, sexual orientation, disability, personal appearance, body size, race, 11 | ethnicity, age, religion, or nationality. 12 | 13 | Examples of unacceptable behavior by participants include: 14 | 15 | - The use of sexualized language or imagery 16 | - Personal attacks 17 | - Trolling or insulting/derogatory comments 18 | - Public or private harassment 19 | - Publishing other's private information, such as physical or electronic addresses, 20 | without explicit permission 21 | - Other unethical or unprofessional conduct 22 | 23 | The Jazzband roadies have the right and responsibility to remove, edit, or reject 24 | comments, commits, code, wiki edits, issues, and other contributions that are not 25 | aligned to this Code of Conduct, or to ban temporarily or permanently any contributor 26 | for other behaviors that they deem inappropriate, threatening, offensive, or harmful. 27 | 28 | By adopting this Code of Conduct, the roadies commit themselves to fairly and 29 | consistently applying these principles to every aspect of managing the jazzband 30 | projects. Roadies who do not follow or enforce the Code of Conduct may be permanently 31 | removed from the Jazzband roadies. 32 | 33 | This code of conduct applies both within project spaces and in public spaces when an 34 | individual is representing the project or its community. 35 | 36 | Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by 37 | contacting the roadies at `roadies@jazzband.co`. All complaints will be reviewed and 38 | investigated and will result in a response that is deemed necessary and appropriate to 39 | the circumstances. Roadies are obligated to maintain confidentiality with regard to the 40 | reporter of an incident. 41 | 42 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 43 | 1.3.0, available at [https://contributor-covenant.org/version/1/3/0/][version] 44 | 45 | [homepage]: https://contributor-covenant.org 46 | [version]: https://contributor-covenant.org/version/1/3/0/ 47 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | [![Jazzband](https://jazzband.co/static/img/jazzband.svg)](https://jazzband.co/) 2 | 3 | This is a [Jazzband](https://jazzband.co/) project. By contributing you agree to abide by the [Contributor Code of Conduct](https://jazzband.co/docs/conduct) and follow the [guidelines](https://jazzband.co/docs/guidelines). 4 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2010-2012, django-queued-storage contributors (see AUTHORS file) 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are 6 | met: 7 | 8 | * Redistributions of source code must retain the above copyright 9 | notice, this list of conditions and the following disclaimer. 10 | * Redistributions in binary form must reproduce the above 11 | copyright notice, this list of conditions and the following 12 | disclaimer in the documentation and/or other materials provided 13 | with the distribution. 14 | * Neither the name of the author nor the names of other 15 | contributors may be used to endorse or promote products derived 16 | from this software without specific prior written permission. 17 | 18 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 19 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 20 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 21 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 22 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 23 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 24 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 25 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 26 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 27 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 28 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | .PHONY: test release 2 | 3 | test: 4 | py.test 5 | 6 | release: 7 | python setup.py sdist bdist_wheel register upload -s 8 | 9 | 10 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | django-queued-storage 2 | ===================== 3 | 4 | .. image:: https://img.shields.io/pypi/v/django-queued-storage.svg 5 | :alt: PyPi page 6 | :target: https://pypi.python.org/pypi/django-queued-storage 7 | 8 | .. image:: https://img.shields.io/travis/jazzband/django-queued-storage.svg 9 | :alt: Travis CI Status 10 | :target: https://travis-ci.org/jazzband/django-queued-storage 11 | 12 | .. image:: https://img.shields.io/coveralls/jazzband/django-queued-storage/master.svg 13 | :alt: Coverage status 14 | :target: https://coveralls.io/r/jazzband/django-queued-storage 15 | 16 | .. image:: https://readthedocs.org/projects/django-queued-storage/badge/?version=latest&style=flat 17 | :alt: ReadTheDocs 18 | :target: https://django-queued-storage.readthedocs.io/en/latest/ 19 | 20 | .. image:: https://img.shields.io/pypi/l/django-queued-storage.svg 21 | :alt: License BSD 22 | 23 | .. image:: https://jazzband.co/static/img/badge.svg 24 | :target: https://jazzband.co/ 25 | :alt: Jazzband 26 | 27 | This storage backend enables having a local and a remote storage 28 | backend. It will save any file locally and queue a task to transfer it 29 | somewhere else using Celery_. 30 | 31 | If the file is accessed before it's transferred, the local copy is 32 | returned. 33 | 34 | Installation 35 | ------------ 36 | 37 | :: 38 | 39 | pip install django-queued-storage 40 | 41 | Configuration 42 | ------------- 43 | 44 | - Follow the configuration instructions for 45 | django-celery_ 46 | - Set up a `caching backend`_ 47 | - Add ``'queued_storage'`` to your ``INSTALLED_APPS`` setting 48 | 49 | .. _django-celery: https://github.com/ask/django-celery 50 | .. _`caching backend`: https://docs.djangoproject.com/en/1.10/topics/cache/#setting-up-the-cache 51 | .. _Celery: http://celeryproject.org/ 52 | -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- 1 | # Makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = sphinx-build 7 | PAPER = 8 | BUILDDIR = _build 9 | 10 | # Internal variables. 11 | PAPEROPT_a4 = -D latex_paper_size=a4 12 | PAPEROPT_letter = -D latex_paper_size=letter 13 | ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 14 | 15 | .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest 16 | 17 | help: 18 | @echo "Please use \`make ' where is one of" 19 | @echo " html to make standalone HTML files" 20 | @echo " dirhtml to make HTML files named index.html in directories" 21 | @echo " singlehtml to make a single large HTML file" 22 | @echo " pickle to make pickle files" 23 | @echo " json to make JSON files" 24 | @echo " htmlhelp to make HTML files and a HTML help project" 25 | @echo " qthelp to make HTML files and a qthelp project" 26 | @echo " devhelp to make HTML files and a Devhelp project" 27 | @echo " epub to make an epub" 28 | @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" 29 | @echo " latexpdf to make LaTeX files and run them through pdflatex" 30 | @echo " text to make text files" 31 | @echo " man to make manual pages" 32 | @echo " changes to make an overview of all changed/added/deprecated items" 33 | @echo " linkcheck to check all external links for integrity" 34 | @echo " doctest to run all doctests embedded in the documentation (if enabled)" 35 | 36 | clean: 37 | -rm -rf $(BUILDDIR)/* 38 | 39 | html: 40 | $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html 41 | @echo 42 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." 43 | 44 | dirhtml: 45 | $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml 46 | @echo 47 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." 48 | 49 | singlehtml: 50 | $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml 51 | @echo 52 | @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." 53 | 54 | pickle: 55 | $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle 56 | @echo 57 | @echo "Build finished; now you can process the pickle files." 58 | 59 | json: 60 | $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json 61 | @echo 62 | @echo "Build finished; now you can process the JSON files." 63 | 64 | htmlhelp: 65 | $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp 66 | @echo 67 | @echo "Build finished; now you can run HTML Help Workshop with the" \ 68 | ".hhp project file in $(BUILDDIR)/htmlhelp." 69 | 70 | qthelp: 71 | $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp 72 | @echo 73 | @echo "Build finished; now you can run "qcollectiongenerator" with the" \ 74 | ".qhcp project file in $(BUILDDIR)/qthelp, like this:" 75 | @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/django-queued-storage.qhcp" 76 | @echo "To view the help file:" 77 | @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/django-queued-storage.qhc" 78 | 79 | devhelp: 80 | $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp 81 | @echo 82 | @echo "Build finished." 83 | @echo "To view the help file:" 84 | @echo "# mkdir -p $$HOME/.local/share/devhelp/django-queued-storage" 85 | @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/django-queued-storage" 86 | @echo "# devhelp" 87 | 88 | epub: 89 | $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub 90 | @echo 91 | @echo "Build finished. The epub file is in $(BUILDDIR)/epub." 92 | 93 | latex: 94 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 95 | @echo 96 | @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." 97 | @echo "Run \`make' in that directory to run these through (pdf)latex" \ 98 | "(use \`make latexpdf' here to do that automatically)." 99 | 100 | latexpdf: 101 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 102 | @echo "Running LaTeX files through pdflatex..." 103 | make -C $(BUILDDIR)/latex all-pdf 104 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 105 | 106 | text: 107 | $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text 108 | @echo 109 | @echo "Build finished. The text files are in $(BUILDDIR)/text." 110 | 111 | man: 112 | $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man 113 | @echo 114 | @echo "Build finished. The manual pages are in $(BUILDDIR)/man." 115 | 116 | changes: 117 | $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes 118 | @echo 119 | @echo "The overview file is in $(BUILDDIR)/changes." 120 | 121 | linkcheck: 122 | $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck 123 | @echo 124 | @echo "Link check complete; look for any errors in the above output " \ 125 | "or in $(BUILDDIR)/linkcheck/output.txt." 126 | 127 | doctest: 128 | $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest 129 | @echo "Testing of doctests in the sources finished, look at the " \ 130 | "results in $(BUILDDIR)/doctest/output.txt." 131 | -------------------------------------------------------------------------------- /docs/backends.rst: -------------------------------------------------------------------------------- 1 | Backends 2 | ======== 3 | 4 | .. currentmodule:: queued_storage.backends 5 | 6 | .. autoclass:: QueuedStorage 7 | :members: 8 | :undoc-members: 9 | :inherited-members: 10 | 11 | .. autoclass:: QueuedFileSystemStorage 12 | :members: 13 | 14 | .. automodule:: queued_storage.backends 15 | :members: 16 | :exclude-members: QueuedStorage, QueuedFileSystemStorage 17 | -------------------------------------------------------------------------------- /docs/changelog.rst: -------------------------------------------------------------------------------- 1 | Changelog 2 | ========= 3 | 4 | v0.8 (2015-12-14) 5 | ----------------- 6 | 7 | - Added Django 1.9 support 8 | 9 | v0.7.2 (2015-12-02) 10 | ------------------- 11 | 12 | - Documentation config fixes. 13 | 14 | v0.7.1 (2015-12-02) 15 | ------------------- 16 | 17 | - Fix dependency on django-appconf. 18 | 19 | - Minor code cleanup. 20 | 21 | v0.7 (2015-12-02) 22 | ----------------- 23 | 24 | - Dropping Django 1.6 support 25 | - Dropping Python 2.6 and 3.2 support 26 | - Switched testing to use tox and py.test 27 | - Added Python 3 support 28 | - Switched to using `setuptools_scm `_ 29 | - Transfered to Jazzband: https://github.com/jazzband/django-queued-storage 30 | - Tests can be found at: http://travis-ci.org/jazzband/django-queued-storage 31 | 32 | v0.6 (2012-05-24) 33 | ----------------- 34 | 35 | - Added ``file_transferred`` signal that is called right after a file has been 36 | transfered from the local to the remote storage. 37 | 38 | - Switched to using `django-discover-runner`_ and Travis for testing: 39 | http://travis-ci.org/jezdez/django-queued-storage 40 | 41 | .. _`django-discover-runner`: http://pypi.python.org/pypi/django-discover-runner 42 | 43 | v0.5 (2012-03-19) 44 | ----------------- 45 | 46 | - Fixed retrying in case of errors. 47 | 48 | - Dropped Python 2.5 support as Celery has dropped it, too. 49 | 50 | - Use django-jenkins. 51 | 52 | v0.4 (2011-11-03) 53 | ----------------- 54 | 55 | - Revised storage parameters to fix an incompatibility with Celery regarding 56 | task parameter passing and pickling. 57 | 58 | It's now *required* to pass the dotted Python import path of the local 59 | and remote storage backend, as well as a dictionary of options for 60 | instantiation of those classes (if needed). Passing storage instances 61 | to the :class:`~queued_storage.backends.QueuedStorage` class is now 62 | considered an error. For example: 63 | 64 | Old:: 65 | 66 | from django.core.files.storage import FileSystemStorage 67 | from mysite.storage import MyCustomStorageBackend 68 | 69 | my_storage = QueuedStorage( 70 | FileSystemStorage(location='/path/to/files'), 71 | MyCustomStorageBackend(spam='eggs')) 72 | 73 | New:: 74 | 75 | my_storage = QueuedStorage( 76 | local='django.core.files.storage.FileSystemStorage', 77 | local_options={'location': '/path/to/files'}, 78 | remote='mysite.storage.MyCustomStorageBackend', 79 | remote_options={'spam': 'eggs'}) 80 | 81 | .. warning:: 82 | 83 | This change is backwards-incompatible if you used the 84 | :class:`~queued_storage.backends.QueuedStorage` API. 85 | 86 | v0.3 (2011-09-19) 87 | ----------------- 88 | 89 | - Initial release. 90 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # django-queued-storage documentation build configuration file, created by 4 | # sphinx-quickstart on Sun Sep 18 19:10:44 2011. 5 | # 6 | # This file is execfile()d with the current directory set to its containing dir. 7 | # 8 | # Note that not all possible configuration values are present in this 9 | # autogenerated file. 10 | # 11 | # All configuration values have a default; values that are commented out 12 | # serve to show the default. 13 | 14 | import sys, os 15 | 16 | # If extensions (or modules to document with autodoc) are in another directory, 17 | # add these directories to sys.path here. If the directory is relative to the 18 | # documentation root, use os.path.abspath to make it absolute, like shown here. 19 | sys.path.insert(0, os.path.abspath('..')) 20 | 21 | os.environ['DJANGO_SETTINGS_MODULE'] = 'tests.settings' 22 | 23 | # -- General configuration ----------------------------------------------------- 24 | 25 | # If your documentation needs a minimal Sphinx version, state it here. 26 | # needs_sphinx = '1.0' 27 | 28 | # Add any Sphinx extension module names here, as strings. They can be extensions 29 | # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. 30 | extensions = ['sphinx.ext.autodoc', 'sphinx.ext.intersphinx'] 31 | 32 | # Add any paths that contain templates here, relative to this directory. 33 | templates_path = ['_templates'] 34 | 35 | # The suffix of source filenames. 36 | source_suffix = '.rst' 37 | 38 | # The encoding of source files. 39 | #source_encoding = 'utf-8-sig' 40 | 41 | # The master toctree document. 42 | master_doc = 'index' 43 | 44 | # General information about the project. 45 | project = u'django-queued-storage' 46 | copyright = u'2012-2015, Jannis Leidel and contributors' 47 | 48 | # The version info for the project you're documenting, acts as replacement for 49 | # |version| and |release|, also used in various other places throughout the 50 | # built documents. 51 | # 52 | try: 53 | from queued_storage import __version__ 54 | # The short X.Y version. 55 | version = '.'.join(__version__.split('.')[:2]) 56 | # The full version, including alpha/beta/rc tags. 57 | release = __version__ 58 | except ImportError: 59 | version = release = 'dev' 60 | 61 | # The language for content autogenerated by Sphinx. Refer to documentation 62 | # for a list of supported languages. 63 | #language = None 64 | 65 | # There are two options for replacing |today|: either, you set today to some 66 | # non-false value, then it is used: 67 | #today = '' 68 | # Else, today_fmt is used as the format for a strftime call. 69 | #today_fmt = '%B %d, %Y' 70 | 71 | # List of patterns, relative to source directory, that match files and 72 | # directories to ignore when looking for source files. 73 | exclude_patterns = ['_build'] 74 | 75 | # The reST default role (used for this markup: `text`) to use for all documents. 76 | #default_role = None 77 | 78 | # If true, '()' will be appended to :func: etc. cross-reference text. 79 | #add_function_parentheses = True 80 | 81 | # If true, the current module name will be prepended to all description 82 | # unit titles (such as .. function::). 83 | #add_module_names = True 84 | 85 | # If true, sectionauthor and moduleauthor directives will be shown in the 86 | # output. They are ignored by default. 87 | #show_authors = False 88 | 89 | # The name of the Pygments (syntax highlighting) style to use. 90 | pygments_style = 'sphinx' 91 | 92 | # A list of ignored prefixes for module index sorting. 93 | #modindex_common_prefix = [] 94 | 95 | 96 | # -- Options for HTML output --------------------------------------------------- 97 | 98 | # The theme to use for HTML and HTML Help pages. See the documentation for 99 | # a list of builtin themes. 100 | import sphinx_rtd_theme 101 | html_theme = 'sphinx_rtd_theme' 102 | html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] 103 | 104 | # Theme options are theme-specific and customize the look and feel of a theme 105 | # further. For a list of options available for each theme, see the 106 | # documentation. 107 | #html_theme_options = {} 108 | 109 | # Add any paths that contain custom themes here, relative to this directory. 110 | #html_theme_path = [] 111 | 112 | # The name for this set of Sphinx documents. If None, it defaults to 113 | # " v documentation". 114 | #html_title = None 115 | 116 | # A shorter title for the navigation bar. Default is the same as html_title. 117 | #html_short_title = None 118 | 119 | # The name of an image file (relative to this directory) to place at the top 120 | # of the sidebar. 121 | #html_logo = None 122 | 123 | # The name of an image file (within the static path) to use as favicon of the 124 | # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 125 | # pixels large. 126 | #html_favicon = None 127 | 128 | # Add any paths that contain custom static files (such as style sheets) here, 129 | # relative to this directory. They are copied after the builtin static files, 130 | # so a file named "default.css" will overwrite the builtin "default.css". 131 | #html_static_path = ['_static'] 132 | 133 | # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, 134 | # using the given strftime format. 135 | #html_last_updated_fmt = '%b %d, %Y' 136 | 137 | # If true, SmartyPants will be used to convert quotes and dashes to 138 | # typographically correct entities. 139 | #html_use_smartypants = True 140 | 141 | # Custom sidebar templates, maps document names to template names. 142 | #html_sidebars = {} 143 | 144 | # Additional templates that should be rendered to pages, maps page names to 145 | # template names. 146 | #html_additional_pages = {} 147 | 148 | # If false, no module index is generated. 149 | #html_domain_indices = True 150 | 151 | # If false, no index is generated. 152 | #html_use_index = True 153 | 154 | # If true, the index is split into individual pages for each letter. 155 | #html_split_index = False 156 | 157 | # If true, links to the reST sources are added to the pages. 158 | #html_show_sourcelink = True 159 | 160 | # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. 161 | #html_show_sphinx = True 162 | 163 | # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. 164 | #html_show_copyright = True 165 | 166 | # If true, an OpenSearch description file will be output, and all pages will 167 | # contain a tag referring to it. The value of this option must be the 168 | # base URL from which the finished HTML is served. 169 | #html_use_opensearch = '' 170 | 171 | # This is the file name suffix for HTML files (e.g. ".xhtml"). 172 | #html_file_suffix = None 173 | 174 | # Output file base name for HTML help builder. 175 | htmlhelp_basename = 'django-queued-storagedoc' 176 | 177 | 178 | # -- Options for LaTeX output -------------------------------------------------- 179 | 180 | # The paper size ('letter' or 'a4'). 181 | #latex_paper_size = 'letter' 182 | 183 | # The font size ('10pt', '11pt' or '12pt'). 184 | #latex_font_size = '10pt' 185 | 186 | # Grouping the document tree into LaTeX files. List of tuples 187 | # (source start file, target name, title, author, documentclass [howto/manual]). 188 | latex_documents = [ 189 | ('index', 'django-queued-storage.tex', u'django-queued-storage Documentation', 190 | u'Jannis Leidel', 'manual'), 191 | ] 192 | 193 | # The name of an image file (relative to this directory) to place at the top of 194 | # the title page. 195 | #latex_logo = None 196 | 197 | # For "manual" documents, if this is true, then toplevel headings are parts, 198 | # not chapters. 199 | #latex_use_parts = False 200 | 201 | # If true, show page references after internal links. 202 | #latex_show_pagerefs = False 203 | 204 | # If true, show URL addresses after external links. 205 | #latex_show_urls = False 206 | 207 | # Additional stuff for the LaTeX preamble. 208 | #latex_preamble = '' 209 | 210 | # Documents to append as an appendix to all manuals. 211 | #latex_appendices = [] 212 | 213 | # If false, no module index is generated. 214 | #latex_domain_indices = True 215 | 216 | 217 | # -- Options for manual page output -------------------------------------------- 218 | 219 | # One entry per manual page. List of tuples 220 | # (source start file, name, description, authors, manual section). 221 | man_pages = [ 222 | ('index', 'django-queued-storage', u'django-queued-storage Documentation', 223 | [u'Jannis Leidel'], 1) 224 | ] 225 | 226 | # Example configuration for intersphinx: refer to the Python standard library. 227 | intersphinx_mapping = { 228 | 'python': ('https://python.readthedocs.io/en/v2.7.2/', None), 229 | 'django': ('https://django.readthedocs.io/en/latest/', None), 230 | 'celery': ('https://celery.readthedocs.io/en/latest/', None), 231 | } 232 | 233 | autodoc_member_order = 'bysource' 234 | -------------------------------------------------------------------------------- /docs/fields.rst: -------------------------------------------------------------------------------- 1 | Fields 2 | ====== 3 | 4 | .. currentmodule:: queued_storage.fields 5 | 6 | .. autoclass:: QueuedFileField 7 | :members: 8 | 9 | .. autoclass:: QueuedFieldFile 10 | :members: 11 | -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../README.rst 2 | 3 | Usage 4 | ----- 5 | 6 | The :class:`~queued_storage.backends.QueuedStorage` can be used as a drop-in 7 | replacement wherever using :class:`django:django.core.files.storage.Storage` 8 | might otherwise be required. 9 | 10 | This example is using django-storages_ for the remote backend:: 11 | 12 | from django.db import models 13 | from queued_storage.backends import QueuedStorage 14 | from storages.backends.s3boto import S3BotoStorage 15 | 16 | queued_s3storage = QueuedStorage( 17 | 'django.core.files.storage.FileSystemStorage', 18 | 'storages.backends.s3boto.S3BotoStorage') 19 | 20 | class MyModel(models.Model): 21 | image = ImageField(storage=queued_s3storage) 22 | 23 | .. _django-storages: http://django-storages.readthedocs.io/en/latest/ 24 | 25 | Settings 26 | -------- 27 | 28 | .. currentmodule:: queued_storage.conf.settings 29 | 30 | .. attribute:: QUEUED_STORAGE_CACHE_PREFIX 31 | 32 | :Default: ``'queued_storage'`` 33 | 34 | The cache key prefix to use when caching the storage backends. 35 | 36 | .. attribute:: QUEUED_STORAGE_RETRIES 37 | 38 | :Default: ``5`` 39 | 40 | How many retries should be attempted before aborting. 41 | 42 | .. attribute:: QUEUED_STORAGE_RETRY_DELAY 43 | 44 | :Default: ``60`` 45 | 46 | The delay between retries in seconds. 47 | 48 | Reference 49 | --------- 50 | 51 | For further details see the reference documentation: 52 | 53 | .. toctree:: 54 | :maxdepth: 1 55 | 56 | backends 57 | fields 58 | tasks 59 | signals 60 | changelog 61 | 62 | Issues 63 | ------ 64 | 65 | For any bug reports and feature requests, please use the 66 | `Github issue tracker`_. 67 | 68 | .. _`Github issue tracker`: https://github.com/jazzband/django-queued-storage/issues 69 | -------------------------------------------------------------------------------- /docs/make.bat: -------------------------------------------------------------------------------- 1 | @ECHO OFF 2 | 3 | REM Command file for Sphinx documentation 4 | 5 | if "%SPHINXBUILD%" == "" ( 6 | set SPHINXBUILD=sphinx-build 7 | ) 8 | set BUILDDIR=_build 9 | set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% . 10 | if NOT "%PAPER%" == "" ( 11 | set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% 12 | ) 13 | 14 | if "%1" == "" goto help 15 | 16 | if "%1" == "help" ( 17 | :help 18 | echo.Please use `make ^` where ^ is one of 19 | echo. html to make standalone HTML files 20 | echo. dirhtml to make HTML files named index.html in directories 21 | echo. singlehtml to make a single large HTML file 22 | echo. pickle to make pickle files 23 | echo. json to make JSON files 24 | echo. htmlhelp to make HTML files and a HTML help project 25 | echo. qthelp to make HTML files and a qthelp project 26 | echo. devhelp to make HTML files and a Devhelp project 27 | echo. epub to make an epub 28 | echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter 29 | echo. text to make text files 30 | echo. man to make manual pages 31 | echo. changes to make an overview over all changed/added/deprecated items 32 | echo. linkcheck to check all external links for integrity 33 | echo. doctest to run all doctests embedded in the documentation if enabled 34 | goto end 35 | ) 36 | 37 | if "%1" == "clean" ( 38 | for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i 39 | del /q /s %BUILDDIR%\* 40 | goto end 41 | ) 42 | 43 | if "%1" == "html" ( 44 | %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html 45 | if errorlevel 1 exit /b 1 46 | echo. 47 | echo.Build finished. The HTML pages are in %BUILDDIR%/html. 48 | goto end 49 | ) 50 | 51 | if "%1" == "dirhtml" ( 52 | %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml 53 | if errorlevel 1 exit /b 1 54 | echo. 55 | echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml. 56 | goto end 57 | ) 58 | 59 | if "%1" == "singlehtml" ( 60 | %SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml 61 | if errorlevel 1 exit /b 1 62 | echo. 63 | echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml. 64 | goto end 65 | ) 66 | 67 | if "%1" == "pickle" ( 68 | %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle 69 | if errorlevel 1 exit /b 1 70 | echo. 71 | echo.Build finished; now you can process the pickle files. 72 | goto end 73 | ) 74 | 75 | if "%1" == "json" ( 76 | %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json 77 | if errorlevel 1 exit /b 1 78 | echo. 79 | echo.Build finished; now you can process the JSON files. 80 | goto end 81 | ) 82 | 83 | if "%1" == "htmlhelp" ( 84 | %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp 85 | if errorlevel 1 exit /b 1 86 | echo. 87 | echo.Build finished; now you can run HTML Help Workshop with the ^ 88 | .hhp project file in %BUILDDIR%/htmlhelp. 89 | goto end 90 | ) 91 | 92 | if "%1" == "qthelp" ( 93 | %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp 94 | if errorlevel 1 exit /b 1 95 | echo. 96 | echo.Build finished; now you can run "qcollectiongenerator" with the ^ 97 | .qhcp project file in %BUILDDIR%/qthelp, like this: 98 | echo.^> qcollectiongenerator %BUILDDIR%\qthelp\django-queued-storage.qhcp 99 | echo.To view the help file: 100 | echo.^> assistant -collectionFile %BUILDDIR%\qthelp\django-queued-storage.ghc 101 | goto end 102 | ) 103 | 104 | if "%1" == "devhelp" ( 105 | %SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp 106 | if errorlevel 1 exit /b 1 107 | echo. 108 | echo.Build finished. 109 | goto end 110 | ) 111 | 112 | if "%1" == "epub" ( 113 | %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub 114 | if errorlevel 1 exit /b 1 115 | echo. 116 | echo.Build finished. The epub file is in %BUILDDIR%/epub. 117 | goto end 118 | ) 119 | 120 | if "%1" == "latex" ( 121 | %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex 122 | if errorlevel 1 exit /b 1 123 | echo. 124 | echo.Build finished; the LaTeX files are in %BUILDDIR%/latex. 125 | goto end 126 | ) 127 | 128 | if "%1" == "text" ( 129 | %SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text 130 | if errorlevel 1 exit /b 1 131 | echo. 132 | echo.Build finished. The text files are in %BUILDDIR%/text. 133 | goto end 134 | ) 135 | 136 | if "%1" == "man" ( 137 | %SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man 138 | if errorlevel 1 exit /b 1 139 | echo. 140 | echo.Build finished. The manual pages are in %BUILDDIR%/man. 141 | goto end 142 | ) 143 | 144 | if "%1" == "changes" ( 145 | %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes 146 | if errorlevel 1 exit /b 1 147 | echo. 148 | echo.The overview file is in %BUILDDIR%/changes. 149 | goto end 150 | ) 151 | 152 | if "%1" == "linkcheck" ( 153 | %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck 154 | if errorlevel 1 exit /b 1 155 | echo. 156 | echo.Link check complete; look for any errors in the above output ^ 157 | or in %BUILDDIR%/linkcheck/output.txt. 158 | goto end 159 | ) 160 | 161 | if "%1" == "doctest" ( 162 | %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest 163 | if errorlevel 1 exit /b 1 164 | echo. 165 | echo.Testing of doctests in the sources finished, look at the ^ 166 | results in %BUILDDIR%/doctest/output.txt. 167 | goto end 168 | ) 169 | 170 | :end 171 | -------------------------------------------------------------------------------- /docs/requirements.txt: -------------------------------------------------------------------------------- 1 | celery 2 | django-celery 3 | django-appconf 4 | django<1.9 5 | -------------------------------------------------------------------------------- /docs/signals.rst: -------------------------------------------------------------------------------- 1 | Signals 2 | ======= 3 | 4 | .. automodule:: queued_storage.signals 5 | -------------------------------------------------------------------------------- /docs/tasks.rst: -------------------------------------------------------------------------------- 1 | Tasks 2 | ===== 3 | 4 | .. currentmodule:: queued_storage.tasks 5 | 6 | .. autoclass:: Transfer 7 | :members: 8 | :undoc-members: 9 | 10 | .. autoclass:: TransferAndDelete 11 | :members: 12 | :undoc-members: 13 | -------------------------------------------------------------------------------- /queued_storage/__init__.py: -------------------------------------------------------------------------------- 1 | # flake8: noqa 2 | import pkg_resources 3 | 4 | __version__ = pkg_resources.get_distribution('django-queued-storage').version 5 | -------------------------------------------------------------------------------- /queued_storage/backends.py: -------------------------------------------------------------------------------- 1 | import six 2 | 3 | from packaging import version 4 | 5 | import django 6 | 7 | from django.core.cache import cache 8 | from django.core.exceptions import ImproperlyConfigured 9 | from django.utils.functional import SimpleLazyObject 10 | from django.utils.http import urlquote 11 | 12 | from .conf import settings 13 | from .utils import import_attribute 14 | 15 | DJANGO_VERSION = django.get_version() 16 | 17 | if version.parse(DJANGO_VERSION) <= version.parse('1.7'): 18 | from django.utils.deconstruct import deconstructible 19 | 20 | 21 | class LazyBackend(SimpleLazyObject): 22 | 23 | def __init__(self, import_path, options): 24 | backend = import_attribute(import_path) 25 | super(LazyBackend, self).__init__(lambda: backend(**options)) 26 | 27 | 28 | class QueuedStorage(object): 29 | """ 30 | Base class for queued storages. You can use this to specify your own 31 | backends. 32 | 33 | :param local: local storage class to transfer from 34 | :type local: str 35 | :param local_options: options of the local storage class 36 | :type local_options: dict 37 | :param remote: remote storage class to transfer to 38 | :type remote: str 39 | :param remote_options: options of the remote storage class 40 | :type remote_options: dict 41 | :param cache_prefix: prefix to use in the cache key 42 | :type cache_prefix: str 43 | :param delayed: whether the transfer task should be executed automatically 44 | :type delayed: bool 45 | :param task: Celery task to use for the transfer 46 | :type task: str 47 | """ 48 | #: The local storage class to use. A dotted path (e.g. 49 | #: ``'django.core.files.storage.FileSystemStorage'``). 50 | local = None 51 | 52 | #: The options of the local storage class, defined as a dictionary. 53 | local_options = None 54 | 55 | #: The remote storage class to use. A dotted path (e.g. 56 | #: ``'django.core.files.storage.FileSystemStorage'``). 57 | remote = None 58 | 59 | #: The options of the remote storage class, defined as a dictionary. 60 | remote_options = None 61 | 62 | #: The Celery task class to use to transfer files from the local 63 | #: to the remote storage. A dotted path (e.g. 64 | #: ``'queued_storage.tasks.Transfer'``). 65 | task = 'queued_storage.tasks.Transfer' 66 | 67 | #: If set to ``True`` the backend will *not* transfer files to the remote 68 | #: location automatically, but instead requires manual intervention by the 69 | #: user with the :meth:`~queued_storage.backends.QueuedStorage.transfer` 70 | #: method. 71 | delayed = False 72 | 73 | #: The cache key prefix to use when saving the which storage backend 74 | #: to use, local or remote (default see 75 | #: :attr:`~queued_storage.conf.settings.QUEUED_STORAGE_CACHE_PREFIX`) 76 | cache_prefix = settings.QUEUED_STORAGE_CACHE_PREFIX 77 | 78 | def __init__(self, local=None, remote=None, 79 | local_options=None, remote_options=None, 80 | cache_prefix=None, delayed=None, task=None): 81 | 82 | self.local_path = local or self.local 83 | self.local_options = local_options or self.local_options or {} 84 | self.local = self._load_backend(backend=self.local_path, 85 | options=self.local_options) 86 | 87 | self.remote_path = remote or self.remote 88 | self.remote_options = remote_options or self.remote_options or {} 89 | self.remote = self._load_backend(backend=self.remote_path, 90 | options=self.remote_options) 91 | 92 | self.task = self._load_backend(backend=task or self.task, 93 | handler=import_attribute) 94 | if delayed is not None: 95 | self.delayed = delayed 96 | if cache_prefix is not None: 97 | self.cache_prefix = cache_prefix 98 | 99 | def _load_backend(self, backend=None, options=None, handler=LazyBackend): 100 | if backend is None: # pragma: no cover 101 | raise ImproperlyConfigured("The QueuedStorage class '%s' " 102 | "doesn't define a needed backend." % 103 | (self)) 104 | if not isinstance(backend, six.string_types): 105 | raise ImproperlyConfigured("The QueuedStorage class '%s' " 106 | "requires its backends to be " 107 | "specified as dotted import paths " 108 | "not instances or classes" % self) 109 | return handler(backend, options) 110 | 111 | def get_storage(self, name): 112 | """ 113 | Returns the storage backend instance responsible for the file 114 | with the given name (either local or remote). This method is 115 | used in most of the storage API methods. 116 | 117 | :param name: file name 118 | :type name: str 119 | :rtype: :class:`~django:django.core.files.storage.Storage` 120 | """ 121 | cache_result = cache.get(self.get_cache_key(name)) 122 | if cache_result: 123 | return self.remote 124 | elif cache_result is None and self.remote.exists(name): 125 | cache.set(self.get_cache_key(name), True) 126 | return self.remote 127 | else: 128 | return self.local 129 | 130 | def get_cache_key(self, name): 131 | """ 132 | Returns the cache key for the given file name. 133 | 134 | :param name: file name 135 | :type name: str 136 | :rtype: str 137 | """ 138 | return '%s_%s' % (self.cache_prefix, urlquote(name)) 139 | 140 | def using_local(self, name): 141 | """ 142 | Determines for the file with the given name whether 143 | the local storage is current used. 144 | 145 | :param name: file name 146 | :type name: str 147 | :rtype: bool 148 | """ 149 | return self.get_storage(name) is self.local 150 | 151 | def using_remote(self, name): 152 | """ 153 | Determines for the file with the given name whether 154 | the remote storage is current used. 155 | 156 | :param name: file name 157 | :type name: str 158 | :rtype: bool 159 | """ 160 | return self.get_storage(name) is self.remote 161 | 162 | def open(self, name, mode='rb'): 163 | """ 164 | Retrieves the specified file from storage. 165 | 166 | :param name: file name 167 | :type name: str 168 | :param mode: mode to open the file with 169 | :type mode: str 170 | :rtype: :class:`~django:django.core.files.File` 171 | """ 172 | return self.get_storage(name).open(name, mode) 173 | 174 | def save(self, name, content, max_length=None): 175 | """ 176 | Saves the given content with the given name using the local 177 | storage. If the :attr:`~queued_storage.backends.QueuedStorage.delayed` 178 | attribute is ``True`` this will automatically call the 179 | :meth:`~queued_storage.backends.QueuedStorage.transfer` method 180 | queuing the transfer from local to remote storage. 181 | 182 | :param name: file name 183 | :type name: str 184 | :param content: content of the file specified by name 185 | :type content: :class:`~django:django.core.files.File` 186 | :rtype: str 187 | """ 188 | cache_key = self.get_cache_key(name) 189 | cache.set(cache_key, False) 190 | 191 | # Use a name that is available on both the local and remote storage 192 | # systems and save locally. 193 | name = self.get_available_name(name) 194 | try: 195 | name = self.local.save(name, content, max_length=max_length) 196 | except TypeError: 197 | # Django < 1.10 198 | name = self.local.save(name, content) 199 | 200 | # Pass on the cache key to prevent duplicate cache key creation, 201 | # we save the result in the storage to be able to test for it 202 | if not self.delayed: 203 | self.result = self.transfer(name, cache_key=cache_key) 204 | return name 205 | 206 | def transfer(self, name, cache_key=None): 207 | """ 208 | Transfers the file with the given name to the remote storage 209 | backend by queuing the task. 210 | 211 | :param name: file name 212 | :type name: str 213 | :param cache_key: the cache key to set after a successful task run 214 | :type cache_key: str 215 | :rtype: task result 216 | """ 217 | if cache_key is None: 218 | cache_key = self.get_cache_key(name) 219 | return self.task.delay(name, cache_key, 220 | self.local_path, self.remote_path, 221 | self.local_options, self.remote_options) 222 | 223 | def get_valid_name(self, name): 224 | """ 225 | Returns a filename, based on the provided filename, that's suitable 226 | for use in the current storage system. 227 | 228 | :param name: file name 229 | :type name: str 230 | :rtype: str 231 | """ 232 | return self.get_storage(name).get_valid_name(name) 233 | 234 | def get_available_name(self, name): 235 | """ 236 | Returns a filename that's free on both the local and remote storage 237 | systems, and available for new content to be written to. 238 | 239 | :param name: file name 240 | :type name: str 241 | :rtype: str 242 | """ 243 | local_available_name = self.local.get_available_name(name) 244 | remote_available_name = self.remote.get_available_name(name) 245 | 246 | if remote_available_name > local_available_name: 247 | return remote_available_name 248 | return local_available_name 249 | 250 | def path(self, name): 251 | """ 252 | Returns a local filesystem path where the file can be retrieved using 253 | Python's built-in open() function. Storage systems that can't be 254 | accessed using open() should *not* implement this method. 255 | 256 | :param name: file name 257 | :type name: str 258 | :rtype: str 259 | """ 260 | return self.get_storage(name).path(name) 261 | 262 | def delete(self, name): 263 | """ 264 | Deletes the specified file from the storage system. 265 | 266 | :param name: file name 267 | :type name: str 268 | """ 269 | return self.get_storage(name).delete(name) 270 | 271 | def exists(self, name): 272 | """ 273 | Returns ``True`` if a file referened by the given name already exists 274 | in the storage system, or False if the name is available for a new 275 | file. 276 | 277 | :param name: file name 278 | :type name: str 279 | :rtype: bool 280 | """ 281 | return self.get_storage(name).exists(name) 282 | 283 | def listdir(self, name): 284 | """ 285 | Lists the contents of the specified path, returning a 2-tuple of lists; 286 | the first item being directories, the second item being files. 287 | 288 | :param name: file name 289 | :type name: str 290 | :rtype: tuple 291 | """ 292 | return self.get_storage(name).listdir(name) 293 | 294 | def size(self, name): 295 | """ 296 | Returns the total size, in bytes, of the file specified by name. 297 | 298 | :param name: file name 299 | :type name: str 300 | :rtype: int 301 | """ 302 | return self.get_storage(name).size(name) 303 | 304 | def url(self, name): 305 | """ 306 | Returns an absolute URL where the file's contents can be accessed 307 | directly by a Web browser. 308 | 309 | :param name: file name 310 | :type name: str 311 | :rtype: str 312 | """ 313 | return self.get_storage(name).url(name) 314 | 315 | def accessed_time(self, name): 316 | """ 317 | Returns the last accessed time (as datetime object) of the file 318 | specified by name. 319 | 320 | :param name: file name 321 | :type name: str 322 | :rtype: :class:`~python:datetime.datetime` 323 | """ 324 | return self.get_storage(name).accessed_time(name) 325 | 326 | def created_time(self, name): 327 | """ 328 | Returns the creation time (as datetime object) of the file 329 | specified by name. 330 | 331 | :param name: file name 332 | :type name: str 333 | :rtype: :class:`~python:datetime.datetime` 334 | """ 335 | return self.get_storage(name).created_time(name) 336 | 337 | def modified_time(self, name): 338 | """ 339 | Returns the last modified time (as datetime object) of the file 340 | specified by name. 341 | 342 | :param name: file name 343 | :type name: str 344 | :rtype: :class:`~python:datetime.datetime` 345 | """ 346 | return self.get_storage(name).modified_time(name) 347 | 348 | def get_accessed_time(self, name): 349 | """ 350 | Django +1.10 351 | Returns the last accessed time (as datetime object) of the file 352 | specified by name. 353 | 354 | :param name: file name 355 | :type name: str 356 | :rtype: :class:`~python:datetime.datetime` 357 | """ 358 | return self.get_storage(name).get_accessed_time(name) 359 | 360 | def get_created_time(self, name): 361 | """ 362 | Django +1.10 363 | Returns the creation time (as datetime object) of the file 364 | specified by name. 365 | 366 | :param name: file name 367 | :type name: str 368 | :rtype: :class:`~python:datetime.datetime` 369 | """ 370 | 371 | return self.get_storage(name).get_created_time(name) 372 | 373 | def get_modified_time(self, name): 374 | """ 375 | Django +1.10 376 | Returns the last modified time (as datetime object) of the file 377 | specified by name. 378 | 379 | :param name: file name 380 | :type name: str 381 | :rtype: :class:`~python:datetime.datetime` 382 | """ 383 | 384 | return self.get_storage(name).get_modified_time(name) 385 | 386 | def generate_filename(self, filename): 387 | return self.get_storage(filename).generate_filename(filename) 388 | 389 | if version.parse(DJANGO_VERSION) <= version.parse('1.7'): 390 | QueuedStorage = deconstructible(QueuedStorage) 391 | 392 | 393 | class QueuedFileSystemStorage(QueuedStorage): 394 | """ 395 | A :class:`~queued_storage.backends.QueuedStorage` subclass which 396 | conveniently uses 397 | :class:`~django:django.core.files.storage.FileSystemStorage` as the local 398 | storage. 399 | """ 400 | def __init__(self, local='django.core.files.storage.FileSystemStorage', *args, **kwargs): 401 | super(QueuedFileSystemStorage, self).__init__(local=local, *args, **kwargs) 402 | 403 | 404 | class QueuedS3BotoStorage(QueuedFileSystemStorage): 405 | """ 406 | A custom :class:`~queued_storage.backends.QueuedFileSystemStorage` 407 | subclass which uses the ``S3BotoStorage`` storage of the 408 | `django-storages `_ app as 409 | the remote storage. 410 | """ 411 | def __init__(self, remote='storages.backends.s3boto.S3BotoStorage', *args, **kwargs): 412 | super(QueuedS3BotoStorage, self).__init__(remote=remote, *args, **kwargs) 413 | 414 | 415 | class QueuedCouchDBStorage(QueuedFileSystemStorage): 416 | """ 417 | A custom :class:`~queued_storage.backends.QueuedFileSystemStorage` 418 | subclass which uses the ``CouchDBStorage`` storage of the 419 | `django-storages `_ app as 420 | the remote storage. 421 | """ 422 | def __init__(self, remote='storages.backends.couchdb.CouchDBStorage', *args, **kwargs): 423 | super(QueuedCouchDBStorage, self).__init__(remote=remote, *args, **kwargs) 424 | 425 | 426 | class QueuedDatabaseStorage(QueuedFileSystemStorage): 427 | """ 428 | A custom :class:`~queued_storage.backends.QueuedFileSystemStorage` 429 | subclass which uses the ``DatabaseStorage`` storage of the 430 | `django-storages `_ app as 431 | the remote storage. 432 | """ 433 | def __init__(self, remote='storages.backends.database.DatabaseStorage', *args, **kwargs): 434 | super(QueuedDatabaseStorage, self).__init__(remote=remote, *args, **kwargs) 435 | 436 | 437 | class QueuedFTPStorage(QueuedFileSystemStorage): 438 | """ 439 | A custom :class:`~queued_storage.backends.QueuedFileSystemStorage` 440 | subclass which uses the ``FTPStorage`` storage of the 441 | `django-storages `_ app as 442 | the remote storage. 443 | """ 444 | def __init__(self, remote='storages.backends.ftp.FTPStorage', *args, **kwargs): 445 | super(QueuedFTPStorage, self).__init__(remote=remote, *args, **kwargs) 446 | 447 | 448 | class QueuedMogileFSStorage(QueuedFileSystemStorage): 449 | """ 450 | A custom :class:`~queued_storage.backends.QueuedFileSystemStorage` 451 | subclass which uses the ``MogileFSStorage`` storage of the 452 | `django-storages `_ app as 453 | the remote storage. 454 | """ 455 | def __init__(self, remote='storages.backends.mogile.MogileFSStorage', *args, **kwargs): 456 | super(QueuedMogileFSStorage, self).__init__(remote=remote, *args, **kwargs) 457 | 458 | 459 | class QueuedGridFSStorage(QueuedFileSystemStorage): 460 | """ 461 | A custom :class:`~queued_storage.backends.QueuedFileSystemStorage` 462 | subclass which uses the ``GridFSStorage`` storage of the 463 | `django-storages `_ app as 464 | the remote storage. 465 | """ 466 | def __init__(self, remote='storages.backends.mongodb.GridFSStorage', *args, **kwargs): 467 | super(QueuedGridFSStorage, self).__init__(remote=remote, *args, **kwargs) 468 | 469 | 470 | class QueuedCloudFilesStorage(QueuedFileSystemStorage): 471 | """ 472 | A custom :class:`~queued_storage.backends.QueuedFileSystemStorage` 473 | subclass which uses the ``CloudFilesStorage`` storage of the 474 | `django-storages `_ app as 475 | the remote storage. 476 | """ 477 | def __init__(self, remote='storages.backends.mosso.CloudFilesStorage', *args, **kwargs): 478 | super(QueuedCloudFilesStorage, self).__init__(remote=remote, *args, **kwargs) 479 | 480 | 481 | class QueuedSFTPStorage(QueuedFileSystemStorage): 482 | """ 483 | A custom :class:`~queued_storage.backends.QueuedFileSystemStorage` 484 | subclass which uses the ``SFTPStorage`` storage of the 485 | `django-storages `_ app as 486 | the remote storage. 487 | """ 488 | def __init__(self, remote='storages.backends.sftpstorage.SFTPStorage', *args, **kwargs): 489 | super(QueuedSFTPStorage, self).__init__(remote=remote, *args, **kwargs) 490 | -------------------------------------------------------------------------------- /queued_storage/conf.py: -------------------------------------------------------------------------------- 1 | from django.conf import settings # noqa 2 | 3 | from appconf import AppConf 4 | 5 | 6 | class QueuedStorageConf(AppConf): 7 | RETRIES = 5 8 | RETRY_DELAY = 60 9 | CACHE_PREFIX = 'queued_storage' 10 | -------------------------------------------------------------------------------- /queued_storage/fields.py: -------------------------------------------------------------------------------- 1 | from django.db.models.fields.files import FileField, FieldFile 2 | 3 | 4 | class QueuedFieldFile(FieldFile): 5 | """ 6 | A custom :class:`~django.db.models.fields.files.FieldFile` which has an 7 | additional method to transfer the file to the remote storage using the 8 | backend's ``transfer`` method. 9 | """ 10 | def transfer(self): 11 | """ 12 | Transfers the file using the storage backend. 13 | """ 14 | return self.storage.transfer(self.name) 15 | 16 | 17 | class QueuedFileField(FileField): 18 | """ 19 | Field to be used together with 20 | :class:`~queued_storage.backends.QueuedStorage` instances or instances 21 | of subclasses. 22 | 23 | Tiny wrapper around :class:`~django:django.db.models.FileField`, 24 | which provides a convenient method to transfer files, using the 25 | :meth:`~queued_storage.fields.QueuedFieldFile.transfer` method, e.g.: 26 | 27 | .. code-block:: python 28 | 29 | from queued_storage.backends import QueuedS3BotoStorage 30 | from queued_storage.fields import QueuedFileField 31 | 32 | class MyModel(models.Model): 33 | image = QueuedFileField(storage=QueuedS3BotoStorage(delayed=True)) 34 | 35 | my_obj = MyModel(image=File(open('image.png'))) 36 | # Save locally: 37 | my_obj.save() 38 | # Transfer to remote location: 39 | my_obj.image.transfer() 40 | """ 41 | attr_class = QueuedFieldFile 42 | -------------------------------------------------------------------------------- /queued_storage/models.py: -------------------------------------------------------------------------------- 1 | # This file intentionally left empty (needs to be present for test detection). 2 | -------------------------------------------------------------------------------- /queued_storage/signals.py: -------------------------------------------------------------------------------- 1 | """ 2 | django-queued-storage ships with a signal fired after a file was transferred 3 | by the Transfer task. It provides the name of the file, the local and the 4 | remote storage backend instances as arguments to connected signal callbacks. 5 | 6 | Imagine you'd want to post-process the file that has been transferred from 7 | the local to the remote storage, e.g. add it to a log model to always know 8 | what exactly happened. All you'd have to do is to connect a callback to 9 | the ``file_transferred`` signal:: 10 | 11 | from django.dispatch import receiver 12 | from django.utils.timezone import now 13 | 14 | from queued_storage.signals import file_transferred 15 | 16 | from mysite.transferlog.models import TransferLogEntry 17 | 18 | 19 | @receiver(file_transferred) 20 | def log_file_transferred(sender, name, local, remote, **kwargs): 21 | remote_url = remote.url(name) 22 | TransferLogEntry.objects.create(name=name, remote_url=remote_url, transfer_date=now()) 23 | 24 | # Alternatively, you can also use the signal's connect method to connect: 25 | file_transferred.connect(log_file_transferred) 26 | 27 | Note that this signal does **NOT** have access to the calling Model or even 28 | the FileField instance that it relates to, only the name of the file. 29 | As a result, this signal is somewhat limited and may only be of use if you 30 | have a very specific usage of django-queued-storage. 31 | """ 32 | from django.dispatch import Signal 33 | 34 | file_transferred = Signal(providing_args=["name", "local", "remote"]) 35 | -------------------------------------------------------------------------------- /queued_storage/tasks.py: -------------------------------------------------------------------------------- 1 | from django.core.cache import cache 2 | 3 | from celery.task import Task 4 | try: 5 | from celery.utils.log import get_task_logger 6 | except ImportError: 7 | from celery.log import get_task_logger 8 | 9 | 10 | from .conf import settings 11 | from .signals import file_transferred 12 | from .utils import import_attribute 13 | 14 | logger = get_task_logger(name=__name__) 15 | 16 | 17 | class Transfer(Task): 18 | """ 19 | The default task. Transfers a file to a remote location. 20 | The actual transfer is implemented in the remote backend. 21 | 22 | To use a different task, pass it into the backend: 23 | 24 | .. code-block:: python 25 | 26 | from queued_storage.backends import QueuedS3BotoStorage 27 | 28 | s3_delete_storage = QueuedS3BotoStorage( 29 | task='queued_storage.tasks.TransferAndDelete') 30 | 31 | # later, in model definition: 32 | image = models.ImageField(storage=s3_delete_storage) 33 | 34 | 35 | The result should be ``True`` if the transfer was successful, 36 | or ``False`` if unsuccessful. In the latter case the task will be 37 | retried. 38 | 39 | You can subclass the :class:`~queued_storage.tasks.Transfer` class 40 | to customize the behaviour, to do something like this: 41 | 42 | .. code-block:: python 43 | 44 | from queued_storage.tasks import Transfer 45 | 46 | class TransferAndNotify(Transfer): 47 | def transfer(self, *args, **kwargs): 48 | result = super(TransferAndNotify, self).transfer(*args, **kwargs) 49 | if result: 50 | # call the (imaginary) notify function with the result 51 | notify(result) 52 | return result 53 | 54 | """ 55 | #: The number of retries if unsuccessful (default: see 56 | #: :attr:`~queued_storage.conf.settings.QUEUED_STORAGE_RETRIES`) 57 | max_retries = settings.QUEUED_STORAGE_RETRIES 58 | 59 | #: The delay between each retry in seconds (default: see 60 | #: :attr:`~queued_storage.conf.settings.QUEUED_STORAGE_RETRY_DELAY`) 61 | default_retry_delay = settings.QUEUED_STORAGE_RETRY_DELAY 62 | 63 | def run(self, name, cache_key, 64 | local_path, remote_path, 65 | local_options, remote_options, **kwargs): 66 | """ 67 | The main work horse of the transfer task. Calls the transfer 68 | method with the local and remote storage backends as given 69 | with the parameters. 70 | 71 | :param name: name of the file to transfer 72 | :type name: str 73 | :param local_path: local storage class to transfer from 74 | :type local_path: str 75 | :param local_options: options of the local storage class 76 | :type local_options: dict 77 | :param remote_path: remote storage class to transfer to 78 | :type remote_path: str 79 | :param remote_options: options of the remote storage class 80 | :type remote_options: dict 81 | :param cache_key: cache key to set after a successful transfer 82 | :type cache_key: str 83 | :rtype: task result 84 | """ 85 | local = import_attribute(local_path)(**local_options) 86 | remote = import_attribute(remote_path)(**remote_options) 87 | result = self.transfer(name, local, remote, **kwargs) 88 | 89 | if result is True: 90 | cache.set(cache_key, True) 91 | file_transferred.send(sender=self.__class__, 92 | name=name, local=local, remote=remote) 93 | elif result is False: 94 | args = [name, cache_key, local_path, 95 | remote_path, local_options, remote_options] 96 | self.retry(args=args, kwargs=kwargs) 97 | else: 98 | raise ValueError("Task '%s' did not return True/False but %s" % 99 | (self.__class__, result)) 100 | return result 101 | 102 | def transfer(self, name, local, remote, **kwargs): 103 | """ 104 | Transfers the file with the given name from the local to the remote 105 | storage backend. 106 | 107 | :param name: The name of the file to transfer 108 | :param local: The local storage backend instance 109 | :param remote: The remote storage backend instance 110 | :returns: `True` when the transfer succeeded, `False` if not. Retries 111 | the task when returning `False` 112 | :rtype: bool 113 | """ 114 | try: 115 | remote.save(name, local.open(name)) 116 | return True 117 | except Exception as e: 118 | logger.error("Unable to save '%s' to remote storage. " 119 | "About to retry." % name) 120 | logger.exception(e) 121 | return False 122 | 123 | 124 | class TransferAndDelete(Transfer): 125 | """ 126 | A :class:`~queued_storage.tasks.Transfer` subclass which deletes the 127 | file with the given name using the local storage if the transfer 128 | was successful. 129 | """ 130 | def transfer(self, name, local, remote, **kwargs): 131 | result = super(TransferAndDelete, self).transfer(name, local, 132 | remote, **kwargs) 133 | if result: 134 | local.delete(name) 135 | return result 136 | -------------------------------------------------------------------------------- /queued_storage/utils.py: -------------------------------------------------------------------------------- 1 | from importlib import import_module 2 | 3 | from django.core.exceptions import ImproperlyConfigured 4 | 5 | 6 | def import_attribute(import_path=None, options=None): 7 | if import_path is None: 8 | raise ImproperlyConfigured("No import path was given.") 9 | try: 10 | dot = import_path.rindex('.') 11 | except ValueError: 12 | raise ImproperlyConfigured("%s isn't a module." % import_path) 13 | module, classname = import_path[:dot], import_path[dot + 1:] 14 | try: 15 | mod = import_module(module) 16 | except ImportError as e: 17 | raise ImproperlyConfigured('Error importing module %s: "%s"' % 18 | (module, e)) 19 | try: 20 | return getattr(mod, classname) 21 | except AttributeError: 22 | raise ImproperlyConfigured( 23 | 'Module "%s" does not define a "%s" class.' % (module, classname)) 24 | 25 | 26 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [tool:pytest] 2 | django_find_project = false 3 | addopts = --cov queued_storage 4 | DJANGO_SETTINGS_MODULE = tests.settings 5 | 6 | [wheel] 7 | universal = 1 8 | 9 | [flake8] 10 | ignore = E124,E501,E127,E128 11 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import codecs 2 | import os 3 | from setuptools import setup 4 | 5 | 6 | def read(*parts): 7 | filename = os.path.join(os.path.dirname(__file__), *parts) 8 | with codecs.open(filename, encoding='utf-8') as fp: 9 | return fp.read() 10 | 11 | 12 | setup( 13 | name="django-queued-storage", 14 | use_scm_version=True, 15 | setup_requires=['setuptools_scm'], 16 | url='https://github.com/jazzband/django-queued-storage', 17 | license='BSD', 18 | description="Queued remote storage for Django.", 19 | long_description=read('README.rst'), 20 | author='Jannis Leidel', 21 | author_email='jannis@leidel.info', 22 | packages=['queued_storage'], 23 | classifiers=[ 24 | 'Development Status :: 4 - Beta', 25 | 'Framework :: Django', 26 | 'Intended Audience :: Developers', 27 | 'License :: OSI Approved :: BSD License', 28 | 'Operating System :: OS Independent', 29 | 'Programming Language :: Python', 30 | 'Programming Language :: Python :: 2.7', 31 | 'Programming Language :: Python :: 3.3', 32 | 'Programming Language :: Python :: 3.4', 33 | 'Topic :: Utilities', 34 | ], 35 | install_requires=[ 36 | 'six>=1.10.0', 37 | 'django-appconf >= 0.4', 38 | 'packaging==16.8', 39 | ], 40 | zip_safe=False, 41 | ) 42 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jazzband/django-queued-storage/1cbb3c973e2dc6b1f4b8823ac3306111581067a5/tests/__init__.py -------------------------------------------------------------------------------- /tests/celeryconfig.py: -------------------------------------------------------------------------------- 1 | BROKER_TRANSPORT = "memory" 2 | CELERY_IGNORE_RESULT = True 3 | CELERYD_LOG_LEVEL = "DEBUG" 4 | CELERY_DEFAULT_QUEUE = "queued_storage" 5 | CELERY_RESULT_BACKEND = "database" 6 | CELERY_RESULT_DBURI = "sqlite://" 7 | 8 | CELERY_ALWAYS_EAGER = True 9 | CELERY_IGNORE_RESULT = True 10 | CELERY_IMPORTS = [ 11 | 'queued_storage.tasks', 12 | ] 13 | -------------------------------------------------------------------------------- /tests/models.py: -------------------------------------------------------------------------------- 1 | from django.db import models 2 | 3 | from queued_storage.fields import QueuedFileField 4 | 5 | 6 | class TestModel(models.Model): 7 | testfile = models.FileField(upload_to='test', null=True) 8 | remote = QueuedFileField(upload_to='test', null=True) 9 | 10 | retried = False 11 | -------------------------------------------------------------------------------- /tests/requirements.txt: -------------------------------------------------------------------------------- 1 | flake8 2 | coverage 3 | celery 4 | SQLAlchemy 5 | anyjson 6 | pytest-cov 7 | pytest-sugar 8 | pytest-assume 9 | pytest-cov 10 | pytest-flake8 11 | 12 | -------------------------------------------------------------------------------- /tests/settings.py: -------------------------------------------------------------------------------- 1 | SITE_ID = 1 2 | 3 | DATABASES = { 4 | 'default': { 5 | 'ENGINE': 'django.db.backends.sqlite3', 6 | 'NAME': ':memory:', 7 | } 8 | } 9 | 10 | INSTALLED_APPS = [ 11 | 'queued_storage', 12 | 'tests', 13 | ] 14 | 15 | 16 | SECRET_KEY = 'top_secret' 17 | 18 | CACHES = { 19 | 'default': { 20 | 'BACKEND': 'django.core.cache.backends.locmem.LocMemCache', 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /tests/tasks.py: -------------------------------------------------------------------------------- 1 | from queued_storage.tasks import Transfer 2 | from queued_storage.utils import import_attribute 3 | 4 | from .models import TestModel 5 | 6 | 7 | def test_task(name, cache_key, 8 | local_path, remote_path, 9 | local_options, remote_options): 10 | local = import_attribute(local_path)(**local_options) 11 | remote = import_attribute(remote_path)(**remote_options) 12 | remote.save(name, local.open(name)) 13 | 14 | 15 | def delay(*args, **kwargs): 16 | test_task(*args, **kwargs) 17 | 18 | test_task.delay = delay 19 | 20 | 21 | class NoneReturningTask(Transfer): 22 | def transfer(self, *args, **kwargs): 23 | return None 24 | 25 | 26 | class RetryingTask(Transfer): 27 | def transfer(self, *args, **kwargs): 28 | if TestModel.retried: 29 | return True 30 | else: 31 | TestModel.retried = True 32 | return False 33 | -------------------------------------------------------------------------------- /tests/test_storages.py: -------------------------------------------------------------------------------- 1 | """ 2 | For simplicity and to avoid requiring a paid-for account on some cloud 3 | storage system testing is conducted against two local storage backends. Since 4 | the QueuedStorage backend is truly agnostic about the local and remote 5 | storage systems, this should work as transparently as using one (or even two!) 6 | remote storage systems. 7 | """ 8 | import os 9 | import shutil 10 | import tempfile 11 | from os import path 12 | from datetime import datetime 13 | from packaging import version 14 | from packaging.specifiers import SpecifierSet 15 | 16 | import django 17 | from django.core.files.base import File 18 | from django.core.files.storage import FileSystemStorage, Storage 19 | from django.test import TestCase 20 | 21 | from queued_storage.backends import QueuedStorage 22 | from queued_storage.conf import settings 23 | 24 | from . import models 25 | 26 | DJANGO_VERSION = django.get_version() 27 | 28 | 29 | class StorageTests(TestCase): 30 | 31 | def setUp(self): 32 | self.old_celery_always_eager = getattr( 33 | settings, 'CELERY_ALWAYS_EAGER', False) 34 | settings.CELERY_ALWAYS_EAGER = True 35 | self.local_dir = tempfile.mkdtemp() 36 | self.remote_dir = tempfile.mkdtemp() 37 | tmp_dir = tempfile.mkdtemp() 38 | self.test_file_name = 'queued_storage.txt' 39 | self.test_file_path = path.join(tmp_dir, self.test_file_name) 40 | with open(self.test_file_path, 'a') as test_file: 41 | test_file.write('test') 42 | self.test_file = open(self.test_file_path, 'r') 43 | self.addCleanup(shutil.rmtree, self.local_dir) 44 | self.addCleanup(shutil.rmtree, self.remote_dir) 45 | self.addCleanup(shutil.rmtree, tmp_dir) 46 | 47 | def tearDown(self): 48 | settings.CELERY_ALWAYS_EAGER = self.old_celery_always_eager 49 | 50 | def test_storage_init(self): 51 | """ 52 | Make sure that creating a QueuedStorage object works 53 | """ 54 | storage = QueuedStorage( 55 | 'django.core.files.storage.FileSystemStorage', 56 | 'django.core.files.storage.FileSystemStorage') 57 | self.assertIsInstance(storage, QueuedStorage) 58 | self.assertEqual(FileSystemStorage, storage.local.__class__) 59 | self.assertEqual(FileSystemStorage, storage.remote.__class__) 60 | 61 | def test_storage_cache_key(self): 62 | storage = QueuedStorage( 63 | 'django.core.files.storage.FileSystemStorage', 64 | 'django.core.files.storage.FileSystemStorage', 65 | cache_prefix='test_cache_key') 66 | self.assertEqual(storage.cache_prefix, 'test_cache_key') 67 | 68 | def test_storage_methods(self): 69 | """ 70 | Make sure that QueuedStorage implements all the methods 71 | """ 72 | storage = QueuedStorage( 73 | 'django.core.files.storage.FileSystemStorage', 74 | 'django.core.files.storage.FileSystemStorage') 75 | 76 | file_storage = Storage() 77 | 78 | for attr in dir(file_storage): 79 | method = getattr(file_storage, attr) 80 | 81 | if not callable(method): 82 | continue 83 | 84 | method = getattr(storage, attr, False) 85 | self.assertTrue(callable(method), 86 | "QueuedStorage has no method '%s'" % attr) 87 | 88 | def test_storage_simple_save(self): 89 | """ 90 | Make sure that saving to remote locations actually works 91 | """ 92 | storage = QueuedStorage( 93 | local='django.core.files.storage.FileSystemStorage', 94 | remote='django.core.files.storage.FileSystemStorage', 95 | local_options=dict(location=self.local_dir), 96 | remote_options=dict(location=self.remote_dir), 97 | task='tests.tasks.test_task') 98 | 99 | field = models.TestModel._meta.get_field('testfile') 100 | field.storage = storage 101 | 102 | obj = models.TestModel() 103 | obj.testfile.save(self.test_file_name, File(self.test_file)) 104 | obj.save() 105 | 106 | self.assertTrue(path.isfile(path.join(self.local_dir, obj.testfile.name))) 107 | self.assertTrue(path.isfile(path.join(self.remote_dir, obj.testfile.name))) 108 | 109 | def test_storage_celery_save(self): 110 | """ 111 | Make sure it actually works when using Celery as a task queue 112 | """ 113 | storage = QueuedStorage( 114 | local='django.core.files.storage.FileSystemStorage', 115 | remote='django.core.files.storage.FileSystemStorage', 116 | local_options=dict(location=self.local_dir), 117 | remote_options=dict(location=self.remote_dir)) 118 | 119 | field = models.TestModel._meta.get_field('testfile') 120 | field.storage = storage 121 | 122 | obj = models.TestModel() 123 | obj.testfile.save(self.test_file_name, File(self.test_file)) 124 | obj.save() 125 | 126 | 127 | self.assertTrue(obj.testfile.storage.result.get()) 128 | self.assertTrue(path.isfile(path.join(self.local_dir, obj.testfile.name))) 129 | self.assertTrue( 130 | path.isfile(path.join(self.remote_dir, obj.testfile.name)), 131 | "Remote file is not available.") 132 | self.assertFalse(storage.using_local(obj.testfile.name)) 133 | self.assertTrue(storage.using_remote(obj.testfile.name)) 134 | 135 | self.assertEqual(self.test_file_name, 136 | storage.get_valid_name(self.test_file_name)) 137 | self.assertEqual(self.test_file_name, 138 | storage.get_available_name(self.test_file_name)) 139 | 140 | subdir_path = os.path.join('test', self.test_file_name) 141 | self.assertTrue(storage.exists(subdir_path)) 142 | self.assertEqual(storage.path(self.test_file_name), 143 | path.join(self.local_dir, self.test_file_name)) 144 | self.assertEqual(storage.listdir('test')[1], [self.test_file_name]) 145 | self.assertEqual(storage.size(subdir_path), 146 | os.stat(self.test_file_path).st_size) 147 | self.assertEqual(storage.url(self.test_file_name), self.test_file_name) 148 | 149 | if version.parse(DJANGO_VERSION) in SpecifierSet("<=2.0"): 150 | self.assertIsInstance(storage.accessed_time(subdir_path), datetime) 151 | self.assertIsInstance(storage.created_time(subdir_path), datetime) 152 | self.assertIsInstance(storage.modified_time(subdir_path), datetime) 153 | else: 154 | self.assertIsInstance(storage.get_accessed_time(subdir_path), datetime) 155 | self.assertIsInstance(storage.get_created_time(subdir_path), datetime) 156 | self.assertIsInstance(storage.get_modified_time(subdir_path), datetime) 157 | 158 | subdir_name = 'queued_storage_2.txt' 159 | testfile = storage.open(subdir_name, 'w') 160 | try: 161 | testfile.write('test') 162 | finally: 163 | testfile.close() 164 | self.assertTrue(storage.exists(subdir_name)) 165 | storage.delete(subdir_name) 166 | self.assertFalse(storage.exists(subdir_name)) 167 | 168 | def test_transfer_and_delete(self): 169 | """ 170 | Make sure the TransferAndDelete task does what it says 171 | """ 172 | storage = QueuedStorage( 173 | local='django.core.files.storage.FileSystemStorage', 174 | remote='django.core.files.storage.FileSystemStorage', 175 | local_options=dict(location=self.local_dir), 176 | remote_options=dict(location=self.remote_dir), 177 | task='queued_storage.tasks.TransferAndDelete') 178 | 179 | field = models.TestModel._meta.get_field('testfile') 180 | field.storage = storage 181 | 182 | obj = models.TestModel() 183 | obj.testfile.save(self.test_file_name, File(self.test_file)) 184 | obj.save() 185 | 186 | obj.testfile.storage.result.get() 187 | 188 | self.assertFalse( 189 | path.isfile(path.join(self.local_dir, obj.testfile.name)), 190 | "Local file is still available") 191 | self.assertTrue( 192 | path.isfile(path.join(self.remote_dir, obj.testfile.name)), 193 | "Remote file is not available.") 194 | 195 | def test_transfer_returns_boolean(self): 196 | """ 197 | Make sure an exception is thrown when the transfer task does not return 198 | a boolean. We don't want to confuse Celery. 199 | """ 200 | storage = QueuedStorage( 201 | local='django.core.files.storage.FileSystemStorage', 202 | remote='django.core.files.storage.FileSystemStorage', 203 | local_options=dict(location=self.local_dir), 204 | remote_options=dict(location=self.remote_dir), 205 | task='tests.tasks.NoneReturningTask') 206 | 207 | field = models.TestModel._meta.get_field('testfile') 208 | field.storage = storage 209 | 210 | obj = models.TestModel() 211 | obj.testfile.save(self.test_file_name, File(self.test_file)) 212 | obj.save() 213 | 214 | self.assertRaises(ValueError, 215 | obj.testfile.storage.result.get, propagate=True) 216 | 217 | def test_transfer_retried(self): 218 | """ 219 | Make sure the transfer task is retried correctly. 220 | """ 221 | storage = QueuedStorage( 222 | local='django.core.files.storage.FileSystemStorage', 223 | remote='django.core.files.storage.FileSystemStorage', 224 | local_options=dict(location=self.local_dir), 225 | remote_options=dict(location=self.remote_dir), 226 | task='tests.tasks.RetryingTask') 227 | field = models.TestModel._meta.get_field('testfile') 228 | field.storage = storage 229 | 230 | self.assertFalse(models.TestModel.retried) 231 | 232 | obj = models.TestModel() 233 | obj.testfile.save(self.test_file_name, File(self.test_file)) 234 | obj.save() 235 | 236 | self.assertTrue(models.TestModel.retried) 237 | 238 | def test_delayed_storage(self): 239 | storage = QueuedStorage( 240 | local='django.core.files.storage.FileSystemStorage', 241 | remote='django.core.files.storage.FileSystemStorage', 242 | local_options=dict(location=self.local_dir), 243 | remote_options=dict(location=self.remote_dir), 244 | delayed=True) 245 | 246 | field = models.TestModel._meta.get_field('testfile') 247 | field.storage = storage 248 | 249 | obj = models.TestModel() 250 | obj.testfile.save(self.test_file_name, File(self.test_file)) 251 | obj.save() 252 | 253 | self.assertIsNone(getattr(obj.testfile.storage, 'result', None)) 254 | 255 | self.assertFalse( 256 | path.isfile(path.join(self.remote_dir, obj.testfile.name)), 257 | "Remote file should not be transferred automatically.") 258 | 259 | result = obj.testfile.storage.transfer(obj.testfile.name) 260 | result.get() 261 | 262 | self.assertTrue( 263 | path.isfile(path.join(self.remote_dir, obj.testfile.name)), 264 | "Remote file is not available.") 265 | 266 | def test_remote_file_field(self): 267 | storage = QueuedStorage( 268 | local='django.core.files.storage.FileSystemStorage', 269 | remote='django.core.files.storage.FileSystemStorage', 270 | local_options=dict(location=self.local_dir), 271 | remote_options=dict(location=self.remote_dir), 272 | delayed=True) 273 | 274 | field = models.TestModel._meta.get_field('remote') 275 | field.storage = storage 276 | obj = models.TestModel() 277 | obj.remote.save(self.test_file_name, File(self.test_file)) 278 | obj.save() 279 | self.assertIsNone(getattr(obj.testfile.storage, 'result', None)) 280 | 281 | result = obj.remote.transfer() 282 | self.assertTrue(result) 283 | self.assertTrue(path.isfile(path.join(self.remote_dir, 284 | obj.remote.name))) 285 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | downloadcache = {distshare} 3 | args_are_paths = false 4 | envlist = 5 | {py35,py36}-django-dev 6 | {py27,py34,py35}-django-111 7 | {py27,py34,py35}-django-{19,110} 8 | {py27,py34,py35}-django-18 9 | {py27,py34}-django-17 10 | # flake8 11 | 12 | [testenv] 13 | basepython = 14 | py27: python2.7 15 | py34: python3.4 16 | py35: python3.5 17 | py36: python3.6 18 | usedevelop = true 19 | setenv = 20 | CELERY_CONFIG_MODULE=tests.celeryconfig 21 | commands = {posargs:py.test} 22 | deps = 23 | django-17: Django>=1.7,<1.8 24 | django-17: pytest-django>3,<=3.1.0 25 | django-18: Django>=1.8,<1.9 26 | django-19: Django>=1.9,<1.10 27 | django-110: Django>=1.10,<1.11 28 | django-111: Django>=1.11,<1.12 29 | django-dev: git+https://github.com/django/django.git#egg=Django 30 | django-{18,19,110,111,dev}: pytest-django>=3.2.0 31 | -rtests/requirements.txt 32 | 33 | [travis] 34 | python = 35 | 2.7: py27 36 | 3.4: py34 37 | 3.5: py35 38 | 3.6: py36 39 | 40 | [travis:env] 41 | DJANGO = 42 | 1.7: django-17 43 | 1.8: django-18 44 | 1.9: django-19 45 | 1.10: django-110 46 | 1.11: django-111 47 | dev: django-dev 48 | 49 | [testenv:flake8] 50 | basepython = python3.5 51 | skip_install = true 52 | deps = 53 | flake8 54 | flake8-docstrings>=0.2.7 55 | flake8-import-order>=0.9 56 | commands = 57 | flake8 raven/ setup.py 58 | 59 | [testenv:readme] 60 | basepython = python3.5 61 | deps = 62 | readme_renderer 63 | commands = 64 | python setup.py check -r -s 65 | 66 | 67 | --------------------------------------------------------------------------------