├── .gitignore
├── .travis.yml
├── CONTRIBUTING.md
├── CONTRIBUTORS.md
├── LICENSE-HEADER.txt
├── LICENSE.txt
├── README.rst
├── cookiecutter.json
├── requirements.txt
├── tests
    └── test_projext_generation.py
├── tox.ini
└── {{ cookiecutter.project_slug }}
    ├── .dockerignore
    ├── .gitignore
    ├── Makefile
    ├── README.md
    ├── config
        └── .gitkeep
    ├── data
        └── .gitkeep
    ├── docker
        ├── Dockerfile
        ├── Dockerfile.dev
        └── Dockerfile.release
    ├── model
        └── .gitkeep
    ├── notebook
        └── .gitkeep
    ├── requirements.txt
    ├── requirements_dev.txt
    ├── scripts
        └── .gitkeep
    ├── tests
        ├── __init__.py
        └── test_sample.py
    └── {{ cookiecutter.project_slug }}
        └── __init__.py


/.gitignore:
--------------------------------------------------------------------------------
1 | *~
2 | .idea
3 | *.iml
4 | .tox
5 | .pytest_cache
6 | __pycache__


--------------------------------------------------------------------------------
/.travis.yml:
--------------------------------------------------------------------------------
1 | sudo: false
2 | language: python
3 | python:
4 |     - "3.4"
5 | install: pip install tox-travis
6 | script: tox
7 | 


--------------------------------------------------------------------------------
/CONTRIBUTING.md:
--------------------------------------------------------------------------------
 1 | # Contributing
 2 | 
 3 | Any contributions are welcomed, and they are greatly appreciated!
 4 | 
 5 | ## Types of Contributions
 6 | 
 7 | There are many types of contributions. The followings are the common types.
 8 | 
 9 | ### Bug Reports, Feature Requests, and Feedback
10 | 
11 | Please create a [project issue][1]! Try to be as descriptive as possible.
12 | 
13 | ### Bug Fixes, New Features and Documentation
14 | 
15 | If you would like to contribute, fork this repository, and send a [pull request][2].
16 | Make sure to add the description and intent of the changes. Before submit a pull request,
17 | run `make test` from root directory to check the updated template.
18 | 
19 | [1]: https://github.com/docker-science/cookiecutter-docker-science/issues/new
20 | [2]: https://github.com/docker-science/cookiecutter-docker-science/compare
21 | 
22 | 
23 | ## Target Users
24 |  - Basic knowledge of Docker environments
25 |     - relationship between Docker images and containers
26 |     - Docker container lifecycle
27 |     - create an execution environment from a single Dockerfile
28 |  - Experience with Makefile
29 |     - add commands to a Makefile.
30 |     - pass variables to a Makefile
31 | 


--------------------------------------------------------------------------------
/CONTRIBUTORS.md:
--------------------------------------------------------------------------------
1 | # Contributors
2 | 
3 | * Takahiko Ito<takahiko03 _at_ gmail.com>
4 | * Yohei Kikuta<diracdiego _at_ gmail.com>
5 | * Jose Nazario
6 | * Masamune Kobayashi
7 | 


--------------------------------------------------------------------------------
/LICENSE-HEADER.txt:
--------------------------------------------------------------------------------
 1 | Cookiecutter Docker Science
 2 | Copyright (c) 2018 Takahiko Ito and contributors
 3 | (see CONTRIBUTORS.md)
 4 | 
 5 | Licensed under the Apache License, Version 2.0 (the "License");
 6 | you may not use this file except in compliance with the License.
 7 | You may obtain a copy of the License at
 8 | 
 9 | http://www.apache.org/licenses/LICENSE-2.0
10 | 
11 | Unless required by applicable law or agreed to in writing, software
12 | distributed under the License is distributed on an "AS IS" BASIS,
13 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | See the License for the specific language governing permissions and
15 | limitations under the License.
16 | 


--------------------------------------------------------------------------------
/LICENSE.txt:
--------------------------------------------------------------------------------
  1 | 
  2 |                                  Apache License
  3 |                            Version 2.0, January 2004
  4 |                         http://www.apache.org/licenses/
  5 | 
  6 |    TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
  7 | 
  8 |    1. Definitions.
  9 | 
 10 |       "License" shall mean the terms and conditions for use, reproduction,
 11 |       and distribution as defined by Sections 1 through 9 of this document.
 12 | 
 13 |       "Licensor" shall mean the copyright owner or entity authorized by
 14 |       the copyright owner that is granting the License.
 15 | 
 16 |       "Legal Entity" shall mean the union of the acting entity and all
 17 |       other entities that control, are controlled by, or are under common
 18 |       control with that entity. For the purposes of this definition,
 19 |       "control" means (i) the power, direct or indirect, to cause the
 20 |       direction or management of such entity, whether by contract or
 21 |       otherwise, or (ii) ownership of fifty percent (50%) or more of the
 22 |       outstanding shares, or (iii) beneficial ownership of such entity.
 23 | 
 24 |       "You" (or "Your") shall mean an individual or Legal Entity
 25 |       exercising permissions granted by this License.
 26 | 
 27 |       "Source" form shall mean the preferred form for making modifications,
 28 |       including but not limited to software source code, documentation
 29 |       source, and configuration files.
 30 | 
 31 |       "Object" form shall mean any form resulting from mechanical
 32 |       transformation or translation of a Source form, including but
 33 |       not limited to compiled object code, generated documentation,
 34 |       and conversions to other media types.
 35 | 
 36 |       "Work" shall mean the work of authorship, whether in Source or
 37 |       Object form, made available under the License, as indicated by a
 38 |       copyright notice that is included in or attached to the work
 39 |       (an example is provided in the Appendix below).
 40 | 
 41 |       "Derivative Works" shall mean any work, whether in Source or Object
 42 |       form, that is based on (or derived from) the Work and for which the
 43 |       editorial revisions, annotations, elaborations, or other modifications
 44 |       represent, as a whole, an original work of authorship. For the purposes
 45 |       of this License, Derivative Works shall not include works that remain
 46 |       separable from, or merely link (or bind by name) to the interfaces of,
 47 |       the Work and Derivative Works thereof.
 48 | 
 49 |       "Contribution" shall mean any work of authorship, including
 50 |       the original version of the Work and any modifications or additions
 51 |       to that Work or Derivative Works thereof, that is intentionally
 52 |       submitted to Licensor for inclusion in the Work by the copyright owner
 53 |       or by an individual or Legal Entity authorized to submit on behalf of
 54 |       the copyright owner. For the purposes of this definition, "submitted"
 55 |       means any form of electronic, verbal, or written communication sent
 56 |       to the Licensor or its representatives, including but not limited to
 57 |       communication on electronic mailing lists, source code control systems,
 58 |       and issue tracking systems that are managed by, or on behalf of, the
 59 |       Licensor for the purpose of discussing and improving the Work, but
 60 |       excluding communication that is conspicuously marked or otherwise
 61 |       designated in writing by the copyright owner as "Not a Contribution."
 62 | 
 63 |       "Contributor" shall mean Licensor and any individual or Legal Entity
 64 |       on behalf of whom a Contribution has been received by Licensor and
 65 |       subsequently incorporated within the Work.
 66 | 
 67 |    2. Grant of Copyright License. Subject to the terms and conditions of
 68 |       this License, each Contributor hereby grants to You a perpetual,
 69 |       worldwide, non-exclusive, no-charge, royalty-free, irrevocable
 70 |       copyright license to reproduce, prepare Derivative Works of,
 71 |       publicly display, publicly perform, sublicense, and distribute the
 72 |       Work and such Derivative Works in Source or Object form.
 73 | 
 74 |    3. Grant of Patent License. Subject to the terms and conditions of
 75 |       this License, each Contributor hereby grants to You a perpetual,
 76 |       worldwide, non-exclusive, no-charge, royalty-free, irrevocable
 77 |       (except as stated in this section) patent license to make, have made,
 78 |       use, offer to sell, sell, import, and otherwise transfer the Work,
 79 |       where such license applies only to those patent claims licensable
 80 |       by such Contributor that are necessarily infringed by their
 81 |       Contribution(s) alone or by combination of their Contribution(s)
 82 |       with the Work to which such Contribution(s) was submitted. If You
 83 |       institute patent litigation against any entity (including a
 84 |       cross-claim or counterclaim in a lawsuit) alleging that the Work
 85 |       or a Contribution incorporated within the Work constitutes direct
 86 |       or contributory patent infringement, then any patent licenses
 87 |       granted to You under this License for that Work shall terminate
 88 |       as of the date such litigation is filed.
 89 | 
 90 |    4. Redistribution. You may reproduce and distribute copies of the
 91 |       Work or Derivative Works thereof in any medium, with or without
 92 |       modifications, and in Source or Object form, provided that You
 93 |       meet the following conditions:
 94 | 
 95 |       (a) You must give any other recipients of the Work or
 96 |           Derivative Works a copy of this License; and
 97 | 
 98 |       (b) You must cause any modified files to carry prominent notices
 99 |           stating that You changed the files; and
100 | 
101 |       (c) You must retain, in the Source form of any Derivative Works
102 |           that You distribute, all copyright, patent, trademark, and
103 |           attribution notices from the Source form of the Work,
104 |           excluding those notices that do not pertain to any part of
105 |           the Derivative Works; and
106 | 
107 |       (d) If the Work includes a "NOTICE" text file as part of its
108 |           distribution, then any Derivative Works that You distribute must
109 |           include a readable copy of the attribution notices contained
110 |           within such NOTICE file, excluding those notices that do not
111 |           pertain to any part of the Derivative Works, in at least one
112 |           of the following places: within a NOTICE text file distributed
113 |           as part of the Derivative Works; within the Source form or
114 |           documentation, if provided along with the Derivative Works; or,
115 |           within a display generated by the Derivative Works, if and
116 |           wherever such third-party notices normally appear. The contents
117 |           of the NOTICE file are for informational purposes only and
118 |           do not modify the License. You may add Your own attribution
119 |           notices within Derivative Works that You distribute, alongside
120 |           or as an addendum to the NOTICE text from the Work, provided
121 |           that such additional attribution notices cannot be construed
122 |           as modifying the License.
123 | 
124 |       You may add Your own copyright statement to Your modifications and
125 |       may provide additional or different license terms and conditions
126 |       for use, reproduction, or distribution of Your modifications, or
127 |       for any such Derivative Works as a whole, provided Your use,
128 |       reproduction, and distribution of the Work otherwise complies with
129 |       the conditions stated in this License.
130 | 
131 |    5. Submission of Contributions. Unless You explicitly state otherwise,
132 |       any Contribution intentionally submitted for inclusion in the Work
133 |       by You to the Licensor shall be under the terms and conditions of
134 |       this License, without any additional terms or conditions.
135 |       Notwithstanding the above, nothing herein shall supersede or modify
136 |       the terms of any separate license agreement you may have executed
137 |       with Licensor regarding such Contributions.
138 | 
139 |    6. Trademarks. This License does not grant permission to use the trade
140 |       names, trademarks, service marks, or product names of the Licensor,
141 |       except as required for reasonable and customary use in describing the
142 |       origin of the Work and reproducing the content of the NOTICE file.
143 | 
144 |    7. Disclaimer of Warranty. Unless required by applicable law or
145 |       agreed to in writing, Licensor provides the Work (and each
146 |       Contributor provides its Contributions) on an "AS IS" BASIS,
147 |       WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
148 |       implied, including, without limitation, any warranties or conditions
149 |       of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
150 |       PARTICULAR PURPOSE. You are solely responsible for determining the
151 |       appropriateness of using or redistributing the Work and assume any
152 |       risks associated with Your exercise of permissions under this License.
153 | 
154 |    8. Limitation of Liability. In no event and under no legal theory,
155 |       whether in tort (including negligence), contract, or otherwise,
156 |       unless required by applicable law (such as deliberate and grossly
157 |       negligent acts) or agreed to in writing, shall any Contributor be
158 |       liable to You for damages, including any direct, indirect, special,
159 |       incidental, or consequential damages of any character arising as a
160 |       result of this License or out of the use or inability to use the
161 |       Work (including but not limited to damages for loss of goodwill,
162 |       work stoppage, computer failure or malfunction, or any and all
163 |       other commercial damages or losses), even if such Contributor
164 |       has been advised of the possibility of such damages.
165 | 
166 |    9. Accepting Warranty or Additional Liability. While redistributing
167 |       the Work or Derivative Works thereof, You may choose to offer,
168 |       and charge a fee for, acceptance of support, warranty, indemnity,
169 |       or other liability obligations and/or rights consistent with this
170 |       License. However, in accepting such obligations, You may act only
171 |       on Your own behalf and on Your sole responsibility, not on behalf
172 |       of any other Contributor, and only if You agree to indemnify,
173 |       defend, and hold each Contributor harmless for any liability
174 |       incurred by, or claims asserted against, such Contributor by reason
175 |       of your accepting any such warranty or additional liability.
176 | 
177 |    END OF TERMS AND CONDITIONS
178 | 
179 |    APPENDIX: How to apply the Apache License to your work.
180 | 
181 |       To apply the Apache License to your work, attach the following
182 |       boilerplate notice, with the fields enclosed by brackets "[]"
183 |       replaced with your own identifying information. (Don't include
184 |       the brackets!)  The text should be enclosed in the appropriate
185 |       comment syntax for the file format. We also recommend that a
186 |       file or class name and description of purpose be included on the
187 |       same "printed page" as the copyright notice for easier
188 |       identification within third-party archives.
189 | 
190 |    Copyright [yyyy] [name of copyright owner]
191 | 
192 |    Licensed under the Apache License, Version 2.0 (the "License");
193 |    you may not use this file except in compliance with the License.
194 |    You may obtain a copy of the License at
195 | 
196 |        http://www.apache.org/licenses/LICENSE-2.0
197 | 
198 |    Unless required by applicable law or agreed to in writing, software
199 |    distributed under the License is distributed on an "AS IS" BASIS,
200 |    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
201 |    See the License for the specific language governing permissions and
202 |    limitations under the License.
203 | 


--------------------------------------------------------------------------------
/README.rst:
--------------------------------------------------------------------------------
  1 | .. |travis| image:: https://travis-ci.org/docker-science/cookiecutter-docker-science.svg?branch=master
  2 |     :target: https://travis-ci.org/docker-science/cookiecutter-docker-science
  3 | 
  4 | |travis|
  5 | 
  6 | Table of Contents
  7 | ------------------
  8 | 
  9 | .. contents:: This article consists of the following sections.
 10 |     :depth: 1
 11 | 
 12 | Features
 13 | --------
 14 | 
 15 |  `Cookiecutter Docker Science <https://docker-science.github.io/>`_ provides the following features.
 16 | 
 17 | * **Improve reproducibility** of the results in machine learning projects with **Docker**
 18 | * Output optimal directories and file template for machine learning projects
 19 | * `Edit codes with favorite editors (Atom, vim, Emacs etc) <https://docker-science.github.io/#edit-codes-with-preferred-editors>`_
 20 | 
 21 | * Provide `make` targets useful for data analysis (Jupyter notebook, test, lint, docker etc)
 22 | 
 23 | Introduction
 24 | ------------
 25 | 
 26 | **NOTE**: please visit `home page <https://docker-science.github.io/>`_ before you get started.
 27 | 
 28 | Many researchers and engineers do their machine learning or data mining experiments.
 29 | For such data engineering tasks, researchers apply various tools and system libraries which are constantly
 30 | updated, installing and updating them cause problems in local environments. Even when we work in hosting
 31 | environments such as EC2, we are not free from this problem. Some experiments succeeded in one
 32 | instance but failed in another one, since library versions of each EC2 instances could be different.
 33 | 
 34 | By contrast, we can creates the identical Docker container in which needed tools with the correct versions are already installed in one command without
 35 | changing system libraries in host machines. This aspect of Docker is important for reproducibility of experiments,
 36 | and keep the projects in continuous integration systems.
 37 | 
 38 | Unfortunately running experiments in a Docker containers is troublesome. Adding a new library into ``requirements.txt``
 39 | or ``Dockerfile`` does not installed as if local machine. We need to create Docker image and container each time.
 40 | We also need to forward ports to see server responses such as Jupyter Notebook UI launch in Docker container in our local PC.
 41 | Cookiecutter Docker Science provides utilities to make working in Docker container simple.
 42 | 
 43 | This project is a tiny template for machine learning projects developed in Docker environments.
 44 | In machine learning tasks, projects glow uniquely to fit target tasks, but in the initial state,
 45 | most directory structure and targets in `Makefile` are common.
 46 | Cookiecutter Docker Science generates initial directories which fits simple machine learning tasks.
 47 | 
 48 | Requirements
 49 | ------------
 50 | 
 51 | * Python 3.5 or later
 52 | * `Cookiecutter 1.6 or later <https://cookiecutter.readthedocs.io/en/latest/installation.html>`_
 53 | * `Docker version 17 or later <https://docs.docker.com/install/#support>`_
 54 | 
 55 | Quick start
 56 | -----------
 57 | 
 58 | To generate project from the cookiecutter-docker-science template, please run the following command.
 59 | 
 60 | ``$cookiecutter git@github.com:docker-science/cookiecutter-docker-science.git``
 61 | 
 62 | Then the cookiecutter command ask for several questions on generated project as follows.
 63 | 
 64 | ::
 65 | 
 66 |     $cookiecutter git@github.com:docker-science/cookiecutter-docker-science.git
 67 |     project_name [project_name]: food-image-classification
 68 |     project_slug [food_image_classification]:
 69 |     jupyter_host_port [8888]:
 70 |     description [Please Input a short description]: Classify food images into several categories
 71 |     Select data_source_type:
 72 |     1 - s3
 73 |     2 - nfs
 74 |     3 - url
 75 |     data_source [Please Input data source]: s3://research-data/food-images
 76 | 
 77 | Then you get the generated project directory, ``food-image-classification``.
 78 | 
 79 | Initial directories and files
 80 | -----------------------------
 81 | 
 82 | The following is the initial directory structure generated in the previous section.
 83 | 
 84 | ::
 85 | 
 86 |     ├── Makefile                          <- Makefile contains many targets such as create docker container or
 87 |     │                                        get input files.
 88 |     ├── config                            <- This directory contains configuration files used in scripts
 89 |     │   │                                    or Jupyter Notebook.
 90 |     │   └── jupyter_config.py
 91 |     ├── data                              <- data directory contains the input resources.
 92 |     ├── docker                            <- docker directory contains Dockerfile.
 93 |     │   ├── Dockerfile                    <- Base Dockerfile contains the basic settings.
 94 |     │   ├── Dockerfile.dev                <- Dockerfile for experiments this Docker image is derived from the base Docker image.
 95 |     │   │                                    This Docker image does not copy the files and directory but used mount the top
 96 |     │   │                                    directory of the host environments.
 97 |     │   └── Dockerfile.release            <- Dockerfile for production this Docker image is derived from the base Docker image.
 98 |     │                                        The Docker image copy the files and directory under the project top directory.
 99 |     ├── model                             <- model directory store the model files created in the experiments.
100 |     ├── my_data_science_project           <- cookie-cutter-docker-science creates the directory whose name is same
101 |     │   │                                    as project name. In this directory users puts python files used in scripts
102 |     │   │                                    or Jupyter Notebook.
103 |     │   └── __init__.py
104 |     ├── notebook                          <- This directory stores the ipynb files saved in Jupyter Notebook.
105 |     ├── requirements.txt                  <- Libraries needed in the project. The library listed in this file
106 |     │                                        are installed in the Docker images for not only development but also production.
107 |     ├── requirements_dev.txt              <- Libraries needed to run experiments. The library listed in this file
108 |     │                                        are installed in the Docker images for developments.
109 |     └── scripts                           <- Users add the script files to generate model files or run evaluation.
110 | 
111 | 
112 | Makefile targets
113 | ----------------
114 | 
115 | Cookiecutter Docker Science provides many Makefile targets to supports experiments in a Docker container. Users can run the target with `make [TARGET]` command.
116 | 
117 | init
118 | ~~~~~
119 | 
120 | After cookiecutter-docker-science generate the directories and files, users first run this command. `init` setups resources for experiments.
121 | Specifically `init` run `init-docker` and `sync-from-source` command.
122 | 
123 | - init-docker
124 | 
125 |   `init-docker` command first creates Docker the images based on `docker/Dockerfile`.
126 | 
127 | - sync-from-source
128 | 
129 |   `sync-from-source` downloads input files which we specified in the project generation.  If you want to change the input files, please modify this target to download the new data source.
130 | 
131 | create-container
132 | ~~~~~~~~~~~~~~~~~
133 | 
134 | `create-container` command creates Docker container based on the created image and login the Docker container.
135 | 
136 | start-container
137 | ~~~~~~~~~~~~~~~~
138 | 
139 | Users can start and login the Docker container with `start container` created by the `create-container`.
140 | 
141 | jupyter
142 | ~~~~~~~
143 | 
144 | `jupyter` target launch Jupyter Notebook server.
145 | 
146 | profile
147 | ~~~~~~~
148 | 
149 | `profile` target shows the misc information of the project such as port number or container name.
150 | 
151 | 
152 | clean
153 | ~~~~~
154 | 
155 | `clean` target removes the artifacts such as models and *.pyc files.
156 | 
157 | - clean-model
158 | 
159 |   `clean-model` command removes model files in `model` directory.
160 | 
161 | - clean-pyc
162 | 
163 |   `clean-pyc` command removes model files of *.pyc, *.pyo and __pycache__.
164 | 
165 | - clean-docker
166 | 
167 |   `clean-docker` command removes the Docker images and container generated with `make init-docker` and `make create-container`.
168 |   When we update Python libraries in `requirements.txt` or system tools in `Dockerfile`, we need to clean Docker the image and container with this target and create the updated image and container with `make init-docker` and `make create-container`.
169 | 
170 | distclean
171 | ~~~~~~~~~
172 | 
173 | `distclean` target removes all reproducible objects. Specifically this target run `clean` target and remove all files in data directory.
174 | 
175 | - clean-data
176 | 
177 |   `clean-data` command removes all datasets in `data` directory.
178 | 
179 | lint
180 | ~~~~~
181 | 
182 | `lint` target check if coding style meets the coding standard.
183 | 
184 | test
185 | ~~~~~
186 | 
187 | `test` target executes tests.
188 | 
189 | 
190 | sync-to-source
191 | ~~~~~~~~~~~~~~
192 | 
193 | `sync-to-remote` target uploads the local files stored in `data` to specified data sources in such as S3 or NFS directories.
194 | 
195 | Working with Docker container
196 | ------------------------------
197 | 
198 | With Cookiecutter Docker Science, data scientists or software engineers do their developments in host environment.
199 | They open Jupyter notebook in the browsers in the host machine connecting the Jupyter server launched in Docker container.
200 | They also writes the ML scripts or library classes in the host machine. The code modification in host environment are
201 | reflected in the container environment. In the containers, they just launch Jupyter server or start ML scripts
202 | with make command.
203 | 
204 | Files and directories
205 | ~~~~~~~~~~~~~~~~~~~~~
206 | 
207 | When you log in a Docker container by ``make create-container`` or ``make start-container`` command, the log in directory is ``/work``.
208 | The directory contains the project top directories in host computer such as ``data`` or ``model``. Actually the Docker container mounts
209 | the project directory to ``/work`` of the container and therefore when you can edit the files in the host environment with your favorite editor
210 | such as Vim, Emacs, Atom or PyCharm. The changes in host environment are reflected in container environment.
211 | 
212 | Jupyter Notebook
213 | ~~~~~~~~~~~~~~~~~
214 | 
215 | We can run a Jupyter Notebook in the Docker container. The Jupyter Notebook uses the default port ``8888`` in **Docker container (NOT host machine)** and
216 | the port is forwarded to the one you specify with ``JUPYTER_HOST_PORT``  in the cookiecutter command. You can see the Jupyter Notebook UI accessing
217 | "http://localhost:JUPYTER_HOST_PORT". When you save notebooks the files are saved in the ``notebook`` directory.
218 | 
219 | Tips
220 | -----
221 | 
222 | Generate Docker Image for production
223 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
224 | 
225 | `make init-docker` command creates a Docker image based on `docker/Dockerfile.dev`, which contains
226 | libraries for developments. The libraries are not needed in production.
227 | 
228 | To create a Docker image for production which does not contain the development
229 | libraries such as Jupyter, we run `make init-docker` command specifying a environment variable `MODE` to `release` as `make init-docker MODE=release`.
230 | 
231 | Override port number for Jupyter Notebook
232 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
233 | 
234 | In the generation of project with cookiecutter, the default port of Jupyter Notebook in host is ``8888``. The number is common and could
235 | have a collision to another server processes.
236 | 
237 | If we already have the container, we first need to remove the current container with ``make clean-container``. And then
238 | we create the Docker container changing the port number with ``make create-container`` command adding the Jupyter port parameter (JUPYTER_HOST_PORT).
239 | For example the following command creates Docker container forwarding Jupyter default port ``8888`` to ``9900`` in host.
240 | 
241 | ::
242 | 
243 |     make create-container JUPYTER_HOST_PORT=9900
244 | 
245 | Then you launch Jupyter Notebook in the Docker container, you can see the Jupyter Notebook in http://localhost:9900
246 | 
247 | Specify suitable Dockerfile in stages
248 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
249 | 
250 | Some projects can have multiple Dockerfiles. ``Dockerfile.gpu`` contains the settings for GPU machines. ``Dockerfile.cpu`` contains settings to be that can be used in production for non-GPU machines.
251 | 
252 | To use one of these specific Dockerfile, override the settings by adding parameters to the make command. For example, when we want to create a container from ``docker/Dockerfile.cpu``, we run ``make create-container DOCKERFILE=docker/Dockerfile.cpu``.
253 | 
254 | 
255 | Show target specific help
256 | ~~~~~~~~~~~~~~~~~~~~~~~~~
257 | 
258 | `help` target flushes the details of specified target. For example, to get the details of `clean` target.
259 | 
260 | :: 
261 | 
262 |     $make help TARGET=clean
263 |     target: clean
264 |     dependencies: clean-model clean-pyc clean-docker
265 |     description: remove all artifacts
266 | 
267 | As we can see, the dependencies and description of the specified target (`clean`) are shown.
268 | 
269 | License
270 | -------
271 | 
272 | Apache version 2.0
273 | 
274 | Contribution
275 | -------------
276 | 
277 | See `CONTRIBUTING.md <CONTRIBUTING.md>`_.
278 | 


--------------------------------------------------------------------------------
/cookiecutter.json:
--------------------------------------------------------------------------------
 1 | {
 2 |     "project_name": "project_name",
 3 |     "project_slug": "{{ cookiecutter.project_name.lower().replace(' ', '_').replace('-', '_') }}",
 4 |     "jupyter_host_port": "8888",
 5 |     "description": "Please Input a short description",
 6 |     "data_source_type": ["s3", "nfs", "url"],
 7 |     "data_source": "Please Input data source",
 8 |     "use_nvidia_docker": ["no", "yes"],
 9 |     "formatter_type": ["black", "autopep8", "yapf"]
10 | }
11 | 


--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | cookiecutter==1.5.0
2 | 
3 | # Testing
4 | pytest==3.0.5
5 | tox==2.5.0
6 | pytest-cookies==0.2.0
7 | binaryornot==0.4.0
8 | tox==2.5.0
9 | 


--------------------------------------------------------------------------------
/tests/test_projext_generation.py:
--------------------------------------------------------------------------------
 1 | # -*- coding: utf-8 -*-
 2 | 
 3 | import os
 4 | import re
 5 | 
 6 | from binaryornot.check import is_binary
 7 | from io import open
 8 | 
 9 | RE_OBJ = re.compile('{{(\s?cookiecutter)[.](.*?)}}')
10 | 
11 | 
12 | def build_files_list(root_dir):
13 |     return [
14 |         os.path.join(dirpath, file_path)
15 |         for dirpath, _, files in os.walk(root_dir)
16 |         for file_path in files
17 |     ]
18 | 
19 | 
20 | def check_paths(paths):
21 |     for path in paths:
22 |         if is_binary(path):
23 |             continue
24 |         for line in open(path, 'r', encoding="latin-1"):
25 |             match = RE_OBJ.search(line)
26 |             msg = 'variable not replaced in {}'
27 |             assert match is None, msg.format(path)
28 | 
29 | 
30 | def test_default_configuration(cookies, context={}):
31 |     result = cookies.bake(extra_context=context)
32 |     assert result.exit_code == 0
33 |     assert result.exception is None
34 |     assert result.project.basename == 'project_name'
35 |     assert result.project.isdir()
36 |     paths = build_files_list(str(result.project))
37 | 
38 |     assert paths
39 |     check_paths(paths)
40 | 
41 | 
42 | def test_custom_configuration(cookies, context={'project_name': 'ml project'}):
43 |     result = cookies.bake(extra_context=context)
44 |     assert result.exit_code == 0
45 |     assert result.exception is None
46 |     assert result.project.basename == 'ml_project'  # note formatted
47 |     assert result.project.isdir()
48 |     paths = build_files_list(str(result.project))
49 | 
50 |     assert paths
51 |     check_paths(paths)
52 | 


--------------------------------------------------------------------------------
/tox.ini:
--------------------------------------------------------------------------------
 1 | [tox]
 2 | skipsdist = true
 3 | envlist = py34
 4 | 
 5 | [testenv]
 6 | passenv = LC_ALL, LANG, HOME
 7 | deps =
 8 |     binaryornot
 9 |     pytest-cookies
10 | commands = py.test {posargs:tests}
11 | 


--------------------------------------------------------------------------------
/{{ cookiecutter.project_slug }}/.dockerignore:
--------------------------------------------------------------------------------
1 | model
2 | data
3 | log
4 | 


--------------------------------------------------------------------------------
/{{ cookiecutter.project_slug }}/.gitignore:
--------------------------------------------------------------------------------
 1 | # emacs
 2 | *~
 3 | 
 4 | # compiled
 5 | *.pyc
 6 | 
 7 | # IntelliJ
 8 | .idea
 9 | *.iml
10 | 
11 | # model
12 | model
13 | 
14 | # data
15 | data
16 | 
17 | # notebook
18 | notebook/.ipynb_checkpoints
19 | 


--------------------------------------------------------------------------------
/{{ cookiecutter.project_slug }}/Makefile:
--------------------------------------------------------------------------------
  1 | .PHONY: clean clean-model clean-pyc docs help init init-docker create-container start-container jupyter test lint profile clean clean-data clean-docker clean-container clean-image sync-from-source sync-to-source
  2 | .DEFAULT_GOAL := help
  3 | 
  4 | ###########################################################################################################
  5 | ## SCRIPTS
  6 | ###########################################################################################################
  7 | 
  8 | define PRINT_HELP_PYSCRIPT
  9 | import os, re, sys
 10 | 
 11 | if os.environ['TARGET']:
 12 |     target = os.environ['TARGET']
 13 |     is_in_target = False
 14 |     for line in sys.stdin:
 15 |         match = re.match(r'^(?P<target>{}):(?P<dependencies>.*)?## (?P<description>.*)$$'.format(target).format(target), line)
 16 |         if match:
 17 |             print("target: %-20s" % (match.group("target")))
 18 |             if "dependencies" in match.groupdict().keys():
 19 |                 print("dependencies: %-20s" % (match.group("dependencies")))
 20 |             if "description" in match.groupdict().keys():
 21 |                 print("description: %-20s" % (match.group("description")))
 22 |             is_in_target = True
 23 |         elif is_in_target == True:
 24 |             match = re.match(r'^\t(.+)', line)
 25 |             if match:
 26 |                 command = match.groups()
 27 |                 print("command: %s" % (command))
 28 |             else:
 29 |                 is_in_target = False
 30 | else:
 31 |     for line in sys.stdin:
 32 |         match = re.match(r'^([a-zA-Z_-]+):.*?## (.*)$$', line)
 33 |         if match:
 34 |             target, help = match.groups()
 35 |             print("%-20s %s" % (target, help))
 36 | endef
 37 | 
 38 | define START_DOCKER_CONTAINER
 39 | if [ `$(DOCKER) inspect -f {% raw -%}{{.State.Running}}{%- endraw %} $(CONTAINER_NAME)` = "false" ] ; then
 40 |         $(DOCKER) start $(CONTAINER_NAME)
 41 | fi
 42 | endef
 43 | 
 44 | ###########################################################################################################
 45 | ## VARIABLES
 46 | ###########################################################################################################
 47 | 
 48 | {%- if cookiecutter.use_nvidia_docker == 'yes' %}
 49 | export DOCKER=nvidia-docker
 50 | {%- else %}
 51 | export DOCKER=docker
 52 | {%- endif %}
 53 | export TARGET=
 54 | export PWD=`pwd`
 55 | export PRINT_HELP_PYSCRIPT
 56 | export START_DOCKER_CONTAINER
 57 | export PYTHONPATH=$PYTHONPATH:$(PWD)
 58 | export PROJECT_NAME={{ cookiecutter.project_slug }}
 59 | export MODE=dev
 60 | export BASE_DOCKERFILE=docker/Dockerfile
 61 | export DOCKERFILE=$(BASE_DOCKERFILE).$(MODE)
 62 | export BASE_IMAGE_NAME=$(PROJECT_NAME)-image-base
 63 | export IMAGE_NAME=$(PROJECT_NAME)-$(MODE)-image
 64 | export CONTAINER_NAME=$(PROJECT_NAME)-$(MODE)-container
 65 | export DATA_SOURCE={{ cookiecutter.data_source }}
 66 | export JUPYTER_HOST_PORT={{ cookiecutter.jupyter_host_port }}
 67 | export JUPYTER_CONTAINER_PORT=8888
 68 | export PYTHON=python3
 69 | 
 70 | ###########################################################################################################
 71 | ## ADD TARGETS SPECIFIC TO "{{ cookiecutter.project_name }}"
 72 | ###########################################################################################################
 73 | 
 74 | 
 75 | ###########################################################################################################
 76 | ## GENERAL TARGETS
 77 | ###########################################################################################################
 78 | 
 79 | help: ## show this message
 80 | 	@$(PYTHON) -c "$$PRINT_HELP_PYSCRIPT" < $(MAKEFILE_LIST)
 81 | 
 82 | init: init-docker sync-from-source ## initialize repository for traning
 83 | 
 84 | sync-from-source: ## download data data source to local envrionment
 85 | {%- if cookiecutter.data_source_type == 's3' %}
 86 | 	-aws s3 sync $(DATA_SOURCE) ./data/
 87 | {%- elif cookiecutter.data_source_type == 'nfs' %}
 88 | 	cp -r $(DATA_SOURCE)/* ./data/
 89 | {%- elif cookiecutter.data_source_type == 'url' %}
 90 | 	wget $(DATA_SOURCE) -P ./data/
 91 | {% endif %}
 92 | 
 93 | init-docker: ## initialize docker image
 94 | 	$(DOCKER) build -t $(BASE_IMAGE_NAME) -f $(BASE_DOCKERFILE) --build-arg UID=$(shell id -u) .
 95 | 	$(DOCKER) build -t $(IMAGE_NAME) -f $(DOCKERFILE) .
 96 | 
 97 | init-docker-no-cache: ## initialize docker image without cache
 98 | 	$(DOCKER) build --no-cache -t $(BASE_IMAGE_NAME) -f $(BASE_DOCKERFILE) --build-arg UID=$(shell id -u) .
 99 | 	$(DOCKER) build --no-cache -t $(IMAGE_NAME) -f $(DOCKERFILE) .
100 | 
101 | sync-to-source: ## sync local data to data source
102 | {%- if cookiecutter.data_source_type == 's3' %}
103 | 	-aws s3 sync ./data/ $(DATA_SOURCE)
104 | {%- elif cookiecutter.data_source_type == 'nfs' %}
105 | 	cp -r ./data/* $(DATA_SOURCE)/
106 | {%- elif cookiecutter.data_source_type == 'url' %}
107 | 	echo "no sync target for url data source..."
108 | {% endif %}
109 | 
110 | create-container: ## create docker container
111 | 	$(DOCKER) run -it -v $(PWD):/work -p $(JUPYTER_HOST_PORT):$(JUPYTER_CONTAINER_PORT) --name $(CONTAINER_NAME) $(IMAGE_NAME)
112 | 
113 | start-container: ## start docker container
114 | 	@echo "$$START_DOCKER_CONTAINER" | $(SHELL)
115 | 	@echo "Launched $(CONTAINER_NAME)..."
116 | 	$(DOCKER) attach $(CONTAINER_NAME)
117 | 
118 | jupyter: ## start Jupyter Notebook server
119 | 	jupyter-notebook --ip=0.0.0.0 --port=${JUPYTER_CONTAINER_PORT}
120 | 
121 | test: ## run test cases in tests directory
122 | 	$(PYTHON) -m unittest discover
123 | 
124 | lint: ## check style with flake8
125 | 	flake8 {{ cookiecutter.project_slug }}
126 | 	mypy {{ cookiecutter.project_slug }}
127 | 
128 | profile: ## show profile of the project
129 | 	@echo "CONTAINER_NAME: $(CONTAINER_NAME)"
130 | 	@echo "IMAGE_NAME: $(IMAGE_NAME)"
131 | 	@echo "JUPYTER_PORT: `$(DOCKER) port $(CONTAINER_NAME)`"
132 | 	@echo "DATA_SOURE: $(DATA_SOURCE)"
133 | 
134 | clean: clean-model clean-pyc clean-docker ## remove all artifacts
135 | 
136 | clean-model: ## remove model artifacts
137 | 	rm -fr model/*
138 | 
139 | clean-pyc: ## remove Python file artifacts
140 | 	find . -name '*.pyc' -exec rm -f {} +
141 | 	find . -name '*.pyo' -exec rm -f {} +
142 | 	find . -name '*~' -exec rm -f {} +
143 | 	find . -name '__pycache__' -exec rm -fr {} +
144 | 
145 | distclean: clean clean-data ## remove all the reproducible resources including Docker images
146 | 
147 | clean-data: ## remove files under data
148 | 	rm -fr data/*
149 | 
150 | clean-docker: clean-container clean-image ## remove Docker image and container
151 | 
152 | clean-container: ## remove Docker container
153 | 	-$(DOCKER) rm $(CONTAINER_NAME)
154 | 
155 | clean-image: ## remove Docker image
156 | 	-$(DOCKER) image rm $(IMAGE_NAME)
157 | 
158 | format:
159 | {%- if cookiecutter.formatter_type == 'black' %}
160 | 	- black scripts
161 | 	- black $(PROJECT_NAME)
162 | {%- elif cookiecutter.formatter_type == 'autopep8' %}
163 | 	- autopep8 --in-place --recursive scripts
164 | 	- autopep8 --in-place --recursive $(PROJECT_NAME)
165 | {%- elif cookiecutter.formatter_type == 'yapf' %}
166 | 	- yapf --in-place --recursive scripts
167 | 	- yapf --in-place --recursive $(PROJECT_NAME)
168 | {%- elif cookiecutter.formatter_type == '' %}
169 | 	echo "Formatter is not set."
170 | {% endif %}
171 | 


--------------------------------------------------------------------------------
/{{ cookiecutter.project_slug }}/README.md:
--------------------------------------------------------------------------------
 1 | # {{ cookiecutter.project_name }}
 2 | 
 3 | {{ cookiecutter.description }}
 4 | 
 5 | ## Requirements
 6 | 
 7 | * [Docker version 17 or later](https://docs.docker.com/install/#support)
 8 | 
 9 | ## Setup development environment
10 | 
11 | We setup the development environment in a Docker container with the following command.
12 | 
13 | - `make init`
14 | 
15 | This command gets the resources for training and testing, and then prepares the Docker image for the experiments.
16 | After creating the Docker image, you run the following command.
17 | 
18 | - `make create-container`
19 | 
20 | The above command creates a Docker container from the Docker image which we create with `make init`, and then
21 | login to the Docker container. Now we made the development environment. For create and evaluate the model,
22 | you run the following command.
23 | 
24 | ## Development with Docker container
25 | 
26 | This section shows how we develop with the created Docker container.
27 | 
28 | ### Edit source code
29 | 
30 | Most of the source codes of this project, `{{ cookiecutter.project_name }}` are stored in the `{{ cookiecutter.project_slug }}` directory.
31 | Generated Docker container mounts the project directory to ``/work`` of the container and therefore
32 | when you can edit the files in the host environment with your favorite editor
33 | such as Vim, Emacs, Atom or PyCharm. The changes in host environment are reflected in the Docker container environment.
34 | 
35 | ### Update dependencies
36 | 
37 | When we need to add libraries in `Dockerfile` or `requirements.txt`
38 | which are added to working environment in the Docker container, we need to drop the current Docker container and
39 | image, and then create them again with the latest setting. To remove the Docker the container and image, run `make clean-docker`
40 | and then `make init-docker` command to create the Docker container with the latest setting.
41 | 
42 | ### Login Docker container
43 | 
44 | Only the first time you need to create a Docker container, from the image created in `make init` command.
45 | `make create-container` creates and launch the {{ cookiecutter.project_slug }} container.
46 | After creating the container, you just need run `make start-container`.
47 | 
48 | ### Logout from Docker container
49 | 
50 | When you logout from shell in Docker container, please run `exit` in the console.
51 | 
52 | ### Run linter
53 | 
54 | When you check the code quality, please run `make lint`
55 | 
56 | ### Run test
57 | 
58 | When you run test in `tests` directory, please run `make test`
59 | 
60 | ### Sync data source to local data directory
61 | 
62 | When you want to download data in remote data sources such as Amazon S3 or NFS, `sync-from-remote` target downloads them.
63 | 
64 | ### Sync local data to remote source
65 | 
66 | When you modify the data in local environment, `sync-to-remote` target uploads the local files stored in `data` to specified data sources such as S3 or NFS directories.
67 | 
68 | ### Show profile of Docker container
69 | 
70 | When you see the status of Docker container, please run `make profile` in host machine.
71 | 
72 | ### Use Jupyter Notebook
73 | 
74 | To launch Jupyter Notebook, please run `make jupyter` in the Docker container. After launch the Jupyter Notebook, you can
75 | access the Jupyter Notebook service in http://localhost:{{ cookiecutter.jupyter_host_port }}.
76 | 
77 | ### Run formatter
78 | When you format project's codes, please run `make format`.
79 | More details of {{ cookiecutter.formatter_type }} in {%- if cookiecutter.formatter_type == 'black' %}https://github.com/psf/black {%- elif cookiecutter.formatter_type == 'autopep8' %} https://github.com/hhatto/autopep8 {%- elif cookiecutter.formatter_type == 'yapf' %} https://github.com/google/yapf {% endif %}
80 | 
81 | 
82 | # Credits
83 | 
84 | This package was created with [Cookiecutter](https://github.com/audreyr/cookiecutter) and the [cookiecutter-docker-science](https://docker-science.github.io/) project template.
85 | 


--------------------------------------------------------------------------------
/{{ cookiecutter.project_slug }}/config/.gitkeep:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/docker-science/cookiecutter-docker-science/494427e0f41a4d4f06ee5d7796eb9356bc149b54/{{ cookiecutter.project_slug }}/config/.gitkeep


--------------------------------------------------------------------------------
/{{ cookiecutter.project_slug }}/data/.gitkeep:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/docker-science/cookiecutter-docker-science/494427e0f41a4d4f06ee5d7796eb9356bc149b54/{{ cookiecutter.project_slug }}/data/.gitkeep


--------------------------------------------------------------------------------
/{{ cookiecutter.project_slug }}/docker/Dockerfile:
--------------------------------------------------------------------------------
 1 | {%- if cookiecutter.use_nvidia_docker == 'yes' -%}
 2 | FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04
 3 | {%- else -%}
 4 | FROM ubuntu:18.04
 5 | {%- endif %}
 6 | 
 7 | RUN apt-get update && apt-get install -y \
 8 |   git \
 9 |   python3.8 \
10 |   python3-pip \
11 |   python3.8-dev
12 | 
13 | RUN pip3 install --upgrade pip
14 | 
15 | COPY ./requirements.txt /requirements.txt
16 | RUN pip install -r /requirements.txt
17 | 
18 | ARG UID
19 | RUN useradd docker -l -u $UID -s /bin/bash -m
20 | USER docker
21 | 
22 | ENV PATH=$PATH:/home/docker/.local/bin
23 | 


--------------------------------------------------------------------------------
/{{ cookiecutter.project_slug }}/docker/Dockerfile.dev:
--------------------------------------------------------------------------------
1 | FROM {{cookiecutter.project_slug}}-image-base
2 | 
3 | COPY ./requirements_dev.txt /requirements_dev.txt
4 | RUN pip install -r /requirements_dev.txt
5 | 
6 | WORKDIR /work
7 | 
8 | ENTRYPOINT ["/bin/bash"]
9 | 


--------------------------------------------------------------------------------
/{{ cookiecutter.project_slug }}/docker/Dockerfile.release:
--------------------------------------------------------------------------------
1 | FROM {{cookiecutter.project_slug}}-image-base
2 | 
3 | COPY . /work
4 | WORKDIR /work
5 | 
6 | ENTRYPOINT ["/bin/bash"]
7 | 


--------------------------------------------------------------------------------
/{{ cookiecutter.project_slug }}/model/.gitkeep:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/docker-science/cookiecutter-docker-science/494427e0f41a4d4f06ee5d7796eb9356bc149b54/{{ cookiecutter.project_slug }}/model/.gitkeep


--------------------------------------------------------------------------------
/{{ cookiecutter.project_slug }}/notebook/.gitkeep:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/docker-science/cookiecutter-docker-science/494427e0f41a4d4f06ee5d7796eb9356bc149b54/{{ cookiecutter.project_slug }}/notebook/.gitkeep


--------------------------------------------------------------------------------
/{{ cookiecutter.project_slug }}/requirements.txt:
--------------------------------------------------------------------------------
1 | awscli
2 | click
3 | {{ cookiecutter.formatter_type }}
4 | 


--------------------------------------------------------------------------------
/{{ cookiecutter.project_slug }}/requirements_dev.txt:
--------------------------------------------------------------------------------
1 | matplotlib
2 | jupyter
3 | notebook
4 | Sphinx
5 | flake8
6 | mypy


--------------------------------------------------------------------------------
/{{ cookiecutter.project_slug }}/scripts/.gitkeep:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/docker-science/cookiecutter-docker-science/494427e0f41a4d4f06ee5d7796eb9356bc149b54/{{ cookiecutter.project_slug }}/scripts/.gitkeep


--------------------------------------------------------------------------------
/{{ cookiecutter.project_slug }}/tests/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/docker-science/cookiecutter-docker-science/494427e0f41a4d4f06ee5d7796eb9356bc149b54/{{ cookiecutter.project_slug }}/tests/__init__.py


--------------------------------------------------------------------------------
/{{ cookiecutter.project_slug }}/tests/test_sample.py:
--------------------------------------------------------------------------------
 1 | import unittest
 2 | 
 3 | 
 4 | class TestSample(unittest.TestCase):
 5 |     def setUp(self):
 6 |         pass
 7 | 
 8 |     def test_add(self):
 9 |         self.assertEqual((3 + 4), 7)
10 | 


--------------------------------------------------------------------------------
/{{ cookiecutter.project_slug }}/{{ cookiecutter.project_slug }}/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/docker-science/cookiecutter-docker-science/494427e0f41a4d4f06ee5d7796eb9356bc149b54/{{ cookiecutter.project_slug }}/{{ cookiecutter.project_slug }}/__init__.py


--------------------------------------------------------------------------------