├── .gitignore
├── CONTRIBUTING.md
├── LICENSE
├── README.md
├── RESOURCE_GUIDE.md
├── cats
├── DATAFLOW_TUTORIAL.md
├── LICENSE
├── README.md
├── nn_demo_part1.ipynb
├── nn_demo_part2.ipynb
├── run_step_2a_query.sh
├── run_step_2b_get_images.sh
├── run_step_3_split_images.sh
├── setup.py
├── step_0_to_0.ipynb
├── step_1_to_3.ipynb
├── step_2a_query.sql
├── step_2b_get_images.py
├── step_3_split_images.py
├── step_4_to_4_part1.ipynb
├── step_4_to_4_part2.ipynb
├── step_5_to_6_part1.ipynb
├── step_5_to_8_part2.ipynb
└── step_8_to_9.ipynb
├── cleanup_notebooks.py
├── imgs
├── step_2b.png
└── step_3.png
├── requirements.txt
├── setup_step_1_cloud_project.sh
└── setup_step_2_install_software.sh
/.gitignore:
--------------------------------------------------------------------------------
1 | *.idea*
2 | *.pyc*
3 | *.egg-info*
4 | *nyc_dataprep*
5 | *~
6 | *DS_Store
7 | *.bmp
8 | *.ipynb_checkpoints*
9 | *.tar.gz
--------------------------------------------------------------------------------
/CONTRIBUTING.md:
--------------------------------------------------------------------------------
1 | # How to Contribute
2 |
3 | While this is meant as a basic tutorial with fixed content, if you would like
4 | to make additions or revisions for the purpose of better teaching data science
5 | on Google Cloud, or if you have fixes for Cloud environment setups, we'd love to
6 | accept your patches and contributions to this project. There are a few small
7 | guidelines you need to follow.
8 |
9 | ## Contributor License Agreement
10 |
11 | Contributions to this project must be accompanied by a Contributor License
12 | Agreement. You (or your employer) retain the copyright to your contribution,
13 | this simply gives us permission to use and redistribute your contributions as
14 | part of the project. Head over to to see
15 | your current agreements on file or to sign a new one.
16 |
17 | You generally only need to submit a CLA once, so if you've already submitted one
18 | (even if it was for a different project), you probably don't need to do it
19 | again.
20 |
21 | ## Code reviews
22 |
23 | All submissions, including submissions by project members, require review. We
24 | use GitHub pull requests for this purpose. Consult
25 | [GitHub Help](https://help.github.com/articles/about-pull-requests/) for more
26 | information on using pull requests.
27 |
28 | ## Notebooks
29 |
30 | For notebook contributions or edits, after making proper changes, please shut
31 | down your notebook editor (e.g. Jupyter, Colab, etc.), and run the python script
32 | in the project's base directory as the FINAL STEP to ensure that outputs and
33 | other editor-specific metadata are removed. This ensures that diffs between
34 | notebooks are clearly marked during code review.
35 |
36 | ```
37 | python cleanup_notebooks.py
38 | ```
39 |
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 |
2 | Apache License
3 | Version 2.0, January 2004
4 | http://www.apache.org/licenses/
5 |
6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
7 |
8 | 1. Definitions.
9 |
10 | "License" shall mean the terms and conditions for use, reproduction,
11 | and distribution as defined by Sections 1 through 9 of this document.
12 |
13 | "Licensor" shall mean the copyright owner or entity authorized by
14 | the copyright owner that is granting the License.
15 |
16 | "Legal Entity" shall mean the union of the acting entity and all
17 | other entities that control, are controlled by, or are under common
18 | control with that entity. For the purposes of this definition,
19 | "control" means (i) the power, direct or indirect, to cause the
20 | direction or management of such entity, whether by contract or
21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the
22 | outstanding shares, or (iii) beneficial ownership of such entity.
23 |
24 | "You" (or "Your") shall mean an individual or Legal Entity
25 | exercising permissions granted by this License.
26 |
27 | "Source" form shall mean the preferred form for making modifications,
28 | including but not limited to software source code, documentation
29 | source, and configuration files.
30 |
31 | "Object" form shall mean any form resulting from mechanical
32 | transformation or translation of a Source form, including but
33 | not limited to compiled object code, generated documentation,
34 | and conversions to other media types.
35 |
36 | "Work" shall mean the work of authorship, whether in Source or
37 | Object form, made available under the License, as indicated by a
38 | copyright notice that is included in or attached to the work
39 | (an example is provided in the Appendix below).
40 |
41 | "Derivative Works" shall mean any work, whether in Source or Object
42 | form, that is based on (or derived from) the Work and for which the
43 | editorial revisions, annotations, elaborations, or other modifications
44 | represent, as a whole, an original work of authorship. For the purposes
45 | of this License, Derivative Works shall not include works that remain
46 | separable from, or merely link (or bind by name) to the interfaces of,
47 | the Work and Derivative Works thereof.
48 |
49 | "Contribution" shall mean any work of authorship, including
50 | the original version of the Work and any modifications or additions
51 | to that Work or Derivative Works thereof, that is intentionally
52 | submitted to Licensor for inclusion in the Work by the copyright owner
53 | or by an individual or Legal Entity authorized to submit on behalf of
54 | the copyright owner. For the purposes of this definition, "submitted"
55 | means any form of electronic, verbal, or written communication sent
56 | to the Licensor or its representatives, including but not limited to
57 | communication on electronic mailing lists, source code control systems,
58 | and issue tracking systems that are managed by, or on behalf of, the
59 | Licensor for the purpose of discussing and improving the Work, but
60 | excluding communication that is conspicuously marked or otherwise
61 | designated in writing by the copyright owner as "Not a Contribution."
62 |
63 | "Contributor" shall mean Licensor and any individual or Legal Entity
64 | on behalf of whom a Contribution has been received by Licensor and
65 | subsequently incorporated within the Work.
66 |
67 | 2. Grant of Copyright License. Subject to the terms and conditions of
68 | this License, each Contributor hereby grants to You a perpetual,
69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable
70 | copyright license to reproduce, prepare Derivative Works of,
71 | publicly display, publicly perform, sublicense, and distribute the
72 | Work and such Derivative Works in Source or Object form.
73 |
74 | 3. Grant of Patent License. Subject to the terms and conditions of
75 | this License, each Contributor hereby grants to You a perpetual,
76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable
77 | (except as stated in this section) patent license to make, have made,
78 | use, offer to sell, sell, import, and otherwise transfer the Work,
79 | where such license applies only to those patent claims licensable
80 | by such Contributor that are necessarily infringed by their
81 | Contribution(s) alone or by combination of their Contribution(s)
82 | with the Work to which such Contribution(s) was submitted. If You
83 | institute patent litigation against any entity (including a
84 | cross-claim or counterclaim in a lawsuit) alleging that the Work
85 | or a Contribution incorporated within the Work constitutes direct
86 | or contributory patent infringement, then any patent licenses
87 | granted to You under this License for that Work shall terminate
88 | as of the date such litigation is filed.
89 |
90 | 4. Redistribution. You may reproduce and distribute copies of the
91 | Work or Derivative Works thereof in any medium, with or without
92 | modifications, and in Source or Object form, provided that You
93 | meet the following conditions:
94 |
95 | (a) You must give any other recipients of the Work or
96 | Derivative Works a copy of this License; and
97 |
98 | (b) You must cause any modified files to carry prominent notices
99 | stating that You changed the files; and
100 |
101 | (c) You must retain, in the Source form of any Derivative Works
102 | that You distribute, all copyright, patent, trademark, and
103 | attribution notices from the Source form of the Work,
104 | excluding those notices that do not pertain to any part of
105 | the Derivative Works; and
106 |
107 | (d) If the Work includes a "NOTICE" text file as part of its
108 | distribution, then any Derivative Works that You distribute must
109 | include a readable copy of the attribution notices contained
110 | within such NOTICE file, excluding those notices that do not
111 | pertain to any part of the Derivative Works, in at least one
112 | of the following places: within a NOTICE text file distributed
113 | as part of the Derivative Works; within the Source form or
114 | documentation, if provided along with the Derivative Works; or,
115 | within a display generated by the Derivative Works, if and
116 | wherever such third-party notices normally appear. The contents
117 | of the NOTICE file are for informational purposes only and
118 | do not modify the License. You may add Your own attribution
119 | notices within Derivative Works that You distribute, alongside
120 | or as an addendum to the NOTICE text from the Work, provided
121 | that such additional attribution notices cannot be construed
122 | as modifying the License.
123 |
124 | You may add Your own copyright statement to Your modifications and
125 | may provide additional or different license terms and conditions
126 | for use, reproduction, or distribution of Your modifications, or
127 | for any such Derivative Works as a whole, provided Your use,
128 | reproduction, and distribution of the Work otherwise complies with
129 | the conditions stated in this License.
130 |
131 | 5. Submission of Contributions. Unless You explicitly state otherwise,
132 | any Contribution intentionally submitted for inclusion in the Work
133 | by You to the Licensor shall be under the terms and conditions of
134 | this License, without any additional terms or conditions.
135 | Notwithstanding the above, nothing herein shall supersede or modify
136 | the terms of any separate license agreement you may have executed
137 | with Licensor regarding such Contributions.
138 |
139 | 6. Trademarks. This License does not grant permission to use the trade
140 | names, trademarks, service marks, or product names of the Licensor,
141 | except as required for reasonable and customary use in describing the
142 | origin of the Work and reproducing the content of the NOTICE file.
143 |
144 | 7. Disclaimer of Warranty. Unless required by applicable law or
145 | agreed to in writing, Licensor provides the Work (and each
146 | Contributor provides its Contributions) on an "AS IS" BASIS,
147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
148 | implied, including, without limitation, any warranties or conditions
149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
150 | PARTICULAR PURPOSE. You are solely responsible for determining the
151 | appropriateness of using or redistributing the Work and assume any
152 | risks associated with Your exercise of permissions under this License.
153 |
154 | 8. Limitation of Liability. In no event and under no legal theory,
155 | whether in tort (including negligence), contract, or otherwise,
156 | unless required by applicable law (such as deliberate and grossly
157 | negligent acts) or agreed to in writing, shall any Contributor be
158 | liable to You for damages, including any direct, indirect, special,
159 | incidental, or consequential damages of any character arising as a
160 | result of this License or out of the use or inability to use the
161 | Work (including but not limited to damages for loss of goodwill,
162 | work stoppage, computer failure or malfunction, or any and all
163 | other commercial damages or losses), even if such Contributor
164 | has been advised of the possibility of such damages.
165 |
166 | 9. Accepting Warranty or Additional Liability. While redistributing
167 | the Work or Derivative Works thereof, You may choose to offer,
168 | and charge a fee for, acceptance of support, warranty, indemnity,
169 | or other liability obligations and/or rights consistent with this
170 | License. However, in accepting such obligations, You may act only
171 | on Your own behalf and on Your sole responsibility, not on behalf
172 | of any other Contributor, and only if You agree to indemnify,
173 | defend, and hold each Contributor harmless for any liability
174 | incurred by, or claims asserted against, such Contributor by reason
175 | of your accepting any such warranty or additional liability.
176 |
177 | END OF TERMS AND CONDITIONS
178 |
179 | APPENDIX: How to apply the Apache License to your work.
180 |
181 | To apply the Apache License to your work, attach the following
182 | boilerplate notice, with the fields enclosed by brackets "[]"
183 | replaced with your own identifying information. (Don't include
184 | the brackets!) The text should be enclosed in the appropriate
185 | comment syntax for the file format. We also recommend that a
186 | file or class name and description of purpose be included on the
187 | same "printed page" as the copyright notice for easier
188 | identification within third-party archives.
189 |
190 | Copyright [yyyy] [name of copyright owner]
191 |
192 | Licensed under the Apache License, Version 2.0 (the "License");
193 | you may not use this file except in compliance with the License.
194 | You may obtain a copy of the License at
195 |
196 | http://www.apache.org/licenses/LICENSE-2.0
197 |
198 | Unless required by applicable law or agreed to in writing, software
199 | distributed under the License is distributed on an "AS IS" BASIS,
200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
201 | See the License for the specific language governing permissions and
202 | limitations under the License.
203 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Step-by-Step Deep Learning Tutorial
2 |
3 | Authors: Cassie Kozyrkov (@kozyrkov) and Brian Foo (@bkungfoo)
4 |
5 | Team: Google Applied AI
6 |
7 | ## Follow along with us!
8 |
9 | Take a look at these **walkthrough slides** with screenshots to guide you along:
10 |
11 | [github.com/kozyrkov/deep-learning-walkthrough](https://github.com/kozyrkov/deep-learning-walkthrough)
12 |
13 | Bonus: slides contain ML hints, summaries, and pitfall alerts.
14 |
15 | ## Create and Setup Cloud Project
16 |
17 | **This tutorial is meant to run fully on the Google Cloud Platform.**
18 |
19 | Starting with your web browser, do the following:
20 |
21 | * Open a browser and sign up for a [google account](https://accounts.google.com).
22 | * Sign into your account.
23 | * [Create a new GCP project](https://cloud.google.com/resource-manager/docs/creating-managing-projects)
24 | and [enable billing](https://cloud.google.com/billing/docs/how-to/modify-project#enable_billing_for_a_project).
25 |
26 | ### Editing Resource Quota
27 |
28 | **Why edit resource quota?** To complete this tutorial, you will need more computing resources than your
29 | Google Cloud Platform account has access to by default. One reason that accounts start out with limits on
30 | resources is that this protects users from being billed unexpectedly for the more expensive options.
31 | For more information, see the [quotas documentation page](https://cloud.google.com/compute/quotas).
32 |
33 | For this project, we will require several types of resources:
34 |
35 | * **Compute Engine VM for Data Science:** We will create a VM that you will log
36 | into and do most of your work and run notebooks. You will have the option of
37 | creating a VM with a GPU choice, or no GPU. A GPU is strongly recommended for
38 | deep network training because it dramatically cuts down the time to completion.
39 | However, the cost is also higher. Please refer to the
40 | [resource guide](RESOURCE_GUIDE.md) for a brief discussion and comparison of
41 | performances and costs.
42 |
43 | * **Cloud resources:** We will be using Dataflow to run distributed
44 | preprocessing jobs. Thus, we need to extend quotas on Cloud resources, such as
45 | CPUs, IP addresses, and total disk space.
46 |
47 | We will be setting quotas for these two types of resources. Note that quotas
48 | only determine the maximum amount of a resource that your project is allowed to
49 | request! It does not mean that your jobs will use this amount necessarily,
50 | but that you are permitted to use up to this amount. The general
51 | guideline is to set higher quotas such that there is no need to readjust them
52 | for compute-intensive tasks.
53 |
54 | **Quota Setup Instructions:**
55 |
56 | To set up the data science VM, we will need to extend the quota for GPUs.
57 | * Select your project from the list on the
58 | [resource quota page](https://console.cloud.google.com/iam-admin/quotas).
59 | * (If this is the first time creating the project, compute engine may still
60 | need to boot up. If the quota page does not have GPU options, click on
61 | "Compute Engine" in the dropdown menu on the top left, and click quota there.
62 | Wait for it to load, and return to the quota page above.)
63 | * If you would like to try out a GPU machine (recommended),
64 | [find a region that has gpu support](https://cloud.google.com/compute/docs/gpus/).
65 | At the time this tutorial was written, valid regions include
66 | us-east1, us-west1, europe-west1, and asia-east1.
67 | * Select your chosen region from the Region dropdown menu.
68 | Then select the following:
69 | * NVIDIA K80 GPUs
70 | * NVIDIA P100 GPUs
71 | * Click "+ edit quotas" at the top of the page. Change the fields above to
72 | the following values:
73 | * NVIDIA K80 GPUs: 1
74 | * NVIDIA P100 GPUs: 1
75 | * Follow the process to submit a request.
76 |
77 | To setup cloud resources for preprocessing jobs, follow a similar request as
78 | above to edit quotas:
79 | * Find a region with
80 | [Dataflow support](https://cloud.google.com/dataflow/docs/concepts/regional-endpoints#supported_regional_endpoints)
81 | At the time this tutorial was written, valid regions include us-central1 and
82 | europe-west1.
83 | * Select this region in the dropdown menu on the
84 | [resource quota page]((https://console.cloud.google.com/iam-admin/quotas)).
85 | * Change the following quotas:
86 | * CPUs: 400
87 | * In-use IP addresses: 100
88 | * Persistent Disk Standard (GB): 65536
89 | * Select region "Global" in the dropdown menu:
90 | * Change the following quotas:
91 | * CPUs (all regions): 400
92 |
93 | After you have completed these steps, you will need to wait until you receive
94 | an email approving of the quota increases. Please note that you may be asked to
95 | provide credit card details to confirm these increases.
96 |
97 | ## Setup Cloud Project Components and API
98 |
99 | *Expected setup time: 5 minutes*
100 |
101 | Click on the ">_" icon at the top right of your web console to open a cloud
102 | shell. Inside the cloud shell, execute the following commands:
103 |
104 | ```
105 | git clone https://github.com/google-aai/sc17.git
106 | cd sc17
107 | ```
108 |
109 | If you happen to have the project files locally, you can also upload locally by
110 | clicking on the 3 vertically arranged dots on the top right of the shell window,
111 | and then click "upload file".
112 |
113 | After you have the proper scripts uploaded, set permissions on the following
114 | script:
115 |
116 | ```
117 | chmod 777 setup_step_1_cloud_project.sh
118 | ```
119 |
120 | Then run the script to create storage, dataset,
121 | and compute VMs for your project (Note: using the "sh" command will fail as it
122 | is missing some necessary syntax in the cloud shell environment.)
123 | ```
124 | ./setup_step_1_cloud_project.sh project-name [gpu-type] [compute-region] [dataflow-region]
125 | ```
126 |
127 | where:
128 | * [project-name] is the ID of the project you created (check the Cloud Dashboard for the ID extension if needed)
129 | * [gpu-type] (optional) is either None, K80, or P100
130 | (default: None)
131 | * [compute-region] (optional) is the region you will create your data science VM
132 | (default: us-east1)
133 | * [dataflow-region] (optional) is where you will run dataflow preprocessing jobs
134 | (default: us-central1)
135 |
136 | If this is your first time setting up the project, you will be prompted during
137 | the course of running the script, such as selecting the number corresponding to
138 | your project. Enter what is needed to allow the script to continue running.
139 |
140 | If the script stops with an error message "ERROR [timestamp]: message" (e.g.
141 | quota limits are too low), use relevant parts of the error message to fix your
142 | project setting if needed, and rerun the script.
143 |
144 | ## Setting up your VM environment
145 |
146 | ### Library installations
147 |
148 | *Expected setup time: 15 minutes*
149 |
150 | From the
151 | [VM instances page](https://console.cloud.google.com/compute/instances),
152 | click the "SSH" text under "Connect" to connect to your compute VM instance.
153 | You may have to click twice if your browser auto-blocks pop-ups.
154 |
155 | In the new window, run git clone to download the project onto your VM, and cd
156 | into it:
157 |
158 | ```
159 | git clone https://github.com/google-aai/sc17.git
160 | cd sc17
161 | ```
162 |
163 | If you happen to have the project files locally, you can also upload locally by
164 | opening your Storage Bucket from the GCP Console menu and dragging your local files
165 | over. Then in your VM window, download them from your storage bucket by running:
166 |
167 | ```
168 | gsutil cp [gs://[bucket-name]/* .]
169 | ```
170 |
171 | Note that tab-complete will work after ```gs://``` if you don't know your bucket name.
172 |
173 | After you have the script files downloaded to your VM, run the following script:
174 |
175 | ```
176 | sh setup_step_2_install_software.sh
177 | ```
178 |
179 | The script should setup opencv dependencies, python, virtual env, and
180 | jupyter. It will also automatically detect the presence of
181 | an NVIDIA GPU and install/link CUDA libraries and tensorflow GPU if necessary.
182 | The script will also prompt you to **provide a password** at some
183 | point. This password is for connecting to jupyter from your web browser. Please
184 | take note of it since you'll be prompted to enter it when you start working in Jupyter.
185 |
186 | To complete and test the setup, reload bashrc to load the newly created virtual
187 | environment:
188 |
189 | ```
190 | . ~/.bashrc
191 | ```
192 |
193 | ## Use Unix Screen to Start a Notebook
194 |
195 | Screen takes a little to get used to, but it will make working on cloud VMs much
196 | more pleasant, especially with a project that needs to run many tasks!
197 |
198 | For those not familiar with the unix screen command, Screen is known as a
199 | "terminal multiplexer", which allows you to run multiple terminal (shell)
200 | instances at the same time in your ssh session. Furthermore, Screen sessions are
201 | NOT tied to your ssh session, which means that if you accidentally log out or
202 | disconnect from your ssh session in the middle of running a long process running
203 | on your VM, you will not lose your work!
204 |
205 | Furthermore, you might want multiple processes running simultaneously and have
206 | an easy way to switch back and forth. A simple example is that you want to leave
207 | your Jupyter notebook open while running a Cloud Dataflow job (which you do not
208 | want abruptly canceled!). Running these in separate terminals is ideal.
209 |
210 | To start screen for the first time, run:
211 |
212 | ```
213 | screen
214 | ```
215 |
216 | and press return. This opens up a screen terminal (defaults to terminal 0).
217 |
218 | Let's create one more Screen terminal (terminal 1) by pressing `Ctrl-a`, and then
219 | `c` (We will write this shorthand as `Ctrl-a c`)
220 |
221 | You can now jump between the two terminals by using `Ctrl-a n`, or access them
222 | directly using `Ctrl-a 0` or `Ctrl-a 1`.
223 |
224 | Go to terminal 0 by typing `Ctrl-a 0`, and then type:
225 |
226 | ```
227 | jupyter notebook
228 | ```
229 |
230 | to start jupyter.
231 |
232 | Finally, detach from both Screen terminals by typing `Ctrl-a d`. If you want to
233 | resume the screen terminals, simply type:
234 |
235 | ```
236 | screen -R
237 | ```
238 |
239 | Fantastic! Now let's do another cool trick: Make sure you are detached from
240 | Screen terminals (type `Ctrl-a d` if necessary), and then exit the machine
241 | by typing:
242 |
243 | ```
244 | exit
245 | ```
246 |
247 | at the command line. You just exited the machine, but the Screen terminals are
248 | still be running, including Jupyter which you started in Screen!
249 |
250 | ## Connecting to Jupyter
251 |
252 | Jupyter is now running inside a Screen terminal even though your ssh session has
253 | ended. Let's try it out through an ssh tunnel (For security reasons, we will
254 | not simply open up a firewall port and show your notebook to the entire world!)
255 |
256 | On your local computer, make sure you have [gcloud sdk installed](https://cloud.google.com/sdk/downloads).
257 | Then run:
258 |
259 | ```
260 | gcloud init
261 | ```
262 |
263 | Follow the instructions and choose your project, and then choose the region
264 | corresponding to where your vm was created. After this has been setup, run:
265 |
266 | ```
267 | gcloud compute config-ssh
268 | ```
269 |
270 | After this runs successfully, you will get this back in your shell:
271 |
272 | ```
273 | You should now be able to use ssh/scp with your instances.
274 | For example, try running:
275 |
276 | $ ssh [instance-name].[zone-name].[project-name]
277 | ```
278 |
279 | Run the suggested command to check that ssh works when connecting to your cloud
280 | VM. Then exit the ssh shell by typing `exit`.
281 |
282 | Now we are ready to connect to Jupyter! Run the same ssh command again, but this
283 | time, add some flags and ports:
284 |
285 | ```
286 | ssh -N -f -L localhost:8888:localhost:5000 [instance-name].[zone-name].[project-name]
287 | ```
288 |
289 | This command basically configures port forwarding, redirecting port 5000 on your
290 | cloud VM to your own computer's port 8888. Now go to your web browser, and type:
291 |
292 | ```
293 | localhost:8888
294 | ```
295 |
296 | If you see a password page for Jupyter, enter your password as prompted.
297 | Once you are in, you can see the notebook view of the directory you
298 | started Jupyter in.
299 |
300 | Before proceeding, please read the [resource guide](RESOURCE_GUIDE.md) to beware
301 | of common pitfalls (such as forgetting to stop your VM when not using it!) and
302 | other ways to save on cost.
303 |
304 | **Hooray! [Let's go detect some cats!](cats/README.md)**
305 |
--------------------------------------------------------------------------------
/RESOURCE_GUIDE.md:
--------------------------------------------------------------------------------
1 | # Resource Guide
2 |
3 | This guide is meant to provide some resource, time, and money tradeoffs such
4 | that you can choose the best configuration for your data science experience!
5 |
6 | ## STOPPING VM INSTANCES
7 |
8 | **Make sure to stop your VM instance when you are not using it!**
9 |
10 | When you start a VM, GCP will charge you an hourly rate (see below). If you are
11 | not actively running tasks on the VM or using the interface, you should stop the
12 | instance to save money. To do so, you can either do it through the
13 | [web ui](https://console.cloud.google.com/compute/instances) by clicking on the
14 | vertical dots at the right of your instance and stopping it, or use the command
15 | line from cloud shell:
16 |
17 | ```
18 | gcloud compute instances stop [vm-instance-name]
19 | ```
20 |
21 | ## GPU or No GPU
22 |
23 | The NVIDIA Tesla K80 is significantly cheaper than the Tesla P100, but not
24 | nearly as powerful. The P100 can speed up deep network training by 10-20x
25 | compared to no GPU. If you plan to spend a lot of time visualizing and debugging
26 | data and less time running massive training jobs, it may be a good idea to start
27 | without a GPU. When you reach a step in the tutorial where you are training and
28 | optimizing deep neural nets is required, a GPU can saves you a lot of time,
29 | and potentially money. To compare costs and performance, consider the
30 | [training task](cats/step_8_to_9.ipynb) used in the `cats`
31 | project:
32 |
33 | * **No GPU:** A 2 CPU VM costs around $0.10 per hour when active. Training on
34 | the full dataset in a notebook (which only runs on 1 CPU) takes about 8 hours.
35 | * **K80**: around $0.55 per hour when active. Training takes about 31 minutes.
36 | * **P100**: around $1.50 per hour when active. Training takes about 25 minutes.
37 |
38 | (It is possible that the default configuration does not optimize for P100, and
39 | further savings can be achieved.)
40 |
41 | ## Dataflow CPUs
42 |
43 | By default, our project is tuned to run tasks that will not incur more than a
44 | few dollars to your account, while ensuring that jobs complete quickly.
45 | If you would like to save money at the cost of extra time, dataflow jobs can be
46 | tuned using flags such as `--num_workers` and `--worker_machine_type`. The:
47 | example below dispatches a job on 20 machines with a worker type that contains
48 | 4 cpus per machine:
49 |
50 | ```
51 | --num_workers 20 \
52 | --worker_machine_type n1-standard-4 \
53 | ```
54 |
55 | If you are running a lightweight task, such as a small sample of cat images,
56 | consider using fewer machines and/or changing the worker machine type to a
57 | simpler worker. This can reduce resources required to boot up and setup
58 | machines, and reduce communication overhead between machines.
--------------------------------------------------------------------------------
/cats/DATAFLOW_TUTORIAL.md:
--------------------------------------------------------------------------------
1 | # Hello Dataflow Image Preprocessing!
2 |
3 | This tutorial walks through the main steps of 2b and 3 in the cats ML example.
4 | It goes over some of the higher level concepts and discusses in some detail how
5 | the dataflow pipelines are built.
6 |
7 | ## Quick Intro to Beam Pipelines
8 |
9 | Cloud Dataflow is a service that runs Apache Beam pipelines on GCP. A beam
10 | pipeline reads entries from a source, and streams data through numerous
11 | processing elements. In python, a processing element is marked by a **unique**
12 | string name (preceded by a "|") and a corresponding transformation process
13 | (preceded by a ">>"). A single pipeline will generally follow this pattern:
14 |
15 | ```
16 | with beam.Pipeline(options=pipeline_options) as p:
17 | (p
18 | | 'step_1_component_name'
19 | >> do_step_1()
20 | | 'step_2_component_name'
21 | >> do_step_2()
22 | ...
23 | | 'step_3_component_name'
24 | >> do_step_n()
25 | )
26 | ```
27 |
28 | where `with beam.Pipeline(...) as p` creates a new pipeline, and each `do_step`
29 | corresponds to an Apache beam processing component.
30 |
31 | When exiting from the `with beam.Pipeline...` block, the pipeline will execute.
32 | Before then, the components that are merely being defined within the block as
33 | part of the pipeline architecture. Essentially, pipelines are *lazy*. They wait
34 | for you to write code for all of the processing steps, and then they execute.
35 | We will show some real examples of this behavior below in step 2b and step 3.
36 |
37 | ## Step 2b: Preprocessing images
38 |
39 | Preprocessing is an important step for both traditional and deep ML.
40 |
41 | In traditional ML, in order to do meaningful feature extraction where resulting
42 | feature vector lengths are equal, images often have to be reshaped and resized
43 | to a specific size. For example, if you wanted to create a feature out of the
44 | mean values of pixels over 8x8 grids along each image, different image sizes
45 | would lead to different feature lengths.
46 |
47 | In deep ML on images, convolutional neural networks (CNNs) are often used as
48 | they can learn both local and global scale model layers, which can then be
49 | aggregated to make an accurate prediction of the image content. CNNs also
50 | require the same image sizes, since they are architected to learn models whose
51 | inputs are of a fixed dimension.
52 |
53 | Step 2b performs multiple steps to ensure that our images
54 | are in a centralized location (Cloud Storage), and that they have been resized
55 | in a manner that is satisfactory for our model building.
56 |
57 | ### Understanding the Beam Pipeline in Step 2b
58 |
59 | In the `step_2b_get_images.py` file, you see the following pipeline:
60 |
61 | ```
62 | bq_source = bigquery.BigQuerySource(query=query)
63 |
64 | with beam.Pipeline(options=pipeline_options) as p:
65 | _ = (p
66 | | 'read_rows_from_cat_info_table'
67 | >> beam.io.Read(bq_source)
68 | | 'fetch_images_from_urls'
69 | >> beam.Map(fetch_image_from_url)
70 | | 'filter_bad_or_absent_images'
71 | >> beam.Filter(filter_bad_or_missing_image)
72 | | 'resize_and_pad_images'
73 | >> beam.Map(resize_and_pad,
74 | output_image_dim=known_args.output_image_dim)
75 | | 'write_images_to_storage'
76 | >> beam.Map(write_processed_image,
77 | output_dir=output_dir)
78 | )
79 | ```
80 |
81 | We will discuss each step below.
82 |
83 | #### 'read_rows_from_cat_info_table'
84 | The **'read_rows_from_cat_info_table'** step reads rows from a source, in this
85 | case, a bigquery source. The bigquery source is defined by a sql query. In the
86 | default setup for the tutorial, the query passed into the BigQuerySource is:
87 |
88 | ```
89 | SELECT ROW_NUMBER() OVER() as index,
90 | original_url,
91 | label,
92 | randnum
93 | FROM [dataset.catinfo]
94 | ```
95 |
96 | The output of reading this source is a collection of dictionary elements, where
97 | each row read from the query is converted to a dictionary, with each key
98 | corresponding to a column name, and each value corresponding to the value in
99 | that column. A row from the bigquery source would look something like this:
100 |
101 | ```
102 | {
103 | 'index': 1234,
104 | 'original_url': http://path.to.picture.com/cat.jpg,
105 | 'label': 1,
106 | 'randnum': 0.2349
107 | }
108 | ```
109 |
110 | ####'fetch_images_from_urls'
111 |
112 | This step is a known as a **transformation** step and follows the syntax:
113 |
114 | ```
115 | beam.Map(some_fn())
116 | ```
117 |
118 | where `some_fn()` is a function that takes, as input, an element from the
119 | pipeline, and returns, as output, an element to transmit downstream to the next
120 | processor. In this step, `some_fn()` is given by
121 |
122 | ```
123 | def fetch_image_from_url(row):
124 | # return a dict with the url field replaced with actual image data
125 | ```
126 |
127 | The function parses out the 'original_url' field from the input row, downloads
128 | the actual image from the url, and decodes it into a raw 3d numpy tensor (width
129 | x height x 3 colors). The function then returns all of the other entries, but
130 | replacing 'original_url' with the entry {'img': [raw_img_tensor]}. If the image
131 | does not exist, it will return nothing, represented by a python "None" object.
132 |
133 | ####'filter_bad_or_absent_images`
134 |
135 | Obviously, we don't want to process any missing images ("None" objects) from the
136 | previous step. This step is known as a *filter* step and follows the syntax:
137 |
138 | ```
139 | beam.Filter(boolean_fn())
140 | ```
141 |
142 | where `boolean_fn()` takes as input an element from the pipeline, and returns
143 | a True (if the element should pass on to the next step) or False (if the element
144 | should be removed) value. In this step, the filter function is:
145 |
146 | ```
147 | def filter_bad_or_missing_image(img_and_metadata):
148 | # return False if bad/missing image, True otherwise
149 | ```
150 |
151 | `filter_bad_or_missing_image` checks for two things:
152 | * If the datum is a "None" object, remove it from the pipeline.
153 | * If the image in the datum meets certain specifications that match the "missing
154 | image" on Flickr, remove it from the pipeline.
155 | * Otherwise, pass the datum on to the next step.
156 |
157 | ####'resize_and_pad_images'
158 |
159 | This is the most involved step in the tutorial. In order to run our machine
160 | learning algorithms to run properly in subsequent steps, the output image has to
161 | be a square image with 128 pixels per side. If the input image happens to be
162 | wider than it is tall, this step will shrink the image proportionally such that
163 | it is at most 128 pixels wide. Then, it will pad the top
164 | and bottom of the image equally with "zeros", or black pixels, until the result
165 | is a perfect 128x128 square. Likewise, if the image is taller than it is wide,
166 | it will perform the same operation but pad the left and right with zeros.
167 |
168 | This is also a transform step, but notice that the beam.Map() takes in two
169 | arguments, i.e.:
170 |
171 | ```
172 | beam.Map(resize_and_pad,
173 | output_image_dim=known_args.output_image_dim)
174 | ```
175 |
176 | beam.Map() can actually take an indefinite number of arguments, but the first
177 | element must always be a function. Every additional element is known as a *side
178 | input* and gives the pipeline writer the flexibility to pass parameters from
179 | outside the pipeline into the function. This becomes clear when we inspect the
180 | `resize_and_pad()` function, as we see that the second argument in beam.Map(),
181 | i.e. `output_image_dim`, is also an argument in resize_and_pad().
182 |
183 | ```
184 | def resize_and_pad(img_and_metadata, output_image_dim=128):
185 | # returns dict with resized image
186 | # default size is 128 if no side input provided
187 | ```
188 |
189 | ####'write_images_to_storage'
190 |
191 |
192 | The final step is also a *transform* step which writes the output image to a
193 | custom-named file on Cloud Storage. The name follows the format:
194 |
195 | ```
196 | [index]_[randnum_times_1000]_[label].png
197 | ```
198 |
199 | where index is prepended with 0s such that it is always at least 6 digits long,
200 | and randnum_times_1000 is an integer between 0 and 999 obtained by multiplying
201 | the random number by 1000 (and stripping the fraction if necessary).
202 |
203 | There are a couple interesting features to note about this function. First, the
204 | function used here also has a side input `output_dir`:
205 |
206 | ```
207 | def write_processed_image(img_and_metadata, output_dir)
208 | ```
209 |
210 | More importantly, because this is the last transformation in the pipeline, there
211 | is no need for this function to return an output, as there are no further steps
212 | to ingest the output data. Instead, the function takes the processed image and
213 | calls the beam.FileSystems API to write the image data into cloud storage.
214 |
215 | For additional reading, see
216 | [detailed documentation for writing beam pipelines here.](https://beam.apache.org/documentation/programming-guide/)
217 |
218 | ### Running Step 2b
219 |
220 | A sample dataflow run command can be found in `run_step_2b_get_images.sh`:
221 |
222 | ```
223 | python -m step_2b_get_images \
224 | --project $PROJECT \
225 | --runner DataflowRunner \
226 | --staging_location gs://$BUCKET/$IMAGE_DIR/staging \
227 | --temp_location gs://$BUCKET/$IMAGE_DIR/temp \
228 | --num_workers 50 \
229 | --worker_machine_type n1-standard-4 \
230 | --setup_file ./setup.py \
231 | --region $DATAFLOW_REGION \
232 | --dataset dataset \
233 | --table $TABLE \
234 | --storage-bucket $BUCKET \
235 | --output-dir $IMAGE_DIR/all_images \
236 | --output-image-dim 128 \
237 | --cloud
238 | ```
239 |
240 | Note that each argument is either pipeline-runner (in this case, DataflowRunner)
241 | specific, or a custom argument that we parse out in `step_2b_get_images.py`. We
242 | have organized the command such that pipeline-runner specific arguments come
243 | first, followed by project-specific arguments
244 |
245 | **Runner-specific arguments**
246 |
247 | These arguments are implicitly parsed by the runner that is specified when
248 | running the job, i.e. DataflowRunner.
249 |
250 | * **--project:** required for read/write permissions and dataflow billing
251 | * **--runner:** for GCP, this is always DataflowRunner
252 | * **--staging_location:** locations for uploading build files to transfer to Cloud
253 | Dataflow workers
254 | * **--temp_location:** location for writing temporary files during pipeline
255 | processing. Often used to store the end output of a pipeline before moving data
256 | to appropriate destination location.
257 | * **--num_workers:** number of Cloud Dataflow VMs/workers to spin up
258 | * **--worker_machine_type:**
259 | [Cloud compute engine VM types](https://cloud.google.com/compute/docs/machine-types)
260 | with different amounts of memory and cpus
261 | * **--setup_file:** a python file that is run by each worker to setup its
262 | environment and libraries. For instance, our project's `setup.py` file requires
263 | workers to install opencv2 libraries and python packages.
264 | * **--region:**
265 | [the region that dataflow will run on](https://cloud.google.com/dataflow/docs/concepts/regional-endpoints#supported_regional_endpoints)
266 |
267 | **Step 2b specific arguments**
268 |
269 | These arguments are parsed out in the `run()` function at the bottom of the
270 | `step_2b_get_images.py` file using the command line argument parser library
271 | `argparse`.
272 |
273 | * **--dataset:** the dataset where the catinfo table is located
274 | * **--table:** "catinfo"
275 | * **--storage-bucket:** cloud storage bucket name to write output images to
276 | * **--output-dir:** the directory to write output images to in the bucket
277 | * **--output-image-dim:** width/height of the output images
278 | * **--cloud:** Currently, a required argument that tells us to write everything
279 | to cloud. The only case for removing this is for experts to debugging python
280 | code locally using a small sample with "DirectRunner". (In the future, we will
281 | provide local run scripts and tests for debugging.)
282 |
283 | ### Visualizing Step 2b
284 |
285 | After you run step 2, you can go to the
286 | [Dataflow Console](https://console.cloud.google.com/dataflow)
287 | and check the progress of your graph. You should see all the steps mentioned
288 | above organized as a linear pipeline like this:
289 |
290 |
291 |
292 |
378 |
379 |