├── Dockerfile ├── LICENSE ├── README.md ├── build.txt ├── jupyter_notebook_config.py ├── notebooks ├── 1_hello_tensorflow.ipynb ├── 2_getting_started.ipynb ├── 3_mnist_from_scratch.ipynb ├── BUILD ├── LICENSE └── tf.contrib.learn Quickstart.ipynb └── simple_console.py /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM ubuntu:16.04 2 | 3 | LABEL mantainer="Eloy Lopez " 4 | 5 | RUN apt-get update && apt-get install -y --no-install-recommends \ 6 | build-essential \ 7 | curl \ 8 | libfreetype6-dev \ 9 | libpng12-dev \ 10 | libzmq3-dev \ 11 | pkg-config \ 12 | python \ 13 | python-dev \ 14 | python-pip \ 15 | python-setuptools \ 16 | python-scipy \ 17 | rsync \ 18 | software-properties-common \ 19 | unzip \ 20 | git \ 21 | && \ 22 | apt-get clean && \ 23 | rm -rf /var/lib/apt/lists/* 24 | 25 | RUN python -m pip install --upgrade pip && \ 26 | pip --no-cache-dir install \ 27 | ipykernel \ 28 | jupyterlab \ 29 | matplotlib \ 30 | numpy \ 31 | sklearn \ 32 | pandas \ 33 | && \ 34 | python -m ipykernel.kernelspec 35 | 36 | ARG WHL_URL=http://ci.tensorflow.org/view/Nightly/job/nightly-pi/lastSuccessfulBuild/artifact/output-artifacts/ 37 | ARG WHL_FILE=tensorflow-1.8.0-cp27-none-linux_armv7l.whl 38 | 39 | RUN pip --no-cache-dir install ${WHL_URL}${WHL_FILE} && \ 40 | rm -f ${WHL_FILE} 41 | 42 | COPY jupyter_notebook_config.py /root/.jupyter/ 43 | 44 | # Copy sample notebooks. 45 | COPY notebooks /notebooks 46 | 47 | # TensorBoard & Jupyter 48 | EXPOSE 6006 8888 49 | 50 | WORKDIR "/notebooks" 51 | 52 | CMD jupyter lab --ip=* --no-browser --allow-root -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 DeftWork 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # DEPRECATED - BE AWARE 2 | 3 | PLEASE NOTE THAT THIS REPOSITORY IS DEPRECATED, WON'T BE LONGER DEVELOPED,IT HAS BEEN REPLACED BY THIS NEW SET OF REPOSITORIES: 4 | 5 | - [Data Science Docker Stack](https://goo.gl/qvx7Vv) 6 | 7 | ## Supercharged Machine Learning ToolBox for ARM 8 | 9 | A [Docker](http://docker.com) image for ARM devices with [Tensorflow 1.8.0](https://www.tensorflow.org/) an open source software library for numerical computation using data flow graphs that will let you play and learn distinct Machine Learning techniques over [JupyterLab](https://github.com/jupyterlab/jupyterlab) an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text. Computational Narratives as the Engine of Collaborative Data Science. All this under Python 2.7 language. 10 | There is very similar image based on Python 3.4 instead of 2.7 [elswork/rpi-tensorflow-py3](https://hub.docker.com/r/elswork/rpi-tensorflow-py3/). 11 | 12 | > Be aware! You should read carefully the usage documentation of every tool! 13 | 14 | ## Latest Changes 15 | 16 | ### 1.8.0 17 | 18 | - Upgraded from Tensorflow 1.7.0 to 1.8.0 19 | 20 | ### 1.7.0 21 | 22 | - Upgraded from Tensorflow 1.7.0rc1 to 1.7.0 23 | - Implementation of the Keras API 24 | 25 | ### 1.7.0 rc1 26 | 27 | - Upgraded from Tensorflow 1.6.0 to 1.7.0rc1 28 | 29 | ### 1.6.0 30 | 31 | - Upgraded from Tensorflow 1.6.0 rc1 to 1.6.0 32 | 33 | ### 1.6.0 rc1 34 | 35 | - Upgraded from Tensorflow 1.5.0 to 1.6.0rc1 36 | - Replace with new latest binary 37 | - Replaced base image with ubuntu:16.04 the same as official tensorflow container 38 | - Change jupyter notebook for jupyterlab 39 | 40 | ### 1.5.0 41 | 42 | - Upgraded from Tensorflow 1.5.0 rc1 to 1.5.0 43 | - Replace with new latest binary 44 | 45 | ### 1.5.0 rc1 46 | 47 | - Upgraded from Tensorflow 1.4.0 to 1.5.0rc1 48 | - Replace with new latest binary 49 | 50 | ### 1.4.0 51 | 52 | - Upgraded from Tensorflow 1.3.0 to 1.4.0 53 | - Replaced Tensorflow local binary to [tensorflow-1.4.0-cp27-none-any.whl](http://ci.tensorflow.org/view/Nightly/job/nightly-pi/lastSuccessfulBuild/artifact/output-artifacts/tensorflow-1.4.0-cp27-none-any.whl) 54 | 55 | ### 1.3.0 56 | 57 | - Upgraded from Tensorflow 1.2.1 to 1.3.0 58 | 59 | ### 1.2.1 60 | 61 | - Replaced base image by a [Docker Official Image](https://github.com/docker-library/official-images) 62 | - Upgraded from Tensorflow 1.1.0 to 1.2.1 63 | 64 | ## Details 65 | 66 | - [GitHub](https://github.com/DeftWork/rpi-tensorflow) 67 | - [Docker Hub](https://hub.docker.com/r/elswork/rpi-tensorflow/) 68 | - [Deft.Work my personal blog](http://deft.work/tensorflow_for_raspberry) 69 | 70 | ## Thanks to 71 | 72 | - [Romilly Cocking for the idea](https://github.com/romilly/rpi-docker-tensorflow) 73 | - Pi tensorflow whl file that i builded thanks to [Sam Abrahm's Step-By-Step Guide for build Tensorflow for Raspberry Pi](https://github.com/samjabrahams/tensorflow-on-raspberry-pi/blob/master/GUIDE.md) 74 | 75 | ## Build Instructions 76 | 77 | Build for arm32v7 architecture 78 | 79 | ```sh 80 | docker build -t elswork/rpi-tensorflow:latest . 81 | ``` 82 | 83 | Build for amd64 architecture 84 | 85 | ```sh 86 | docker build -t elswork/rpi-tensorflow:latest \ 87 | --build-arg WHL_URL=https://storage.googleapis.com/tensorflow/linux/cpu/ \ 88 | --build-arg WHL_FILE=tensorflow-1.8.0-cp27-none-linux_x86_64.whl . 89 | ``` 90 | 91 | ## My Real Usage Example 92 | 93 | In order everyone could take full advantages of the usage of this docker container, I'll describe my own real usage setup. 94 | For amd64 architecture replace latest by amd64 tag. 95 | 96 | ```sh 97 | docker run -d -p 8888:8888 elswork/rpi-tensorflow:latest 98 | ``` 99 | 100 | A more complex sample: 101 | 102 | ```sh 103 | docker run -d -p 8888:8888 -p 0.0.0.0:6006:6006 \ 104 | --restart=unless-stopped elswork/rpi-tensorflow:latest 105 | ``` 106 | 107 | Point your browser to `http://localhost:8888` 108 | 109 | First time you open it, you should provide a Token to log on you cand find it with this command: 110 | 111 | ```sh 112 | docker logs container_name 113 | ``` 114 | 115 | With the second example you can run TensorBoard executing this command in the container: 116 | 117 | ```sh 118 | tensorboard --logdir=path/to/log-directory --host=0.0.0.0 119 | ``` 120 | 121 | And pointing your browser to `http://localhost:6006` 122 | -------------------------------------------------------------------------------- /build.txt: -------------------------------------------------------------------------------- 1 | Sending build context to Docker daemon 181.3 MB 2 | Step 1/12 : FROM ubuntu:16.04 3 | ---> e6bce28de62e 4 | Step 2/12 : LABEL mantainer "Eloy Lopez " 5 | ---> Using cache 6 | ---> 03abb53cea31 7 | Step 3/12 : RUN apt-get update && apt-get install -y --no-install-recommends build-essential curl libfreetype6-dev libpng12-dev libzmq3-dev pkg-config python python-dev python-pip python-setuptools python-scipy rsync software-properties-common unzip git && apt-get clean && rm -rf /var/lib/apt/lists/* 8 | ---> Using cache 9 | ---> 97eb94af6ae9 10 | Step 4/12 : RUN python -m pip install --upgrade pip && pip --no-cache-dir install ipykernel jupyterlab matplotlib numpy sklearn pandas && python -m ipykernel.kernelspec 11 | ---> Using cache 12 | ---> 8c342676ffc9 13 | Step 5/12 : ARG WHL_URL=http://ci.tensorflow.org/view/Nightly/job/nightly-pi/lastSuccessfulBuild/artifact/output-artifacts/ 14 | ---> Using cache 15 | ---> 3683a1454892 16 | Step 6/12 : ARG WHL_FILE=tensorflow-1.8.0-cp27-none-linux_armv7l.whl 17 | ---> Running in 41c6a397e272 18 | ---> d097d968a9c1 19 | Removing intermediate container 41c6a397e272 20 | Step 7/12 : RUN pip --no-cache-dir install ${WHL_URL}${WHL_FILE} && rm -f ${WHL_FILE} 21 | ---> Running in 8d6d0befbf11 22 | Collecting tensorflow==1.8.0 from http://ci.tensorflow.org/view/Nightly/job/nightly-pi/lastSuccessfulBuild/artifact/output-artifacts/tensorflow-1.8.0-cp27-none-linux_armv7l.whl 23 | Downloading http://ci.tensorflow.org/view/Nightly/job/nightly-pi/lastSuccessfulBuild/artifact/output-artifacts/tensorflow-1.8.0-cp27-none-linux_armv7l.whl (67.0MB) 24 | Collecting mock>=2.0.0 (from tensorflow==1.8.0) 25 | Downloading https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl (56kB) 26 | Collecting backports.weakref>=1.0rc1 (from tensorflow==1.8.0) 27 | Downloading https://files.pythonhosted.org/packages/88/ec/f598b633c3d5ffe267aaada57d961c94fdfa183c5c3ebda2b6d151943db6/backports.weakref-1.0.post1-py2.py3-none-any.whl 28 | Collecting astor>=0.6.0 (from tensorflow==1.8.0) 29 | Downloading https://files.pythonhosted.org/packages/b2/91/cc9805f1ff7b49f620136b3a7ca26f6a1be2ed424606804b0fbcf499f712/astor-0.6.2-py2.py3-none-any.whl 30 | Collecting wheel (from tensorflow==1.8.0) 31 | Downloading https://files.pythonhosted.org/packages/81/30/e935244ca6165187ae8be876b6316ae201b71485538ffac1d718843025a9/wheel-0.31.1-py2.py3-none-any.whl (41kB) 32 | Collecting protobuf>=3.4.0 (from tensorflow==1.8.0) 33 | Downloading https://files.pythonhosted.org/packages/bf/dc/42d35afffe88ce301c90457eb60d145dd64144f283ef4bf927c1187ef114/protobuf-3.5.2.post1-py2.py3-none-any.whl (388kB) 34 | Collecting numpy>=1.13.3 (from tensorflow==1.8.0) 35 | Downloading https://files.pythonhosted.org/packages/b0/2b/497c2bb7c660b2606d4a96e2035e92554429e139c6c71cdff67af66b58d2/numpy-1.14.3.zip (4.9MB) 36 | Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow==1.8.0) (1.11.0) 37 | Collecting grpcio>=1.8.6 (from tensorflow==1.8.0) 38 | Downloading https://files.pythonhosted.org/packages/a9/20/16514bfadc37f2c5b286af4e641549df36926f85fbe247465d2455a83de4/grpcio-1.12.0-cp27-cp27mu-linux_armv7l.whl (8.0MB) 39 | Collecting tensorboard<1.9.0,>=1.8.0 (from tensorflow==1.8.0) 40 | Downloading https://files.pythonhosted.org/packages/4d/1e/3bfb48ff165e331c0c5fdca5de79d497fa5d71f3cb2eee2733ff22e898df/tensorboard-1.8.0-py2-none-any.whl (3.1MB) 41 | Collecting absl-py>=0.1.6 (from tensorflow==1.8.0) 42 | Downloading https://files.pythonhosted.org/packages/57/8d/6664518f9b6ced0aa41cf50b989740909261d4c212557400c48e5cda0804/absl-py-0.2.2.tar.gz (82kB) 43 | Requirement already satisfied: enum34>=1.1.6 in /usr/local/lib/python2.7/dist-packages (from tensorflow==1.8.0) (1.1.6) 44 | Collecting gast>=0.2.0 (from tensorflow==1.8.0) 45 | Downloading https://files.pythonhosted.org/packages/5c/78/ff794fcae2ce8aa6323e789d1f8b3b7765f601e7702726f430e814822b96/gast-0.2.0.tar.gz 46 | Collecting termcolor>=1.1.0 (from tensorflow==1.8.0) 47 | Downloading https://files.pythonhosted.org/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz 48 | Collecting funcsigs>=1; python_version < "3.3" (from mock>=2.0.0->tensorflow==1.8.0) 49 | Downloading https://files.pythonhosted.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl 50 | Collecting pbr>=0.11 (from mock>=2.0.0->tensorflow==1.8.0) 51 | Downloading https://files.pythonhosted.org/packages/2d/9d/7bfab757977067556c7ca5fe437f28e8b8843c95564fca504de79df63b25/pbr-4.0.3-py2.py3-none-any.whl (98kB) 52 | Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages (from protobuf>=3.4.0->tensorflow==1.8.0) (20.7.0) 53 | Requirement already satisfied: futures>=2.2.0 in /usr/local/lib/python2.7/dist-packages (from grpcio>=1.8.6->tensorflow==1.8.0) (3.2.0) 54 | Collecting bleach==1.5.0 (from tensorboard<1.9.0,>=1.8.0->tensorflow==1.8.0) 55 | Downloading https://files.pythonhosted.org/packages/33/70/86c5fec937ea4964184d4d6c4f0b9551564f821e1c3575907639036d9b90/bleach-1.5.0-py2.py3-none-any.whl 56 | Collecting markdown>=2.6.8 (from tensorboard<1.9.0,>=1.8.0->tensorflow==1.8.0) 57 | Downloading https://files.pythonhosted.org/packages/6d/7d/488b90f470b96531a3f5788cf12a93332f543dbab13c423a5e7ce96a0493/Markdown-2.6.11-py2.py3-none-any.whl (78kB) 58 | Collecting html5lib==0.9999999 (from tensorboard<1.9.0,>=1.8.0->tensorflow==1.8.0) 59 | Downloading https://files.pythonhosted.org/packages/ae/ae/bcb60402c60932b32dfaf19bb53870b29eda2cd17551ba5639219fb5ebf9/html5lib-0.9999999.tar.gz (889kB) 60 | Collecting werkzeug>=0.11.10 (from tensorboard<1.9.0,>=1.8.0->tensorflow==1.8.0) 61 | Downloading https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl (322kB) 62 | Installing collected packages: funcsigs, pbr, mock, backports.weakref, astor, wheel, protobuf, numpy, grpcio, html5lib, bleach, markdown, werkzeug, tensorboard, absl-py, gast, termcolor, tensorflow 63 | Found existing installation: numpy 1.11.0 64 | Uninstalling numpy-1.11.0: 65 | Successfully uninstalled numpy-1.11.0 66 | Running setup.py install for numpy: started 67 | Running setup.py install for numpy: still running... 68 | Running setup.py install for numpy: still running... 69 | Running setup.py install for numpy: still running... 70 | Running setup.py install for numpy: still running... 71 | Running setup.py install for numpy: still running... 72 | Running setup.py install for numpy: still running... 73 | Running setup.py install for numpy: still running... 74 | Running setup.py install for numpy: still running... 75 | Running setup.py install for numpy: finished with status 'done' 76 | Found existing installation: html5lib 1.0.1 77 | Uninstalling html5lib-1.0.1: 78 | Successfully uninstalled html5lib-1.0.1 79 | Running setup.py install for html5lib: started 80 | Running setup.py install for html5lib: finished with status 'done' 81 | Found existing installation: bleach 2.1.3 82 | Uninstalling bleach-2.1.3: 83 | Successfully uninstalled bleach-2.1.3 84 | Running setup.py install for absl-py: started 85 | Running setup.py install for absl-py: finished with status 'done' 86 | Running setup.py install for gast: started 87 | Running setup.py install for gast: finished with status 'done' 88 | Running setup.py install for termcolor: started 89 | Running setup.py install for termcolor: finished with status 'done' 90 | Successfully installed absl-py-0.2.2 astor-0.6.2 backports.weakref-1.0.post1 bleach-1.5.0 funcsigs-1.0.2 gast-0.2.0 grpcio-1.12.0 html5lib-0.9999999 markdown-2.6.11 mock-2.0.0 numpy-1.14.3 pbr-4.0.3 protobuf-3.5.2.post1 tensorboard-1.8.0 tensorflow-1.8.0 termcolor-1.1.0 werkzeug-0.14.1 wheel-0.31.1 91 | ---> 02b75f54490b 92 | Removing intermediate container 8d6d0befbf11 93 | Step 8/12 : COPY jupyter_notebook_config.py /root/.jupyter/ 94 | ---> 47efcfe4481f 95 | Removing intermediate container fd76e8f05047 96 | Step 9/12 : COPY notebooks /notebooks 97 | ---> d44636c1b26e 98 | Removing intermediate container c07a95449224 99 | Step 10/12 : EXPOSE 6006 8888 100 | ---> Running in 518f07b3604d 101 | ---> 921ed131649a 102 | Removing intermediate container 518f07b3604d 103 | Step 11/12 : WORKDIR "/notebooks" 104 | ---> 2834c535492a 105 | Removing intermediate container eeaf4b0c31e3 106 | Step 12/12 : CMD jupyter lab --ip=* --no-browser --allow-root 107 | ---> Running in 07b8aa4d6279 108 | ---> 73eeff7267df 109 | Removing intermediate container 07b8aa4d6279 110 | Successfully built 73eeff7267df 111 | -------------------------------------------------------------------------------- /jupyter_notebook_config.py: -------------------------------------------------------------------------------- 1 | # Copyright 2015 The TensorFlow Authors. All Rights Reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | # ============================================================================== 15 | import os 16 | from IPython.lib import passwd 17 | 18 | c.NotebookApp.ip = '*' 19 | c.NotebookApp.port = int(os.getenv('PORT', 8888)) 20 | c.NotebookApp.open_browser = False 21 | c.MultiKernelManager.default_kernel_name = 'python2' 22 | 23 | # sets a password if PASSWORD is set in the environment 24 | if 'PASSWORD' in os.environ: 25 | password = os.environ['PASSWORD'] 26 | if password: 27 | c.NotebookApp.password = passwd(password) 28 | else: 29 | c.NotebookApp.password = '' 30 | c.NotebookApp.token = '' 31 | del os.environ['PASSWORD'] 32 | -------------------------------------------------------------------------------- /notebooks/1_hello_tensorflow.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "version": "0.3.2", 7 | "views": {}, 8 | "default_view": {}, 9 | "name": "Untitled", 10 | "provenance": [] 11 | } 12 | }, 13 | "cells": [ 14 | { 15 | "cell_type": "markdown", 16 | "metadata": { 17 | "id": "a3bskVXPvchm", 18 | "colab_type": "text" 19 | }, 20 | "source": [ 21 | "# Hello, TensorFlow\n", 22 | "## A beginner-level, getting started, basic introduction to TensorFlow" 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": { 28 | "id": "Rb5rSpcZvYbX", 29 | "colab_type": "text" 30 | }, 31 | "source": [ 32 | "TensorFlow is a general-purpose system for graph-based computation. A typical use is machine learning. In this notebook, we'll introduce the basic concepts of TensorFlow using some simple examples.\n", 33 | "\n", 34 | "TensorFlow gets its name from [tensors](https://en.wikipedia.org/wiki/Tensor), which are arrays of arbitrary dimensionality. A vector is a 1-d array and is known as a 1st-order tensor. A matrix is a 2-d array and a 2nd-order tensor. The \"flow\" part of the name refers to computation flowing through a graph. Training and inference in a neural network, for example, involves the propagation of matrix computations through many nodes in a computational graph.\n", 35 | "\n", 36 | "When you think of doing things in TensorFlow, you might want to think of creating tensors (like matrices), adding operations (that output other tensors), and then executing the computation (running the computational graph). In particular, it's important to realize that when you add an operation on tensors, it doesn't execute immediately. Rather, TensorFlow waits for you to define all the operations you want to perform. Then, TensorFlow optimizes the computation graph, deciding how to execute the computation, before generating the data. Because of this, a tensor in TensorFlow isn't so much holding the data as a placeholder for holding the data, waiting for the data to arrive when a computation is executed." 37 | ] 38 | }, 39 | { 40 | "cell_type": "markdown", 41 | "metadata": { 42 | "id": "E8FhiMivhcYB", 43 | "colab_type": "text" 44 | }, 45 | "source": [ 46 | "## Adding two vectors in TensorFlow\n", 47 | "\n", 48 | "Let's start with something that should be simple. Let's add two length four vectors (two 1st-order tensors):\n", 49 | "\n", 50 | "$\\begin{bmatrix} 1. & 1. & 1. & 1.\\end{bmatrix} + \\begin{bmatrix} 2. & 2. & 2. & 2.\\end{bmatrix} = \\begin{bmatrix} 3. & 3. & 3. & 3.\\end{bmatrix}$" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "metadata": { 56 | "id": "2iv3XQ6k3eF1", 57 | "colab_type": "code", 58 | "colab": { 59 | "autoexec": { 60 | "startup": false, 61 | "wait_interval": 0 62 | }, 63 | "output_extras": [ 64 | { 65 | "item_id": 1 66 | } 67 | ] 68 | }, 69 | "cellView": "both", 70 | "executionInfo": { 71 | "elapsed": 131, 72 | "status": "ok", 73 | "timestamp": 1446243605678, 74 | "user": { 75 | "color": "#1FA15D", 76 | "displayName": "Michael Piatek", 77 | "isAnonymous": false, 78 | "isMe": true, 79 | "permissionId": "00327059602783983041", 80 | "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", 81 | "sessionId": "7391995727249e65", 82 | "userId": "106975671469698476657" 83 | }, 84 | "user_tz": 420 85 | }, 86 | "outputId": "e21e1144-736a-4b1f-df78-a9ceab9d4c61" 87 | }, 88 | "source": [ 89 | "import tensorflow as tf\n", 90 | "\n", 91 | "with tf.Session():\n", 92 | " input1 = tf.constant([1.0, 1.0, 1.0, 1.0])\n", 93 | " input2 = tf.constant([2.0, 2.0, 2.0, 2.0])\n", 94 | " output = tf.add(input1, input2)\n", 95 | " result = output.eval()\n", 96 | " print result" 97 | ], 98 | "outputs": [ 99 | { 100 | "output_type": "stream", 101 | "text": [ 102 | "[ 3. 3. 3. 3.]\n" 103 | ], 104 | "name": "stdout" 105 | } 106 | ], 107 | "execution_count": 0 108 | }, 109 | { 110 | "cell_type": "markdown", 111 | "metadata": { 112 | "id": "dqLV5GXT3wLy", 113 | "colab_type": "text" 114 | }, 115 | "source": [ 116 | "What we're doing is creating two vectors, [1.0, 1.0, 1.0, 1.0] and [2.0, 2.0, 2.0, 2.0], and then adding them. Here's equivalent code in raw Python and using numpy:" 117 | ] 118 | }, 119 | { 120 | "cell_type": "code", 121 | "metadata": { 122 | "id": "7DzDJ7sW79ao", 123 | "colab_type": "code", 124 | "colab": { 125 | "autoexec": { 126 | "startup": false, 127 | "wait_interval": 0 128 | }, 129 | "output_extras": [ 130 | { 131 | "item_id": 1 132 | } 133 | ] 134 | }, 135 | "cellView": "both", 136 | "executionInfo": { 137 | "elapsed": 152, 138 | "status": "ok", 139 | "timestamp": 1446242020458, 140 | "user": { 141 | "color": "#1FA15D", 142 | "displayName": "Michael Piatek", 143 | "isAnonymous": false, 144 | "isMe": true, 145 | "permissionId": "00327059602783983041", 146 | "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", 147 | "sessionId": "7391995727249e65", 148 | "userId": "106975671469698476657" 149 | }, 150 | "user_tz": 420 151 | }, 152 | "outputId": "cf89e613-06e5-4435-bea3-9f48a4eff943" 153 | }, 154 | "source": [ 155 | "print [x + y for x, y in zip([1.0] * 4, [2.0] * 4)]" 156 | ], 157 | "outputs": [ 158 | { 159 | "output_type": "stream", 160 | "text": [ 161 | "[3.0, 3.0, 3.0, 3.0]\n" 162 | ], 163 | "name": "stdout" 164 | } 165 | ], 166 | "execution_count": 0 167 | }, 168 | { 169 | "cell_type": "code", 170 | "metadata": { 171 | "id": "MDWJf0lHAF4E", 172 | "colab_type": "code", 173 | "colab": { 174 | "autoexec": { 175 | "startup": false, 176 | "wait_interval": 0 177 | }, 178 | "output_extras": [ 179 | { 180 | "item_id": 1 181 | } 182 | ] 183 | }, 184 | "cellView": "both", 185 | "executionInfo": { 186 | "elapsed": 97, 187 | "status": "ok", 188 | "timestamp": 1446242021921, 189 | "user": { 190 | "color": "#1FA15D", 191 | "displayName": "Michael Piatek", 192 | "isAnonymous": false, 193 | "isMe": true, 194 | "permissionId": "00327059602783983041", 195 | "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", 196 | "sessionId": "7391995727249e65", 197 | "userId": "106975671469698476657" 198 | }, 199 | "user_tz": 420 200 | }, 201 | "outputId": "66d8c4a2-92b7-4048-b365-39dc42dff2bc" 202 | }, 203 | "source": [ 204 | "import numpy as np\n", 205 | "x, y = np.full(4, 1.0), np.full(4, 2.0)\n", 206 | "print \"{} + {} = {}\".format(x, y, x + y)" 207 | ], 208 | "outputs": [ 209 | { 210 | "output_type": "stream", 211 | "text": [ 212 | "[ 1. 1. 1. 1.] + [ 2. 2. 2. 2.] = [ 3. 3. 3. 3.]\n" 213 | ], 214 | "name": "stdout" 215 | } 216 | ], 217 | "execution_count": 0 218 | }, 219 | { 220 | "cell_type": "markdown", 221 | "metadata": { 222 | "id": "I52jQOyO8vAn", 223 | "colab_type": "text" 224 | }, 225 | "source": [ 226 | "## Details of adding two vectors in TensorFlow\n", 227 | "\n", 228 | "The example above of adding two vectors involves a lot more than it seems, so let's look at it in more depth.\n", 229 | "\n", 230 | ">`import tensorflow as tf`\n", 231 | "\n", 232 | "This import brings TensorFlow's public API into our IPython runtime environment.\n", 233 | "\n", 234 | ">`with tf.Session():`\n", 235 | "\n", 236 | "When you run an operation in TensorFlow, you need to do it in the context of a `Session`. A session holds the computation graph, which contains the tensors and the operations. When you create tensors and operations, they are not executed immediately, but wait for other operations and tensors to be added to the graph, only executing when finally requested to produce the results of the session. Deferring the execution like this provides additional opportunities for parallelism and optimization, as TensorFlow can decide how to combine operations and where to run them after TensorFlow knows about all the operations. \n", 237 | "\n", 238 | ">>`input1 = tf.constant([1.0, 1.0, 1.0, 1.0])`\n", 239 | "\n", 240 | ">>`input2 = tf.constant([2.0, 2.0, 2.0, 2.0])`\n", 241 | "\n", 242 | "The next two lines create tensors using a convenience function called `constant`, which is similar to numpy's `array` and numpy's `full`. If you look at the code for `constant`, you can see the details of what it is doing to create the tensor. In summary, it creates a tensor of the necessary shape and applies the constant operator to it to fill it with the provided values. The values to `constant` can be Python or numpy arrays. `constant` can take an optional shape parameter, which works similarly to numpy's `fill` if provided, and an optional name parameter, which can be used to put a more human-readable label on the operation in the TensorFlow operation graph.\n", 243 | "\n", 244 | ">>`output = tf.add(input1, input2)`\n", 245 | "\n", 246 | "You might think `add` just adds the two vectors now, but it doesn't quite do that. What it does is put the `add` operation into the computational graph. The results of the addition aren't available yet. They've been put in the computation graph, but the computation graph hasn't been executed yet.\n", 247 | "\n", 248 | ">>`result = output.eval()`\n", 249 | "\n", 250 | ">>`print result`\n", 251 | "\n", 252 | "`eval()` is also slightly more complicated than it looks. Yes, it does get the value of the vector (tensor) that results from the addition. It returns this as a numpy array, which can then be printed. But, it's important to realize it also runs the computation graph at this point, because we demanded the output from the operation node of the graph; to produce that, it had to run the computation graph. So, this is the point where the addition is actually performed, not when `add` was called, as `add` just put the addition operation into the TensorFlow computation graph." 253 | ] 254 | }, 255 | { 256 | "cell_type": "markdown", 257 | "metadata": { 258 | "id": "H_5_2YY3ySr2", 259 | "colab_type": "text" 260 | }, 261 | "source": [ 262 | "## Multiple operations\n", 263 | "\n", 264 | "To use TensorFlow, you add operations on tensors that produce tensors to the computation graph, then execute that graph to run all those operations and calculate the values of all the tensors in the graph.\n", 265 | "\n", 266 | "Here's a simple example with two operations:" 267 | ] 268 | }, 269 | { 270 | "cell_type": "code", 271 | "metadata": { 272 | "id": "-kQmn3U_yXX8", 273 | "colab_type": "code", 274 | "colab": { 275 | "autoexec": { 276 | "startup": false, 277 | "wait_interval": 0 278 | }, 279 | "output_extras": [ 280 | { 281 | "item_id": 1 282 | } 283 | ] 284 | }, 285 | "cellView": "both", 286 | "executionInfo": { 287 | "elapsed": 101, 288 | "status": "ok", 289 | "timestamp": 1446242580297, 290 | "user": { 291 | "color": "#1FA15D", 292 | "displayName": "Michael Piatek", 293 | "isAnonymous": false, 294 | "isMe": true, 295 | "permissionId": "00327059602783983041", 296 | "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", 297 | "sessionId": "7391995727249e65", 298 | "userId": "106975671469698476657" 299 | }, 300 | "user_tz": 420 301 | }, 302 | "outputId": "e96a6e27-665e-47d3-822e-47aeb66fc7f8" 303 | }, 304 | "source": [ 305 | "import tensorflow as tf\n", 306 | "\n", 307 | "with tf.Session():\n", 308 | " input1 = tf.constant(1.0, shape=[4])\n", 309 | " input2 = tf.constant(2.0, shape=[4])\n", 310 | " input3 = tf.constant(3.0, shape=[4])\n", 311 | " output = tf.add(tf.add(input1, input2), input3)\n", 312 | " result = output.eval()\n", 313 | " print result" 314 | ], 315 | "outputs": [ 316 | { 317 | "output_type": "stream", 318 | "text": [ 319 | "[ 6. 6. 6. 6.]\n" 320 | ], 321 | "name": "stdout" 322 | } 323 | ], 324 | "execution_count": 0 325 | }, 326 | { 327 | "cell_type": "markdown", 328 | "metadata": { 329 | "id": "Hod0zvsly8YT", 330 | "colab_type": "text" 331 | }, 332 | "source": [ 333 | "This version uses `constant` in a way similar to numpy's `fill`, specifying the optional shape and having the values copied out across it.\n", 334 | "\n", 335 | "The `add` operator supports operator overloading, so you could try writing it inline as `input1 + input2` instead as well as experimenting with other operators." 336 | ] 337 | }, 338 | { 339 | "cell_type": "code", 340 | "metadata": { 341 | "id": "yS2WElRfxz53", 342 | "colab_type": "code", 343 | "colab": { 344 | "autoexec": { 345 | "startup": false, 346 | "wait_interval": 0 347 | }, 348 | "output_extras": [ 349 | { 350 | "item_id": 1 351 | } 352 | ] 353 | }, 354 | "cellView": "both", 355 | "executionInfo": { 356 | "elapsed": 156, 357 | "status": "ok", 358 | "timestamp": 1446242664353, 359 | "user": { 360 | "color": "#1FA15D", 361 | "displayName": "Michael Piatek", 362 | "isAnonymous": false, 363 | "isMe": true, 364 | "permissionId": "00327059602783983041", 365 | "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", 366 | "sessionId": "7391995727249e65", 367 | "userId": "106975671469698476657" 368 | }, 369 | "user_tz": 420 370 | }, 371 | "outputId": "9818bf3c-5659-4a87-8b5d-40a28f1a2677" 372 | }, 373 | "source": [ 374 | "with tf.Session():\n", 375 | " input1 = tf.constant(1.0, shape=[4])\n", 376 | " input2 = tf.constant(2.0, shape=[4])\n", 377 | " output = input1 + input2\n", 378 | " print output.eval()" 379 | ], 380 | "outputs": [ 381 | { 382 | "output_type": "stream", 383 | "text": [ 384 | "[ 3. 3. 3. 3.]\n" 385 | ], 386 | "name": "stdout" 387 | } 388 | ], 389 | "execution_count": 0 390 | }, 391 | { 392 | "cell_type": "markdown", 393 | "metadata": { 394 | "id": "zszjoYUjkUNU", 395 | "colab_type": "text" 396 | }, 397 | "source": [ 398 | "## Adding two matrices" 399 | ] 400 | }, 401 | { 402 | "cell_type": "markdown", 403 | "metadata": { 404 | "id": "EWNYBCB6kbri", 405 | "colab_type": "text" 406 | }, 407 | "source": [ 408 | "Next, let's do something very similar, adding two matrices:\n", 409 | "\n", 410 | "$\\begin{bmatrix}\n", 411 | " 1. & 1. & 1. \\\\\n", 412 | " 1. & 1. & 1. \\\\\n", 413 | "\\end{bmatrix} + \n", 414 | "\\begin{bmatrix}\n", 415 | " 1. & 2. & 3. \\\\\n", 416 | " 4. & 5. & 6. \\\\\n", 417 | "\\end{bmatrix} = \n", 418 | "\\begin{bmatrix}\n", 419 | " 2. & 3. & 4. \\\\\n", 420 | " 5. & 6. & 7. \\\\\n", 421 | "\\end{bmatrix}$" 422 | ] 423 | }, 424 | { 425 | "cell_type": "code", 426 | "metadata": { 427 | "id": "tmWcCxSilYkg", 428 | "colab_type": "code", 429 | "colab": { 430 | "autoexec": { 431 | "startup": false, 432 | "wait_interval": 0 433 | }, 434 | "output_extras": [ 435 | { 436 | "item_id": 1 437 | } 438 | ] 439 | }, 440 | "cellView": "both", 441 | "executionInfo": { 442 | "elapsed": 1540, 443 | "status": "ok", 444 | "timestamp": 1446242690334, 445 | "user": { 446 | "color": "#1FA15D", 447 | "displayName": "Michael Piatek", 448 | "isAnonymous": false, 449 | "isMe": true, 450 | "permissionId": "00327059602783983041", 451 | "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", 452 | "sessionId": "7391995727249e65", 453 | "userId": "106975671469698476657" 454 | }, 455 | "user_tz": 420 456 | }, 457 | "outputId": "f3a2e904-790b-42e1-9ca4-2f3c54d7f4a8" 458 | }, 459 | "source": [ 460 | "import tensorflow as tf\n", 461 | "import numpy as np\n", 462 | "\n", 463 | "with tf.Session():\n", 464 | " input1 = tf.constant(1.0, shape=[2, 3])\n", 465 | " input2 = tf.constant(np.reshape(np.arange(1.0, 7.0, dtype=np.float32), (2, 3)))\n", 466 | " output = tf.add(input1, input2)\n", 467 | " print output.eval()" 468 | ], 469 | "outputs": [ 470 | { 471 | "output_type": "stream", 472 | "text": [ 473 | "[[ 2. 3. 4.]\n", 474 | " [ 5. 6. 7.]]\n" 475 | ], 476 | "name": "stdout" 477 | } 478 | ], 479 | "execution_count": 0 480 | }, 481 | { 482 | "cell_type": "markdown", 483 | "metadata": { 484 | "id": "JuU3Bmglq1vd", 485 | "colab_type": "text" 486 | }, 487 | "source": [ 488 | "Recall that you can pass numpy or Python arrays into `constant`.\n", 489 | "\n", 490 | "In this example, the matrix with values from 1 to 6 is created in numpy and passed into `constant`, but TensorFlow also has `range`, `reshape`, and `tofloat` operators. Doing this entirely within TensorFlow could be more efficient if this was a very large matrix.\n", 491 | "\n", 492 | "Try experimenting with this code a bit -- maybe modifying some of the values, using the numpy version, doing this using, adding another operation, or doing this using TensorFlow's `range` function." 493 | ] 494 | }, 495 | { 496 | "cell_type": "markdown", 497 | "metadata": { 498 | "id": "gnXnpnuLrflb", 499 | "colab_type": "text" 500 | }, 501 | "source": [ 502 | "## Multiplying matrices" 503 | ] 504 | }, 505 | { 506 | "cell_type": "markdown", 507 | "metadata": { 508 | "id": "Ho-QNSOorj0y", 509 | "colab_type": "text" 510 | }, 511 | "source": [ 512 | "Let's move on to matrix multiplication. This time, let's use a bit vector and some random values, which is a good step toward some of what we'll need to do for regression and neural networks." 513 | ] 514 | }, 515 | { 516 | "cell_type": "code", 517 | "metadata": { 518 | "id": "uNqMaFR8sIY5", 519 | "colab_type": "code", 520 | "colab": { 521 | "autoexec": { 522 | "startup": false, 523 | "wait_interval": 0 524 | }, 525 | "output_extras": [ 526 | { 527 | "item_id": 1 528 | } 529 | ] 530 | }, 531 | "cellView": "both", 532 | "executionInfo": { 533 | "elapsed": 132, 534 | "status": "ok", 535 | "timestamp": 1446242872027, 536 | "user": { 537 | "color": "#1FA15D", 538 | "displayName": "Michael Piatek", 539 | "isAnonymous": false, 540 | "isMe": true, 541 | "permissionId": "00327059602783983041", 542 | "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", 543 | "sessionId": "7391995727249e65", 544 | "userId": "106975671469698476657" 545 | }, 546 | "user_tz": 420 547 | }, 548 | "outputId": "fc0e29a0-306c-4709-c181-1108d5a21d88" 549 | }, 550 | "source": [ 551 | "#@test {\"output\": \"ignore\"}\n", 552 | "import tensorflow as tf\n", 553 | "import numpy as np\n", 554 | "\n", 555 | "with tf.Session():\n", 556 | " input_features = tf.constant(np.reshape([1, 0, 0, 1], (1, 4)).astype(np.float32))\n", 557 | " weights = tf.constant(np.random.randn(4, 2).astype(np.float32))\n", 558 | " output = tf.matmul(input_features, weights)\n", 559 | " print \"Input:\"\n", 560 | " print input_features.eval()\n", 561 | " print \"Weights:\"\n", 562 | " print weights.eval()\n", 563 | " print \"Output:\"\n", 564 | " print output.eval()" 565 | ], 566 | "outputs": [ 567 | { 568 | "output_type": "stream", 569 | "text": [ 570 | "Input:\n", 571 | "[[ 1. 0. 0. 1.]]\n", 572 | "Weights:\n", 573 | "[[-0.8187139 -0.81037313]\n", 574 | " [-0.31439888 -2.36761999]\n", 575 | " [-1.3127892 -0.33629459]\n", 576 | " [-1.23475349 -1.19031894]]\n", 577 | "Output:\n", 578 | "[[-2.05346727 -2.00069213]]\n" 579 | ], 580 | "name": "stdout" 581 | } 582 | ], 583 | "execution_count": 0 584 | }, 585 | { 586 | "cell_type": "markdown", 587 | "metadata": { 588 | "id": "JDAVTPhb22AP", 589 | "colab_type": "text" 590 | }, 591 | "source": [ 592 | "Above, we're taking a 1 x 4 vector [1 0 0 1] and multiplying it by a 4 by 2 matrix full of random values from a normal distribution (mean 0, stdev 1). The output is a 1 x 2 matrix.\n", 593 | "\n", 594 | "You might try modifying this example. Running the cell multiple times will generate new random weights and a new output. Or, change the input, e.g., to \\[0 0 0 1]), and run the cell again. Or, try initializing the weights using the TensorFlow op, e.g., `random_normal`, instead of using numpy to generate the random weights.\n", 595 | "\n", 596 | "What we have here is the basics of a simple neural network already. If we are reading in the input features, along with some expected output, and change the weights based on the error with the output each time, that's a neural network." 597 | ] 598 | }, 599 | { 600 | "cell_type": "markdown", 601 | "metadata": { 602 | "id": "XhnBjAUILuy8", 603 | "colab_type": "text" 604 | }, 605 | "source": [ 606 | "## Use of variables\n", 607 | "\n", 608 | "Let's look at adding two small matrices in a loop, not by creating new tensors every time, but by updating the existing values and then re-running the computation graph on the new data. This happens a lot with machine learning models, where we change some parameters each time such as gradient descent on some weights and then perform the same computations over and over again." 609 | ] 610 | }, 611 | { 612 | "cell_type": "code", 613 | "metadata": { 614 | "id": "vJ_AgZ8lLtRv", 615 | "colab_type": "code", 616 | "colab": { 617 | "autoexec": { 618 | "startup": false, 619 | "wait_interval": 0 620 | }, 621 | "output_extras": [ 622 | { 623 | "item_id": 1 624 | } 625 | ] 626 | }, 627 | "cellView": "both", 628 | "executionInfo": { 629 | "elapsed": 180, 630 | "status": "ok", 631 | "timestamp": 1446244201894, 632 | "user": { 633 | "color": "#1FA15D", 634 | "displayName": "Michael Piatek", 635 | "isAnonymous": false, 636 | "isMe": true, 637 | "permissionId": "00327059602783983041", 638 | "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", 639 | "sessionId": "7391995727249e65", 640 | "userId": "106975671469698476657" 641 | }, 642 | "user_tz": 420 643 | }, 644 | "outputId": "8d3aadaa-2b34-4642-889b-e3daaf5ee693" 645 | }, 646 | "source": [ 647 | "#@test {\"output\": \"ignore\"}\n", 648 | "import tensorflow as tf\n", 649 | "import numpy as np\n", 650 | "\n", 651 | "with tf.Session() as sess:\n", 652 | " # Set up two variables, total and weights, that we'll change repeatedly.\n", 653 | " total = tf.Variable(tf.zeros([1, 2]))\n", 654 | " weights = tf.Variable(tf.random_uniform([1,2]))\n", 655 | " \n", 656 | " # Initialize the variables we defined above.\n", 657 | " tf.initialize_all_variables().run()\n", 658 | " \n", 659 | " # This only adds the operators to the graph right now. The assignment\n", 660 | " # and addition operations are not performed yet.\n", 661 | " update_weights = tf.assign(weights, tf.random_uniform([1, 2], -1.0, 1.0))\n", 662 | " update_total = tf.assign(total, tf.add(total, weights))\n", 663 | " \n", 664 | " for _ in range(5):\n", 665 | " # Actually run the operation graph, so randomly generate weights and then\n", 666 | " # add them into the total. Order does matter here. We need to update\n", 667 | " # the weights before updating the total.\n", 668 | " sess.run(update_weights)\n", 669 | " sess.run(update_total)\n", 670 | " \n", 671 | " print weights.eval(), total.eval()" 672 | ], 673 | "outputs": [ 674 | { 675 | "output_type": "stream", 676 | "text": [ 677 | "[[-0.41494703 0.47648168]] [[-0.41494703 0.47648168]]\n", 678 | "[[ 0.35746408 0.99504066]] [[-0.05748296 1.47152233]]\n", 679 | "[[-0.46462393 -0.80201006]] [[-0.52210689 0.66951227]]\n", 680 | "[[-0.99513483 -0.42322445]] [[-1.51724172 0.24628782]]\n", 681 | "[[ 0.13371086 -0.85545826]] [[-1.38353086 -0.60917044]]\n" 682 | ], 683 | "name": "stdout" 684 | } 685 | ], 686 | "execution_count": 0 687 | }, 688 | { 689 | "cell_type": "markdown", 690 | "metadata": { 691 | "id": "kSYJr89aM_n0", 692 | "colab_type": "text" 693 | }, 694 | "source": [ 695 | "This is more complicated. At a high level, we create two variables and add operations over them, then, in a loop, repeatedly execute those operations. Let's walk through it step by step.\n", 696 | "\n", 697 | "Starting off, the code creates two variables, `total` and `weights`. `total` is initialized to \\[0, 0\\] and `weights` is initialized to random values between -1 and 1.\n", 698 | "\n", 699 | "Next, two assignment operators are added to the graph, one that updates weights with random values from [-1, 1], the other that updates the total with the new weights. Again, the operators are not executed here. In fact, this isn't even inside the loop. We won't execute these operations until the `eval` call inside the loop.\n", 700 | "\n", 701 | "Finally, in the for loop, we run each of the operators. In each iteration of the loop, this executes the operators we added earlier, first putting random values into the weights, then updating the totals with the new weights. This call uses `eval` on the session; the code also could have called `eval` on the operators (e.g. `update_weights.eval`).\n", 702 | "\n", 703 | "It can be a little hard to wrap your head around exactly what computation is done when. The important thing to remember is that computation is only performed on demand.\n", 704 | "\n", 705 | "Variables can be useful in cases where you have a large amount of computation and data that you want to use over and over again with just a minor change to the input each time. That happens quite a bit with neural networks, for example, where you just want to update the weights each time you go through the batches of input data, then run the same operations over again." 706 | ] 707 | }, 708 | { 709 | "cell_type": "markdown", 710 | "metadata": { 711 | "id": "fL3WfAbKzqr5", 712 | "colab_type": "text" 713 | }, 714 | "source": [ 715 | "## What's next?\n", 716 | "\n", 717 | "This has been a gentle introduction to TensorFlow, focused on what TensorFlow is and the very basics of doing anything in TensorFlow. If you'd like more, the next tutorial in the series is Getting Started with TensorFlow, also available in the [notebooks directory](..)." 718 | ] 719 | } 720 | ] 721 | } 722 | -------------------------------------------------------------------------------- /notebooks/2_getting_started.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "version": "0.3.2", 7 | "views": {}, 8 | "default_view": {}, 9 | "name": "Untitled", 10 | "provenance": [] 11 | } 12 | }, 13 | "cells": [ 14 | { 15 | "cell_type": "markdown", 16 | "metadata": { 17 | "id": "6TuWv0Y0sY8n", 18 | "colab_type": "text" 19 | }, 20 | "source": [ 21 | "# Getting Started in TensorFlow\n", 22 | "## A look at a very simple neural network in TensorFlow" 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": { 28 | "id": "u9J5e2mQsYsQ", 29 | "colab_type": "text" 30 | }, 31 | "source": [ 32 | "This is an introduction to working with TensorFlow. It works through an example of a very simple neural network, walking through the steps of setting up the input, adding operators, setting up gradient descent, and running the computation graph. \n", 33 | "\n", 34 | "This tutorial presumes some familiarity with the TensorFlow computational model, which is introduced in the [Hello, TensorFlow](../notebooks/1_hello_tensorflow.ipynb) notebook, also available in this bundle." 35 | ] 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "metadata": { 40 | "id": "Dr2Sv0vD8rT-", 41 | "colab_type": "text" 42 | }, 43 | "source": [ 44 | "## A simple neural network\n", 45 | "\n", 46 | "Let's start with code. We're going to construct a very simple neural network computing a linear regression between two variables, y and x. The function it tries to compute is the best $w_1$ and $w_2$ it can find for the function $y = w_2 x + w_1$ for the data. The data we're going to give it is toy data, linear perturbed with random noise.\n", 47 | "\n", 48 | "This is what the network looks like:" 49 | ] 50 | }, 51 | { 52 | "cell_type": "code", 53 | "metadata": { 54 | "id": "q09my4JYtKXw", 55 | "colab_type": "code", 56 | "colab": { 57 | "autoexec": { 58 | "startup": false, 59 | "wait_interval": 0 60 | }, 61 | "output_extras": [ 62 | {} 63 | ] 64 | }, 65 | "cellView": "both", 66 | "outputId": "22b94683-437b-45ed-f6b8-4abf3b76ce38" 67 | }, 68 | "source": [ 69 | "from IPython.display import Image\n", 70 | "import base64\n", 71 | "Image(data=base64.decodestring(\"iVBORw0KGgoAAAANSUhEUgAAAJYAAABkCAYAAABkW8nwAAAO90lEQVR4Xu2dT5Dc1J3Hv+YQT8VJZUhVdprLWs4FTSrGGv4ql9CuHBCH4GaTFCLZwnIcjOAy8l6Q/1SlU4XHcg6xJgtY2OOik2KxSGoTGWrXzYFC2T2MDAtWitRavmQ0e9k2SYGowom4hNRPtqA9TE+rW3/cPfPepcfup6f3fu/Tv9/T+/PVpo8//vhjsMQsULAFNjGwCrYoKy6xAAOLgVCKBRhYpZiVFcrAYgyUYgEGVilmZYUysBgDpViAgVWKWVmhDCzGQCkWGEuwrly5gtf++zW887/vYOn/lnD5T5cT40x9ZQrb/nEbxDtFiHeI2LJlSylGY4X2t8BYgUVAvfzqy3i5/TI+vPLhmq37wpYv4AHpATxw3wMMsP4cFJ5jbMAiqA4eOYg/Lv8xMcL26e34+vTXk8+vbv1q8n/03TsX38EfLv4h+aRE380dmmNwFY7O2gWOBVgE1Y/2/yjxUls+vwXaY1oS7tZK3v94MJ8zceUvV0Dea+H4AoOrQrhGHqxuT0Xjp0P7D2HqH6Yymejyu5dx5PiRZBxGnmt+bj7TdSxTfgv0ASuAzglwmyE8pfbZu3VaEDkDdT+AweevzGolvPjvL+LMb84knmr+yHxmqNKyCK7ZQ7OJ5yIo+3m6clqx8UrNB1bso2W64FQN9cnijdcdAvNAQWGRPBcLicX3Ua8S84FVcj3PnjuLhRcWkgH63OG5XHc7+NTBZEBP47NvffNbucpiF/e3QCaw2g0NfNvES5c+wtQ9u2G0LCj8BLAiFEaeBU0zYJ9fxkfYjKl7FZgtCzIHIA7QUmXov/g9LmMztt6rwLBMyFROj3TkZ0fgveXh4X96GN//zvf7t2aNHGlI7VlW0pYmRC+AKUwAsQu5thOuvIjQEjGBGJ7CQYptdOw6etc6VzXXzcUZwJrGseWt2P28DV2I4OgyDgQKFgMTYtQ1xqq10eDuR6j8Fi1NxGTkwpAfRos7h05bQscQIFgibEeHMBHCVhs4EBtY8lQQd6ulvbN78e6f302mC7Z/bXsuo9NkKk1X9PZ+IUyeR0sN4GscYl8DPzOP5VuPYynQwMU+dL4O3wzRbpQQ93O1bvQuzgRWS0p/tQA6Nuqcilq7A5u3Px28T7qw7BB1VUHqhEKTB2+pCAIVHZVD3dPgujpE6peOBzesQRS5nr/+b//g24nF7JN27qkCGq/J++RknHXm5JlVeiKGr/MQPQMdV0ZkCRBbNUwEMYzQhRyZEHgHOv29ynPM6HXtja1Rf7B4AZ7RgZv+SuMAOj+NtrYEX3avfyqMfDi2DdcLEAQBvPOX8MGtR3Ex0MEFJiRxP373wWZsvaeBhixDVRrg1/jxlwEWPV3ap+xVrR57Cjgpht2xEDV4mLIFvqkiaoUwwzp4U4Hv9/awN7YrR+vuGcAS4ZsdtKV0VNEFVqMLrIkWJGEPPP4hKA0RgiCAc1XsdJQErGQ2Ig7hOQ5sx4Hz0u+wvHX2akjtMWCpNhQCiCicq+AcCx1Fh9B2IegcNN6B4Teg1z0EeknzKqPFRe7a9AeLm4ajXvzUoJEDqUahMESrKxSqbQHbDBGLoXUNlBiuUsNOT8fFQEVsNdHmdOjStTgSGOCnLTQuBDBosLxKqnTwntw/glPnoHMS4E6iFVjgbBGcwUGMPAjtawP73GZf/wVkAutYtAvPezYUPoKjipBdGZ5vQOgavGteHbfsiXD09TZUIUbg6JD3vITlrU/iYthErPOYaQk44ZhocDF8U0HDqsEOHfQaC7/2X68lyzJVTjd0WiJu2XMem++7+tAxSd52+hguTe3GYtjq6V3XPyqDtbA/WLyAtqRg0rHhLceo3avCsk0kjqd7uoEL0FJkaC/9Hh/gS9ixS0dTCaDKHVidNhoTNN2gQP/FedAmly/t2IWm2YK2xswqDbj3antzz5oToD/915/i5smbcdo8vfaDQGiC37YfEyeW4KtcMu2g1HbCrp9Dx5Fw3ZCw04ZSb0Jse6CsLH1qgZFfK0znn+hpznzKHGpJRzus4YJ/AX/78G94ofUC7r777pwMxAhdE6pyAK8u78CJJZ+BtcKiIw8Wea0DTx34ZCH5oHYwM1y0TjhnziXbaWgB+4cP/RCPPfYYtm/fjpMnT+Kmm24aDrDYhdpoQdAbaMtNSB4Da6UhRx4sqnB3SCTPNbtvtu9iMoU/Wg5Kt9p0h8DTp09j3759ePrpp/H4448PB1fylOtC5jTUGVifseFYgJXClXou+jcN6Gk2nj7JG1Gi7TG0Hkiz7OlGP/ru6OGjq46rnnjiCSwuLibe66677hocMAZWT5uNDVgpXGfbZ5OtybQNZq1EE6G0NXmXtGvNwbrv+4n3uu222wYPjwys9QFW2goKjbQ4Tdth6CAFeSpK5J3oQMUwhynS8PjMM89AVdVs3ouBtb7Aytbrw+WiMZfnednCIwOLgTUIZml43LFjB5577rnhnx4Huek6yztWY6yqbb+wsJBMTwwUHquu5Ijej4GVoWMoPJ4/fz7xXkM9PWa4x3rLwsDK2KMXLlxIvBeFR5qe2LRpU8YrN2Y2BtaA/U7hkaYnnn322exPjwPeYz1kZ2AN2YtpeCTvdeeddw5Zyvq9jIGVo28pPJL3ok2NLDxeb0gGVg6w0kvT8HjixIlkHJY1lauaE8GRangwsvD/noKqt+kzsLJSkCEfzdi/8cYbifdaKzxWoppDmxJ5FT54NH06YZShAQVmYWAVaEwqKg2PMzMzyfTEyqfHqlRzAoOH6OqwJnXoNQeBSWcjq0sMrJJsferUqSQsdofHylRzYg8aLyG0QtiTOvhGhFZglyKD0Mt8DKySwEqLpfD45ptvYn5+Hr/+z19/sukwj2pOP72vyJXBy4BNME340Pg6AiNAu8IDkQysksGi4t9++2189wffxee++DkIO4TcqjlrSw504Eg81FobYetq+KOwKDgagjVOnRdtBgZW0RZdpbw0BL73/nv4yZM/6bv7tVeVxkk1h4FVAVgbUTWHgVUBWGUcvCVV6EP/cuiztQ9NCNsMiIshrPSIeaK3oUNIlXQqaDMDqwIjlyEV0Fv6MoQlbENT/FTIhWSXOF2AF5jocei8cCswsAo36WcLLEPchO7yyr+9smrt6TQ3geQmcgcd2CQbIHoIDKGyuSwG1joEi06oU+jj3RAWR2HQgFiiTuxqJmRgVQBWGaGQDo78/OjPe9T+qpfSeBeeqIM3JPip4k8F7aVbMLAqMHSlg/dr7YkcCZxWg1Jz0G5UL7/EwKoArBuhmoNEbupBvPrRDhxf8qFVLFrCwKoArFQi4P3o/VwTpCmgdBi3r2oOIrQbNdwfGljytZ46r2U1n4FVlmW7yn3rrbfwvX/+XrKkMyPM5FLNIS2KbCrSNI8loKX48G6AxhIDq2SwaIcDgWWaJn71H78qRDWnlxbF1aaQxJILj6TRjRhm0L4hYrwMrJLAos1+BBXtyaLty5SKVs1Zverx1RB4dhIPPe/CVioeXF2rFAOrYLDIOxFQd9xxRwLVytSt90XfFaGaU3ATCimOgVWIGa8WkoY9AorA6pUIrqJVcwpsRiFFMbAKMONqYS9LsWWo5mS5bxV5GFg5rExhj8ZPdHBitbCXo+ixv5SBNWQXpmGPvNXtt98+ZCnr9zIG1oB9O2zYG/A2Y5+dgZWxC1nYy2goNt2Q3VA0jqIDESzsZbcZ81hr2CoNe/T56KOPZrcqy8m2zazGAAt7+X8ZzGOtsCELe/mhohLGEqwyVFpY2CsGqLSUsQKrDJUWFvaKBWrswCpDpYWFvXKgKiYUxh5U/huwhd8idBqYRARX4bHTldd8Le8gTSpapYWWX0is47qnveTdi02I6aFOejlAbSdcOT2fF8NTOEixDTqnV6Uk0CC2GpW8hYTCyFXA72yj8XoAAzoE+nsxgNnrZc8DtL7bU9HJlDwqLY9855FkbY8ktS3LWlGLECbPo6UG8DUOsa+Bn5nH8q3HsRRo4GISL6vDN0O0e70SdoB2rfeshYBF71Juyzzu90TcF59FIC8WJvSVvgiT9nnPH5nP/K7CtOPonYWzh2aTF2Fu+usmvPjLF3us7cXwdR6iZ6DjyogsAWKrhokghhG6kCMTAu9Ap7+r1l0cQwoLAote4+ugwT+IsxO78XrQKkTkqzsEkqeily8Nk0il5cfHfowv3/xlLBxf6Pk2sNhTwEkx7I6FqMHDlC3wTRVRK4QZ1sGbCnxfrfxgwjBtvtHXFAZW7OsQZo7hEm7Fkxf8nm+mH6TBlau0RG00OBWcY6Gj6BDaLgSdDn46MPwG9Hr15/MGsdco5S0GrDiAIU7D5M/AgIo9gY6Lng4+5wi3jIOea59wieCQzgEnAe4kWoEFzhbBGRzEyIPQDmBWpaoxSpQMUZdCwCLh1OlmDWcCBzJsSNzDiIyL8LR8Ur1lHE2nPeZzh+d6mooENW7Zcx6b7zuHTlvCJB1Nnz6GS1O7sUhKxDl/LEP00Vhekh8sUjThNUyYAdxr59dCSwSvAWbg5Xq7exkqLfRO6TMnz/TurNAEv20/Jk4swaf2xC6U2k7Y9XPoOBIm6crYh6UoaLodABOoSU3YlpLbQ48lQT0qnR+sEq1RBlj0dGmfsnPVOtB51IMmfEdGLQ7RkkSYkps8VbJ01QIjDdaNCIVZwOi4DnxOgsRRXIzhazwakY3gmphsljLWe56RBqv6wfvg3R0HFqS6CcHxC5kQHrwGo3nFSIN1Q1RaBuinyDchSyYmDRcthWPLPF22G2mwuo+k55kgHUylJRtZoa1A0kI0bAdGPRnSszQuYFE90yUdepoznzKHWtLRDmsglZY8cHZTE7UVCGqEpmtDScZZLK20wEh7LKpst9YBKQUf1A5mhovWCefMuU9eM9JbWnEQMAIY/DQOXLr+mqmHXkfIdj18YpSRByuFa6+2F1f+cgXkuWb3zfZdN6Twt/DCQuKpsgmVDQIXy9vPAmMB1krPRf9eryot/TpsXL4fG7BSuNa7Ssu4gNOvnmMFVtqY9azS0q/DxuX7sQRrXIy7kevJwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3Ixf9d0NIelzdt4X5AAAAAElFTkSuQmCC\"), embed=True)" 72 | ], 73 | "outputs": [ 74 | { 75 | "output_type": "execute_result", 76 | "execution_count": 1, 77 | "metadata": {}, 78 | "data": { 79 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAJYAAABkCAYAAABkW8nwAAAO90lEQVR4Xu2dT5Dc1J3Hv+YQT8VJ\nZUhVdprLWs4FTSrGGv4ql9CuHBCH4GaTFCLZwnIcjOAy8l6Q/1SlU4XHcg6xJgtY2OOik2KxSGoT\nGWrXzYFC2T2MDAtWitRavmQ0e9k2SYGowom4hNRPtqA9TE+rW3/cPfPepcfup6f3fu/Tv9/T+/PV\npo8//vhjsMQsULAFNjGwCrYoKy6xAAOLgVCKBRhYpZiVFcrAYgyUYgEGVilmZYUysBgDpViAgVWK\nWVmhDCzGQCkWGEuwrly5gtf++zW887/vYOn/lnD5T5cT40x9ZQrb/nEbxDtFiHeI2LJlSylGY4X2\nt8BYgUVAvfzqy3i5/TI+vPLhmq37wpYv4AHpATxw3wMMsP4cFJ5jbMAiqA4eOYg/Lv8xMcL26e34\n+vTXk8+vbv1q8n/03TsX38EfLv4h+aRE380dmmNwFY7O2gWOBVgE1Y/2/yjxUls+vwXaY1oS7tZK\n3v94MJ8zceUvV0Dea+H4AoOrQrhGHqxuT0Xjp0P7D2HqH6Yymejyu5dx5PiRZBxGnmt+bj7TdSxT\nfgv0ASuAzglwmyE8pfbZu3VaEDkDdT+AweevzGolvPjvL+LMb84knmr+yHxmqNKyCK7ZQ7OJ5yIo\n+3m6clqx8UrNB1bso2W64FQN9cnijdcdAvNAQWGRPBcLicX3Ua8S84FVcj3PnjuLhRcWkgH63OG5\nXHc7+NTBZEBP47NvffNbucpiF/e3QCaw2g0NfNvES5c+wtQ9u2G0LCj8BLAiFEaeBU0zYJ9fxkfY\njKl7FZgtCzIHIA7QUmXov/g9LmMztt6rwLBMyFROj3TkZ0fgveXh4X96GN//zvf7t2aNHGlI7VlW\n0pYmRC+AKUwAsQu5thOuvIjQEjGBGJ7CQYptdOw6etc6VzXXzcUZwJrGseWt2P28DV2I4OgyDgQK\nFgMTYtQ1xqq10eDuR6j8Fi1NxGTkwpAfRos7h05bQscQIFgibEeHMBHCVhs4EBtY8lQQd6ulvbN7\n8e6f302mC7Z/bXsuo9NkKk1X9PZ+IUyeR0sN4GscYl8DPzOP5VuPYynQwMU+dL4O3wzRbpQQ93O1\nbvQuzgRWS0p/tQA6Nuqcilq7A5u3Px28T7qw7BB1VUHqhEKTB2+pCAIVHZVD3dPgujpE6peOBzes\nQRS5nr/+b//g24nF7JN27qkCGq/J++RknHXm5JlVeiKGr/MQPQMdV0ZkCRBbNUwEMYzQhRyZEHgH\nOv29ynPM6HXtja1Rf7B4AZ7RgZv+SuMAOj+NtrYEX3avfyqMfDi2DdcLEAQBvPOX8MGtR3Ex0MEF\nJiRxP373wWZsvaeBhixDVRrg1/jxlwEWPV3ap+xVrR57Cjgpht2xEDV4mLIFvqkiaoUwwzp4U4Hv\n9/awN7YrR+vuGcAS4ZsdtKV0VNEFVqMLrIkWJGEPPP4hKA0RgiCAc1XsdJQErGQ2Ig7hOQ5sx4Hz\n0u+wvHX2akjtMWCpNhQCiCicq+AcCx1Fh9B2IegcNN6B4Teg1z0EeknzKqPFRe7a9AeLm4ajXvzU\noJEDqUahMESrKxSqbQHbDBGLoXUNlBiuUsNOT8fFQEVsNdHmdOjStTgSGOCnLTQuBDBosLxKqnTw\nntw/glPnoHMS4E6iFVjgbBGcwUGMPAjtawP73GZf/wVkAutYtAvPezYUPoKjipBdGZ5vQOgavGte\nHbfsiXD09TZUIUbg6JD3vITlrU/iYthErPOYaQk44ZhocDF8U0HDqsEOHfQaC7/2X68lyzJVTjd0\nWiJu2XMem++7+tAxSd52+hguTe3GYtjq6V3XPyqDtbA/WLyAtqRg0rHhLceo3avCsk0kjqd7uoEL\n0FJkaC/9Hh/gS9ixS0dTCaDKHVidNhoTNN2gQP/FedAmly/t2IWm2YK2xswqDbj3antzz5oToD/9\n15/i5smbcdo8vfaDQGiC37YfEyeW4KtcMu2g1HbCrp9Dx5Fw3ZCw04ZSb0Jse6CsLH1qgZFfK0zn\nn+hpznzKHGpJRzus4YJ/AX/78G94ofUC7r777pwMxAhdE6pyAK8u78CJJZ+BtcKiIw8Wea0DTx34\nZCH5oHYwM1y0TjhnziXbaWgB+4cP/RCPPfYYtm/fjpMnT+Kmm24aDrDYhdpoQdAbaMtNSB4Da6Uh\nRx4sqnB3SCTPNbtvtu9iMoU/Wg5Kt9p0h8DTp09j3759ePrpp/H4448PB1fylOtC5jTUGVifseFY\ngJXClXou+jcN6Gk2nj7JG1Gi7TG0Hkiz7OlGP/ru6OGjq46rnnjiCSwuLibe66677hocMAZWT5uN\nDVgpXGfbZ5OtybQNZq1EE6G0NXmXtGvNwbrv+4n3uu222wYPjwys9QFW2goKjbQ4Tdth6CAFeSpK\n5J3oQMUwhynS8PjMM89AVdVs3ouBtb7Aytbrw+WiMZfnednCIwOLgTUIZml43LFjB5577rnhnx4H\nuek6yztWY6yqbb+wsJBMTwwUHquu5Ijej4GVoWMoPJ4/fz7xXkM9PWa4x3rLwsDK2KMXLlxIvBeF\nR5qe2LRpU8YrN2Y2BtaA/U7hkaYnnn322exPjwPeYz1kZ2AN2YtpeCTvdeeddw5Zyvq9jIGVo28p\nPJL3ok2NLDxeb0gGVg6w0kvT8HjixIlkHJY1lauaE8GRangwsvD/noKqt+kzsLJSkCEfzdi/8cYb\nifdaKzxWoppDmxJ5FT54NH06YZShAQVmYWAVaEwqKg2PMzMzyfTEyqfHqlRzAoOH6OqwJnXoNQeB\nSWcjq0sMrJJsferUqSQsdofHylRzYg8aLyG0QtiTOvhGhFZglyKD0Mt8DKySwEqLpfD45ptvYn5+\nHr/+z19/sukwj2pOP72vyJXBy4BNME340Pg6AiNAu8IDkQysksGi4t9++2189wffxee++DkIO4Tc\nqjlrSw504Eg81FobYetq+KOwKDgagjVOnRdtBgZW0RZdpbw0BL73/nv4yZM/6bv7tVeVxkk1h4FV\nAVgbUTWHgVUBWGUcvCVV6EP/cuiztQ9NCNsMiIshrPSIeaK3oUNIlXQqaDMDqwIjlyEV0Fv6MoQl\nbENT/FTIhWSXOF2AF5jocei8cCswsAo36WcLLEPchO7yyr+9smrt6TQ3geQmcgcd2CQbIHoIDKGy\nuSwG1joEi06oU+jj3RAWR2HQgFiiTuxqJmRgVQBWGaGQDo78/OjPe9T+qpfSeBeeqIM3JPip4k8F\n7aVbMLAqMHSlg/dr7YkcCZxWg1Jz0G5UL7/EwKoArBuhmoNEbupBvPrRDhxf8qFVLFrCwKoArFQi\n4P3o/VwTpCmgdBi3r2oOIrQbNdwfGljytZ46r2U1n4FVlmW7yn3rrbfwvX/+XrKkMyPM5FLNIS2K\nbCrSNI8loKX48G6AxhIDq2SwaIcDgWWaJn71H78qRDWnlxbF1aaQxJILj6TRjRhm0L4hYrwMrJLA\nos1+BBXtyaLty5SKVs1Zverx1RB4dhIPPe/CVioeXF2rFAOrYLDIOxFQd9xxRwLVytSt90XfFaGa\nU3ATCimOgVWIGa8WkoY9AorA6pUIrqJVcwpsRiFFMbAKMONqYS9LsWWo5mS5bxV5GFg5rExhj8ZP\ndHBitbCXo+ixv5SBNWQXpmGPvNXtt98+ZCnr9zIG1oB9O2zYG/A2Y5+dgZWxC1nYy2goNt2Q3VA0\njqIDESzsZbcZ81hr2CoNe/T56KOPZrcqy8m2zazGAAt7+X8ZzGOtsCELe/mhohLGEqwyVFpY2CsG\nqLSUsQKrDJUWFvaKBWrswCpDpYWFvXKgKiYUxh5U/huwhd8idBqYRARX4bHTldd8Le8gTSpapYWW\nX0is47qnveTdi02I6aFOejlAbSdcOT2fF8NTOEixDTqnV6Uk0CC2GpW8hYTCyFXA72yj8XoAAzoE\n+nsxgNnrZc8DtL7bU9HJlDwqLY9855FkbY8ktS3LWlGLECbPo6UG8DUOsa+Bn5nH8q3HsRRo4GIS\nL6vDN0O0e70SdoB2rfeshYBF71Juyzzu90TcF59FIC8WJvSVvgiT9nnPH5nP/K7CtOPonYWzh2aT\nF2Fu+usmvPjLF3us7cXwdR6iZ6DjyogsAWKrhokghhG6kCMTAu9Ap7+r1l0cQwoLAote4+ugwT+I\nsxO78XrQKkTkqzsEkqeily8Nk0il5cfHfowv3/xlLBxf6Pk2sNhTwEkx7I6FqMHDlC3wTRVRK4QZ\n1sGbCnxfrfxgwjBtvtHXFAZW7OsQZo7hEm7Fkxf8nm+mH6TBlau0RG00OBWcY6Gj6BDaLgSdDn46\nMPwG9Hr15/MGsdco5S0GrDiAIU7D5M/AgIo9gY6Lng4+5wi3jIOea59wieCQzgEnAe4kWoEFzhbB\nGRzEyIPQDmBWpaoxSpQMUZdCwCLh1OlmDWcCBzJsSNzDiIyL8LR8Ur1lHE2nPeZzh+d6mooENW7Z\ncx6b7zuHTlvCJB1Nnz6GS1O7sUhKxDl/LEP00Vhekh8sUjThNUyYAdxr59dCSwSvAWbg5Xq7exkq\nLfRO6TMnz/TurNAEv20/Jk4swaf2xC6U2k7Y9XPoOBIm6crYh6UoaLodABOoSU3YlpLbQ48lQT0q\nnR+sEq1RBlj0dGmfsnPVOtB51IMmfEdGLQ7RkkSYkps8VbJ01QIjDdaNCIVZwOi4DnxOgsRRXIzh\nazwakY3gmphsljLWe56RBqv6wfvg3R0HFqS6CcHxC5kQHrwGo3nFSIN1Q1RaBuinyDchSyYmDRct\nhWPLPF22G2mwuo+k55kgHUylJRtZoa1A0kI0bAdGPRnSszQuYFE90yUdepoznzKHWtLRDmsglZY8\ncHZTE7UVCGqEpmtDScZZLK20wEh7LKpst9YBKQUf1A5mhovWCefMuU9eM9JbWnEQMAIY/DQOXLr+\nmqmHXkfIdj18YpSRByuFa6+2F1f+cgXkuWb3zfZdN6Twt/DCQuKpsgmVDQIXy9vPAmMB1krPRf9e\nryot/TpsXL4fG7BSuNa7Ssu4gNOvnmMFVtqY9azS0q/DxuX7sQRrXIy7kevJwNrIvV9i2xlYJRp3\nIxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3Ixf9d0NIelzdt4X5\nAAAAAElFTkSuQmCC\n", 80 | "text/plain": [ 81 | "" 82 | ] 83 | } 84 | } 85 | ], 86 | "execution_count": 1 87 | }, 88 | { 89 | "cell_type": "markdown", 90 | "metadata": { 91 | "id": "fBQq_R8B8rRf", 92 | "colab_type": "text" 93 | }, 94 | "source": [ 95 | "Here is the TensorFlow code for this simple neural network and the results of running this code:" 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "metadata": { 101 | "id": "Dy8pFefa_Ho_", 102 | "colab_type": "code", 103 | "colab": { 104 | "autoexec": { 105 | "startup": false, 106 | "wait_interval": 0 107 | }, 108 | "output_extras": [ 109 | { 110 | "item_id": 1 111 | } 112 | ] 113 | }, 114 | "cellView": "both", 115 | "executionInfo": { 116 | "elapsed": 665, 117 | "status": "ok", 118 | "timestamp": 1446658971218, 119 | "user": { 120 | "color": "#1FA15D", 121 | "displayName": "Michael Piatek", 122 | "isAnonymous": false, 123 | "isMe": true, 124 | "permissionId": "00327059602783983041", 125 | "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", 126 | "sessionId": "4896c353dcc58d9f", 127 | "userId": "106975671469698476657" 128 | }, 129 | "user_tz": 480 130 | }, 131 | "outputId": "5a95f8c8-0c32-411d-956d-bb81aeed8e50" 132 | }, 133 | "source": [ 134 | "#@test {\"output\": \"ignore\"}\n", 135 | "import tensorflow as tf\n", 136 | "import numpy as np\n", 137 | "import matplotlib.pyplot as plt\n", 138 | "\n", 139 | "%matplotlib inline\n", 140 | "\n", 141 | "# Set up the data with a noisy linear relationship between X and Y.\n", 142 | "num_examples = 50\n", 143 | "X = np.array([np.linspace(-2, 4, num_examples), np.linspace(-6, 6, num_examples)])\n", 144 | "X += np.random.randn(2, num_examples)\n", 145 | "x, y = X\n", 146 | "x_with_bias = np.array([(1., a) for a in x]).astype(np.float32)\n", 147 | "\n", 148 | "losses = []\n", 149 | "training_steps = 50\n", 150 | "learning_rate = 0.002\n", 151 | "\n", 152 | "with tf.Session() as sess:\n", 153 | " # Set up all the tensors, variables, and operations.\n", 154 | " input = tf.constant(x_with_bias)\n", 155 | " target = tf.constant(np.transpose([y]).astype(np.float32))\n", 156 | " weights = tf.Variable(tf.random_normal([2, 1], 0, 0.1))\n", 157 | " \n", 158 | " tf.initialize_all_variables().run()\n", 159 | " \n", 160 | " yhat = tf.matmul(input, weights)\n", 161 | " yerror = tf.sub(yhat, target)\n", 162 | " loss = tf.nn.l2_loss(yerror)\n", 163 | " \n", 164 | " update_weights = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)\n", 165 | " \n", 166 | " for _ in range(training_steps):\n", 167 | " # Repeatedly run the operations, updating the TensorFlow variable.\n", 168 | " update_weights.run()\n", 169 | " losses.append(loss.eval())\n", 170 | "\n", 171 | " # Training is done, get the final values for the graphs\n", 172 | " betas = weights.eval()\n", 173 | " yhat = yhat.eval()\n", 174 | "\n", 175 | "# Show the fit and the loss over time.\n", 176 | "fig, (ax1, ax2) = plt.subplots(1, 2)\n", 177 | "plt.subplots_adjust(wspace=.3)\n", 178 | "fig.set_size_inches(10, 4)\n", 179 | "ax1.scatter(x, y, alpha=.7)\n", 180 | "ax1.scatter(x, np.transpose(yhat)[0], c=\"g\", alpha=.6)\n", 181 | "line_x_range = (-4, 6)\n", 182 | "ax1.plot(line_x_range, [betas[0] + a * betas[1] for a in line_x_range], \"g\", alpha=0.6)\n", 183 | "ax2.plot(range(0, training_steps), losses)\n", 184 | "ax2.set_ylabel(\"Loss\")\n", 185 | "ax2.set_xlabel(\"Training steps\")\n", 186 | "plt.show()" 187 | ], 188 | "outputs": [ 189 | { 190 | "output_type": "display_data", 191 | "metadata": {}, 192 | "data": { 193 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlsAAAEPCAYAAAB1MgENAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYVNW1///36oFuZpFmUFBQQGVwQkVQgx0cUENEo5HM\nIRqvSuKAibZmuOD3l1wFjWg0XjVxilfjnGCIUVDpoAmIiiIWCCjz1NDMMz2s3x+nWsumm56q6lRV\nf17Pc56qOnXqnFVAb1bvvc/a5u6IiIiISGJkhR2AiIiISCZTsiUiIiKSQEq2RERERBJIyZaIiIhI\nAinZEhEREUkgJVsiIiIiCRSXZMvMHjGzEjP7KGZfBzObamYLzew1M2sfj2uJiCSSmXU3szfNLGJm\n88zs2uj+iWa2wMw+NLMXzaxdzGduNbPF0ffPDS96EUlF8erZegwYXm3fLcDr7n408CZwa5yuJSKS\nSOXAje7eHxgC/NTMjgGmAv3d/QRgMdE2zcz6AZcBfYHzgQfMzEKJXERSUlySLXd/G9hcbfdI4Ino\n8yeAi+JxLRGRRHL3de7+YfT5DmAB0M3dX3f3yuhhs4Du0ecXAs+4e7m7LyNIxAYlOWwRSWGJnLPV\n2d1LIGi8gM4JvJaISNyZWU/gBOCdam9dDrwSfd4NWBnz3uroPhERILkT5LUukIikDTNrA7wAXB/t\n4ara/0ugzN3/ElpwIpJWchJ47hIz6+LuJWbWFVhf00FmpiRMJEO4e0bMVTKzHIJE60l3nxyzfzRw\nATAs5vDVwGExr7tH99V0XrV3IhmgoW1dPHu2LLpVeRkYHX3+Q2By9Q9UcfeU2MaNGxd6DKkWS6rE\noVhSOw73jMshHgXmu/u9VTvM7DzgJuBCd98bc+zLwLfMrIWZHQH0BmbXduKw/54y5d9bc/0O6R5/\nJnyHxohLz5aZPQ0UAh3NbAUwDrgDeN7MLgeWE9ytIyKS0szsdOC7wDwz+4BgCsQvgd8DLYBp0ZsN\nZ7n7GHefb2bPAfOBMmCMN7ZFFpGMFJdky92/U8tbZ8fj/CIiyeLu/waya3irzwE+cztwe8KCEpG0\npgryMQoLC8MO4XOpEkuqxAGKpSapEoc0D5nw7y3dv0O6xw+Z8R0aysLu7TYz9biLZAAzwzNkgnyi\nqL0TSX+NaevUsyUiIiKSQEq2RERERBJIyZaIiIhIAinZEhEREUkgJVsiIiIiCaRkS0RERCSBlGyJ\niIiIJJCSLREREZEEUrIlIiIikkBKtkREREQSSMmWiIiISAIp2RIRERFJICVbIiJJpHWoRZofJVsi\nIkm0c2fYEYhIsinZEhFJopKSsCMQkWRTsiUikkTr14cdgYgkm5ItEZEkUrIl0vwo2RIRSSINI4o0\nPzlhByAijROJRJgybQoAI84ZQf/+/UOOSOpDPVsizY96tkTSUCQS4eY7b2Zm2Uxmls3k5jtvJhKJ\nhB2W1IOSLZHmR8mWSJqJRCL8fNzPWdpyKbmdc+lyTBdy++d+3sslqU3DiCLNj5ItkTRS1aO1onIF\n67PW88aCN9i8eXPYYWUUM+tuZm+aWcTM5pnZddH9HcxsqpktNLPXzKx9zGduNbPFZrbAzM490PnV\nsyXS/CjZEkkjU6ZNIbd/Lt3O7Ma+8n3kbMvh45kfUxYpY8Q5I8IOL1OUAze6e39gCPATMzsGuAV4\n3d2PBt4EbgUws37AZUBf4HzgATOz2k6uni2R5kfJlkia2V65nZIWJZze/3Q6retEjw09mHjTRE2Q\njxN3X+fuH0af7wAWAN2BkcAT0cOeAC6KPr8QeMbdy919GbAYGFTb+dWzJdL8KNkSSSPHDj6WpSVL\n6bStE3l78+jZrid33XaXEq0EMbOewAnALKCLu5dAkJABnaOHdQNWxnxsdXRfjbZuhfLyREQrIqlK\npR9E0sSSzUt4c9ub3PPNe1jwzgIARtykkg+JYmZtgBeA6919h5lVX0K6UUtKt2gxnqIiaNsWCgsL\nKSwsbGqoIpJAxcXFFBcXN+kc5iEvQW9mHnYMIqlu+Zbl3Df7PkafMJoBnQeEHU6NzAx3r3WuUjox\nsxxgCvBPd783um8BUOjuJWbWFZju7n3N7BbA3X1C9LhXgXHu/k4N5/Vjj3WefBKOPz5530dE4qcx\nbZ2GEUVS3Kptq7h/9v18/7jvp2yilYEeBeZXJVpRLwOjo89/CEyO2f8tM2thZkcAvYHZtZ24c2fN\n2xJpbjSMKJLC1m5fy72z7uVbA77F8V3VFZIMZnY68F1gnpl9QDBc+AtgAvCcmV0OLCe4AxF3n29m\nzwHzgTJgzIG66zt31h2JIs2Nki2RFFWyo4R7Zt3DN/t/k5MOPSnscJoNd/83kF3L22fX8pnbgdvr\nc/4uXdSzJdLcaBhRJAWV7ipl0qxJjDxmJIO61VpFQNKQhhFFmh/1bImkmI27NnL3zLu5oM8FnHbY\nafu9P2fOHJ599jXWrVuNWTZdunRl1KjhDBw4MIRopaE6d4ZFi8KOQkSSScmWSArZsmcLk2ZN4uwj\nz2Zoj6H7vT9nzhyuvnoSe/aczpIls4DzOeKIQ5k+fRIPPjhWCVca0DCiSPOjYUSRFLFt7zbunnk3\nQ3sMZdgRw2o85tlnXyM7ezS7dm0mN/c6cnMvYffuvmRnj+bZZ19LarzSOBpGFGl+1LMlEqJIJMKU\naVPY63tZ120dI04cwbm9DriOsaS5Ll10N6JIc6OeLZGQRCIRbr7zZt7a9xZ/2f0XZhfPpse+Hgf8\nzKhRw6moeJxWrTpQVvZ7yspepGXLBVRUPM6oUcOTFLk0RadOQc+WajmLNB+qIC/NQtWkciBlJpNP\nuGcCb+97m5IOJbTPa0+rda04rcVpFN1QdMDPpeoE+UyqIJ8oVe1d27awahW0bx92RCLSUI1p6zSM\nKBmvalJ5dvZogJSZTF7mZSwpX0LXFl054qAjWF9Sv4k8AwcODD12aZqqSfJKtkSaBw0jSsarmlRe\nUHAWBQVnpcRk8r3le9nSYwu5q3NpU9KG9QvXUxYpY8Q5I0KNS5JDk+RFmhf1bIkkWVlFGQ+8+wDH\nHnkso48azT9e/wcAI24aQf/+/UOOTpJBS/aINC9KtiTjjRo1nOnTJ1FaGrwOJpOPDSWW8spy/ve9\n/6VdXju+f/z3ybIsBgzQ4tLNjWptiTQvSrYk4w0cOJAHHxwbM0E+nPlaFZUVPPz+w+Rl5/GjE39E\nlmkUv7nSMKJI86JkS5qFsCeVV3olf5rzJ9ydK066QolWM9elCyxYEHYUIpIsavFFEqzSK3nsg8fY\nW7GXq06+ipws/Y7T3KlnS6R5UbIlkkDuzpNzn2Tb3m1cc/I1SrQEULIl0twkvOU3s2XAVqASKHP3\nQYm+pkgqcHeenvc0G3Zt4NpB15KbnRt2SJIitGSPSPOSjF+zK4FCd9+chGuJ1CqZVeTdneciz7Fq\n2yquH3w9eTl5CbuWpB/1bIk0LwlfrsfMlgInu/vGWt7Xcj2ScNWryFdUPB7XKvJVC0oDfO3sr7Ew\nayELNy7khsE30Cq3VVyukeq0XE/dqtq7ykrIz4cdO6BFi7CjEpGGSNXlehyYZmYVwMPu/sckXFPk\nS2KryAOUlgb74pFsVS0onds/GCZ8/oHnGfLVIdwx8o5mk2hJw2RlQUEBbNgA3bqFHY2IJFoykq3T\n3X2tmXUiSLoWuPvbsQeMHz/+8+eFhYUUFhYmISyR+JgybQq5/XPpckwXVmxdwY59O+i0qhOtW7QO\nO7SEKi4upri4OOww0lbVUKKSLZHMl/Bky93XRh83mNlfgUFArcmWSCIko4r8qm2rWL9zPb1ye5Fn\nmT9Hq/ovRrfddlt4waQhLdkj0nwkNNkys1ZAlrvvMLPWwLmAWmSJq/pMfE9EFfmqeVola0tYsm4J\ne/ftpXdub5gfrHMociBaskek+Uh0z1YX4K9m5tFrPeXuUxN8TWlGqk98nz59Uq0T3+NZRT4SiXDV\nr66ipP0mdufvojxnDxfvuJieXXpqQWmpF92RKNJ8JDTZcvelwAmJvIY0b4mc+H4gDz3+EJGyZVir\nw9nbch85nxm5B+dSdENRg86TzHIUUn9m9ggwAihx9+Oi+04B/gDkAmXAGHd/L/rercDlQDlwfX1+\nqVStLZHmQxXkRRph7txFVLQ5mPK2u+iwbwjZFUcyd+6iBp2jqlduxoxBzJgxiKuvnsScOXMSFLE0\n0GPA8Gr7JgK/cvcTgXHAnQBm1g+4DOgLnA88YGZ13hauni2R5kPJlqS1UaOGU1HxOKWlb1Ba+kZ0\n4nv1/yPj76AjD2afl5K3vAvla7fjn1TQ+7BjGnSO2F65goKzyM4e/Xkvl4Qresd09ULMa4H20ecH\nAaujzy8EnnH3cndfBiwmuBHogJRsiTQfWqhN0lpTJr43dghv7rq5tDs9m5739WPXlsMB6JRfwU9+\n8oNGfANJI7cA/zaz3wEGnBbd3w2YGXPc6ui+A9IwokjzoWRL0l5jJr43ZGJ9bHX4YwYdw1s73uI3\nX/8NG/tvjEnWrmlwDMkoRyFx9Qhwrbv/zcwuBR4FzmnoSapK3WzdCitXFgKF8YtQROIuHjUFE75c\nT50BaLmeZqshPUvxnkheVHQ7M2YMiplY/wZDh85mwoRbv3RcbHX47ZXbWVqylAdGPcB5p54Xl/gy\naYJ8pi3XY2Y9gL/HTJDf5u7tYt7f4u4HmdktgLv7hOj+V4Fx7v5ODef8vL3bswfatw8e657hJSKp\nIlWX6xHZT0N6lhpybLxVVYfP75nP0tKlHMmRzJ0590vJVlPii2c5Cok7i25VFpvZme7+LzM7i2Bu\nFsDLwFNmNolg+LA3MLuuk+fnQ+vWwZI9nTvHO3QRSSVKtiQUDSnZkIjyDg0ZwttZuZOlpUs5puAY\n9u7Ym5T4JFxm9jTB+F5HM1tBcPfhfxHcadgC2BN9jbvPN7PngPl8URKiXt31AwfCe+/BBRck4EuI\nSMpQsiXNUn0n1g88bSAPPfMQPenJ3h17KYuUqTp8M+Du36nlrVNrOf524PaGXmfwYJg1S8mWSKZT\nsiWhqG/P0pw5cygpWceyZb9m587FtG7dJ24Tyesawlu5dSVTt0zlrkvuYvG7wYhRTdXhNdFdGmvw\nYPj978OOQkQSTRPkJTR1TQ6PnQu1Y8dOSkvv5dxz+3L99ZcnfIhuzfY1TJo5iW8f+20GHlL3tTJp\nontjZdoE+USo3t5t2AB9+sCmTZClqociaUET5CWt1NWz9OWin9CmTWu6dp2d8ESmZEcJ9866l2/2\n/2a9Ei3QRHdpnE6doKAAFiwALacpkrmUbEmzF1tH67Shp/GPTf9g5DEjGdStziLgIk02ZEgwb0vJ\nlkjmUse1pKy6luKZM2cORUW3U1R0e6PXFKyqozWzbCb/2vcvfvTkj+iX04/TDjut7g+LxEHVJHkR\nyVzq2ZKUdaA7BuNVe2vKtCns7LaT0q3b2JC1jq6tO7H2w7Uq6i1JM3gwPPhg2FGISCIp2ZKUVttc\nqHjUtopEIrz8ysu8v2keNqgDeXYYK5avZnXf1XV/WCROjjsOli6FbdugXbu6jxeR9KNkS1JWIu/w\nqxo+XNFpDfsOLyP70620phcVS7uzs4tG1yV5cnPhxBPh3XfhrLPCjkZEEkH/q0hKqhomnDFjEDNm\nDOLqqyd9aV5WXfO56jJl2hSsn1Heq4zWLXqRn3MoWYvzOLzVlRQUdEnEVxKpleZtiWQ2JVuSkr5c\n9uEssrNHf97LBV/M5xo6dDZDh85u8Hytfb6PJeVL6NGlO/l79pCb3YH83O7k5RU3KGkTiQclWyKZ\nTcOIknYaM7wYiUR45P8eYcHiBfTp1YetvbaS/2k+XXK6kH9IPivnr6PwK635yU+uqbG4anMvWCqJ\nNWQIXH01uIOpLKxIxlEFeYmreCUm1e82rKh4nAcfDJbAqWn/ga4TiUQYM24Mn+R+grUzduzcQfec\n7txx0R0sXLYQgBHn7L8Mz4HiUMK1P1WQr9uB2rvDD4fp06FXryQHJSINogryEqp4lWOA2ss+FBXd\n3uC7EKdMm8KGThvI75bPnvw95Jfmk7Uoi4XLFlJ0Q9EB44jHXY8i9TF4MMycqWRLJBMp2ZK4iXdi\nEs8lcBxnu28nz/LIL8vHUAeMpJaqeVvf+17YkYhIvCnZkrQyatRwpk+fRGlp8Dq4C3HsfsfFLsHT\np2cfyueXU766nBbegj0L9tCjQw9GnDMibtcTaarBg+Hpp8OOQkQSQXO2JG6SNb+prnlhVTW0cvvn\n4u4sWbmE0044jZaftmTh4oX07dWXK35wRY1ztBpzPQlozlbdDtTe7d4dLEq9YQO0apXkwESk3hrT\n1inZkrhKhcRkwj0TmFk2k85Hd2bhxoVsXruZb7X8Fr8Y+4ukx9KcKNmqW13t3aBBcPfdcMYZSQxK\nRBpEE+QldPGcZ9UU7s7iTYspqyyjZ05Psi077JBE6jRkSDBvS8mWSGZRsiUZo2qe1rq161i4cSE5\n5TkcmXMkFfMrGHFT3fOzRMI2eDC8+GLYUYhIvGkYUTJC1TytnH45rK5YzcalG/l6x6/TrUu3Wmto\nSXxpGLFudbV3S5fCaafBmjUqbiqSqjSMKM1SJBLh5+N+zvLK5bTPaU9WhywGZA+gW4tuddbREkkl\nRxwBBx0E770Hp5wSdjQiEi9aG1HSWlWP1vJOy1l72Frmz59Pt73dNEdL0tbFF8NLL4UdhYjEk5It\nSVtVPVpLWy4lv08+la0qyc/JZ9GbiyiLlNWrjpZIqvnGN4J5W5pdIZI5lGxJWqrq0VpRuYJ1uev4\nrPQzTj7kZDpaRw7POpyJN03UPC1pNDN7xMxKzOyjavuvNbMFZjbPzO6I2X+rmS2OvnduU6590kmw\nZw/Mn9+Us4hIKlGyJWkntker5cCWVOyuIH9TPmsXrOWI3Udw1213KdGSpnoMGB67w8wKga8Dx7r7\nscBd0f19gcuAvsD5wANmjZ/ebhb0bmkoUSRzKNmStBLbo1WSU8KiXYs4qedJFKwvoMeGHurRkrhw\n97eBzdV2XwPc4e7l0WOiizgxEnjG3cvdfRmwGBjUlOt/4xvw17825Qwikkp0N6KEpjHV5qdMm0Ju\n/1w653Zm5fyVtCxvybrN6zii3RFKtCTRjgKGmtn/ALuBn7v7+0A3YGbMcauj+xrt9NNh1aqgFMQR\nRzTlTCKSCpRsSSiqr6M4ffqkA66jWFWwtPjtYtYfuZ6KIys489gzWTJ9STBH6zYlWpJwOUAHdx9s\nZqcAzwNHNvQk48eP//x5YWEhhYWF+x2TnQ0jRwa9Wzfe2Oh4RSQOiouLKS4ubtI5VNRUQlFUdDsz\nZgyioOAsAEpL32Do0NlMmHDrfsfefffd/OZPt2NHGwd1a8vaLWs5seuJdOrSibJImXq0UkSmFTU1\nsx7A3939uOjrV4AJ7v6v6OvFwGDgSgB3vyO6/1VgnLu/U8M5693e/fOf8Nvfwttvx+PbiEi8NKat\n05wtSWl33303N99+K1ta7WBr+30sy1nBYe0Oo92ydgzJHaJESxLJoluVvwHDAMzsKKCFu28EXgZG\nmVkLMzsC6A3MburFhw2DSATWrm3qmUQkbEq2JBSjRg2nouJxSkvfoLT0DSoqHmfUqC/d/MV9991H\n0T23UHFcOX5MOZW+E9vYhtK1myk8o5CiG4qUaElCmNnTwH+Ao8xshZn9CHgUONLM5gFPAz8AcPf5\nwHPAfOAVYEw8uuvz8uCCC2Dy5KaeSUTCpmFECc2BJshPnjyZ79zwHXafvBvv4pANbMjFFmVz0K42\nvPV8sRKtFJNpw4iJ0ND27oUX4OGHYerUBAYlIg3SmLZOyZaknEgkwsU/vJhlOcvgBCjrVBbchL8g\ni+yPcph46+3cqFnDKUfJVt0a2t7t2AGHHgrLlsHBBycuLhGpP83ZkrRXVUdri2+h8tBKyveUk7sh\nFysxchZnK9GSZqVNm2Du1pQpYUciIk2h0g+SMiKRCFeNvYpFGxeR0z4Hb+Fk7ciCT6BlSUvuuOkO\nrr322lo/35i6XSKprmqtxB/8IOxIRKSxNIwoKSESiTBm3BjeL3uffQfto5JKWi5sSe6OXAryC7jz\n13cycuTIWj9fvW7Xtm2/45RT+tClS1clXkmSisOIZtYLWOXue6PL7RwH/Nndt4QUT4Pbu61boUcP\nWLQIOndOUGAiUm8aRpS0NWXaFFa2WYn3goruFWTlZ0EL6Ny+M3994q8HTLQAnn32NbKzR1NQcBa5\nuR1YvjyXqVMHMmPGIK6+ehJz5sxJzheRVPMiUGFmvYGHgcMI7iRMG+3bw0UXwZNPhh2JiDSWki1J\nCatXr2b1hg3syXWy93WgcgvkVuRywdALGnzX4Zo1rwE/oFWr0ygoOIvs7NGfDy9Ks1MZXcvwYuA+\nd78JOCTkmBrsiivgT38CDQKIpCclWxKKOXPmcPbZX6dDjwJ6DOjBv995D1q1JGtJHrYsF/u4Nbkb\n89m6vpKiotvr7JmKrdu1a9cyYAOHHqoxF6HMzL4N/BCommaeG2I8jXLGGVBZCTNn1n2siKSehM/Z\nMrPzgHsIErtH3H1Ctfc1Z6uZmTNnDl/72g9YlzsfBkVraJVBx2XDaLf7eHbsWUDl3n2wbwd9+vwP\nABUVjx9w7cSq8z777GusW7ea999fQ9u2P6n3Z6XpUnTOVj/gamCmu/8lWuH9surtUBLjaXR7N3Ei\nfPIJPPponIMSkQZJuTpbZpYFLALOAtYA7wLfcvdPYo5RstXMXHHFDTz61wfgrDI4OgvMYYWT+4/W\nnNQnKJe9bNmvKSi4hZ49LwQOvHZiTXRnYvKlYrIVy8w6AIe5+0chxtDo9m7dOjjmGFixAtq1i3Ng\nIlJvjWnrEl36YRCw2N2XA5jZM8BI4JMDfkrSQmMTmk9XfgK50R6tbKDSYCfk5+UwdGiwpNzRR5/C\nwoWtGx3bwIEDlWAJZlYMXEjQ1r0PrDezf7t72hVr69oVvvpVePZZuPLKsKMRkYZI9JytbsDKmNer\novskzVWVWpgxY1CD7/g7/vijsINzYA+w0uETh9kwqO9XmDDhViZMuJXrrvthnWsnitRDe3ffBnyD\noOTDqcDZIcfUaFdcAY88EnYUItJQKVHUdPz48Z8/LywspLCwMLRYpH5iSy0AlJYG++rTm/T9732f\n5zY+x/ollfhMoMzomtebiRNv+/yYgQMH8uCDY2N6zjTnKtUUFxdTXFwcdhh1yTGzQ4DLgF+GHUxT\nnXceXHUVfPwxDBgQdjQiUl+JTrZWA4fHvO4e3fclscmWZLZdZbuYunUqt3zrFj6b9hkftVhM78OO\n4Sc/+cF+yZSGAlNb9V+MbrvtttoPDs//A14D/u3u75rZkcDikGNqtJwcGD066N2aNCnsaESkvhI9\nQT4bWEgwQX4tMBv4trsviDlGE+TTUPWK7fW5429P+R4mzZxEr4N78c1+38QsZedSSyOk+gT5VBCP\n9u6zz2DwYFi1CvLy4hSYiNRbyt2NCJ+XfriXL0o/3FHtfSVbaaohE+T3lu/l3nfupXu77nx7wLeV\naGWgVEy2zKw7cB9wenTXW8D17r4qpHji0t4NGwZXXw2XXRaHoESkQVIy2aozACVbGW9fxT7un30/\nBa0K+P5x31eilaFSNNmaRrA8T9ViN98Dvuvu54QUT1zau2eegQcfhNSfMieSeZRsZbh0rB1VVlHG\nA+8+QNu8tow+YTRZpkULMlWKJlsfuvsJde1LYjxxae/KyqBXL3jpJTj55DgEJiL1poWoM1hTSi2E\npbyynIfff5iWuS2VaElYNprZ98wsO7p9D9gYdlBNlZsLN9wAd90VdiQiUh8pUfpB6taUUgvJFIlE\nmDJtCpVeye5euznkkEO44sQrlGhJWC4nmLM1CXDgP8DoMAOKlx//GH77W1i2DHr2DDsaETkQ/Q8o\ncROJRLj5zpv5z77/8Nye53jx9Rc5Lf80srOyww5Nmil3X+7uF7p7J3fv7O4XAZeEHVc8tGsXJFwq\nASGS+pRspYlRo4andEX1yZMnc/EPL2b2ztl8lvUZLTu3pFf3Xrz6xqthhyZSXdot1VOb666DJ5+E\nTZvCjkREDkTDiGkizIrqdU3Mnzx5MleOv5JdbXexp+MetqzYQmFeoYYOJVWl1CT+pujWDS68EB56\nCG6t3xrtIhIC3Y0oB1Sf4qXnX3o+H3T8gMrOlWxZvgUq4aC9B3FK61OYeNNE+vfvH07wklSpeDdi\nTcxshbsfXveRCbl23Nu7efNg+HBYulRFTkWSQXcjStzFTswvKDiL7OzRn/dyRSIRJtwzgU+XfcrO\n8p3QDg49/FDyVuRx0NKDlGhJaMxsu5ltq2HbDhxaj88/YmYlZvZRDe/9zMwqzezgmH23mtliM1tg\nZufG+esc0LHHwnHHwVNPJfOqItIQSrakUWInw9tZxq6Nu6h4t4K9G/bScntL7vz1nUq0JDTu3tbd\n29WwtXX3+kyfeAzYb1JktCL9OcDymH19CRa67gucDzxgSa7ce9NNQRmIyspkXlVE6kvJlhxQ9Yn5\nO3bcw+7Ktfx83M/Zc/ge9h66l3a92nFirxPpuKQjJ2w8gT+O/yMjR45s1PXmzJlDUdHtFBXdnvJ1\nxCRzufvbwOYa3poE3FRt30jgGXcvd/dlBAtdD0pshF82bFgwhPjKK8m8qojUlybIywHFTswvLS1h\n4abNrOiczYrPVrB6zWq6dezGoB6D2Lx1M0O+N4SiG4oafa3q88OmT59U5+LWIsliZhcCK919XrWO\nq27AzJjXq6P7khgb/OIXMH48fO1rwWsRSR1KtqROAwcOZODAgUy4ZwIbywrockwXNuVsYtncZZQv\nLGfzvs2URcoYcdOIJl0nXQq3SvNjZi2BXxAMITbJ+PHjP39eWFhIYWFhU08JwCWXwB13wIsvwqWX\nxuWUIgIUFxdT3MSFSJVsSYOt2b6G7S23M7DjQNosb8OQo4Yw4qYRmqMlmawX0BOYG52P1R2YY2aD\nCHqyYu9u7B7dV6PYZCuesrLgf/4Hrr8eLroIctS6i8RF9V+KbrvttgafQz+OUqeqJXhKSkqY/8l8\nNvTcRIceGWcDAAAgAElEQVStB9N6c2vu+s1d9O/f//O5VtD4RbJHjRrO9OmTKC0NXgeFW8fG86uI\nNIRFN9z9Y6Dr52+YLQUGuvtmM3sZeMrM7iYYPuwNzA4hXs49Fw45BJ54Aq64IowIRKQmqrPVzNVV\nsLTqrsPc/rms2LqCjz/5hIPmDSPfD6F93jqeeCLI8OuqxRWveCR1pUudrfows6eBQqAjUAKMc/fH\nYt5fApzs7puir28FrgDKgOvdfWot5014ezdrFnzzm7B4MeTnJ/RSIs1SY9o6JVvNWF0FSyORCD8f\n93OWd1rOoQMP5aOVEfZ93J1Dll5K7+5FlJa+wdChwS/wM2YMiplrFeyfMEElrZuTTEq2EiVZ7d3I\nkXDmmXBjxixMJJI6VNRUGuRABUsnT57MJWMu4f0N77O2ci2zl86m7e72ZJW1CDVmEanbb38LEybA\ntm1hRyIioGRLajB58mR+fNOPWd15NeX9y9m+Zzs5G3LwDfuo/GQZLSoLvrQYdqovki3S3AwYECzh\n87vfhR2JiICGEZu1moYRb7rpQn59/69Z7aspO6qMioMraLe9Hdnzszmp00lcPupy3nvvU+DLc6o0\n10o0jFi3ZLZ3S5fCySfDggXQuXNSLinSLGjOljRYbJJ08sm9efTZR3l/w/uUH1HO9uztZJNN7rZc\nuq3vxosPvKjyDlIrJVt1S3Z7d8MNsGMH/OlPSbukSMZTsiWNVnXX4dKWS1lXsY5te7fR1tpSvqSc\n/HX5/OnOPzV6CR5pHpRs1S3Z7d3WrdCvHzz3HJx+etIuK5LRNEFeGqXqrsOlLZdScEwB5QXltM1v\nS+7SXLpZNyVaImmqfftg3tY110BZWdjRiDRf6tlq5iZPnkzRxCI27dlE2dFlVPSsYEDnAWxZuIUe\nG3pw12131WvoUHO2RD1bdQujvXMPip2edx787GdJvbRIRtIwYjMRr8QmEolwyZhL2HrMVipzKykt\nLSWnPJc2Fe3oy1E89JuH6p1oxauoqaQvJVt1C6u9W7wYhgyBDz6Aww5L+uVFMoqGEZuBqsRmxoxB\nzJgxiKuvnsScOXMada4p06aQ1S+L7MOy2XPoHiw3h8r3W1IxuydbP+3A3r1763WeA9XrEpHw9ekD\nP/1pMGFeRJJPyVaaiXdi07FTR7bt2kb5lkpsd2taVHThhD6P0abNDUqYRDLILbfARx/BK6+EHYlI\n86Nkqxk7fejprNm+hj45fchf3JKc99rQr8sE2rZtWHkHFTUVSX35+fCHPwQ9XLt2hR2NSPOiOVtp\npinzoyKRCFOmTQHgK2d+hX9s+ge9s3qzfu56Vq9ezfRXl9KmzQ0NPm9VXJog37xpzlbdUqG9+973\ngrsU//CHUMMQSVuaIN9MNCaxqaqjlds/lzIvY9GaRRRdWMSPh/24SecVqaJkq26p0N5t2QInnAD3\n3w8jRoQaikhaUrIlNaqqo7W803KOPvVoVuxdQYvNLbgo/yKKbiiq9XNKvqQhlGzVLVXau7ffhm9+\nM7g7sWvXsKMRSS+6G1H2U9WjtaJyBRt9I28ufJOW3pIu2V0O+Ll43vUoIqnljDPgyith9GiorAw7\nGpHMp2Qrw02ZNoXc/rkcNewo9pTvIWtXFhvnbqQsUsaIc2ofQ1A5B5HM9utfB0OK998fdiQimS8n\n7AAkcSKRCMVvF7PMl5EzOId+/fqx9e2t9MjqwcTbJmpRaZFmLDcXnnoKBg+Gr34Vjj027IhEMpeS\nrQwViUS46ldXsaLVGtbkrySr2OiS050eWYfWawmeUaOGM336JEpLg9dBOYexSYhcRJKlVy+46y74\n9rfhnXegdeuwIxLJTBpGzFAPPf4QH5cvpbRjCyq9G2U7WrD5nZb1rgw/cOBAHnxwLEOHzmbo0Nla\nfkckQ/3gB3DyyZq/JZJIuhsxQ31l+Lm8330x3qITFRuOxjcuod2ithx1+E0MHTqbCRNuDTtEyTC6\nG7Fuqdre7dkDw4bB8OEwblzY0Yiktsa0dRpGzEDlleWUH1MGiwyraIuXlcLCClrnHBN2aCKSgvLz\n4aWX4NRToV+/oCyEiMSPkq00VlMdrIrKCv74/h/56plD2P5ma0q3dWTT5m3klLfkoN7HaO6ViNSo\na1f429/g3HOhd2848cSwIxLJHBpGTFM1LdvzwP9ezwd8wN6KvVx98tV89OFHPPvsa5SUrMO9gq5d\nu6k4qSSMhhHrlg7t3QsvwI03wuzZKngqUhNVkG9GiopuZ8aMQRQUnAXAhtJpHPzVhxh+4VcYc8oY\ncrNzG31uVY6XxsikZMvMHgFGACXuflx030Tg68Be4DPgR+6+LfrercDlQDlwvbtPreW8adHe3XYb\n/POf8Prr0KZN2NGIpBZVkG9mdu1axqerJrB41R2sKXiG3baLa065psmJlirHi/AYMLzavqlAf3c/\nAVgM3ApgZv2Ay4C+wPnAA2aW1knnf/839O8PI0fC7t1hRyOS/pRspaFIJMKqDR+zsHQsq9v/hVXH\nPcqGrOe58rjv0SK7RZPOrcrxIuDubwObq+173d2riiPMArpHn18IPOPu5e6+jCARG5SsWBPBDB5+\nGLp0gUsugXpUixGRA1CylWaq1jr8IO8DWg/PpazDIloWbOOUASewcu3KsMMTaS4uB16JPu8GxP7w\nrY7uS2vZ2fDEE8Gdit/5DpSXhx2RSPrS3YgprPrcqby8PH4+7uesqFxB1kFZZB+aTV6XFvTY3J2D\nux4cl2uqcrzIgZnZL4Eyd/9LYz4/fvz4z58XFhZSWFgYn8ASIDcX/vIXuPhi+OEP4c9/DpIwkeak\nuLiY4uLiJp1DE+RTVPW7DXfsuIf2vTezqesmNlduZs/mPZQfUk6rva0oWF9Az3Y9mXjTgdc7rO/E\nd02Ql8bIpAnyAGbWA/h71QT56L7RwJXAMHffG913C+DuPiH6+lVgnLu/U8M507K9270bvvY16NED\n/vhHyNGv6dKM6W7EDFL9bsO5C68k77R/c+KwAUyPTGd3xW46LexE3rY8Lhh8AVf84Io6E63qpSK0\nBI/EUwYmWz0Jkq1jo6/PA34HDHX3jTHH9QOeAk4lGD6cBvSpqWFL5/Zux45g/lZeHjzzDLRqFXZE\nIuFQBfkMtX17hG0755K9tZSS3SV0OrQTFQsrODLvSO56oO5FpeHLE98BSkuDfUq2RPZnZk8DhUBH\nM1sBjAN+AbQApkVvNpzl7mPcfb6ZPQfMB8qAMWmbUR1Amzbw97/DFVfA2WcHzzt2DDsqkfSQsAny\nZjbOzFaZ2Zzodl6irpWJRo0aTkXF46xY8QhzV19BZe/llO3bw3sfv0fBxgJ67+rNXbfVL9ESkYZx\n9++4+6Hunufuh7v7Y+7ex917uPvA6DYm5vjb3b23u/etrcZWJmjRIpg0/5WvwBlnwPLlYUckkh4S\nfTfi3TEN06sJvlZGGThwIDfddCHbKiaQ1XkJR519GIeeeSjd13Wn9fzWdc7Pqq4qeSstfYPS0jei\nE9+rlxESETmwrCyYMAGuuipIuObODTsikdSXsDlbZjYO2OHuv6vjuEzscW+yyZMnUzSxiE05m9jb\nYy/e0jlzwJlk78lmSO4Qim4oavA5NfFdEinT5mwlQqa1d88+Cz/9KdxzD3z3u2FHI5IcKTVBPpps\njQa2Au8BP3P3rTUcl1GNTzxEIhEuGXMJW4/ZSnmrcjZv3Ey7snYcsveQet11KBIGJVt1y8T2bu5c\nuPRSGD4cfve7YAK9SCZL+gR5M5sGdIndBTjwS+AB4P+5u5vZb4C7gStqOk861Z1JhinTppDVLwvr\nZlS2qKTAC6j8oJLDOx3eLBIt9cClh3jUnpH0d/zx8O67MHo0nHkmPP88HHZY2FGJpJaklH6oqV5N\nzHsZ95teU024ZwIvbXqJBeULaNeqHRUrK2j/cXtefPjFZpFoqURFelLPVt0yub1zhzvvhLvvhkcf\nhQsuCDsikcRIqYWozaxrzMtvAB8n6lqZpu+pfdmwYwMDcgbQbnU78j7Mo1+XU/jzn1/O+EWhtTaj\nSHoyg5tvDuZx/eQn8KMfwZYtYUclkhoSeTfiRDP7yMw+BM4EtOZLPSwsXciM7TP4w6g/MPLgkZzd\n5mza7jqWtWtHM2PGIK6+elLGJ1wikr7OPBPmzQuKnh57LLzySt2fEcl0qiCfQj7d9CkPvvcg/3XS\nf3FUx6OA/SvJl5a+wdChs5kw4dYwQ00YDSOmLw0j1q25tXdvvhkUQS0sDCbPHxyfJVxFQpVSw4jS\nMEs3L+XB9x7kihOv+DzRCtOcOXMoKrqdoqLbk9qTNnDgQB58cCxDh85m6NDZSrRE0tiwYUEvV9u2\n0Lcv3H8/lJeHHZVI8qlnKwWs2LqC37/ze0afMJoBnQd86b0wenrUuySNoZ6tujXn9m7ePBg7Ftau\nhUmT4Nxzw45IpHFSqs5WvQNoxo0PwKptq7h31r1897jvckLXE2o8JtmlEJrb0KXEh5KtujX39s4d\nXn4ZfvazoKfrt7+F4/a7R10ktWkh6jSzdvta7p11L6MGjKo10YJgaE29SiKS7sxg5Eg47zz4wx+C\nQqhDhsB//zecUHsTKJL2NGcrJOt3rueeWfdwab9LOfnQk8MO50u0jqKIJFJeHtx4I3z2WbCo9QUX\nBEnY+++HHZlIYmgYMUkikQhTpk0B4PShpzNl0xS+ftTXOf3w00OOrGaq4i4NpWHEujWX9q6hdu+G\nP/4RJk6EXr3g2mvhoosgR2MvkoI0ZytFRSIRbr7zZnL757LP97Fo7SJ+eeEv+dFXfxR2aCJxo2Sr\nbs2hvWuKsjL461/hvvtg2TK45hq48kro1CnsyES+oNIPKWrKtCnk9s/loN4Hsa79Orp06cL6uevD\nDktEJKXk5sJll8FbbwUT6T/7DPr0gW98AyZPhn37wo5QpHGUbCVJmZcxb/08urbpSqds/ZomInIg\nJ54IjzwCy5fD174WrLnYrRv89KcwcyZUVoYdoUj9KdlKgmFfHcbiNYvJ3ZxLizUtKIuUMeKcEWGH\nJSKS8tq3D6rQ/+tfMHs2dOkCP/4xHHYYjBkDr78eDD+KpDLN2UqA2Mnww746jNe2vka7Pe3YFdmF\nmTHinBH0798/5ChF4ktztuqWie1dWBYtCuZ3vfQSfPppUEbi3HPhnHOCHjCRRNEE+RQQOxm+witY\ntHoRV11wFWPPHYuZ/h+SzKVkq26Z1t6litWr4Z//hGnTgp6uQw4Jkq5hw+C006Bjx7AjlEyiZCsF\nTLhnAjPLZtLxqI58vP5jKkoruDT/Us4989wGlVJQ6QVJN0q26pZp7V0qqqiAOXNg6tRg6HHWrGDI\n8Ywzgu3UU6F3b8jSJBppJCVbKWDCPRP4975/s+HgDbTMaUnb9W3psaEHs/61ud5rDWptQklHSrbq\nlmntXTooL4ePPoK33w7ucnz3XdiyBQYOhJNOCrbjjw/uelRdL6kPLdeTAoafNZwn/vgEeRV5dM7u\nTPn8cnZmZ5GdPTpmrUF49tnXak2enn32tQYdLyIiNcvJCRKrgQPhuuuCfaWlQbX6996DZ5+FX/0K\n1qyBo4+GAQPg2GOD50cdFRRZbdEi3O8g6U/JVhyVV5bz1q63GHXuKFosbkGWZTHiphH8+c8vhx3a\nAWnIUuTLzOwRYARQ4u7HRfd1AJ4FegDLgMvcfWv0vVuBy4Fy4Hp3nxpG3FI/BQXBhPrhMauQ7dwJ\n8+fDxx8H24wZwST8FSuge/cg8TriiC9vPXtChw7Bmo8iB6JhxDipqKzgofcfIsuyuHLglWRnZX/+\nXkOHBZM5jKghS4mXTBpGNLMzgB3An2OSrQnARnefaGZFQAd3v8XM+gFPAacA3YHXgT41NWyZ0t41\nJ/v2wdKlsHhx8LhkSfC4dGlQA6ysLEjGDjss2Lp1g65dg0n6VY+dO0Pr1krKMoXmbIWk0iv505w/\nUVZRxlUnX0VO1v4dhg3tPUpWb1NR0e3MmDEoZsjyDYYOnc2ECbcm5HqSuTIp2QIwsx7A32OSrU+A\nM929xMy6AsXufoyZ3QK4u0+IHvdPYLy7v1PDOdO+vZMv274dVq2ClSuDbc0aWLsW1q0LHteuhfXr\nwT1YdqhTpyD56tgRDj74i8eDD4aDDvpia98+eFSSlno0ZysElV7J4x8+zu6y3Yw5ZQw5WTk1JkpV\nW3019HgRSbjO7l4C4O7rzKxzdH83YGbMcauj+6QZaNsW+vYNtgPZuTNIujZsCLZNm4Jt40ZYuDB4\nvmVLsG3d+sXzPXugTRto1y64Vrt2QQLWps0XW+vW0KrVF1vLll/e8vO/eKza8vKCxxYtgue5ubpD\nM5GUbDWBu/N/H/0fW/Zs4dpB15KbnbvfsNz06ZNSelhu1KjhTJ8+idLS4HVFxeOMGjU23KBE0kOj\nuqjGjx//+fPCwkIKCwvjFI6kstatv5jr1RDl5bBjB2zbFvSibdsWJG47dnz5cdeu4Pn69cHrPXtg\n9+5gq3q+Zw/s3Rs8Vm379n2x5eYGyVeLFl88z82tecvJ+eIxdsvO3v+xti0ra//nWVkH3sy+/Fh9\nX/X3q79X0wa1v5edDXl5xRQXFzfp71/DiI3k7vzl47+wettqrjv1OvJy8oD0HJbTBHmJh2YwjLgA\nKIwZRpzu7n1rGEZ8FRinYURJJ+7B/LO9e4PHffu+eNy3L0j6ysqCrep5efmXt7KyoM5ZRcUX+6pe\nV98qK/d/HbtVVAQx1bSv+v7YfbGPscdVva6+VX332racHPj737/8Z6VhxCRxd56f/zwrtq7ghsE3\nfJ5opSsNWYrUyKJblZeB0cAE4IfA5Jj9T5nZJILhw97A7OSFKdJ0Zl/0akn8KdlqIHfnr5/8lcUb\nFzN2yFjyc/K/9L6G5UTSn5k9DRQCHc1sBTAOuAN43swuB5YDlwG4+3wzew6YD5QBY9R9JSKxNIzY\nQH9f+Hc+WPcBPxvyM1q3aF3jMRqWk+Yo04YREyHd2jsR2Z9KPyTYPxf/k3dWv8PPhvyMtnltww5H\nJKUo2apbOrV3IlKzxrR1utGznqZ9No3/rPwPYwePVaIlIiIi9aZkqx6mL51O8bJibhxyI+3z24cd\njoiIiKQRJVt1eGv5W0z9bCo3DrmRDi07hB2OiIiIpBklWwcwc+VM/rH4H4wdMpaOrTqGHY6IiIik\nISVbtXh39bv87ZO/ccPgG+jcunPdHxARERGpgZKtGsxZO4fnIs9x/eDr6dqma9jhiIiISBpTslXN\nRyUf8fS8p7nu1Os4tO2hYYcjIiIiaU7JVozI+gh/nvtnfjropxzW/rCwwxEREZEMoGQr6pPST3js\nw8e45uRr6HlQz7DDERERkQyhCvJRCzYsIDsrm6M6HhV2KCJpSRXk65Yq7Z2INJ6W6xGR0CjZqpva\nO5H0p+V6RERERFKMki0RERGRBFKyJSIiIpJASrZEREREEkjJloiIiEgCKdkSERERSSAlWyIiIiIJ\npGRLREREJIGUbImIiIgkkJItERERkQRqUrJlZpea2cdmVmFmA6u9d6uZLTazBWZ2btPCFBFJDdG2\nLWJmH5nZU2bWwsw6mNlUM1toZq+ZWfuw4xSR1NHUnq15wMXAv2J3mllf4DKgL3A+8ICZpfyaacXF\nxWGH8LlUiSVV4gDFUpNUiaO5MLMewJXAie5+HJADfBu4BXjd3Y8G3gRuDS/KxMmEf2/p/h3SPX7I\njO/QUE1Kttx9obsvBqonUiOBZ9y93N2XAYuBQU25VjKk0j+AVIklVeIAxVKTVImjGdkG7ANam1kO\n0BJYTdDmPRE95gngonDCS6xM+PeW7t8h3eOHzPgODZWoOVvdgJUxr1dH94mIpC133wz8DlhB0K5t\ndffXgS7uXhI9Zh3QObwoRSTV5NR1gJlNA7rE7gIc+KW7/z1RgYmIpBozOxIYC/QAtgLPm9l3CdrE\nWNVfi0gzZu5NbxPMbDrwM3efE319C+DuPiH6+lVgnLu/U8Nn1SiJZAh3T/m5mU1hZpcB57j7ldHX\n3wcGA8OAQncvMbOuwHR371vD59XeiWSAhrZ1dfZsNUDshV8GnjKzSQTDh72B2TV9KNMbZxHJKAuB\nX5tZPrAXOAt4F9gBjAYmAD8EJtf0YbV3Is1Tk5ItM7sIuA8oAKaY2Yfufr67zzez54D5QBkwxuPR\nhSYiEiJ3n2tmfwbeByqAD4CHgbbAc2Z2ObCc4G5sEREgTsOIIiIiIlKzlKkgb2bXRgugzjOzO1Ig\nnp+ZWaWZHRzS9SdG/zw+NLMXzaxdCDGcZ2afmNkiMytK9vVj4uhuZm9GC0nOM7PrwoolGk+Wmc0x\ns5dDjqO9mT0f/XcSMbNTQ4pjvyKfYcSRylLlZ6khzOwRMysxs49i9qVN8dba2o00+w55ZvaOmX0Q\n/R7/E92fNt8B9m8z0zD+ZWY2N/r3MDu6r0HfISWSLTMrBL4OHOvuxwJ3hRxPd+AcguGAsEwF+rv7\nCQR1ypJaJNHMsoD7geFAf+DbZnZMMmOIUQ7c6O79gSHAT0KMBeB6giHysN0LvBKdiH08sCDZAdRS\n5PNbyY4jlaXYz1JDPEYQc6x0Kt5aW7uRNt/B3fcCX3X3E4HjgGFmdjpp9B2iqreZ6RZ/JcENMCe6\ne1XN0AZ9h5RItoBrgDvcvRzA3UtDjmcScFOYAbj76+5eGX05C+ie5BAGAYvdfbm7lwHPEBRuTDp3\nX+fuH0af7yBIKkKp2xZNxC8A/hTG9WPiaAd8xd0fA4gWEN4WQijVi3y2AtaEEEcqS5mfpYZw97eB\nzdV2p03x1lraje6k0XcAcPdd0ad5BP9nbyaNvkMtbWbaxB9l7J8vNeg7pEqydRQw1Mxmmdl0Mzs5\nrEDM7EJgpbvPCyuGGlwO/DPJ16xemHYVKVCY1sx6AicA+5URSZKqRDzsyY5HAKVm9li0e/5hM2uZ\n7CBqKPK5JVrkU76Qkj9LjdQ5HYu3xrQbs0izArTRIbgPgHVAsbvPJ72+Q01tZjrFD0Hs08zsXTP7\ncXRfg75DPEs/HJDVXhz1V9E4Orj7YDM7BXgOODKkWH5BMIQY+16y4/i8YKyZ/RIoc/enExVHujCz\nNsALwPXR31STff2vASXu/mF06DvM2/hzgIHAT9z9PTO7h6Bbe1wyg7D9i3y+YGbf0b/XZiPsXzrq\nVL3dsP1rnaX0d4iOcJwY7c1+Ldr2pMV3qKHNrE1Kxh/jdHdfa2adgKlmtpAG/h0kLdly93Nqe8/M\nrgZeih73bnRiekd335jMWMxsANATmGtmRtDl/L6ZDXL39cmKIyae0QTdr8Pife16WA0cHvO6e3Rf\nKKJDVC8AT7p7jTWMkuB04EIzu4BgTby2ZvZnd/9BCLGsIuiBfS/6+gUgjInXJwP/dvdNAGb2EnAa\noGTrCyn1s9REJWbWJaZ4a9zbxXiqpd1Iq+9Qxd23mdkrBD9z6fIdamoznwTWpUn8ALj72ujjBjP7\nG8HUgAb9HaTKMOLfiCYUZnYUkJuoROtA3P1jd+/q7ke6+xEE/6GdmIhEqy5mdh5B1+uF0UmSyfYu\n0NvMekTvLvsWQbHasDwKzHf3e8MKwN1/4e6Hu/uRBH8eb4aUaBHtvl4Z/XmBoLhmGJP2FwKDzSw/\n+gvKWYQwUT/FpdrPUkMY+xesHh19Xmvx1hRSU7uRNt/BzAqq7nKLThM4h6C2W1p8h1razO8DfycN\n4gcws1bR3lHMrDVwLjCPBv4dJK1nqw6PAY+a2TyCqsyh/AdWAye8oaL7gBYE48QAs9x9TLIu7u4V\nZvZTgrsis4BH3D2U/0Sjd998F5gXnbvgwC/c/dUw4kkh1xGs1JALLAF+lOwADlDkU6JS6WepIczs\naaAQ6GhmKwiGqO8gWA8y5Yu31tZuEFT5T5cCtIcAT0R/kcki6KF7I/p90uU71OQO0if+LsBfo8PP\nOcBT7j7VzN6jAd9BRU1FREREEihVhhFFREREMpKSLREREZEEUrIlIiIikkBKtkREREQSSMmWiIiI\nSAIp2RIRERFJICVbIiKStszsYDP7ILpG6FozWxXzul61JM3sETPrU8cxY8zs2/GJusbzXxxTpFgy\njOpsiYhIRjCz/wZ2uPvdNbxnnsL/4UWXsXkhxOXIJIHUsyUiIpni8xU/zKyXmUXM7P/M7GOgq5k9\nZGazzWyemf0q5ti3zOw4M8s2s81mdruZfWhm/zazgugx/5+ZXRdz/O1m9o6ZLTCzwdH9rczsBTP7\n2MyeN7N3zey4/YI0uzMa24fR85xBsA7u3dEeucPNrLeZvRo9R7GZ9Y5+9kkze8DM3jOzT6JLu2Fm\nA6LfbU70vD0T9qcsDZYqy/WIiIjE29HA99z9AwAzK3L3LWaWDUw3sxfc/ZNqn2kPTHf3W83sd8Dl\nwMSaTu7up5rZ1wmWMjofuBZY6+6XRpOs96t/xsw6A+e7e//o63Yxi0w/7+4vR/e/CVzh7kvN7DTg\nD8Dw6Gm6u/vJ0WHH182sFzAGuNPdn48u4RXWUnNSAyVbIiKSqT6rSrSivhtdyy6HYN3BfkD1ZGuX\nu0+NPn8fOKOWc78Uc0yP6PMzCNb9w90/MrNIDZ/bBFSY2cPAK8CU6gdEF58eDLwYXRcRvjwS9Vz0\nGoui61b2Af4D/Drao/WSu39WS9wSAg0jiohIptpZ9SQ6DHcdUOjuxwOvAfk1fGZfzPMKau+U2FuP\nY/brXXL3cuBk4G/ARcA/avncBncf6O4nRrfjY09T7Vh39/+Lnm8v8Gp0aFJShJItERHJVLHJTjtg\nG7DDzA7hiyG5A32mof4NjAIws2OBvvud3KwN0N7dXwFuBE6IvrU9GiPuvgVYa2YXRT9j1eZ+fTO6\n/yigO7DYzI5w9yXu/nuC3rL95opJeDSMKCIimerzHiB3n2NmC4AFwHLg7ZqOq/a8zvNWcx/wRHRC\n/vzotrXaMe2Bl8wsjyCxGxvd/xfgITO7kaCH6lvAg2Y2HsgF/g/4KHrsajN7D2gNXOnu5Wb2nWhp\nig7DR7AAAAB7SURBVDJgNcE8MkkRKv0gIiISB9GJ9znuvjc6bPka0MfdK+N4jSeJmUgv6UE9WyIi\nIvHRBngjppjqf8Uz0YpSD0kaUs+WiIiISAJpgryIiIhIAinZEhEREUkgJVsiIiIiCaRkS0RERCSB\nlGyJiIiIJJCSLREREZEE+v8BDWQFu2iG7q0AAAAASUVORK5CYII=\n", 194 | "text/plain": [ 195 | "" 196 | ] 197 | } 198 | } 199 | ], 200 | "execution_count": 0 201 | }, 202 | { 203 | "cell_type": "markdown", 204 | "metadata": { 205 | "id": "vNtkU8h18rOv", 206 | "colab_type": "text" 207 | }, 208 | "source": [ 209 | "In the remainder of this notebook, we'll go through this example in more detail." 210 | ] 211 | }, 212 | { 213 | "cell_type": "markdown", 214 | "metadata": { 215 | "id": "r6rsv-q5gnn-", 216 | "colab_type": "text" 217 | }, 218 | "source": [ 219 | "## From the beginning\n", 220 | "\n", 221 | "Let's walk through exactly what this is doing from the beginning. We'll start with what the data looks like, then we'll look at this neural network, what is executed when, what gradient descent is doing, and how it all works together." 222 | ] 223 | }, 224 | { 225 | "cell_type": "markdown", 226 | "metadata": { 227 | "id": "UgtkJKqAjuDj", 228 | "colab_type": "text" 229 | }, 230 | "source": [ 231 | "## The data\n", 232 | "\n", 233 | "This is a toy data set here. We have 50 (x,y) data points. At first, the data is perfectly linear." 234 | ] 235 | }, 236 | { 237 | "cell_type": "code", 238 | "metadata": { 239 | "id": "-uoBWol3klhA", 240 | "colab_type": "code", 241 | "colab": { 242 | "autoexec": { 243 | "startup": false, 244 | "wait_interval": 0 245 | }, 246 | "output_extras": [ 247 | { 248 | "item_id": 1 249 | } 250 | ] 251 | }, 252 | "cellView": "form", 253 | "executionInfo": { 254 | "elapsed": 398, 255 | "status": "ok", 256 | "timestamp": 1446659128547, 257 | "user": { 258 | "color": "#1FA15D", 259 | "displayName": "Michael Piatek", 260 | "isAnonymous": false, 261 | "isMe": true, 262 | "permissionId": "00327059602783983041", 263 | "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", 264 | "sessionId": "4896c353dcc58d9f", 265 | "userId": "106975671469698476657" 266 | }, 267 | "user_tz": 480 268 | }, 269 | "outputId": "efef4adf-42de-4e6f-e0c3-07ddd3083d85" 270 | }, 271 | "source": [ 272 | "#@test {\"output\": \"ignore\"}\n", 273 | "num_examples = 50\n", 274 | "X = np.array([np.linspace(-2, 4, num_examples), np.linspace(-6, 6, num_examples)])\n", 275 | "plt.figure(figsize=(4,4))\n", 276 | "plt.scatter(X[0], X[1])\n", 277 | "plt.show()" 278 | ], 279 | "outputs": [ 280 | { 281 | "output_type": "display_data", 282 | "metadata": {}, 283 | "data": { 284 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQMAAAEACAYAAAC3RRNlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEGpJREFUeJzt3XuMHeV5x/Hv4xinBgIN2Yq0WCFFJK1EUzC3otAWC4JB\nRHESkUq0qRKIBG2jAE1cZBKoQL0kBGSRtEr/IBdE2iDUEkhMLrJNqS1BBeF+dYA2KbcALYUWIVyu\nT/8447Ase9a7O+/MOTPn+5Esn4Nn552F9Y9553nP80ZmIklLRn0BksaDYSAJMAwkVQwDSYBhIKli\nGEgCCoVBRHw2Iu6NiLsi4lsRsazEeSW1p3YYRMS+wKnAysz8TWApcFLd80pq19IC53gWeBHYLSJe\nBXYFflbgvJJaVPvOIDOfAdYDDwOPAf+TmdfWPa+kdpWYJuwHfBrYF/gVYPeI+IO655XUrhLThEOB\nGzLzaYCIuAp4L3D59IMiwg9BSCOSmbGzY0pUE+4HjoiIX4iIAI4Btg25oNZ+nXfeeY7X0fH6/L2N\nYrz5KvHM4E7gm8CtwJ1AAJfUPa+kdpWYJpCZFwEXlTiXpNHo7QrEVatWOV5Hx+vz9zaK8eYrFjKn\nqDVQRLY1lqTXRATZ0gNEST1gGEgCDANJFcNAEmAYSKoYBpIAw0BSxTCQBBgGkiqGgSTAMJBUMQwk\nAYaBpIphIAkwDCRVDANJgGEgqVJqr8U9I+KfImJbtefib5U4r6T2FGmICnwZ+EFm/l5ELGWwxZqk\nDimxo9IewO9k5qUAmflyZj5b+8qkntm4cSOrV5/I6tUnsnHjxlFfzhvUbogaEQcy2CfhPuBA4Bbg\nzMzcPuM4G6JqYm3cuJEPf/jjbN/+RQCWL1/H1VdfxnHHHdf42G02RF0KHAx8JTMPBp4Hzi5wXqk3\n1q+/pAqCjwODUFi/frz2GirxzOBR4JHMvKV6fyWwbrYDzz///J+/XrVq1dj2j5dK2bhxI+vXX8Kt\nt94JrGllzC1btrBly5YFf12RfRMiYitwamY+EBHnAbtm5roZxzhN0ER5/dTgbuCrwN8A4zlNKFVN\nOAP4VkTsAvwEOKXQeaXOev3UYGCvvf6SQw45kLVr2wmChSi11+KdwGElziV12Y5pAcBTT/33jD99\nD4cc8lM2bfp2+xc2D6XuDKSJN7NisGzZn7Js2Vm8+OLgz5cvX8fatZeN8ArnZhhIhcycFrz4Iqxc\n+VWmpjYAjOXUYDrDQGrQ1NTeYzstmMkwkGra8Zzgqaee7NS0YCa3ZJdqmO05wQEHHMjU1NtYu/a0\nsZgWtF1alCbSbM8JpqY2dGZqMJ39DKRF2PGho8HKwn7wzkBaoNdPDX6VwZq7ga49J5jOMJAWqGsr\nC+fLMJBqG++VhfNlGEjzMH2Z8VFHHcz1169je9Wxo8tTg+ksLUo7MVtjknPOOZ2tW28DGJsS4jCW\nFqVCZj4j2L4dtm7tZvlwLpYWpSH6WD6ci3cG0iz6Wj6ci2EgzaKv5cO5GAZSpcuNSUowDCS635ik\nBMNAovuNSUooFgYRsYTBBiqPZmY7PaGlBnWpMUkJJe8MzmSwq9IeBc8pNaovjUlKKBIGEbECOAH4\na+AzJc4pNW225wQrV15aNSbp/7RgplJ3BhcDZwF7Fjqf1Lg+NSYpoXYYRMT7gScz846IWAUMXQPt\n9moaB6PY8qxNI9teLSI+D/wh8DKwHHgLcFVmfmzGcX5QSSM3LluetWm+H1Qq+qnFiDgKWDtbNcEw\n0DhYvfpENm9ew2srC/+Mvfb6TrWycLw/fbhYfmpRmpf+ryycr6JhkJlbga0lzynVNQmNSUqwuYl6\nreuNSUpwmiAxOY1JSrC5iXpp0hqTlOCdgXpnEhuTlGAYqHcmsTFJCYaBemHSG5OUYBio82xMUoZh\noM6zMUkZhoF6adIak5RgGKizbExSlisQ1UmzPSc44IADq8Yk/V9VuBCuQFSv2ZikPMNAnTF3+VB1\nGQbqBMuHzTMM1AmWD5tnGKizLB+WZRhorFk+bI+lRY0ty4dlWFpU51k+bFft5iYRsSIirouIeyPi\n7og4Y+dfJQ1nY5LRKHFn8DLwmWoTld2BWyNiU2b+uMC5NWFsTDI6tcMgM58AnqhePxcR24B9AMNA\nC2ZjktEp+swgIt4JHATcVPK8mmQ2JmlLsTCopghXAmdm5nOzHeNei5qN+xqUNbK9FgEiYinwPeCH\nmfnlIcdYWtQbuK9B89ouLX4DuG9YEEjDuK/B+ChRWjwS+ChwdETcHhG3RcTx9S9NfWb5cPyUqCbc\nALypwLVoQlg+HE+uQFTrLB+OJ8NArXBfg/FnGKhxNibpBsNAjbMxSTcYBhoJG5OMH8NAjbExSbfY\n3ESNsDHJ+LC5iUbKxiTdU3sFojSdKwu7yzsDFePKwm4zDFSMKwu7zTBQg1xZ2CWGgWqxMUl/WFrU\notmYpBssLapxNibpF0uLWjDLh/3knYEWxPJhfxkGWhDLh/1lGGinbEwyGYqEQdUA9UsMnkF8PTO/\nWOK8Gj0bk0yO2qXFiFgCPAAcA/wMuBk4aeZei5YWu2n16hPZvHkNr00LLqsak+wNWD7sgjZLi4cD\nD2bmQ9XAVwAfxL0We8vGJP1UIgz2AR6Z9v5RBgGhDrMxyeRp9QGiey12w2zPCVauvLRqTGLFYNyN\nbK/FiDgCOD8zj6/enw3kzIeIPjPojtmeExx7rCsLu6rNZwY3A/tHxL7A48BJwO8XOK9aNHf5UJOg\nxPZqr0TEp4BNvFZa3Fb7ytQay4cCP7UoLB/2nZ9aVC2WDyePYTDBLB9qOqcJE8p9DSaH0wTNyX0N\nNJPNTSaMjUk0jHcGE8TGJJqLYTBBbEyiuRgGE83GJHqNYdBz7mug+bK02GPuayCwtCjc10ALY2mx\nhywfajG8M+gZy4daLMOgZywfarEMgx5wXwOVYBh0nI1JVIph0HGzfeBo0JhkA4BTA82bYdBDNibR\nYhgGHWVjEpVWawViRFwIfAB4Afh34JTMfHbIsa5ALMTGJFqI+a5ArBsG7wOuy8xXI+ICBvslfHbI\nsYZBIe5roIWYbxjUWoGYmddm5qvV2xuBFXXOp7m5slBNKvnM4BPAFQXPp2lcWaim7TQMImIzsPf0\nfwQkcE5mXlMdcw7wUmZePte53Gtx8VxZqPka5V6LJwOnAkdn5gtzHOczgxp8TqDFauUjzBFxPHAW\n8LtzBYEWx8YkalPdasKDwDJgx4L4GzPzk0OO9c5gAWxMolJauTPIzHfV+XoNZ2MStc3mJmPG8qFG\nxeXIY8TyoUbJMBgjlg81SobBiNmYROPCMBghG5NonBgGI2RjEo0Tw2DM2JhEo2IYjICNSTSO3F6t\nZTYmUdvcXm1MzfacYGrKlYUaPcOgBXOXD6XxYBg0zPKhusIwaJjlQ3WFYTAClg81jgyDhlg+VNdY\nWmyA5UONE0uLI2T5UF1kc5OCbEyiLvPOoBAbk6jrioRBRKwFLgKmMvPpEufsGhuTqOtqh0FErACO\nBR6qfzl9YmMSdUuJO4OLGeydsKHAuTrFfQ3UJ3U3UVkDPJKZd0fstHLRKzPLh9dfv2NfA1cWqpvq\n7LV4LvA5BlOE6X82VJ/2WnRfA42r1vdajIjfAK4FnmcQAiuAx4DDM/M/Zzm+F4uOdkwNbr31Tp5+\n+s9x70ONu8YXHWXmPcDbpw34U+DgzHxmseccd5YP1Wcl1xkkO5kmdJ3lQ/VZsTDIzP1KnWucuK+B\nJoUrEOdgYxJNEsNgDjYm0SQxDBbIxiTqK8NgFjYm0SSyuckMNiZR39jcZJFsTKJJZXOTio1JNOm8\nM8CVhRIYBoArCyUwDIZwZaEmz8SGgY1JpNebyNLizPLh8uU7GpPcBmAJUb1iaXEONiaR3miiSouW\nD6XhJubOwPKhNLeJCQPLh9Lceh0GNiaR5q+3YWBjEmlhSuyodDrwSeBl4PuZeXbtqyrAxiTSwtTd\nRGUV8AHgPZn5ckRMFbmqhtiYRBqu7p3BnwAXZObLAJn5VP1LqsfGJNLi1FqBGBG3A98Fjge2A2dl\n5i1Djm18BaKNSaQ3KrYCcSfbqy0F3pqZR0TEYcA/AiNrmW5jEmnxdhoGmXnssD+LiD8GrqqOuzki\nXo2It2XmzDoe0K+9FqVx1fpeiwARcRqwT2aeFxHvBjZn5r5Djm19mrB8+TquvtqqgSbbfKcJdcNg\nF+AbwEHAC8DazNw65NhWPrU4faGRzwmklsJgIcbpI8zSJJlvGEzUpxYlDWcYSAIMA0kVw0ASYBhI\nqhgGkgDDQFLFMJAEGAaSKoaBJMAwkFQxDCQBhoGkimEgCTAMJFUMA0mAYSCpYhhIAgwDSZVaYRAR\nh0XEjyLi9ur3Q0tdmKR21b0zuBA4NzNXAucBF9W/pDIW0zfe8cZjvD5/b6MYb77qhsHjwJ7V618E\nHqt5vmL6/h+4z+P1+XsbxXjzVXfj1bOBGyJiPYNt195b/5IkjULdvRZPB07PzO9ExEcYbKgydDs2\nSeOr7o5Kz2bmHtPe/29m7jnkWHdQkUakyC7MO/FgRByVmVsj4hjggToXI2l06obBHwFfiYhlwP8B\np9W/JEmj0Npei5LGW6srECPiLyLizoi4IyKujYgVDY93YURsq8b7dkTssfOvWvRYH4mIeyLilYg4\nuMFxjo+IH0fEAxGxrqlxqrG+HhFPRsRdTY4zbbwVEXFdRNwbEXdHxBkNj/fmiLipWjR3b0R8vsnx\nqjGXRMRtEbGh6bGq8f6j+jt3e0T8aM6DM7O1X8Du016fDnyt4fHeByypXl8AfKHBsX4NeBdwHXBw\nQ2MsAf4N2BfYBbgD+PUGv6ffBg4C7mrp5+PtwEE7flaA+5v8/qpxdq1+fxNwI3Bkw+N9GvgHYENL\n/05/Arx1Pse2emeQmc9Ne7sb8FTD412bma9Wb28EGrsTycz7M/NBBqXXphwOPJiZD2XmS8AVwAeb\nGiwzrweeaer8s4z3RGbeUb1+DtgG7NPwmM9XL9/MIGwb+36rO+ETgK81NcZswzLPGUDrH1SKiL+K\niIeBk4EvtDj0J4AftjheE/YBHpn2/lEa/ssyKhHxTgZ3JTc1PM6SiLgdeALYkpn3NTjcxcBZDNbp\ntCWBzRFxc0ScOteBdasJbzDHIqVzMvOazDwXOLea734JOKXJ8apjzgFeyszLmx5L9UXE7sCVwJkz\n7iaLq+4cV1bPkzbtKJWXHici3g88mZl3RMQqmr2DnO7IzHw8In6JQShsq+743qB4GGTmfFcgXg78\noOnxIuJkBrdmRzc9VgseA94x7f0KxujzICVExFIGQfD3mfndtsbNzGcj4vvAoUDxMACOBNZExAnA\ncuAtEfHNzPxYA2P9XGY+Xv3+XxFxNYOp5qxh0HY1Yf9pbz/E4AFYk+Mdz+C2bE1mvtDkWDOHbui8\nNwP7R8S+1dqOk4Cmn0oH7f1fDAZL2u/LzC83PVBETEXEntXr5QyW0jfyM5mZn8vMd2Tmfgz+u13X\ndBBExK7VXRYRsRuwGrhn2PFtPzO4ICLuquZoq4C1DY/3twyeSm+uyjl/19RAEfGhiHgEOAL4XkQU\nfz6Rma8AnwI2AfcCV2TmttLj7BARlwP/Crw7Ih6OiFpTunmMdyTwUeDoqhR2WxXoTfll4F+qn8cb\nGTzh/+cGx2vb3sD1076/azJz07CDXXQkCbDtmaSKYSAJMAwkVQwDSYBhIKliGEgCDANJFcNAEgD/\nDzcvNav5fpAxAAAAAElFTkSuQmCC\n", 285 | "text/plain": [ 286 | "" 287 | ] 288 | } 289 | } 290 | ], 291 | "execution_count": 0 292 | }, 293 | { 294 | "cell_type": "markdown", 295 | "metadata": { 296 | "id": "AId3xHBNlcnk", 297 | "colab_type": "text" 298 | }, 299 | "source": [ 300 | "Then we perturb it with noise:" 301 | ] 302 | }, 303 | { 304 | "cell_type": "code", 305 | "metadata": { 306 | "id": "fXcGNNtjlX63", 307 | "colab_type": "code", 308 | "colab": { 309 | "autoexec": { 310 | "startup": false, 311 | "wait_interval": 0 312 | }, 313 | "output_extras": [ 314 | { 315 | "item_id": 1 316 | } 317 | ] 318 | }, 319 | "cellView": "form", 320 | "executionInfo": { 321 | "elapsed": 327, 322 | "status": "ok", 323 | "timestamp": 1446659134929, 324 | "user": { 325 | "color": "#1FA15D", 326 | "displayName": "Michael Piatek", 327 | "isAnonymous": false, 328 | "isMe": true, 329 | "permissionId": "00327059602783983041", 330 | "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", 331 | "sessionId": "4896c353dcc58d9f", 332 | "userId": "106975671469698476657" 333 | }, 334 | "user_tz": 480 335 | }, 336 | "outputId": "231c945e-e4a4-409e-b75b-8a8fe1fdfc30" 337 | }, 338 | "source": [ 339 | "#@test {\"output\": \"ignore\"}\n", 340 | "X += np.random.randn(2, num_examples)\n", 341 | "plt.figure(figsize=(4,4))\n", 342 | "plt.scatter(X[0], X[1])\n", 343 | "plt.show()" 344 | ], 345 | "outputs": [ 346 | { 347 | "output_type": "display_data", 348 | "metadata": {}, 349 | "data": { 350 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQkAAAEACAYAAACgZ4OsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEptJREFUeJzt3X+MpVV9x/H3B5dJx6rYFbNGNqjUX2S1wFbRipWNZHYo\nhtUV02B/iJq4bbBI64QssCZLSmyKdKOGYixRYW3cUH/Rbi3MMgZ3G0ypRUC3gLCt1gKKBrES063L\nj2//uHfWu8OdZ+7Mvec553nu55Xc7Nx7n5nnTLL3O+d8z/eco4jAzGwxR+VugJmVzUHCzCo5SJhZ\nJQcJM6vkIGFmlRwkzKxS8iAh6RJJd0v6tqTPSZpIfU8zG52kQULSi4D3AadExG8Aq4BzU97TzEZr\nVeKf/xhwCPhVSU8BzwR+kPieZjZCSXsSEfFTYAfw38BDwP9ExFdT3tPMRiv1cOME4M+AFwEvBJ4l\n6fdS3tPMRiv1cOM1wNcj4lEASV8G3gDsmr9AkhePmGUUEap6P/Xsxn3A6yX9iiQBZwD3LrwoIhr1\n2L59e/Y2tL3NTWtvU9s8iNQ5iW8BnwW+CXwLEHBNynua2WilHm4QEVcCV6a+j5ml4YrLFdiwYUPu\nJixb09rctPZCM9s8CA06LknWAClyt8FsXEkiMicuzazhHCTMrJKDhJlVcpAws0oOEmZWyUHCzCo5\nSJhZJQcJM6vkIGFmlRwkzKySg4SZVXKQMLNKDhJmVslBwswq1XE4zzGSviDp3u4hPa9LfU8zG506\nehIfB26MiBOBk+izx6VZ2+3Zs4eNG89h48Zz2LNnT+7mLEvSTWckPQe4MyJ+veIabzpjrbZnzx42\nbz6PgwevAGBycis33LCT6enpzC0rY9OZlwCPSLpW0h2SrpE0mfieZkXZseOaboA4D+gEix07mrMf\ndOqNcFcB64H3R8Ttkj4GXAxs773osssuO/z1hg0bWrtXoFlue/fuZe/evcv6ntTDjTXAv0TECd3n\nbwS2RsTZPdd4uGGt1vThRvKNcCXtA94XEfdL2g48MyK29rzvIGGtt2fPnsNDjJmZLUUECCgnSJwE\nfAo4Gvgu8J6I+FnP+w4SVqxSP9yjUkSQWIqDhJUq5TChlODjIGE2hI0bz2FubhOdWQmAnUxN7ebm\nm7801M8tKUcxSJBIfsyfmR3pyClROHiw81qpQxkHCbMevcOA009fz623buXgwc57k5NbmZnZmbF1\neThImHUtHAbceutWtm27gH37dgMwMzOaIcHMzBZuvfW8xgQf5yRs7CyWNEyVg1hOG+rmnIS10jAf\nsKf3Fs7LkjScnp4uNgfxNBGR9dFpgtlgZmdnY3JyTcB1AdfF5OSamJ2dHfj7p6be3v3e6D6ui6mp\nt4/kZzdR9/NX+Rl1T8IaJeXMwPT0NDfcsLOnl1JG6XRuDhI2VpZKGjZqGFATJy6tUUZRiFRK0rAE\nrri0Vqr6kLcxAKT8nQYJEk5cWmu0MfGY+ndigMSlexLWGnXWOdQl9e9UwvZ1ZtZwnt2w1mhaufMg\nSvidPNywxnHist7EZR07Ux0F3A48GBGb+rzvIGEDK2kvhjYoJSdxIXBPDfexMdD07embKGmQkLQW\nOIvOHpdm1kCpexIfBS4CPJ6wkZiZ2cLk5FZgJ7Czm8jbsuT3NfmYvdySzW5Iegvwo4i4S9IGYNFx\njw/nsUGtZBFWKcvDS1DU4TyS/gL4A+AJYBJ4NvDliHjXguucuLSk2lhkNSpZE5cRcWlEHB+d07vO\nBW5ZGCDMrHwuprLWK6EgqclcTGUr1qTCpSa1tU5FFFMtxUGimVzU1A4OEpaMk4HtUErFpZk1mBOX\ntiJOBo4P9yRsxV75ypeyevXlnHLKtc5HtJh7ErZsC5OWBw9uzdwiS8k9CVu23CsxvQ6jXg4S1ijz\nvZi5uU3MzW1i8+bzDgcKB49EltopN/UD75bdODl3pV7smL7UbZqdnY2pqbcfvldb4GP+LIUSj8NL\nefzfuK8idZCwFcl1HN5iU68pcyIpA1ATOEhYo1T1Yly3kYbLsq01Ui3iavM6Fa/dMBuRtq4idZAw\ns0pe4GVmQ0u+pb6kWyTdLWm/pA+kvJ+ZjV7S4YakFwAviM6O2c8Cvgm8NSK+03ONhxtmmWQfbkTE\nwxFxV/frnwP3AselvKdZPy7ZXrnaEpeSXgzsBV7VDRjzr7snYUm1eQpzWIP0JGoppuoONb4IXNgb\nIOb5cB5LadwrJnsVdTjP4RtIq4CvADdFxMf7vO+ehCXl/TgXV0pP4jPAPf0ChFkdvNXecFLPbpwG\n/DOwn86hwQFcGhGzPde4JzHm6qhmbGvF5LBccWnFc1Ixr+xToDbeBpl2zL0V3ii1dZrVS8UtiXHb\nqKXVv+9SW1elfuDt61ppsW3mFsq5Fd4oDfr7loYBtq/zcMOymt9EZmpqN1NTu5f917etXfyiLBVF\nUj9wT6KV6ughlNQLKakty8EAPQkHCRvYcneMTr3DdGld/CbuqD1IkHDi0gaynMTcuNYk5NocOLml\nokjqB+5JNEKqROQwf32b2sUvCe5JWN2Ws5hq2GnDEs//aCMHCRtIivUPo1id2doufkEcJGwgS/3V\nns9DPPLIj5iYuIhDhzqvezFV83nthg1t4bBhYuJPWbfuJI499nmViUuv28jPC7ysFsPs1zCuMyGl\nKGU/CbNFOadQPpdl29BmZrYwObkV2Ans7OYhtuRu1ki47Lue7evOBD5GJyB9OiKuWPC+hxst0MZh\nwzjkTLLnJCQdBdwPnAH8APg34NzwuRvWAOOwN2YJm86cChyIiO9HxOPA9cBbE9/TzEYodeLyOOCB\nnucP0gkcZsXzBrodnt0wW4TLvjtSB4mHgON7nq/tvnYEH87TPm1JZLZtira4w3kkPQO4j07i8ofA\nN4B3RsS9Pdc4cdkyy50VaEtAaaJBEpd1LAU/k06gOABc3Of9Uax4tYIsZzMYL/fOixL2uIyI2Yh4\nRUS8LCL+MvX9rFnq2lLfRVEr58SljVxpswKt3u6+Dkt1NVI/8HCjlQbdcaqO4UZpe2GWBO9MZbkM\nOivgacbyeam4td44rMFYqexrNwbhIGF18DRrfw4SZlaphAVelpin9iw19yQazGNtG5Z7Eg2ykh7B\nKAuR3COxxThIFGC+RzA3t4m5uU1s3nze4Q9qHR/eqvubuZiqAIsV+yxVaDSqQiQXG40vXEzVbEud\ncOVCJKuDg0QBFlvrMEh+YRT7HczMbGHfvnM5dOiTAExMfIeZmeuH+pnWHg4SBajqEdS3UOpo4I+7\nX1+U6B7WRJ4CLVwdlYLjsCu09ecTvFqgbdunWfMkCxKSPgKcDfwC+E/gPRHxWKr72cqVtv+DlSVl\nncTNwLqIOJnO1nWXJLxXI5RasDSfE5ma2s3U1G5XbdoRaslJSHobcE5E/GGf98YiJ+ESaitRSWXZ\n7wVuquleRaprL0ezURsqSEiak/Ttnsf+7r9n91yzDXg8InYN3dqMSh0qmKU2VOIyIqaq3pf0buAs\n4M1V15V+OM8oNlJ1ctBKUNThPJLOBHYAb4qIn1RcV3xOYlR1BN4dyUqTu07iKmACmJMEcFtEnJ/w\nfsVbTs2DA4qVIlmQiIiXpfrZdat7qOBzIqwkLsseUJ1/2V0mbXXJPdxoFZdH27jyzlQjMOrp0ZmZ\nLUxObgV2Aju7w5stQ/9cs5XwcGNIqSopnbi0OvjcjRr0yx+sXn05u3Zd7Q+2Fa+ksuyx8uijz/dm\nstYa7kkMaeFwA+ZzCQ97RsKK555EDeaXWa9efTnwSToBwsMMaw8HiRGYnp5m166rmZz8HvAwnpGw\nNvFwY4Q8I2FN49kNM6vknISZDc1BoiG86Y3l4uFGA3h/TEvFOYmW8KpQS8U5CTMbWvIgIWlG0lOS\nVqe+10o0YazvVaGWU9LhhqS1wKeAVwC/GRGP9rkm23CjSWN912BYCtlzEpK+APw5sJsCg0Tusb4/\n+JZb1p2pJG0CHoiI/d2NcK2H97G0phgqSEiaA9b0vgQE8CHgUmBqwXt95Tp3I+dZGEee6AUHD3Ze\nc5CwlIo5d0PSq4CvAv9LJzisBR4CTo2IHy+4NusUaK4u//r1b+TOO58EXghsoY6l5R7e2EKDDDeI\niOQP4HvAry3yXoyb2dnZmJh4fsB13cexMTHx3JidnU16z8nJNYfvOTm5Jun9rBm6n7/Kz29du2UH\nFcONcbNjxzUcOnQlv0yYwrp11yb9y+7hja1ULUEiIk6o4z5Nduyxz8vdBLO+fO5GBjkSpj6w2FbK\nazcyyZFEdOLSFspeTDWIcQ0STeLg0l4OEja0JpWu2/J5FegimrCoqxRHzop0gsV8r8LGw9glLl0O\nbbY8Y9eTKOkvYxN6NF6mbmPXkyhFU3o084cP/TJxWV4bLa2xS1yWkojLvUzdDDIvFS+V/zKaLc/Y\n9SRK0a9Hs23bBezbdwfgegSrh+skCtdbpHT66ev58Ievyj4MsvHiINEgzlFYDi6mMrOhjV3islRe\npWml8nCjIF5IZXXLnpOQdAFwPvAE8E8RcXGfaxwkzDLJmpOQtAE4G3h1RLwa+KtU9ypVE8quzZaS\nrCch6e+Av4mIW5a4rpU9iVIqO82q5J7deDnwJkm3SfqapNckvFdxSlpIZjaMlIfzrKKzjf7rJb0W\n+DzQd0PcXIfzmI2bYg7nAZB0I3BFROzrPv8P4HUR8ZMF13m4YZZJ1tkNSVuA4yJiu6SXA3MR8aI+\n17UySICnNK18uYPE0cBngJOBXwAz872KBddlDxL+MNu4yl4nMYjcQcLDAhtnDhID8MIqG2e5p0DN\nrAXGfoGXF1aZVRv74QY4cWnjyzkJM6vknISZDc1BwswqOUiYWSUHCTOr5CBRGG9UY6Xx7EZBXCJu\ndfMUaMO4RNzq5ilQMxva2Jdll8Ql4lYiDzcK4xJxq5NzEmZWKfe5G6+V9A1Jd3b/Havdss3aImXi\n8iPAhyLiFGA7cGXCe5lZIimDxA+BY7pfPxd4KOG9zCyRlBvhHg98nc45HALeEBEP9LnOOQmzTAbJ\nSaQ8nOcC4IKI+HtJ76Czc/ZUv5/jw3nM6lHa4TyPRcRzep7/LCKO6XOdexJmmeSuuDwg6fRuQ84A\n7k94LzNLJGXF5R8BV0uaAP4P2JLwXmaWiIupzMZY7uGGmbWAg4SZVXKQMLNKDhJmVslBwswqOUiY\nWSUHCTOr5CBhZpUcJMyskoOEmVVykDCzSg4SZlbJQcLMKjlImFklBwkzqzRUkJD0Dkn/LulJSesX\nvHeJpAOS7pW0cbhmmlkuw/Yk9gObgX29L0o6Efhd4ETgd4BPSKrc2KJJlruRaAma1uamtRea2eZB\nDBUkIuK+iDhAZ5fsXm8Fro+IJyLiv4ADwKnD3KskTfzP0LQ2N6290Mw2DyJVTuI4oPeMjYe6r5lZ\nwyy5EW7F2RrbIuIfUzXMzMowko1wJX0NmImIO7rPLwYiIq7oPp8FtkfEv/b5Xu+Ca5ZR0hO8Fui9\n0W7gc5I+SmeY8VLgG/2+aakGmllew06Bvk3SA8Drga9IugkgIu4BPg/cA9wInO99882aKfu5G2ZW\ntqIqLiXNSHpK0urcbVmKpI90C8XukvQlSc9Z+rvqJ+lMSd+RdL+krbnbsxRJayXdIuluSfslfSB3\nmwYh6ShJd0janbstg5B0jKQvdP8P3y3pdYtdW0yQkLSWzqnj38/dlgHdDKyLiJPp1IFckrk9TyPp\nKOCvgWlgHfBOSa/M26olPQF8MCLWAb8FvL8BbQa4kM7wuik+DtwYEScCJwH3LnZhMUEC+ChwUe5G\nDCoivhoRT3Wf3gaszdmeRZwKHIiI70fE48D1dArdihURD0fEXd2vf07nP2/RNTbdP3BnAZ/K3ZZB\ndHu9vx0R1wJ0ix4fW+z6IoKEpE3AAxGxP3dbVui9wE25G9HHwqK2Byn8A9dL0ouBk4GnTZ0XZv4P\nXFMSfC8BHpF0bXeIdI2kycUuTnmq+BEqirI+BFxKZ6jR+152gxSSSdoGPB4RuzI0sbUkPQv4InBh\nt0dRJElvAX4UEXdJ2kAh/3eXsApYD7w/Im6X9DHgYmD7YhfXIiKm+r0u6VXAi4FvdReBrQW+KenU\niPhxXe3rZ7E2z5P0bjrdzDfX0qDlewg4vuf52u5rRZO0ik6A+NuI+Ifc7VnCacAmSWcBk8CzJX02\nIt6VuV1VHqTTc7+9+/yLwKJJ7eKmQCV9D1gfET/N3ZYqks4EdgBvioif5G5PP5KeAdwHnAH8kE5B\n2zsjYtEkVQkkfRZ4JCI+mLstyyHpdDqVx5tyt2UpkvYB74uI+yVtB54ZEX0DRW09iWUImtFluwqY\nAOa6q+Bvi4jz8zbpSBHxpKQ/oTMTcxTw6QYEiNOA3wf2S7qTzv+HSyNiNm/LWucDdKqijwa+C7xn\nsQuL60mYWVmKmN0ws3I5SJhZJQcJM6vkIGFmlRwkzKySg4SZVXKQMLNKDhJmVun/AUYHTBJb9HcU\nAAAAAElFTkSuQmCC\n", 351 | "text/plain": [ 352 | "" 353 | ] 354 | } 355 | } 356 | ], 357 | "execution_count": 0 358 | }, 359 | { 360 | "cell_type": "markdown", 361 | "metadata": { 362 | "id": "3dc1cl5imNLM", 363 | "colab_type": "text" 364 | }, 365 | "source": [ 366 | "## What we want to do\n", 367 | "\n", 368 | "What we're trying to do is calculate the green line below:" 369 | ] 370 | }, 371 | { 372 | "cell_type": "code", 373 | "metadata": { 374 | "id": "P0m-3Mf8sQaA", 375 | "colab_type": "code", 376 | "colab": { 377 | "autoexec": { 378 | "startup": false, 379 | "wait_interval": 0 380 | }, 381 | "output_extras": [ 382 | { 383 | "item_id": 1 384 | } 385 | ] 386 | }, 387 | "cellView": "form", 388 | "executionInfo": { 389 | "elapsed": 414, 390 | "status": "ok", 391 | "timestamp": 1446659137254, 392 | "user": { 393 | "color": "#1FA15D", 394 | "displayName": "Michael Piatek", 395 | "isAnonymous": false, 396 | "isMe": true, 397 | "permissionId": "00327059602783983041", 398 | "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", 399 | "sessionId": "4896c353dcc58d9f", 400 | "userId": "106975671469698476657" 401 | }, 402 | "user_tz": 480 403 | }, 404 | "outputId": "74e74f19-6ff8-4a8c-81c7-9021a08b78b5" 405 | }, 406 | "source": [ 407 | "#@test {\"output\": \"ignore\"}\n", 408 | "weights = np.polyfit(X[0], X[1], 1)\n", 409 | "plt.figure(figsize=(4,4))\n", 410 | "plt.scatter(X[0], X[1])\n", 411 | "line_x_range = (-3, 5)\n", 412 | "plt.plot(line_x_range, [weights[1] + a * weights[0] for a in line_x_range], \"g\", alpha=0.8)\n", 413 | "plt.show()" 414 | ], 415 | "outputs": [ 416 | { 417 | "output_type": "display_data", 418 | "metadata": {}, 419 | "data": { 420 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQMAAAEACAYAAAC3RRNlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHvdJREFUeJzt3Xl4VPX1+PH3CRCNRVCgUpWK4iOlRWWxipVa0BDCIsHt\nZ22lLLbybRECGBYXFBBbQeQBRCoWGkWptrUUxQoJQYQWLVKURREBFUGggKDsS0jm/P6YgULMJDOZ\nO3eZnNfzzNNMcrn3tM2cfO7nc+75iKpijDFpXgdgjPEHSwbGGMCSgTEmwpKBMQawZGCMibBkYIwB\n4kgGIvJHEdkpImtO+d65IrJARNaLSKGI1E1OmMaYZItnZPAckF3me/cDC1X1e8Ai4AGnAjPGuEvi\nKToSkcbA66p6ZeT9x0A7Vd0pIt8BFqtqs+SEaoxJpkTnDM5T1Z0AqroDOC/xkIwxXnB6AtFqm40J\nqJoJ/vudItLwlNuEXdEOFBFLFMZ4RFWlsmPiHRlI5HXCXKB35OtewGuVBBSo18iRIz2PIZXjtZjd\necUqnqXFl4B3gKYiskVE+gBjgSwRWQ9kRt4bYwIo5tsEVf15lB91cCgWY4yHrAKxAu3bt/c6hLgE\nLV6wmP0krjqDhC4kom5dyxjzPyKCJmEC0RiToiwZGGMASwbGmAhLBsYYwJKBMSbCkoExBrBkYIyJ\nsGRgjAEsGRhjIiwZGGMASwbGmAhLBsYYwJKBMSbCkoExBnAoGYjIAyKyVkTWiMifRCTdifMaY9yT\ncDKI7KVwD9BKw/sp1ATuTPS8xhh3JdodGWA/UAx8S0RCwFnAdgfOa4xxUcIjA1X9GpgAbAG2AXtV\ndWGi5zXGuCvhkYGINAEGA42BfcDfROTnqvpS2WNHjRp18uv27dunbC85Y7y0ePFiFi9eHPe/S7gH\noojcAWSp6j2R978A2qhq/zLHWQ9EYzzgZg/E9cC1InKmiAjh/RPWOXBeY4yLnJgzWA28ALwHrCa8\n49IfEj2vMcZd1ird+FZhYSETJoT/ruTl9SU7O9vjiIIp1tsESwbGlwoLC7nlll4cOTIOgIyM4cyZ\nM9MSQhVYMjCB1rHjbRQV5RDezxdgJllZc1mwYLaXYQWSbaJijA8UFhbSseNtdOx4G4WFhV6HUyEn\nKhCNcVxeXl+WLu3FkSPh9xkZw8nLm+ltUHEqe6uzdGkvX9/q2G2C8a2gTyD65VYn1tsEGxkY38rO\nzg5cAggySwbGJEnQbnXsNsGYJPLDrY4tLRpjAFtaNMbEyZKBMQawZGCMibBkYIwPFH1axPYD3nYL\ntGRgjIdUlRdXv8ikdydxtOSop7FYMjCBFKSa/2hCGmLisom8vuF18nPyaXJuE0/jsaVFEzip8Hhz\ncWkxoxaP4stDXzIhewJ1zqiTtGu5urQoInVF5BURWRfZTKWNE+c1pjwTJvwhkgh6AeGkcKKwJwgO\nFh8kd34uJaESpnadmtREEA+nypEnA/NU9f+JSE3CeycYY8r48tCX5Bbk0rJhS4a2HUqa+OdO3YlW\n6XWA61W1N4CqlhDeWMWYpAhazf8Jn+/9nNz5udzc7Gb6tOxDuH+wfzjRKr0F4QaoHwEtgBXAQFU9\nUuY4mzMwjvFDzX881uxcw5AFQxhwzQC6fa+bq9d27dkEEbkKWAb8SFVXiMgkYJ+qjixznI4c+b9v\n2SYqprr45+Z/8uiSR3n0hke57rvXJf16ZTdRGT16tGvJoCHwb1VtEnn/Y2C4qnYrc5yNDEy1M2fd\nHKa9N42J2RP5wbd/4EkMrjU3UdWdIvKFiDRV1Q2EN1H5KNHzGhNkqsr096fzxsY3mN5tOhfVvcjr\nkCrl1GpCLvAnEakFfAb0cei8xgROaaiUsUvHsm73Op7r/hz1Mup5HVJMrOjIGAcdLTnKg28+yLGS\nY4zvOJ6zanm/ym79DIxx2b6j+/jNG7+hdnptJnWa5ItEEA/rgWiMA7Yf2M6A+QNo17gd/a/p76ti\nolhZMjAmQRv2bGBQwSB6tujJnZff6XU4VWbJwJgE/Gfbf3hw0YMMu24YWZdmeR1OQiwZGFNFCz5d\nwPh3xjM2cyxXXXCV1+EkzJKBMVXw0gcvMWvNLH7f5fdcVv8yr8NxRPBmOYyJUTIaoIQ0xKRlk5jz\n8Rzyu+enTCIAqzMwKSoZDVCOlx5n9JLRbD+wnUmdJvmmD0FlrM7A+Fqy25Y53QDlUPEhBhYM5GjJ\nUZ7p+kxgEkE8bM7AuC5oW5XvObyHAfMHcPl5l3P/j+8PZA1BTFTVlVf4UsaoZmXdqvC8gkZez2tW\n1q2OXqOgoEAzMhpGrvO8ZmQ01IKCgrjPs3nvZu32Ujed/t50DYVCjsbolshnr9LPqI0MTErKzs5m\nzpyZpzRAiX/k8eGuD7mv8D76Xd2Pm5vdnIwwfcUmEI3rgtDdeOmWpYxaPIqR7UZyfePrvQ4nIbYL\ns/E1P7ctm7t+Lk8vf5oJHSdwRcMrvA4nYZYMjImTqpK/Mp/X1r/GlM5TaHxOY69DcoQtLRoTh5CG\nGLt0LG9uepP87vm+SgRu7R7lWDIQkTQReV9E5jp1TpM6/Lwd2rGSYwwrGsYX+79gerfpNDirgdch\nnXRifqWoKIeiohxuuaVX8v73i2XJIZYXMBiYBcyN8vNkrJqYAHBqmS8Z9h3dp3e/erc+9OZDWlxS\n7HU43+DEMiwxLi06tb1aI6ALMMOJ85nU4pft0MqOTnYc3MEv5/6SKxpewaM3PEqtGrVcj8lPnKoz\nmAgMBeo6dD5jKhXPikTZ5cx/fngXVw5vwqAbB/HzK37uSrxV4ebuUU5sr9YV2Kmqq0SkPRB11nLU\nqFEnv7ZNVKqPsr/QaWmDadcuL6FzxlvSfNro5Pz3ONbhO3w17yg/H+jfRABVK54qu4lKzGK5l6jo\nBfwO2EK4Rfp/gYPAC+UcV9XbJpMCHnvsMU1Lq69wrUJewvMG8d5Lnzy+SZHyiw7KBY9oWlp938xd\nJBNuzRmo6oOqepGGd1S6E1ikqj0TPa9JLUuWvE8oNAH4N/Ckg/MGhcBtwDR2794Z9ai8vL7I5ffC\nj/JgXgfY/iyhUO9AbeWebFZnYHyrouXIvLy+pKcPAnoAOcCvWbt2Q7nLbqrKhnM2UOe6M2FuTdiz\nHJgJBL+60FGxDB+ceGG3CdVavMuLsRzfqlW7Sm8Vjpce14cXPay9X+2tf/vH33y7xJlM2FOLxk/i\nnQg7fTkSjhwJf+/Uf9OgQf0Kr3n4+GGGFQ2jVlotnun6DGfWPJPac2on9CRjSoslYzjxwkYGJg6x\nTBBWNHrYc3iP9vh7Dx2zZIyWlJZ84/wFBQWalXWrZmXdmvKjA2IcGVgyML4U621FQUGBtmrVTuvV\nu1RbtWqrBQUFumXvFu3+cnd9dsWz5TYk8XNFZDJYMjCBF8tf77If7DMa1dfWE1vr7I9mRz2vG52W\n/CTWZGBzBsa3srOzK72nP21u4bvvcOyGhqQtrc2tg251J8gUYsnApIamr0ObKVD4M869cmWFh7pZ\n4hsk1tzEBFpBQQE5I3/K8Usvgfk9yDj2ZEwt1Pzcaclp1unIpLyQhhj/9njmr5oP82qTXnJGyn+w\nq8KSgUlpxaXFjFg0gv3H9vNkxyepnV7b65B8y9qemZS1/9h++s/rTw2pwVOdn7JE4BBLBiZQdh3a\nxa/m/opmDZrx28zfkl4j3euQUoatJpjA+OzrzxgwfwB3Nr+THlf2QKTSka+JgyUDEwgr/7uS4QuH\nM/jawXS+rLPX4aQku00wvlLeY8uLNi1i2MJhjLlhjCWCJLLVBOMb5W27NnjGL3lf32di9kSaNWiW\ntOumcs1BrKsJTjxz0AhYBKwFPgByoxyXhKprk0pOf2YgpFz9Cz1vwAW6dd/WpF2zOjy0hIvPJpQA\n92m4IWpt4D0RWaCqHztwblMdpZXA9b+Fcz/jBxtbc2GdC5N2qVj6JlQXCScDVd0B7Ih8fVBE1gEX\nApYMTFzy8vryr2U9OfrjWaDCmW9u5P5XXvA6rGrD0dUEEbkYaAm86+R5TfVw9U+upu3jV7Dp/W00\n+aIZQ155Iel/oe2hpf9xbAIxcouwGBijqq+V83N16lom9Wzbv43+8/vTsUlHfv3DX7taQ2ATiJHj\nnPiAikhN4B/AfFWdHOUYHTly5Mn3tomKOWHdl+sYXDiYX7X+Fbf/4Havwwm8spuojB492tVk8AKw\nW1Xvq+AYGxmYb1i2dRkjFo3goesf4oZLbvA6nJTk2oNKItIWuAu4UURWRrZl75ToeU3s/LzdeUXm\nbZzHI289wpMdn7RE4AexrD868cLqDBJWXk/AIK6Th0Ihnblqpnb9U1f99KtPvQ4n5WENUVNLtA99\n0Jp7loZKdfzb4/WOV+7QnQd3eh1OtRBrMrAHlQIiWnFMkBSXFvPIW4/w1ZGvmNFtBmefcXbSrpXq\nKwTJYA8qBVxeXl8yMoYT3jtwZmSdvK/XYX3DgWMH6D+vP6rK012eTloiKCwspHXr9nTpchdFRZdQ\nVJTDLbf0CtRcimdiGT448cJuExJS0dyA33cH2nlwp/70lZ/qE0uf0NJQadKuU/Z/I2ioUOD7W6dk\nI8bbBHtqMUCCOPTd9PUmcgtyuf37t9OzRc+kFhN17HgbRUU5nLiVCo+W5gI5ZGXNZcGC2Um7tp/F\nurRocwYBEsumIn6yesdqhhYNZdC1g+hyWRePotherUuM42FzBqZcidYuLP58MUOKhjC6/WjXEkHZ\n+ZO0tMG0alUjpn0UDDZnYL4p0dqF2R/N1uwXs3XtrrVJjLJ8fp8/8QI2Z2Cqqrx771juuVWVZ997\nloJPCni6y9M0qtMo6bGaytmcgXFVaaiU3/3rd2z8aiP53fOpl1HP65BMnCwZmG+I9xn/I8eP8MCb\nD1AaKmXaTdM4q9ZZLkVqnGS3CaZcsS5j7j26l0EFg2hctzEPt3uYmmn298VvbK9Fk3TbD2yn/7z+\nZF6SSb+r+9mmJj5lcwYmqdbvXs+gwkH0admHO5rf4XU4xgGWDEzclm9bzkOLHuL+tveT2STT63CM\nQ6zoKAUls9lJwScFjFg0gnEdxlkiSDFOtT3rBEwinFz+qKrjyjnG5gxcUN6uRE5V4M1aM4uXP3yZ\npzo9xaX1Lk34fMYdrk0gikgasAHIBLYD/wHu1DKbqFgycEdVC4YqEtIQk5ZNYtnWZUzpPIWGtRs6\nEqtxh5sTiNcAG1V1c+TCfwa6Y5uopITi0mJGLR7Fl4e+ZEbODOqcUcfrkEySODFncCHwxSnvt0a+\nZxwSzxyAk81ODhUfYuD8gZSESpjadaolghRnqwk+V3YOYOnSXhXOAWRnZzNnzsxTCoaqNl+w+/Bu\ncufn0qJhC4a2HUqa2FxzqnMiGWwDLjrlfaPI975h1KhRJ7+2TVRiU5WNQU/te3BiVAGxN0TZvHcz\nA+YP4OZmN9OnZR8rJgqYspuoxCyWRxsregE1gE+AxkA6sAr4fjnHOflUZrWRSPfjqjyKvHrHas16\nIUtf+/g1J8I3PoCbrdKBTsB6YCNwf5Rjkv/fOgUl0lsg3kSy5PMlmjkzU5duXupU+MYHYk0GjswZ\nqGoB8D0nzmVO59QcQGVe/fhVnlnxDJM7Tab5ec0dP7/xP3tQKYWUfdIQqLQASVWZ/v503tj4BlM6\nT+GiuhdVeE5rHxY8sdYZWNuzFBHtdqKiNmAlpSX62JLH9K7Zd+mew3tiPqcJFmx7tdQU7cMd7/zA\nkeNHdHDBYO33j356qPhQuccEbes2U75Yk4HVGQRI2ZqDJUt+QfPmTWnQoCG7d++J+Tz7ju5jcOFg\nGtVpxLgO46hVo1ayQjZBEkvGcOKFjQwSVt5farhW4XlNTz9H09O/XemQfvv+7XrrX27VycsmV7q7\nkd0mpAZsZFBdXAD0orgYWrWaToMGc4HyVx027NnAoIJB9GzRkzsvv7PSM7u1kmH8wVYTAqTsbQIM\nAWYB2VT2dOKK7St44M0HGHbdMLIuzXIpYuMH1gMxRZ1Y6tu9ew9r166muHgSUHHfggWfLmD8O+MZ\nmzmWqy64yu2QjccsGVQDsdQAvPTBS8xaM4vJnSZzWf3Lkn494z+WDKq5kIaY8u4U/rXlX0zpPIXz\nzz4/ofM51UHJEor7rOioGisuKdYRb47QPq/20b1H9jpyTidqDmx1whvEuJpgD6kHXNnGJ4ePH2Zg\nwUAOHz/MM12foe6Zdb0O8aTTH8cOjzJOjBKM92xpMcDKDt3/teIX/OjRy+nQogPD2w6nRloNx64V\n75ZrJoBiGT448cJuExx32tC97mblZy31+z1aaSgUSsr1Et3u3G4TvIEVHVUj530IHfNgxfU0+u62\npHUmOrWDUlX/vRUx+ZetJgRYYWEhOf1/RvF158Pi7mR8me/YHgkmdcS6mpDQBKKIPCEi60RklYjM\nFhFrn+uiYxcf4/J7L+HafeeT9b31lghMQhIaGYhIB2CRqoZEZCzhe5MHohxrIwOHqCr5K/N5bf1r\nTOk8hcbnNPY6JONjrmyioqoLT3m7DLgtkfOZyoU0xBNvP8GanWvI755Pg7MaeB2SSRFOTiDeDfzZ\nwfOZMo6VHGPEohEcOn6I6d2m8630b3kdkkkhlSYDESkCTt1cTwAFHlLV1yPHPAQcV9WXKjqX7ZtQ\ndfuP7WdwwWC+U/s7/C7zd9aQxERV1X0TnNh4tTdwD3Cjqh6r4LjAzxl4VVe/4+AOBswfQNvvtiW3\nTa7tbmTi4sqzCYT3S1gL1I/hWEcLKdzmVcHMxj0btfOszjpr9aykX8ttiRYxmdjgRkNUwpumbAbe\nj7x+X8GxLvzXdtapv6ytWrVzvTnoe9vf0w4vdNCCjan3QbFqRPfEmgwSXU1I7AF5Hytb95+Wlufq\n9Rd+tpBxb4/jtzf+lmsuvMbVa7uhKntImuSycuQoyv6yhkIfkJY2mFAo/PNkPqjzlw//wszVM5na\nZSpN6zdNyjWMKcuSQcyuoEWLH1TYcDRRqsrU/0zlrc/fYkbODC44+4Jyj0uFBiH2FKQPxXIv4cSL\ngM0ZuH1Pe7z0uD6y6BHt/Wpv/frI176JK5lsAtEdxDhnYA8qVcCtv8CHjx9meNFwaqbV5PEOj3Nm\nzTOjHtu69Y9ZubKUcIv0vsCOCrsiR5MKowsTG2t7FhB7Du/RHn/voWOWjNGS0pIKjy0oKNC0tHNP\njgqgoUKetR8zFcL2WvS/LXu3aPeXu+uzK56NqSFJeX0I09Lqx/1Btj0Uq5dYk4GVsnnkoy8/4p7X\n76Fni570vaovIvKNfoaxaNHichviG2fEkjGceGEjg5Pe3vK2Zs7M1CWfLzn5vViG7k4N7+02oXrB\nJhD96R8b/sFT7z7Fkx2f5MqGV578fseOt1FUlMOJuoZTt0s7dbKvXbvWLFnyPpDYxJ9NIFYfrvQz\nMLFTVZ5f9Tx///jvPHvTs1xy7iUx/buylZBLl1Zt85KyEu1naFKPJQMXhDTEk+88ycodK8nPyefb\n3/r2N46JVoRjZbvGLTaBmGTFpcXcv/B+Pv3qU6Z3m15uIoD/dQ7OyppLVtZc62doXGdzBkl04NgB\n8hbkUT+jPqNvGE16jfS4z+HUHoem+rKNVz2269AuBswfQJsL2zDo2kEVNiSpbDLPJvtMIiwZeOiz\nrz8jd34uP23+U3pc2aPCTU3sL79JNksGHlm1YxXDioYx+NrBdL6sc6XHV7SkaIwTXNlE5ZSL5YlI\nSETqOXG+oFq0aRFDi4Yy5oYxMSUCY/wk4WQgIo2ALMLtz1JGvKXBr6x9hfHvjGdK5ym0adQm5uvk\n5fUlI2M4MBOYGVlS7Fv1wI2pqljKFCt6Aa8AVwCbgHoVHOdoiWUyxVOuGwqFdOryqXrLn2/Rrfu2\nVvl69ly/SRbcKEcWkRygvareJyKbgKtU9asox2oi13JTrKXBg+77JSsyVvDJV58wudNkzs0417OY\njYnGsXLkCjZRGQE8SPgW4dSfRRX0TVROm/mvWcyiM27npps68vLdL5NRK8Pr8IwBPNhERUQuBxYC\nhwkngUbANuAaVd1VzvGBGRlEW+6bMOEP4RHDmTnQeSB8dZDMdGXhgjkeR2xMdElfTVDVD1X1O6ra\nRFUvAbYCrcpLBEFTYWnw2V9D97th67WwJIc0q+g2KcKxOgMR+Qz4YSrMGUQz49UZ/N+cewmtuAc+\nupr09EE0b96CBg3qW2Wg8S1X6wwAIiOEchNBKnh367v85cBfeLzrGLIu/C+tWk0HarFyZR+KinK4\n5ZZeMXcnckJVuiIZU6FYlhyceBGgpcWy3tjwhma9kKUr/7vy5Pe87CNonYpMPHBje7VUp6q8uOZF\n/rr2r0y7aRpNzm3idUiAbU1mksOSQRQhDTHx3xNZvn05+d3zOe9b5532c9sRyKQae1CpHMWlxYx8\nayR7juxhQscJnH3G2eUe59Wjxfako4mHPbVYRQeLD5JXmMc5Z57DmBvHVKkhiRusx4GJlSWDKth1\naBe583O56vyryLsur8KGJMYEhXVHjtOmrzeRW5DL7d+/nZ4telbYkMSYVGTJAFizcw1DFgxhYJuB\ndG3a1etwjPFEtR8HL/l8CXkL8hjVfhRdm3a1Yh5TfcVSjODECx8WHc3+aLZmv5ita3etVVX/FvNY\nvwOTCGwX5uhCoZBO+8807f5yd92yd8vJ7/txd2K/JigTHLEmg2o3Z1AaKuXxpY+zYc8G8rvnUy/D\n320brdrQuKVaJYOjJUd5YOEDlIRKmHbTNM6qddZpP7eqQlOdVZs6g71H9zK4cDAX1bmIh9s9TM20\n8vOg34p5rNrQJMqKjk6x/cB2+s/rT+YlmfS7ul/gagj8lqBMsFgyiNiwZwODCgbRu2Vv7mh+h+vX\nN8ZrrjU3EZEBIrJORD4QkbGJns9Jy7ct595595L3ozxLBMZUIqEJRBFpD3QDrlDVEhFp4EhUDij8\npJAJ/57AuA7jaH1+a6/DMcb3El1N+A0wVlVLAFR1d+IhJW7Wmlm8/OHLPNP1GS6td6nX4RgTCIne\nJjQFfiIiy0TkLRH5oRNBVdWJhiRz188lPyffEoExcUh0E5WawLmqeq2IXA38FYjaGyyZm6gUlxYz\nevFodh7ayYycGdQ5o45j5zYmSFzfRAVAROYB41R1SeT9J0AbVd1TzrFJW004VHyIIQuGUDu9No/d\n+Bhn1DwjKdcxJojcWk14FbgxcsGmQK3yEkEy7T68m3tev4eLz7mYcVnjLBEYU0WJTiA+B+SLyAfA\nMaBn4iHFbuv+rfR7ox83N7uZPi37BK6YyBg/CXTR0YFjB1i+bTmZTTIdPa+TrHrQeM0qEH2g7HMF\n6elDad68KQ0aNLTEYFzj+vZqbglSJ6LTHz/uRXHxeFauLPVkOzZjKhOoZHDiL21RUU6AP1AXAOHR\nwonbB2P8IFD9DILW6KNsfwQYAszyMCJjogvUyCBosrOzmTNnJllZc2nV6jnS00uAHcDMSOOUvl6H\naMxJgZpADHqjD1tZMF5I2dUE+0AZE5+UTQbGmPik7NKiMSY5LBkYYwBLBsaYCEsGxhjAkoExJsKS\ngTEGsGRgjIlIKBmIyNUislxEVkb+09OGqMaYqkt0ZPAEMEJVWwEjgfGJh+QfVWkq6aWgxQsWs58k\nmgz+C9SNfH0OsC3B8/lK0P5PD1q8YDH7SaKPMN8PvC0iEwi3UL8u8ZCMMV5IdN+EAcAAVX1VRG4H\n8oGsZARqjEmuRPdN2K+qdU55v09V60Y51p5SMsYjsTyolOhtwkYRaaeqS0QkE9iQSDDGGO8kmgz+\nD5gqIunAUcBa9xgTUK71MzDG+JsnFYgikiciIRGp58X1YyUiT4jIOhFZJSKzRcS3u7mKSCcR+VhE\nNojIcK/jqYiINBKRRSKyVkQ+EJFcr2OKlYikicj7IjLX61hiISJ1ReSVyO/xWhFpE+1Y15OBiDQi\nvOKw2e1rV8ECoLmqtgQ2Ag94HE+5RCQNeBrIBpoDPxORZt5GVaES4D5VbQ78CLjX5/GeaiDwkddB\nxGEyME9Vvw+0ANZFO9CLkcFEYKgH142bqi5U1VDk7TKgkZfxVOAaYKOqblbV48Cfge4exxSVqu5Q\n1VWRrw8S/gW90NuoKhf5Q9YFmOF1LLGIjGSvV9XnAFS1RFX3Rzve1WQgIjnAF6r6gZvXdcjdwHyv\ng4jiQuCLU95vJQAfLgARuRhoCbzrbSQxOfGHLCgTbZcAu0XkucitzR9EJCPawY5volJJkdKDnF6U\n5PlyYwXxPqSqr0eOeQg4rqoveRBiyhKR2sDfgIGREYJviUhXYKeqrhKR9vjgdzcGNYHWwL2qukJE\nJhGuGh4Z7WBHqWq5FYgicjlwMbBawnunNwLeE5FrVHWX03HEKlq8J4hIb8JDwxtdCahqtgEXnfK+\nET5/TkREahJOBC+q6mtexxODtkCOiHQBMoCzReQFVe3pcVwV2Up4JL4i8v5vQNTJZc+WFkVkE9Ba\nVb/2JIAYiEgnYALwE1Xd43U80YhIDWA9kEn44bHlwM9UNepkkddE5AVgt6re53Us8RKRdkCequZ4\nHUtlRGQJcI+qbhCRkcBZqlpuQvByr0XF/0OtKUA6UBQezLBMVft5G9I3qWqpiPQnvPqRBvzR54mg\nLXAX8IGIrCT8u/CgqhZ4G1lKygX+JCK1gM+APtEOtKIjYwxgbc+MMRGWDIwxgCUDY0yEJQNjDGDJ\nwBgTYcnAGANYMjDGRFgyMMYA8P8BB4YEUxpBpuwAAAAASUVORK5CYII=\n", 421 | "text/plain": [ 422 | "" 423 | ] 424 | } 425 | } 426 | ], 427 | "execution_count": 0 428 | }, 429 | { 430 | "cell_type": "markdown", 431 | "metadata": { 432 | "id": "VYUr2uPA9ah8", 433 | "colab_type": "text" 434 | }, 435 | "source": [ 436 | "Remember that our simple network looks like this:" 437 | ] 438 | }, 439 | { 440 | "cell_type": "code", 441 | "metadata": { 442 | "id": "gt8UuSQA9frA", 443 | "colab_type": "code", 444 | "colab": { 445 | "autoexec": { 446 | "startup": false, 447 | "wait_interval": 0 448 | }, 449 | "output_extras": [ 450 | { 451 | "item_id": 1 452 | } 453 | ] 454 | }, 455 | "cellView": "form", 456 | "executionInfo": { 457 | "elapsed": 170, 458 | "status": "ok", 459 | "timestamp": 1446659140755, 460 | "user": { 461 | "color": "#1FA15D", 462 | "displayName": "Michael Piatek", 463 | "isAnonymous": false, 464 | "isMe": true, 465 | "permissionId": "00327059602783983041", 466 | "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", 467 | "sessionId": "4896c353dcc58d9f", 468 | "userId": "106975671469698476657" 469 | }, 470 | "user_tz": 480 471 | }, 472 | "outputId": "080025d5-d110-4975-e105-7635afaa3ce9" 473 | }, 474 | "source": [ 475 | "from IPython.display import Image\n", 476 | "import base64\n", 477 | "Image(data=base64.decodestring(\"iVBORw0KGgoAAAANSUhEUgAAAJYAAABkCAYAAABkW8nwAAAO90lEQVR4Xu2dT5Dc1J3Hv+YQT8VJZUhVdprLWs4FTSrGGv4ql9CuHBCH4GaTFCLZwnIcjOAy8l6Q/1SlU4XHcg6xJgtY2OOik2KxSGoTGWrXzYFC2T2MDAtWitRavmQ0e9k2SYGowom4hNRPtqA9TE+rW3/cPfPepcfup6f3fu/Tv9/T+/PVpo8//vhjsMQsULAFNjGwCrYoKy6xAAOLgVCKBRhYpZiVFcrAYgyUYgEGVilmZYUysBgDpViAgVWKWVmhDCzGQCkWGEuwrly5gtf++zW887/vYOn/lnD5T5cT40x9ZQrb/nEbxDtFiHeI2LJlSylGY4X2t8BYgUVAvfzqy3i5/TI+vPLhmq37wpYv4AHpATxw3wMMsP4cFJ5jbMAiqA4eOYg/Lv8xMcL26e34+vTXk8+vbv1q8n/03TsX38EfLv4h+aRE380dmmNwFY7O2gWOBVgE1Y/2/yjxUls+vwXaY1oS7tZK3v94MJ8zceUvV0Dea+H4AoOrQrhGHqxuT0Xjp0P7D2HqH6Yymejyu5dx5PiRZBxGnmt+bj7TdSxTfgv0ASuAzglwmyE8pfbZu3VaEDkDdT+AweevzGolvPjvL+LMb84knmr+yHxmqNKyCK7ZQ7OJ5yIo+3m6clqx8UrNB1bso2W64FQN9cnijdcdAvNAQWGRPBcLicX3Ua8S84FVcj3PnjuLhRcWkgH63OG5XHc7+NTBZEBP47NvffNbucpiF/e3QCaw2g0NfNvES5c+wtQ9u2G0LCj8BLAiFEaeBU0zYJ9fxkfYjKl7FZgtCzIHIA7QUmXov/g9LmMztt6rwLBMyFROj3TkZ0fgveXh4X96GN//zvf7t2aNHGlI7VlW0pYmRC+AKUwAsQu5thOuvIjQEjGBGJ7CQYptdOw6etc6VzXXzcUZwJrGseWt2P28DV2I4OgyDgQKFgMTYtQ1xqq10eDuR6j8Fi1NxGTkwpAfRos7h05bQscQIFgibEeHMBHCVhs4EBtY8lQQd6ulvbN78e6f302mC7Z/bXsuo9NkKk1X9PZ+IUyeR0sN4GscYl8DPzOP5VuPYynQwMU+dL4O3wzRbpQQ93O1bvQuzgRWS0p/tQA6Nuqcilq7A5u3Px28T7qw7BB1VUHqhEKTB2+pCAIVHZVD3dPgujpE6peOBzesQRS5nr/+b//g24nF7JN27qkCGq/J++RknHXm5JlVeiKGr/MQPQMdV0ZkCRBbNUwEMYzQhRyZEHgHOv29ynPM6HXtja1Rf7B4AZ7RgZv+SuMAOj+NtrYEX3avfyqMfDi2DdcLEAQBvPOX8MGtR3Ex0MEFJiRxP373wWZsvaeBhixDVRrg1/jxlwEWPV3ap+xVrR57Cjgpht2xEDV4mLIFvqkiaoUwwzp4U4Hv9/awN7YrR+vuGcAS4ZsdtKV0VNEFVqMLrIkWJGEPPP4hKA0RgiCAc1XsdJQErGQ2Ig7hOQ5sx4Hz0u+wvHX2akjtMWCpNhQCiCicq+AcCx1Fh9B2IegcNN6B4Teg1z0EeknzKqPFRe7a9AeLm4ajXvzUoJEDqUahMESrKxSqbQHbDBGLoXUNlBiuUsNOT8fFQEVsNdHmdOjStTgSGOCnLTQuBDBosLxKqnTwntw/glPnoHMS4E6iFVjgbBGcwUGMPAjtawP73GZf/wVkAutYtAvPezYUPoKjipBdGZ5vQOgavGteHbfsiXD09TZUIUbg6JD3vITlrU/iYthErPOYaQk44ZhocDF8U0HDqsEOHfQaC7/2X68lyzJVTjd0WiJu2XMem++7+tAxSd52+hguTe3GYtjq6V3XPyqDtbA/WLyAtqRg0rHhLceo3avCsk0kjqd7uoEL0FJkaC/9Hh/gS9ixS0dTCaDKHVidNhoTNN2gQP/FedAmly/t2IWm2YK2xswqDbj3antzz5oToD/915/i5smbcdo8vfaDQGiC37YfEyeW4KtcMu2g1HbCrp9Dx5Fw3ZCw04ZSb0Jse6CsLH1qgZFfK0znn+hpznzKHGpJRzus4YJ/AX/78G94ofUC7r777pwMxAhdE6pyAK8u78CJJZ+BtcKiIw8Wea0DTx34ZCH5oHYwM1y0TjhnziXbaWgB+4cP/RCPPfYYtm/fjpMnT+Kmm24aDrDYhdpoQdAbaMtNSB4Da6UhRx4sqnB3SCTPNbtvtu9iMoU/Wg5Kt9p0h8DTp09j3759ePrpp/H4448PB1fylOtC5jTUGVifseFYgJXClXou+jcN6Gk2nj7JG1Gi7TG0Hkiz7OlGP/ru6OGjq46rnnjiCSwuLibe66677hocMAZWT5uNDVgpXGfbZ5OtybQNZq1EE6G0NXmXtGvNwbrv+4n3uu222wYPjwys9QFW2goKjbQ4Tdth6CAFeSpK5J3oQMUwhynS8PjMM89AVdVs3ouBtb7Aytbrw+WiMZfnednCIwOLgTUIZml43LFjB5577rnhnx4Huek6yztWY6yqbb+wsJBMTwwUHquu5Ijej4GVoWMoPJ4/fz7xXkM9PWa4x3rLwsDK2KMXLlxIvBeFR5qe2LRpU8YrN2Y2BtaA/U7hkaYnnn322exPjwPeYz1kZ2AN2YtpeCTvdeeddw5Zyvq9jIGVo28pPJL3ok2NLDxeb0gGVg6w0kvT8HjixIlkHJY1lauaE8GRangwsvD/noKqt+kzsLJSkCEfzdi/8cYbifdaKzxWoppDmxJ5FT54NH06YZShAQVmYWAVaEwqKg2PMzMzyfTEyqfHqlRzAoOH6OqwJnXoNQeBSWcjq0sMrJJsferUqSQsdofHylRzYg8aLyG0QtiTOvhGhFZglyKD0Mt8DKySwEqLpfD45ptvYn5+Hr/+z19/sukwj2pOP72vyJXBy4BNME340Pg6AiNAu8IDkQysksGi4t9++2189wffxee++DkIO4TcqjlrSw504Eg81FobYetq+KOwKDgagjVOnRdtBgZW0RZdpbw0BL73/nv4yZM/6bv7tVeVxkk1h4FVAVgbUTWHgVUBWGUcvCVV6EP/cuiztQ9NCNsMiIshrPSIeaK3oUNIlXQqaDMDqwIjlyEV0Fv6MoQlbENT/FTIhWSXOF2AF5jocei8cCswsAo36WcLLEPchO7yyr+9smrt6TQ3geQmcgcd2CQbIHoIDKGyuSwG1joEi06oU+jj3RAWR2HQgFiiTuxqJmRgVQBWGaGQDo78/OjPe9T+qpfSeBeeqIM3JPip4k8F7aVbMLAqMHSlg/dr7YkcCZxWg1Jz0G5UL7/EwKoArBuhmoNEbupBvPrRDhxf8qFVLFrCwKoArFQi4P3o/VwTpCmgdBi3r2oOIrQbNdwfGljytZ46r2U1n4FVlmW7yn3rrbfwvX/+XrKkMyPM5FLNIS2KbCrSNI8loKX48G6AxhIDq2SwaIcDgWWaJn71H78qRDWnlxbF1aaQxJILj6TRjRhm0L4hYrwMrJLAos1+BBXtyaLty5SKVs1Zverx1RB4dhIPPe/CVioeXF2rFAOrYLDIOxFQd9xxRwLVytSt90XfFaGaU3ATCimOgVWIGa8WkoY9AorA6pUIrqJVcwpsRiFFMbAKMONqYS9LsWWo5mS5bxV5GFg5rExhj8ZPdHBitbCXo+ixv5SBNWQXpmGPvNXtt98+ZCnr9zIG1oB9O2zYG/A2Y5+dgZWxC1nYy2goNt2Q3VA0jqIDESzsZbcZ81hr2CoNe/T56KOPZrcqy8m2zazGAAt7+X8ZzGOtsCELe/mhohLGEqwyVFpY2CsGqLSUsQKrDJUWFvaKBWrswCpDpYWFvXKgKiYUxh5U/huwhd8idBqYRARX4bHTldd8Le8gTSpapYWWX0is47qnveTdi02I6aFOejlAbSdcOT2fF8NTOEixDTqnV6Uk0CC2GpW8hYTCyFXA72yj8XoAAzoE+nsxgNnrZc8DtL7bU9HJlDwqLY9855FkbY8ktS3LWlGLECbPo6UG8DUOsa+Bn5nH8q3HsRRo4GISL6vDN0O0e70SdoB2rfeshYBF71Juyzzu90TcF59FIC8WJvSVvgiT9nnPH5nP/K7CtOPonYWzh2aTF2Fu+usmvPjLF3us7cXwdR6iZ6DjyogsAWKrhokghhG6kCMTAu9Ap7+r1l0cQwoLAote4+ugwT+IsxO78XrQKkTkqzsEkqeily8Nk0il5cfHfowv3/xlLBxf6Pk2sNhTwEkx7I6FqMHDlC3wTRVRK4QZ1sGbCnxfrfxgwjBtvtHXFAZW7OsQZo7hEm7Fkxf8nm+mH6TBlau0RG00OBWcY6Gj6BDaLgSdDn46MPwG9Hr15/MGsdco5S0GrDiAIU7D5M/AgIo9gY6Lng4+5wi3jIOea59wieCQzgEnAe4kWoEFzhbBGRzEyIPQDmBWpaoxSpQMUZdCwCLh1OlmDWcCBzJsSNzDiIyL8LR8Ur1lHE2nPeZzh+d6mooENW7Zcx6b7zuHTlvCJB1Nnz6GS1O7sUhKxDl/LEP00Vhekh8sUjThNUyYAdxr59dCSwSvAWbg5Xq7exkqLfRO6TMnz/TurNAEv20/Jk4swaf2xC6U2k7Y9XPoOBIm6crYh6UoaLodABOoSU3YlpLbQ48lQT0qnR+sEq1RBlj0dGmfsnPVOtB51IMmfEdGLQ7RkkSYkps8VbJ01QIjDdaNCIVZwOi4DnxOgsRRXIzhazwakY3gmphsljLWe56RBqv6wfvg3R0HFqS6CcHxC5kQHrwGo3nFSIN1Q1RaBuinyDchSyYmDRcthWPLPF22G2mwuo+k55kgHUylJRtZoa1A0kI0bAdGPRnSszQuYFE90yUdepoznzKHWtLRDmsglZY8cHZTE7UVCGqEpmtDScZZLK20wEh7LKpst9YBKQUf1A5mhovWCefMuU9eM9JbWnEQMAIY/DQOXLr+mqmHXkfIdj18YpSRByuFa6+2F1f+cgXkuWb3zfZdN6Twt/DCQuKpsgmVDQIXy9vPAmMB1krPRf9eryot/TpsXL4fG7BSuNa7Ssu4gNOvnmMFVtqY9azS0q/DxuX7sQRrXIy7kevJwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3Ixf9d0NIelzdt4X5AAAAAElFTkSuQmCC\"), embed=True)" 478 | ], 479 | "outputs": [ 480 | { 481 | "output_type": "execute_result", 482 | "execution_count": 6, 483 | "metadata": {}, 484 | "data": { 485 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAJYAAABkCAYAAABkW8nwAAAO90lEQVR4Xu2dT5Dc1J3Hv+YQT8VJ\nZUhVdprLWs4FTSrGGv4ql9CuHBCH4GaTFCLZwnIcjOAy8l6Q/1SlU4XHcg6xJgtY2OOik2KxSGoT\nGWrXzYFC2T2MDAtWitRavmQ0e9k2SYGowom4hNRPtqA9TE+rW3/cPfPepcfup6f3fu/Tv9/T+/PV\npo8//vhjsMQsULAFNjGwCrYoKy6xAAOLgVCKBRhYpZiVFcrAYgyUYgEGVilmZYUysBgDpViAgVWK\nWVmhDCzGQCkWGEuwrly5gtf++zW887/vYOn/lnD5T5cT40x9ZQrb/nEbxDtFiHeI2LJlSylGY4X2\nt8BYgUVAvfzqy3i5/TI+vPLhmq37wpYv4AHpATxw3wMMsP4cFJ5jbMAiqA4eOYg/Lv8xMcL26e34\n+vTXk8+vbv1q8n/03TsX38EfLv4h+aRE380dmmNwFY7O2gWOBVgE1Y/2/yjxUls+vwXaY1oS7tZK\n3v94MJ8zceUvV0Dea+H4AoOrQrhGHqxuT0Xjp0P7D2HqH6Yymejyu5dx5PiRZBxGnmt+bj7TdSxT\nfgv0ASuAzglwmyE8pfbZu3VaEDkDdT+AweevzGolvPjvL+LMb84knmr+yHxmqNKyCK7ZQ7OJ5yIo\n+3m6clqx8UrNB1bso2W64FQN9cnijdcdAvNAQWGRPBcLicX3Ua8S84FVcj3PnjuLhRcWkgH63OG5\nXHc7+NTBZEBP47NvffNbucpiF/e3QCaw2g0NfNvES5c+wtQ9u2G0LCj8BLAiFEaeBU0zYJ9fxkfY\njKl7FZgtCzIHIA7QUmXov/g9LmMztt6rwLBMyFROj3TkZ0fgveXh4X96GN//zvf7t2aNHGlI7VlW\n0pYmRC+AKUwAsQu5thOuvIjQEjGBGJ7CQYptdOw6etc6VzXXzcUZwJrGseWt2P28DV2I4OgyDgQK\nFgMTYtQ1xqq10eDuR6j8Fi1NxGTkwpAfRos7h05bQscQIFgibEeHMBHCVhs4EBtY8lQQd6ulvbN7\n8e6f302mC7Z/bXsuo9NkKk1X9PZ+IUyeR0sN4GscYl8DPzOP5VuPYynQwMU+dL4O3wzRbpQQ93O1\nbvQuzgRWS0p/tQA6Nuqcilq7A5u3Px28T7qw7BB1VUHqhEKTB2+pCAIVHZVD3dPgujpE6peOBzes\nQRS5nr/+b//g24nF7JN27qkCGq/J++RknHXm5JlVeiKGr/MQPQMdV0ZkCRBbNUwEMYzQhRyZEHgH\nOv29ynPM6HXtja1Rf7B4AZ7RgZv+SuMAOj+NtrYEX3avfyqMfDi2DdcLEAQBvPOX8MGtR3Ex0MEF\nJiRxP373wWZsvaeBhixDVRrg1/jxlwEWPV3ap+xVrR57Cjgpht2xEDV4mLIFvqkiaoUwwzp4U4Hv\n9/awN7YrR+vuGcAS4ZsdtKV0VNEFVqMLrIkWJGEPPP4hKA0RgiCAc1XsdJQErGQ2Ig7hOQ5sx4Hz\n0u+wvHX2akjtMWCpNhQCiCicq+AcCx1Fh9B2IegcNN6B4Teg1z0EeknzKqPFRe7a9AeLm4ajXvzU\noJEDqUahMESrKxSqbQHbDBGLoXUNlBiuUsNOT8fFQEVsNdHmdOjStTgSGOCnLTQuBDBosLxKqnTw\nntw/glPnoHMS4E6iFVjgbBGcwUGMPAjtawP73GZf/wVkAutYtAvPezYUPoKjipBdGZ5vQOgavGte\nHbfsiXD09TZUIUbg6JD3vITlrU/iYthErPOYaQk44ZhocDF8U0HDqsEOHfQaC7/2X68lyzJVTjd0\nWiJu2XMem++7+tAxSd52+hguTe3GYtjq6V3XPyqDtbA/WLyAtqRg0rHhLceo3avCsk0kjqd7uoEL\n0FJkaC/9Hh/gS9ixS0dTCaDKHVidNhoTNN2gQP/FedAmly/t2IWm2YK2xswqDbj3antzz5oToD/9\n15/i5smbcdo8vfaDQGiC37YfEyeW4KtcMu2g1HbCrp9Dx5Fw3ZCw04ZSb0Jse6CsLH1qgZFfK0zn\nn+hpznzKHGpJRzus4YJ/AX/78G94ofUC7r777pwMxAhdE6pyAK8u78CJJZ+BtcKiIw8Wea0DTx34\nZCH5oHYwM1y0TjhnziXbaWgB+4cP/RCPPfYYtm/fjpMnT+Kmm24aDrDYhdpoQdAbaMtNSB4Da6Uh\nRx4sqnB3SCTPNbtvtu9iMoU/Wg5Kt9p0h8DTp09j3759ePrpp/H4448PB1fylOtC5jTUGVifseFY\ngJXClXou+jcN6Gk2nj7JG1Gi7TG0Hkiz7OlGP/ru6OGjq46rnnjiCSwuLibe66677hocMAZWT5uN\nDVgpXGfbZ5OtybQNZq1EE6G0NXmXtGvNwbrv+4n3uu222wYPjwys9QFW2goKjbQ4Tdth6CAFeSpK\n5J3oQMUwhynS8PjMM89AVdVs3ouBtb7Aytbrw+WiMZfnednCIwOLgTUIZml43LFjB5577rnhnx4H\nuek6yztWY6yqbb+wsJBMTwwUHquu5Ijej4GVoWMoPJ4/fz7xXkM9PWa4x3rLwsDK2KMXLlxIvBeF\nR5qe2LRpU8YrN2Y2BtaA/U7hkaYnnn322exPjwPeYz1kZ2AN2YtpeCTvdeeddw5Zyvq9jIGVo28p\nPJL3ok2NLDxeb0gGVg6w0kvT8HjixIlkHJY1lauaE8GRangwsvD/noKqt+kzsLJSkCEfzdi/8cYb\nifdaKzxWoppDmxJ5FT54NH06YZShAQVmYWAVaEwqKg2PMzMzyfTEyqfHqlRzAoOH6OqwJnXoNQeB\nSWcjq0sMrJJsferUqSQsdofHylRzYg8aLyG0QtiTOvhGhFZglyKD0Mt8DKySwEqLpfD45ptvYn5+\nHr/+z19/sukwj2pOP72vyJXBy4BNME340Pg6AiNAu8IDkQysksGi4t9++2189wffxee++DkIO4Tc\nqjlrSw504Eg81FobYetq+KOwKDgagjVOnRdtBgZW0RZdpbw0BL73/nv4yZM/6bv7tVeVxkk1h4FV\nAVgbUTWHgVUBWGUcvCVV6EP/cuiztQ9NCNsMiIshrPSIeaK3oUNIlXQqaDMDqwIjlyEV0Fv6MoQl\nbENT/FTIhWSXOF2AF5jocei8cCswsAo36WcLLEPchO7yyr+9smrt6TQ3geQmcgcd2CQbIHoIDKGy\nuSwG1joEi06oU+jj3RAWR2HQgFiiTuxqJmRgVQBWGaGQDo78/OjPe9T+qpfSeBeeqIM3JPip4k8F\n7aVbMLAqMHSlg/dr7YkcCZxWg1Jz0G5UL7/EwKoArBuhmoNEbupBvPrRDhxf8qFVLFrCwKoArFQi\n4P3o/VwTpCmgdBi3r2oOIrQbNdwfGljytZ46r2U1n4FVlmW7yn3rrbfwvX/+XrKkMyPM5FLNIS2K\nbCrSNI8loKX48G6AxhIDq2SwaIcDgWWaJn71H78qRDWnlxbF1aaQxJILj6TRjRhm0L4hYrwMrJLA\nos1+BBXtyaLty5SKVs1Zverx1RB4dhIPPe/CVioeXF2rFAOrYLDIOxFQd9xxRwLVytSt90XfFaGa\nU3ATCimOgVWIGa8WkoY9AorA6pUIrqJVcwpsRiFFMbAKMONqYS9LsWWo5mS5bxV5GFg5rExhj8ZP\ndHBitbCXo+ixv5SBNWQXpmGPvNXtt98+ZCnr9zIG1oB9O2zYG/A2Y5+dgZWxC1nYy2goNt2Q3VA0\njqIDESzsZbcZ81hr2CoNe/T56KOPZrcqy8m2zazGAAt7+X8ZzGOtsCELe/mhohLGEqwyVFpY2CsG\nqLSUsQKrDJUWFvaKBWrswCpDpYWFvXKgKiYUxh5U/huwhd8idBqYRARX4bHTldd8Le8gTSpapYWW\nX0is47qnveTdi02I6aFOejlAbSdcOT2fF8NTOEixDTqnV6Uk0CC2GpW8hYTCyFXA72yj8XoAAzoE\n+nsxgNnrZc8DtL7bU9HJlDwqLY9855FkbY8ktS3LWlGLECbPo6UG8DUOsa+Bn5nH8q3HsRRo4GIS\nL6vDN0O0e70SdoB2rfeshYBF71Juyzzu90TcF59FIC8WJvSVvgiT9nnPH5nP/K7CtOPonYWzh2aT\nF2Fu+usmvPjLF3us7cXwdR6iZ6DjyogsAWKrhokghhG6kCMTAu9Ap7+r1l0cQwoLAote4+ugwT+I\nsxO78XrQKkTkqzsEkqeily8Nk0il5cfHfowv3/xlLBxf6Pk2sNhTwEkx7I6FqMHDlC3wTRVRK4QZ\n1sGbCnxfrfxgwjBtvtHXFAZW7OsQZo7hEm7Fkxf8nm+mH6TBlau0RG00OBWcY6Gj6BDaLgSdDn46\nMPwG9Hr15/MGsdco5S0GrDiAIU7D5M/AgIo9gY6Lng4+5wi3jIOea59wieCQzgEnAe4kWoEFzhbB\nGRzEyIPQDmBWpaoxSpQMUZdCwCLh1OlmDWcCBzJsSNzDiIyL8LR8Ur1lHE2nPeZzh+d6mooENW7Z\ncx6b7zuHTlvCJB1Nnz6GS1O7sUhKxDl/LEP00Vhekh8sUjThNUyYAdxr59dCSwSvAWbg5Xq7exkq\nLfRO6TMnz/TurNAEv20/Jk4swaf2xC6U2k7Y9XPoOBIm6crYh6UoaLodABOoSU3YlpLbQ48lQT0q\nnR+sEq1RBlj0dGmfsnPVOtB51IMmfEdGLQ7RkkSYkps8VbJ01QIjDdaNCIVZwOi4DnxOgsRRXIzh\nazwakY3gmphsljLWe56RBqv6wfvg3R0HFqS6CcHxC5kQHrwGo3nFSIN1Q1RaBuinyDchSyYmDRct\nhWPLPF22G2mwuo+k55kgHUylJRtZoa1A0kI0bAdGPRnSszQuYFE90yUdepoznzKHWtLRDmsglZY8\ncHZTE7UVCGqEpmtDScZZLK20wEh7LKpst9YBKQUf1A5mhovWCefMuU9eM9JbWnEQMAIY/DQOXLr+\nmqmHXkfIdj18YpSRByuFa6+2F1f+cgXkuWb3zfZdN6Twt/DCQuKpsgmVDQIXy9vPAmMB1krPRf9e\nryot/TpsXL4fG7BSuNa7Ssu4gNOvnmMFVtqY9azS0q/DxuX7sQRrXIy7kevJwNrIvV9i2xlYJRp3\nIxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3Ixf9d0NIelzdt4X5\nAAAAAElFTkSuQmCC\n", 486 | "text/plain": [ 487 | "" 488 | ] 489 | } 490 | } 491 | ], 492 | "execution_count": 6 493 | }, 494 | { 495 | "cell_type": "markdown", 496 | "metadata": { 497 | "id": "Ft95NDUZy4Rr", 498 | "colab_type": "text" 499 | }, 500 | "source": [ 501 | "That's equivalent to the function $\\hat{y} = w_2 x + w_1$. What we're trying to do is find the \"best\" weights $w_1$ and $w_2$. That will give us that green regression line above.\n", 502 | "\n", 503 | "What are the best weights? They're the weights that minimize the difference between our estimate $\\hat{y}$ and the actual y. Specifically, we want to minimize the sum of the squared errors, so minimize $\\sum{(\\hat{y} - y)^2}$, which is known as the *L2 loss*. So, the best weights are the weights that minimize the L2 loss." 504 | ] 505 | }, 506 | { 507 | "cell_type": "markdown", 508 | "metadata": { 509 | "id": "RHDGz_14vGNg", 510 | "colab_type": "text" 511 | }, 512 | "source": [ 513 | "## Gradient descent\n", 514 | "\n", 515 | "What gradient descent does is start with random weights for $\\hat{y} = w_2 x + w_1$ and gradually moves those weights toward better values.\n", 516 | "\n", 517 | "It does that by following the downward slope of the error curves. Imagine that the possible errors we could get with different weights as a landscape. From whatever weights we have, moving in some directions will increase the error, like going uphill, and some directions will decrease the error, like going downhill. We want to roll downhill, always moving the weights toward lower error.\n", 518 | "\n", 519 | "How does gradient descent know which way is downhill? It follows the partial derivatives of the L2 loss. The partial derivative is like a velocity, saying which way the error will change if we change the weight. We want to move in the direction of lower error. The partial derivative points the way.\n", 520 | "\n", 521 | "So, what gradient descent does is start with random weights and gradually walk those weights toward lower error, using the partial derivatives to know which direction to go." 522 | ] 523 | }, 524 | { 525 | "cell_type": "markdown", 526 | "metadata": { 527 | "id": "W7SgnPAWBX2M", 528 | "colab_type": "text" 529 | }, 530 | "source": [ 531 | "## The code again\n", 532 | "\n", 533 | "Let's go back to the code now, walking through it with many more comments in the code this time:" 534 | ] 535 | }, 536 | { 537 | "cell_type": "code", 538 | "metadata": { 539 | "id": "4qtXAPGmBWUW", 540 | "colab_type": "code", 541 | "colab": { 542 | "autoexec": { 543 | "startup": false, 544 | "wait_interval": 0 545 | }, 546 | "output_extras": [ 547 | { 548 | "item_id": 1 549 | } 550 | ] 551 | }, 552 | "cellView": "both", 553 | "executionInfo": { 554 | "elapsed": 718, 555 | "status": "ok", 556 | "timestamp": 1446659172854, 557 | "user": { 558 | "color": "#1FA15D", 559 | "displayName": "Michael Piatek", 560 | "isAnonymous": false, 561 | "isMe": true, 562 | "permissionId": "00327059602783983041", 563 | "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", 564 | "sessionId": "4896c353dcc58d9f", 565 | "userId": "106975671469698476657" 566 | }, 567 | "user_tz": 480 568 | }, 569 | "outputId": "0664707f-ea8a-453b-fc3f-48d5ca0f76dc" 570 | }, 571 | "source": [ 572 | "#@test {\"output\": \"ignore\"}\n", 573 | "import tensorflow as tf\n", 574 | "import numpy as np\n", 575 | "import matplotlib.pyplot as plt\n", 576 | "\n", 577 | "# Set up the data with a noisy linear relationship between X and Y.\n", 578 | "num_examples = 50\n", 579 | "X = np.array([np.linspace(-2, 4, num_examples), np.linspace(-6, 6, num_examples)])\n", 580 | "# Add random noise (gaussian, mean 0, stdev 1)\n", 581 | "X += np.random.randn(2, num_examples)\n", 582 | "# Split into x and y\n", 583 | "x, y = X\n", 584 | "# Add the bias node which always has a value of 1\n", 585 | "x_with_bias = np.array([(1., a) for a in x]).astype(np.float32)\n", 586 | "\n", 587 | "# Keep track of the loss at each iteration so we can chart it later\n", 588 | "losses = []\n", 589 | "# How many iterations to run our training\n", 590 | "training_steps = 50\n", 591 | "# The learning rate. Also known has the step size. This changes how far\n", 592 | "# we move down the gradient toward lower error at each step. Too large\n", 593 | "# jumps risk inaccuracy, too small slow the learning.\n", 594 | "learning_rate = 0.002\n", 595 | "\n", 596 | "# In TensorFlow, we need to run everything in the context of a session.\n", 597 | "with tf.Session() as sess:\n", 598 | " # Set up all the tensors.\n", 599 | " # Our input layer is the x value and the bias node.\n", 600 | " input = tf.constant(x_with_bias)\n", 601 | " # Our target is the y values. They need to be massaged to the right shape.\n", 602 | " target = tf.constant(np.transpose([y]).astype(np.float32))\n", 603 | " # Weights are a variable. They change every time through the loop.\n", 604 | " # Weights are initialized to random values (gaussian, mean 0, stdev 0.1)\n", 605 | " weights = tf.Variable(tf.random_normal([2, 1], 0, 0.1))\n", 606 | "\n", 607 | " # Initialize all the variables defined above.\n", 608 | " tf.initialize_all_variables().run()\n", 609 | " \n", 610 | " # Set up all operations that will run in the loop.\n", 611 | " # For all x values, generate our estimate on all y given our current\n", 612 | " # weights. So, this is computing y = w2 * x + w1 * bias\n", 613 | " yhat = tf.matmul(input, weights)\n", 614 | " # Compute the error, which is just the difference between our \n", 615 | " # estimate of y and what y actually is.\n", 616 | " yerror = tf.sub(yhat, target)\n", 617 | " # We are going to minimize the L2 loss. The L2 loss is the sum of the\n", 618 | " # squared error for all our estimates of y. This penalizes large errors\n", 619 | " # a lot, but small errors only a little.\n", 620 | " loss = tf.nn.l2_loss(yerror)\n", 621 | "\n", 622 | " # Perform gradient descent. \n", 623 | " # This essentially just updates weights, like weights += grads * learning_rate\n", 624 | " # using the partial derivative of the loss with respect to the\n", 625 | " # weights. It's the direction we want to go to move toward lower error.\n", 626 | " update_weights = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)\n", 627 | " \n", 628 | " # At this point, we've defined all our tensors and run our initialization\n", 629 | " # operations. We've also set up the operations that will repeatedly be run\n", 630 | " # inside the training loop. All the training loop is going to do is \n", 631 | " # repeatedly call run, inducing the gradient descent operation, which has the effect of\n", 632 | " # repeatedly changing weights by a small amount in the direction (the\n", 633 | " # partial derivative or gradient) that will reduce the error (the L2 loss).\n", 634 | " for _ in range(training_steps):\n", 635 | " # Repeatedly run the operations, updating the TensorFlow variable.\n", 636 | " sess.run(update_weights)\n", 637 | " \n", 638 | " # Here, we're keeping a history of the losses to plot later\n", 639 | " # so we can see the change in loss as training progresses.\n", 640 | " losses.append(loss.eval())\n", 641 | "\n", 642 | " # Training is done, get the final values for the charts\n", 643 | " betas = weights.eval()\n", 644 | " yhat = yhat.eval()\n", 645 | "\n", 646 | "# Show the results.\n", 647 | "fig, (ax1, ax2) = plt.subplots(1, 2)\n", 648 | "plt.subplots_adjust(wspace=.3)\n", 649 | "fig.set_size_inches(10, 4)\n", 650 | "ax1.scatter(x, y, alpha=.7)\n", 651 | "ax1.scatter(x, np.transpose(yhat)[0], c=\"g\", alpha=.6)\n", 652 | "line_x_range = (-4, 6)\n", 653 | "ax1.plot(line_x_range, [betas[0] + a * betas[1] for a in line_x_range], \"g\", alpha=0.6)\n", 654 | "ax2.plot(range(0, training_steps), losses)\n", 655 | "ax2.set_ylabel(\"Loss\")\n", 656 | "ax2.set_xlabel(\"Training steps\")\n", 657 | "plt.show()" 658 | ], 659 | "outputs": [ 660 | { 661 | "output_type": "display_data", 662 | "metadata": {}, 663 | "data": { 664 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlUAAAEPCAYAAABr+zG+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl8VPW9//HXJwtLIKwBQVAEEdlETBUXLM2VKlZR22uv\n2NpatNdb6lqtGm17L1j7uwgu1KU+qPe617Vqi3LdEIm4VFHjggNaXNjXsIU1ZPn8/pgJjiEJSWY5\nk5n38/GYR2bOnDnnM0K+fviez/l8zd0RERERkdhkBR2AiIiISDpQUiUiIiISB0qqREREROJASZWI\niIhIHCipEhEREYkDJVUiIiIicRCXpMrMrjezkJl9bGaPmFmbeBxXRCRRzKyvmb0aGbsWmtnlke1d\nzexlM/vMzF4ys85Rn7nezJaY2WIzOyW46EUkFcWcVJlZP+Ai4Ch3HwHkAOfGelwRkQSrAq5y92HA\n8cAlZjYYuA54xd0PB14Frgcws6HAOcAQ4HvA3WZmgUQuIikpHjNV5cAeoIOZ5QB5wOo4HFdEJGHc\nfa27fxh5vh1YDPQFzgIejOz2IPD9yPMzgcfdvcrdlwJLgFFJDVpEUlrMSZW7bwZuBZYDq4At7v5K\nrMcVEUkWMzsEGAm8DRzg7usgnHgBPSO79QFWRH1sVWSbiAgQn8t/A4ArgX7AgUBHM/txrMcVEUkG\nM+sIPAVcEZmxqrt2l9byEpEmyYnDMY4G3nT3TQBm9gxwAvBo9E5mpoFJJE24e1rUEkVKFp4CHnb3\nWZHN68zsAHdfZ2a9gPWR7auAg6I+3jeyre4xNdaJpInmjnXxqKn6DDjOzNpFijbHEq5NqC+4lHhM\nnjw58BhSKQ7FktpxpFosaeY+YJG73x617VlgYuT5z4BZUdvPNbM2ZtYfGAgsqO+gQf8ZpdPft0yM\nPx2+Q2uP371lY13MM1Xu/pGZPQS8D1QDHwD3xHpcEZFEMrPRwHnAQjP7gPBlvt8A04AnzexCYBnh\nO/5w90Vm9iSwCKgELvaWjrwikpbicfkPd78ZuDkexxIRSQZ3fxPIbuDt7zbwmanA1IQFJSKtWkZ2\nVC8qKgo6BCB14gDFUp9UiQNSKxZJf63971trjx9a/3do7fG3lCVr9trMNFMukgbMDE+TQvVE0Fgn\nkh5aMtZl5EyViIiISLwpqRIRERGJAyVVIimuoqqixbf3iohI8iipEklhVTVV3LXgLhasqrcdkoiI\npBAlVSIpyt154MMH6NCmA8f0OSbocEREZD+UVImkqGcWP8OmXZv4+VE/J8v0qyoikuo0UoukoHlf\nzeOjdR9xyTGXkJudG3Q4IiLSBEqqRFLMB2s+4MXPX+TyYy+nQ5sOQYcjIiJNpKRKJIV8sekL/vLx\nX7j4mIspyCsIOhwREWkGJVUiKWLd9nXMfG8mFxx1Af269As6HBERaSYlVSIpoLyinDveuYPvD/4+\nw3sODzocERFpASVVIgGrqKrgrgV3cVzf4xh98OigwxERkRZSUiUSoBqv4Z7376Fvp76MHzQ+6HBE\nRCQGSqpEAuLuPPLxIzjOeUech1mzFkMXEZEUo6RKJCDPL3me5VuX84tv/YLsrOygwxERkRgpqRIJ\nwFsr3uLNFW9y2bGX0TanbdDhiIhIHCipEkmy0PoQzyx+hsuPvZxObTsFHY6IiMRJXJIqM+tsZn81\ns8VmFjKzY+NxXJF0s3zrcu774D4mHT2JXh17BR1ORjOze81snZl9HLVthJm9ZWYfmdksM+sY9d71\nZrYkMs6d0tix3RMZuYikKvM4/Pab2QPAa+5+v5nlAHnuXl5nH4/HuURKS0t54omXAJgwYRyFhYUB\nR9Q0G3duZPqb05kwfAKFvVtHzPUxM9y91VfVm9mJwHbgIXcfEdm2ALjK3d8ws4nAAHf/LzMbCjwC\nHAP0BV4BDqtvUDMz377d6aAVhkRatZaMdTHPVJlZJ+Db7n4/gLtX1U2oROKltLSUSZNmMH/+KObP\nH8WkSTMoLS0NOqz92rFnB3e8cwfjBo5r1QlVOnH3N4DNdTYfFtkO4cTp7MjzM4HHI+PbUmAJMKqh\nY2/bFudgRaRViMflv/5AmZndb2alZnaPmbWPw3FF9vHEEy+RnT2RgoKxFBSMJTt74t5Zq1RVWV3J\n3e/ezREHHMFJ/U8KOhxpXMjMzow8P4fwrBRAH2BF1H6rItvqpaRKJDPlxOkYhcAl7v6emf0RuA6Y\nHIdji7Rq7s59H9xHl3ZdOHvI2fv/gATtQuBOM/tP4FlgT0sOcsstU+jdO/y8qKiIoqKieMUnIglS\nUlJCSUlJTMeIR1K1Eljh7u9FXj8FFNe345QpU/Y+10AjLTFhwjjmzZtBWVn4dXX1A0yYcGWwQdUR\nCoWYPWc2ALlDcqnJr+GKY69otc094zHQtBbu/k9gHICZHQacHnlrFXBQ1K59I9vq9eMfT+E730lU\nlCKSCHXzkhtuuKHZx4hXofprwEXu/k8zm0y4UL24zj4qVJe4eOSRR7jjjicAuPzyCZx33nlxPX4s\nhfChUIhrb76W3GG5rK9ez4a1G3jsgsc4+sij4xpjkNKlUB3AzA4BnnP3IyKve7j7BjPLAu4H5rn7\nA1GF6scSvuw3h0YK1Z97zhmvVYdEWrWWjHXxmKkCuBx4xMxygS+BC+J0XJFvKC0t5fbbXyQ7+woA\nbr/9AYYMGbI38Yn1zsDaQvjs7IkAzJs3g5kzr2zycWbPmU3usFyyDspi55adHJZ1GHPnzU2rpCpd\nmNmjQBHQ3cyWEy5ZyDezSwAHnnH3BwDcfZGZPQksAiqBixv7V6JqqkQyU1ySKnf/iPCtxiIJFV2o\nDlBWFt5WWFgYc0K0v+M31faa7WzYvIHhPYezY9uOJn9Oksvdf9zAW3c0sP9UYGpTjq2kSiQzqaO6\npI1UuDPw2BOP5ct1X1JQXsCOL3dQGapk/Mm6DpRplFSJZKZ4Xf4TSYpEF6rHcvwtu7fw8taX+cP3\n/8CK0vDd9+OvGc+wYcPiFp+0DkqqRDJTXArVm3QiFapLnDRUN1X38l919QPNvvzX2PEbs7tqN7e8\ndQuFvQs57bDTmnW+1iadCtUTwcz8qqucW28NOhIRiUVLxjolVZJWgljCprqmmrsW3EX3vO6cd8R5\nrbZ1QlMpqWqcmflFFzn33BN0JCISiyDv/hNJCYWFhQlLpOpL2Nydhz9+mOysbH58xI8xs1a7NqHE\njy7/iWQmJVUiTdDQnYUrO6xkzbY1XHX8VWRZVlzuQJTWr1yrn4pkJCVVIk1QX6uFW/46kwEn96R4\ndDFtc9o2uF9zWzJI66eZKpHMpJYKIi2ws+OXfJ79KZcfezn5bfODDkdSjJIqkcykmSqRJpgwYRzP\nPz+ZVRsfp6rDNvYMfZf7xs2gZ4ee++yX6msTSuIpqRLJTEqqROoRXWx+9NED+cd7/2Bb3kJqRi5k\nV8E2Dll7MId2O3SfzxUWFjJz5pVRheqqp8pESqpEMpNaKojUEV1svnPnUpbt/C+6DWzLjn472N1t\nN0f1O4ou5V04Pvd4in9VvN/jpRu1VGicmXn79s7OnUFHIiKxaMlYp5oqkTqii833ZJWRNfgQdloF\ne7ruoW37tuzatCvoECXFVVRAVVXQUYhIsimpEtkPx/HDHVtlZK3MYtvybVrTTxrVsSNs3x50FCKS\nbEqqROqYMGEcZWU38nrpKFasepid2z8mv2MeI3qMoMsnXTi1y6lMv2a61vSTBuXnq65KJBOpUF2a\nJRO6hb/55pus3PMO1X2q4VDDKmDUulEcOfRIxl+oBZJl/5RUiWQmJVXSZJnQLTwUCvH7//k9NSfW\nkNM9h6rtVXSo6cjmTZszsihdWkZJlUhmUlIlTZYJ3cJnz5mNHW5YX6Mmv4bcmlwqP6+Envv/rEgt\nJVUimUk1VSJ19OnXh6rKKtgGVZurqFpWRfuaAoqLp1JaWhp0eNIKKKkSyUxKqqTJJkwYR3X1A5SV\nzaWsbG6kW/i4oMOKq29/59uUV5dzePbhdPqyE3kf5NHDh7NmzUTmzx/FpEkzlFjJfimpEslMcbv8\nZ2ZZwHvASnc/M17HldSRrt3CQ6EQs+fMptIrWdt3LcXji9m8cDP0gX92WsOiRWek9SXPTGVm9wLj\ngXXuPiKy7RjgT0AuUAlc7O7vRd67HrgQqAKucPeXGzq2kiqRzBTPmqorgEVApzgeU1JMYWFhWiUU\noVCIa2++luyh2XxV9RU5JTlM+uUkho8dDkBx8dSAI5QEuh+4E3goatt04Hfu/rKZfQ+4GfgXMxsK\nnAMMAfoCr5jZYQ0tE5GfD+XliQ1eRFJPXJIqM+sLnAb8P+CqeBxTJFGi20LsqllDztActvTYQmfv\nTPec7vzfK//H8OHhpEoLJKcvd3/DzPrV2bwG6Bx53gVYFXl+JvC4u1cBS81sCTAKeKe+Y2umSiQz\nxWumagZwDV8PRiIpqW5biBUbrqbLuZV06NqBI3oeQdnGsm/sn66XPKVB1wFvmtmtgAEnRLb3Af4R\ntd+qyLZ65efDunUJi1FEUlTMSZWZnU64JuFDMysiPBDVa8qUKXufFxUVUVRUFOvpRZrliSdeoqKi\niD1Z7wFgvY9i1bJnGd3pOMq2lIWXn7nmm8vPpNslz+YqKSmhpKQk6DCS5V7gMnf/u5n9ELgPOLm5\nB3n11SksWwZTpmisE2kt4jHWWQMlAU0/gNl/Az8hXLzZHsgHnnH38+vs11D5gUjS/Pznv+KZ+W/T\n9sheVLbbyo49nzHWRzNm9NEAjD9ZHdP3pyUrt6eqyOW/56IK1cvdvVPU+1vcvYuZXQe4u0+LbH8R\nmOzu+1z+MzN/8knn8cfh6aeT9EVEJO5aMtbFPFPl7r8BfhMJ4DvAr+smVCK1YlnmJh5L5HToVgOH\nr6Sqex678pfR5ssaDj20t7qlZy7jm7PrS8zsO+7+mpmNBZZEtj8LPGJmMwhf9hsILGjooJ06qaZK\nJBOpo7okTSzL3MRjiZxQKMSSr5bQqWc22woWcpAdQO8hB8Oer+/yS9f1DGVfZvYoUAR0N7PlwGTg\nP4C7zawNsDvyGndfZGZPEr7DubbVQoNT7ypUF8lMcU2q3P014LV4HlPSRyzL3MS6RE5t64Qd/XZQ\ntrGMNmva0LfvAVQsrGDe51/RseNZQHquZyj1c/cfN/DWsQ3sPxVoUo8NJVUimUkzVZIRZs+ZTdbQ\nLHZ33c3IbSPZ8sYW2pS34ZBuR7Coo5p7SnwpqRLJTEqqJGla2vOptLSUtWtXsXTpf7N9+w46duzQ\n7H5RNV7DsqpldMvtxqCBg1hfvZ7jc49n05qaln4dkQYpqRLJTEqqJGla0vMpupaqR4/hbNhwE9/6\n1jFcfnnTL9G5O1mDs6h6oYpOOZ1Yv2H93tYJFRUVau4pcVebVLmDpcV9kiLSFDG3VGjyidRSQVqg\nuHgq8+ePiro8N5cxYxYwbdr1TT7GC0te4P0173Nal9OY8+oc4JutE+JxV2EmSaeWColQO9a1bQtb\nt0K7dkFHJCItEUhLBZFU9vbKt3l9+etcO/paurTrQuGIfROmTG/uKYlRO1ulpEokc2QFHYBIYyZM\nGEd19QOUlc2lrGxu5PLcuCZ9dvGGxTy16CkuG3UZXdp1SXCkIt+kuiqRzKOZKklpza3DCoVCzJ4z\nm601W1ndazW/Hfdbeuf3Tla4InspqRLJPEqqJOU19fJcbS8qH+osqVxCz5Ke7DlyD3RPQpAideTn\nQ3l50FGISDLp8p+kjdlzZmNDjXWd1zHokEH0OKwHs+fMDjosyVCaqRLJPEqqJG1UezVLq5bStX1X\n+uT3CTocyXBKqkQyj5IqSQvuTuXASmyN0WFtB9Z/FulFdfL4oEOTDKWkSiTzqKZK0sIzi58hr1se\nD//Hw7w0N1zUPv6ar3tRiSRbp05KqkQyjZKqNJVJDS1f/epVPl73MdeOvpYObTow8oiRQYckopkq\nkQykpCoNRS/tAjBv3gxmzmz6si6pbtasWcx8eCYAp5x9CmsL1u5NqERSRX4+bNgQdBQikkxKqtLQ\nE0+8RHb2xKilXcLbmpJUpfoM16xZs7hoykVkj8qmKruK155/jTvOvIPueeqbIKklPx++/DLoKEQk\nmVSoLnvVznDNnz+K+fNHMWnSDEpLS4MOa69QKMQ1N17DrvxdtOnRhppDasjrlMfTTzwddGgi+9Dl\nP5HMo5mqNDRhwjjmzZtBWVn4dXhplyv3+7lYZrgSrbax55b+W6hoV8GaVWsosAK8Sot0S2pSUiWS\neZRUpaHmLu2Saupegmzbti1XT76aZT2Wcejhh/Le2vdgF2x7bRvtt7Vn0pRJAUcssi8lVSKZR0lV\nmmrq0i7RWjrDFU91i+yff34ynQduZrNtZlPNJtaUr+GQnoew5cMtdN3ZlZun3MxZZ52V1BhFmkJJ\nlUjmibmmysz6mtmrZhYys4Vmdnk8ApPkq53hGjNmAWPGLAjkjsHoS5AFBWPZWtGLdZ03MeSkIVTu\nrKS6opqclTmM6jCKvz34NyVU0mJmdq+ZrTOzj6O2PW5mpZHHV2ZWGvXe9Wa2xMwWm9kp+zu+kiqR\nzBOPmaoq4Cp3/9DMOgLvm9nL7v5pHI4tSdaSGa5k2N5lO71H9Kb67WoOsUOYfsN0NfaUWN0P3Ak8\nVLvB3c+tfW5mtwBbIs+HAOcAQ4C+wCtmdpi7N1jUp6RKJPPEnFS5+1pgbeT5djNbDPQBlFRJs9Ve\ngvznP99k3fbZVFV+RcGmDny57EsOyz0M8mH6NUqoJHbu/oaZ9Wtkl3OAosjzs4DH3b0KWGpmS4BR\nwDsNfVhJlUjmiWtNlZkdAoykkYFGWqdk9a8qLCykqKg3M578A1ZotC1ow9pd2zhjxRkcPfhoLT0j\nSWFm3wbWunttp6k+wD+idlkV2dagjh1h506oqYEsNa8RyQhxS6oil/6eAq5w9+317TNlypS9z4uK\niigqKorX6SWBktmhPRQKcf9z95M12sg5JIfdlbvJ35jP9rLtFP+qOO7nk/0rKSmhpKQk6DCS7UfA\nYy39cO1Yl50NL7xQxOmnF8UnKhFJmHiMddZISUDTD2KWA8wGXnD32xvYp7HyA0lhxcVTmT9/VFT/\nqrmMGbOAadOub9HxGpv1mvbHadz6t1vZNngbfpCTtTuL7M+yOdFP5IWnXoj9y0jMzAx3t6DjiIfI\n5b/n3H1E1LZswjNRhe6+OrLtOsDdfVrk9YvAZHffZ1Y+eqzr3Rveew/6NDqnJSKpqCVjXbwmpe8D\nFjWUUInUakrX9oOHHkzljkp8rVP9VTX+kTPpp+pFJQlhkUe0k4HFtQlVxLPAuWbWxsz6AwOBBfs7\neKdOqqsSySTxaKkwGjgPOMnMPojcinxq7KFJqpgwYRzV1Q9QVjaXsrK5kf5V41p0rLotE7KzJ+6d\ntQIYN3Yc29tsp39Bfzov7Eyn0k5MvWyqWidI3JnZo8BbwCAzW25mF0TemkCdS3/uvgh4ElgEPA9c\n3JSpdxWri2SWeNz99yaQHYdYJEUlq0O7u/NOxTv86Ls/ou3nbbFDjPEnB1OYnuoLS0vs3P3HDWy/\noIHtU4GpzTmHkiqRzKKO6tIk8epf1VDXdnfnr4v+yvY92/nt6b8lJyu4v5rJLMyX9KakSiSzKKmS\npGpo1uuVL19h8YbFXDP6mkATKkjthaWldVFSJZJZlFRJ0tWd9Xpv9Xu88uUrXDv6WvJy8wKMTCS+\nlFSJZBYlVZJ0oVCI2XNmA3DEcUfwavmr/Oq4X9GtfbeAIwtLhYWlJT0oqRLJLEqqJKlmzZpF8W3F\nZA3NoluPbvz5yT/zx3/7I3079Q06tL2SVZgv6U9JlUhmUVIlSRMKhSieXszW4VvJ7ZPLyp0rGdRx\nEIvfWcyZx58ZdHjfkKoLS0vrkp8Py5YFHYWIJItWpJKkmT1nNtkHZNOmYxt2td1F+/bt2blhZ9Bh\niSSMZqpEMouSKkmqHv17UL6pHC9zbKVRs6iG8SePDzoskYRQUiWSWZRUSdKc/t3T2VSxiR4H9KDn\nkp50+aQL066aFkhzT5FkUFIlkllUUyVx1Vgn8i9yvuCkfzmJHst7kNMrJ7Bu6SLJoqRKJLMoqZK4\naawT+evLXufd1e9y4/gbyW+bH2icIsmSnw/l5UFHISLJoqRK4qahTuQ5fXJ49rNnuWb0NUqoJKN0\n6qSZKpFMoqRK4m7bthDrts5mx47P+WpzNQ9++CCXjrqUnh16Bh2aSFLp8p9IZlFSJd/QWE3U/kyY\nMI5nnrmKlTu/hJ5G9sE7eSunLbd3vJ3+XfsnKmSRlKWkSiSzKKlKA7EkQnWP01BNVFO0bduWym5L\nyTp6M1l52ThV9OjYl8/f/RxGtygkkVatbVuoqYE9e6BNm6CjEZFEU0uFVq42EZo/fxTz549i0qQZ\nlJaWtuhY0TVRBQVjyc6euDdZa4rZc2aTV5hHwbButB/chryCPPZ8vqdFsYg0lZkdamZtI8+LzOxy\nM+sSdFwAZpqtEskkSqpauVgToXjr1aMX5dvLqdldQ9aWLKrXVau5pyTa00C1mQ0E7gEOAh4NNqSv\nKakSyRxKqgQIz3itW7eWpUv/k2XLZlJWNpfq6geYMGFck49x+ndPZ/PGzRS0LaDH6h50+bQL065V\nc09JuBp3rwJ+ANzp7tcAvQOOaS8lVSKZQzVVrdyECeOYN28GZWXh1+FE6Momfba2Fmvt2lW8//5q\n8vMvoaBgLBs23M5RRw3hiiuaXk8FsKrdKkYXjabXyl7kds1l/KVq7ilJUWlmPwJ+BpwR2ZYbYDzf\noKRKJHPEJakys1OBPxKe+brX3afF47iyf4WFhcyceWVUoXrTEqHoovRly56kvPwUCguP5pBDOtOx\nYwd69VrQrIRqwaoFzFs6jxvH30jX9l1b+nVEWuICYBLw/9z9KzPrDzy8vw+Z2b3AeGCdu4+I2n4Z\ncDFQBfyfu18X2X49cGFk+xXu/nJTglNSJZI5Yk6qzCwLuAsYC6wG3jWzWe7+aazHlqYpLCxs9h1/\n0bVYGzYsoLy8B6tXr6dz585N+nwoFGLabdN4+6O36di3IwefcjB3//RuJVSSdO6+CLgcwMy6AvlN\n/Ifd/cCdwEO1G8ysiPBs1xHuXmVmBZHtQ4BzgCFAX+AVMzvM3X1/J1FSJZI54lFTNQpY4u7L3L0S\neBw4Kw7HlSQ58MBxwEPs3PlWk2qpQqEQ5196Pk+88QRLhy3lo4M/Yt5f5vHuq+8mL2iRCDMrMbNO\nZtYNKAX+x8xu29/n3P0NYHOdzb8EborUaOHukQvrnAU87u5V7r4UWEJ47NsvJVUimSMeSVUfYEXU\n65WRbZLCJkwYR3X1A5SVzaWycjP9+lVyyimljBmzoNHeVKFQiKsnX82nyz6FYZB1WBZtu7elZkAN\nMx+eCYQvLRYXT6W4eGqL2zuINENndy8H/hV4yN2PBb7bwmMNAsaY2dtmNs/MvhXZXnecW0UTxzkl\nVSKZI6mF6lOmTNn7vKioiKKiomSeXqLsW4v1h/1eQgyFQlx787Us67GMyqGVVLapJHtDDlkdHas2\nIPYGopJ6SkpKKCkpCTqMxuSYWW/Cl+d+G+uxgK7ufpyZHQP8FRjQ3INEj3WbNhWxbVtRjGGJSKLF\nY6yzJpQENH4As+OAKe5+auT1dYDXrWkws6aUH0iKqp2hWtZjGX2G9uGNL95g9/rd4YsgfXPILjWm\nX3ET69ZVMH/+qKhFlecyZswCpk27PtgvIHFjZri7BR1HLTP7N+A/gTfd/ZdmNgC42d3PbsJn+wHP\n1Raqm9nzwDR3fy3yeglwHHARgLvfFNn+IjDZ3d+p55jfGOumToWtW+Gmm2L8oiKSVC0Z6+IxU/Uu\nMDAyOK0BzgV+FIfjSoqonaFaXrOcTTWbWLtqLXm5+VSt64AthY5rB9Mj7xTWrasIOlTJQO7+V8Iz\nSrWvvwT2m1BFWORR6+/AScBrZjYIaOPuG83sWeCRSK1WH2AgsKApJ+jUCVas2P9+ItL6xVxT5e7V\nwKXAy0CIcDHn4liPK6lj9pzZ5A7LZejYoVRtq6KyshL/3Om4fCAnDHuN0SPfoFu38OJ+0bVaLWkg\nKtJcZtbXzP5mZusjj6fNrG8TPvco8BYwyMyWm9kFwH3AADNbSLgr+/mw9w7DJ4FFwPPAxU2deldN\nlUjmiEtNlbu/CBwej2NJ6trddTc9R/bEFzjddnSjvF03KirWUlGxdm/T0Zb2zRKJwf2EE6B/i7z+\nSWTbyY19yN1/3MBbP21g/6nA1OYG16lT+PKfiKS/mGuqmnwi1VS1WqFQiEm3T2Jtv7UMzB1I1qIs\npl8znYqKiqjkaVxck6fabu+JOLbEJgVrqj5095H725bEeL4x1n30EZx3HnzySRDRiEhLtWSsU1Il\n9QqFQsyeMxuAI48/kse/eJxea3vRNasr409O7PIzde8grK5+QHcQppAUTKrmEp6Zeiyy6UfABe4+\nNqB4vjHW7dwJ3buHLwHmaGEwkVYjqEJ1STO1hem5w3LZ7bv58+N/5rZ/u43v//j7STl/dLd3gLKy\n8DYlVdKACwl3Rp8BOOE6qYlBBhQtLw969YKlS2HgwKCjEZFEUlIl31DbOmF5zXIOa3cY63LX0aem\nD58t+AxOCDq62OiSYnpy92XAmdHbzOxXhNcjTQmDB8OnnyqpEkl38eioLmkiFApx8eSLed/eZ1nO\nMua+O5e8nXl0z+6e1DgScQdh7SXF+fNHMX/+KCZNmqFu7+ntqqADiFabVIlIetNMVZqJZTbm3r/c\ny6e5n5IzKIeK3RX4emfNC2vo0qcL468Zn6iQ95GIOwh1STHjpEzNF4STqne1NKZI2lNSlUZiWSKm\ntjB9R/8d5LXNo3P7zuxetZuczTlM/+P0hBam16ewsFAJj8Qipe6KGTwYHn446ChEJNGUVKWRls7G\n1BamVx9aTUV5BRXLKyjIKiB7eTbjxzZ8p19rqlGaMGEc8+bNoKws/Lq2r5a0Xma2jfqTJwPaJzmc\nRunyn0hmUFKV4aLX9Cs4soB1y9bBR1BdXs3gvoP5+fk/r/dzrW3hZDUlTT/unh90DE3VsydUV4f/\noVNQEHTQ56HpAAAgAElEQVQ0IpIoSqrSSHNnY2bNmkXx9GI27d5EZZdKVm5YydEDjmZ9xXr6bejH\nLTfc0uAsVWusUdIlRQmK2dezVSeeGHQ0IpIoSqrSSHNmY2bNmsW//+e/s/vw3eS2zWXbjm10WteJ\n9ZvX039Xf6bfkPw6KpF0pqRKJP0pqUpBsdQqNWU2JhQKUTy9mN0jdlN1YBUVVJC/Pp+cD3Po178f\n028IL0FTXDy1wRhUoyTSPKqrEkl/SqpSTDJqlWbPmU32Adl07NaRje02YnuMqh1VHNDuAG654RYq\nKir2G4NqlESaZ/BgeP31oKMQkURSUpViklWr1KN/D1ZtXkXHNh2pXFtJu8/aMe3GaQwbNozi4qlN\nikE1SiJNp5kqkfSnpCqD1PaiWrtmLavLV9OrZy+y/plFzboapt04jbPOOivoEEXS1oABsHIl7N4N\n7doFHY2IJIKSqhSTqFql6EWSV3dZTdWmKs6wM/BDnR1ds3jrrUUcdNBBFBYWql5KJAFyc6F/f/j8\ncxg+POhoRCQRzD05jYfNzJN1rtauvkL1WBttTvvjNP5R+Q+qDqxizfY19N7amwEbBvD2a5v31k5V\nVz+wt3aqNTX2lOQyM9w9pZaBSSWNjXU/+AGcdx788IdJDkpEmq0lY52SqhTRWBJTt3g9OvnZn9pL\nfiVvlFDWv4zKQys58oAj2fr5VjbP2c2ebddE1U7NZcyYBUybdn3cv5+kDyVVjWtsrLv+eujQAX73\nuyQHJSLN1pKxTpf/UsD+7vhrafH6rFmzKL6tmKyhWeQeksuStUsY6SPZum0rlaFKBh40mEWLEvjF\nROQbBg+GOXOCjkJEEiUrlg+b2XQzW2xmH5rZ02bWKV6BZZLopKmgYCzZ2RP3zlq1VG0vqq2Dt7Kt\nzza+avMVA3sNpPPSzhyfezzTr5nOJZecT3X1A5SVzaWsbG6kdmpcXL6TSKozs3vNbJ2ZfRy1bbKZ\nrTSz0sjj1Kj3rjezJZEx75SWnFN3AIqkt1hnql4GrnP3GjO7Cbg+8pA4aknh+L0P3cum3ZvYs2wP\nHAB5eXlUraqi6MQiin9VvHc/9ZqSDHY/cCfwUJ3tt7n7bdEbzGwIcA4wBOgLvGJmhzW3puHww+Gz\nz8A9vHSNiKSXmJIqd38l6uXbwNmxhZOZ9pc0NbfRZigU4vm3n6fmiBp2tNuBlRo57XOoWV/D+EvH\nf2Nf9ZqSTOXub5hZv3reqi/dOQt43N2rgKVmtgQYBbzTnHN26QIdO8KqVdC3b/NjFpHUFs+aqguB\nx+N4vIzRlKSpqclPKBTi6slXs2fAHvYU7KFju45UbqzEPjKm3TxN6/mJ7N+lZvZT4D3g1+6+FegD\n/CNqn1WRbc1WewlQSZVI+tlvUmVmc4ADojcBDvzW3Z+L7PNboNLdH23sWFOmTNn7vKioiKKiouZH\nnKbiMWNUW5i+kY3s7r0bM6Ov96W6oJpTzz61VTX3VEuH1FFSUkJJSUnQYSTL3cDv3d3N7A/ArcC/\nN/cgjY11tUnVd78bc6wiEkfxGOtibqlgZhOBi4CT3L2ikf3UUiGBQqEQZ//H2WwdvpWKjhVsW72N\nzu0707NtT/rv6s/0a6a3mlmqWFpISOKlU0uFyOW/59x9RGPvmdl1gLv7tMh7LwKT3X2fy3/7G+tu\nvx2WLIG77orb1xCRBGjJWBfr3X+nAtcAZzaWUEli1V7y27R7E5VeSdYBWXTv2Z2sL7Lot6Ffq0qo\nIDF3Q4o0wIiqoTKzXlHv/SvwSeT5s8C5ZtbGzPoDA4EFLTmh7gAUSV+x1lTdCbQB5lj4Vpa33f3i\nmKOSJqu95Lep5yZ2j9jNzm076f5pdyzX6EY3brnhlnoTKl1ek0xnZo8CRUB3M1sOTAb+xcxGAjXA\nUuAXAO6+yMyeBBYBlcDFLZ16V1Ilkr7UUb0Vi77kV1NQw8ZdG8nblEe7he3o1q4b066tf5HkVL+8\nlurxZbp0uvyXCPsb62pqID8f1q4N/xSR1JT0y38SnFmzZvGDn/2AFWtWsKd6D1Wdq+iW1412O9rx\nrf7f4ul7nm6wMD3VL6/V3g05ZswCxoxZoIRK0kpWFgwaFO5XJSLpRcvUtEKzZs3ioikXsWvYLiq2\nVrBz4046fdWJNjVt6Ly+M7fcXf8lv9ZE/bMkndVeAjz66KAjEZF40kxVKzTz4Zlkj8qmx1E9yBqZ\nRVaHLKpeqqLzJ52ZdtX+e1FNmDBOy9OIBGjIEFi4MOgoRCTeVFPVCn3vh9/jg+4fwGFQXVXNrk92\nceCnB/K3B//W5BkqFapLS6mmqnFNGeveew/OPTfcWkHL1YikppaMdUqqWqG///3vnP+/58MgyNuR\nR82CGv5nyv+0quae0nopqWpcU8Y69/AlwIcegmOPTVJgItIsSqrSWCgUYvac2QB0G9GNV5e8ysZX\nNpLt2Uz66SQlVJI0Sqoa19Sx7sYbYf16uPPOJAQlIs2mpCpN1faiyhqaRYceHdi0bRMPn/8wJxx1\nQtChSQZSUtW4po51X34Jxx0XXlw5NzcJgYlIs6ilQhoKhUIUTy9m6+CtbO2zlc+qPqNXfi9ef+31\noEMTkRgMGAADB8KcOUFHIiLxoqQqxc2eM5vsA7LJzs9md5vd5Ofls3nD5qDDEpE4+MlP4C9/CToK\nEYkXJVWtQLf+3SgvL6dNWRtqVtRQs6iG8SePDzosEYnROefA88/Dtm1BRyIi8aCkKsWdfNLJrKte\nR/+u/en2ebcm96ISkdRXUADf/jb8/e9BRyIi8aBC9RRUe6dftVdT3r+cbm27UfNpDWbG+JPHM2zY\nMPWZksCoUL1xzR3rnngC7rsPXkqdlaJEBN39lxZCoRDX3nwtOUNzWF69nMrVlTz2i8c4YvgRe/cp\nLS3l/PP/wKZN4TYK3brN4qGHfqfESpJCSVXjmjvW7dwJffrA4sXQq1cCAxORZtHdf61cKBTi6slX\n81X7ryjLL6NNQRsGHDSA5195/hv73XHHgyxdegq7dp3Grl2nsXTpKdxxx4NJjbW0tJTi4qkUF0+l\ntLQ0qecWSSd5eXDWWfD440FHIiKxUlKVImbNmsXZ/3E273/1PqtrVvPJik84MPdAsmzfP6LFi78C\netCmTfgBPSLbkqO0tJRJk2Ywf/4o5s8fxaRJM5RYicRAdwGKpAclVSkgFApRfFsxW4dvxQud7eXb\nyVmTw2fvfEZlqHKfO/0GDz4YeIg9e+ayZ89c4KHItuR44omXyM6eSEHBWAoKxpKdPXFvfZeINN+/\n/AusXg2ffhp0JCISCyVVKWD2nNlkDc0i+6BsqgdW0y2vG9kfZ9NvQz+mXzN9nzv9rrjiQvr1q6R9\n+ydp3/5J+vWr5IorLgwoehGJVXY2XHghTJsWdCQiEoucoAOQsO49urNy50o65nXEc5zO7Tpzyw23\n1Ns6obCwkIcf/kPU3X+/SGqR+oQJ45g3bwZlZeHX1dUPMGHClUk7v0g6uvZaGDIE3n47vHyNiLQ+\nuvsvBSz4cAHnPXAe+Z3y2b1hNzWLaph21bSUXiRZLR0yl+7+a1wsY91f/gJ//CO880549kpEghNY\nSwUz+zVwM1Dg7psa2EdJVT32VO/htn/cRt7OPHYt2gWwtxeVSCpKl6TKzO4FxgPr3H1Enff2GdPM\n7HrgQqAKuMLdX27guC0e69zDzUB/9jO46KIWHUJE4iSQpMrM+gL/CxwOfEtJVdPVeA0z35tJ+5z2\nTBw5EbNW//8pyQBplFSdCGwHHopOquob08xsCPAocAzQF3gFOKy+QS3Wse6DD+B73wv3reratcWH\nEZEYBdWnagZwTRyOk1Hcncc/eZyKqgp+euRPlVCJJJm7vwHUtzp5fWPaWcDj7l7l7kuBJcCoRMR1\n1FHwr/8K//VfiTi6iCRSTIXqZnYmsMLdFyopaJ6XvniJLzZ9wdUnXE1OVviPQXVKIsFqZEzrA/wj\n6vWqyLaEuPFGGDoU/v3f4cgjE3UWEYm3/SZVZjYHOCB6E+DA74DfACfXea9BU6ZM2fu8qKiIoqKi\npkeaRt5Z+Q6vLX2N4hOLaZ/bHvi6oWZ29kQA5s2bwcyZV8acWClRk1iVlJRQUlISdBgJZ2bt2XdM\na5FYx7ru3eGGG+Cyy+C110D/ZhVJvHiMdS2uqTKz4YTrCnYSTqb6Ev7X2yh3X1/P/hlbU1W7QDLA\n0GOH8tq217jq+Ks4MP/AvfsUF09l/vxRFBSMBaCsbC5jxixg2rTrW3zeuoladfUDcUnUJLOlS00V\ngJn1A55z9xGNjWmEC9Rx95sin3sRmOzu79RzzLiMddXVcOyxcMEFcMklMR9ORJqpJWNdiy//ufsn\nwN7lP83sK6DQ3eurUchYtQsk5w7LZVfNLv781z9z5zl3fiOhSpTozucAZWXhbUqqRPayyKPRMc3M\nngUeMbPbCF/2GwgsSGRg2dnh9QBPOCFcZ3XCCYk8m4jEQzw7qjv7ufyXie596F6Wli9lxRcrWGbL\nOKjnQXzy9if77Ddhwjiqqx+grGwuZWVzIw01xwFavFgkEczsUeAtYJCZLTezC+rssndMc/dFwJPA\nIuB54OJkTL0PHAj33w/nnANr1iT6bCISKzX/TKBQKMTZF5/NliFb2N11N22+asPwPsMZd9A4in9V\nvM/+9dU/xXIJT5f/JBHS6fJfIiRirPv97+Hll+HVV6FNm7geWkQaEFjzzyadKAOTqml/nMaLm17k\ng8oPyM3LJXtlNl0+6cLT9zzd5OaesdZaqVBd4k1JVeMSMdbV1MD3vw8HHwx33RXXQ4tIA5JaUyX7\n5+5sab+FQ3seSnZ5NtuztnPqmFOT2i29sLBQiZRIK5eVBQ8/DMccAw8+GO64LiKpR0lVArUb1o5d\n/7eLQdmDyMrNonJXJT+/9OfNOoYWLxYRgM6d4e9/h6Ii6N8fxowJOiIRqUuX/xJk3lfzmLd0Hmd0\nO4NX570KtHxNP13Ck1Siy3+NS/RY9+qrcO65MGsWHH98wk4jkvFUU5UiPljzAY998hjXjr6WgryC\noMMRiSslVY1Lxlj34otw/vnwf/8XviQoIvGnpCoFfLn5S/604E9cfuzl9OvSD0jeTJNmtCQZlFQ1\nLllj3XPPhZexeeklGDky4acTyThKqgK2bvs6bnnrFn428mcM7zkcSF5bA7VPkGRRUtW4ZI51Tz8N\nl14Kc+bA8OFJOaVIxmjJWBfP5p8ZrbyinDveuYOzBp+1N6ECuP32+1i5Mo8NGxaQm9uV7OyJe2eT\n4im6e3pBwdiEnUdEUsfZZ8Ntt8Epp8C77wYdjYjo7r84qKiq4K4Fd3Fc3+M48eAT924vLS3l5ZcX\ns23bxeza1YWyshkcfPDoACMVkXTzox9BXh6cdhr87//CWWcFHZFI5lJSFaMar+Ge9++hT34fxg8a\n/433nnjiJQoKrmDXroOAvlRVfZ8NG25lwoT4d+9T6wWRzHXWWdCnT/jn0qVwxRVBRySSmZRUxcDd\neXThozjOT0b8BLN9L7127NiBESMGsHr1enbu3M4ppxyTkDqnwsJCZs68MqpQXfVUIpnk6KPhrbfg\n9NPhiy9gxozwoswikjwqVI/B80uep3RNKVefcDXtctrt837QxeN17wYEdHegxEyF6o0LeqzbuhV+\n+MPwGoEPPQTduwcWikirprv/kuitFW8x+5+zKR5dTOd2nRvcL6g2B3UTuvLyWzFrR37+JYDuDpSW\nU1LVuFQY6yor4Te/gSeegL/8Rd3XRVpCa/8lyaINi3hm8TP8+vhfN5pQQXBr70XfDQiwbNmTwAn0\n71+7MHN4HyVVIuknNxduvhlOOgkmTIBJk+B3v9PlQJFEU0uFZlqxdQX3lt7LpKMn0Tu/d9DhiIg0\n6Hvfg9JSmD8fxo6FlSuDjkgkvSmpaoaNOzdy14K7OG/EeQzsNjDocBo1YcI4qqsfoKxsLmVlc+na\ndQXdus3a+zp8d+C4oMMUkQTr3Rtefjncy+qoo+BPf4Lq6qCjEklPqqlqoh17dnDzWzfz7YO/zdgB\nY4MOp0lUqC6JoJqqxqXyWLdoEfziF7BnD/z5z1reRqQxKlRPkMrqSm5/53b6de7Hvw37t6DDEQmU\nkqrGpfpYV1MD998fLmT/yU/ghhugY8egoxJJPYEsU2Nml5nZYjNbaGY3xXq8VOPu3P/h/XRu25kf\nDv1h0OGISJyY2b1mts7MPo7a9nsz+8jMPjSzV8ysb9R715vZksh4d0owUccuKwt+/nP45JPwDSuH\nHRa+JLhnT9CRibR+MSVVZlYEnAEc4e5HALfEI6hU8tSipyivKGfiyIn1NvcUkVbrfqBuYeF0dz/S\n3UcCs4DJAGY2FDgHGAJ8D7jbWvmA0KMHPPggPP88zJ4NgweH2y+o3kqk5WKdqfolcJO7VwG4e1ns\nIaWOuV/OJbQhxC+P/iW52blBhyMiceTubwCb62zbHvWyA7Ax8vxM4HF3r3L3pcASYFQy4ky0o46C\nF14IXxK8++5wndVf/wpVVUFHJtL6xJpUDQLGmNnbZjbPzI6OR1Cp4P3V7/PyFy9z2ajL6NCmQ9Dh\niEiSmNkfzGw5MBGYGtncB1gRtduqyLa08Z3vwJtvwn//N9x+e/iy4B13wPbt+/+siITtN6kyszlm\n9nHUY2Hk55mEm4d2dffjgGuBJxMdcDIs2biExz55jEtHXUr3PK3xIJJJ3P137n4w4cuDfww6nmQy\ngzPOgDfegMceg9dfh0MOgeuuCy/ULCKN229HdXc/uaH3zGwS8Exkv3fNrMbMurv7xvr2nzJlyt7n\nRUVFFBUVNTfehFuzbQ1/fv/PXHjUhRzU+aCgwxEJXElJCSUlJUGHEYRHgecjz1cB0QNC38i2erWG\nsW5/jjsufBnwyy/hzjvhmGPgyCPhwgvhBz+A9u2DjlAkvuIx1sXUUsHM/gPo4+6TzWwQMMfd+zWw\nb0rfZgywdfdWpr05jTMGncHxBx0fdDgiKSmdWiqY2SHAc5EbbTCzge7+eeT5ZcAod/9ppFD9EeBY\nwpf95gCH1TeotYaxriV274Znn4X77oN334VzzoFzz4UTT9TyN5Kekt6nysxygfuAkUAF8Gt3f62B\nfVN6oNldtZtb3rqFwt6FnHbYaUGHI5Ky0iWpMrNHgSKgO7CO8J1+pwOHA1XAl8Av3X19ZP/rgZ8D\nlcAV7v5yA8dN6bEuHpYvh4cfhqeegtWrwzNXP/xhuC4rV/f0SJpQ888Wqq6p5k/v/olu7btx3hHn\nqXWCSCPSJalKlFQe6xLhiy/g6afDj88/h+9+F8aNCz/6pFUpv2QaJVUt4O489NFDbNuzjYuPuZgs\n03KIIo1RUtW4VB3rkmHVqvA6gy++CK+8El538JRT4NvfhtGjoWfPoCMUaTolVS3w3GfPsXD9Qn59\n/K9pm9M26HBEUp6Sqsal6liXbNXV8N57MHdu+G7Ct96CXr3CNVgnnABHHw1Dh0LOfm+XEgmGkqpm\nemP5G7yw5AWKTyymU9tOQYfTInUXTdYiyZJoSqoal4pjXSqorg4vjfP66/D22/D+++HarBEj4Fvf\nCjcdHT4chg2D/PygoxVRUtUsn6z/hAc/fJCrT7iaAzoeEHQ4LVJaWsqkSTPIzp4IQHX1A8yceaUS\nK0koJVWNS7WxLpWVl8MHH4QTrI8+glAIFi8OL6EzfDgcfjgMGhRuRDpoEBx4YHjtQpFkaMlYl5ET\nr8u2LOP+D+7nklGXtNqECuCJJ14iO3siBQVjgfDiqE888ZKSKhFpFTp1Ct8x+J3vfL2tuhq++io8\nq/XZZ+H2DY8+Cv/8ZzgJ69cv3JA0+mffvuGi+AMPhLaq4pAAZVxSVbazjD+9+yd+euRPGdB1QNDh\niIhIlOxsGDgw/Khr+/ZwZ/fax7Jl4bqtlSvDRfJr10LnzuEE64AD9n107w4FBV//7NQp3EVeJF4y\nKqmq8Rr+tOBPnHbYaYzsNTLocGI2YcI45s2bQVlkGevq6geYMOHKYIMSEUmQjh3DlwWHD6///Zoa\n2LAhnGCtW/f1Y/Vq+PBD2LgxPKNf+3PXrnAS1qULdO0a/tmlSzjZ6tQp/F6nTuEar44dv3506BB+\n5OV9/bN9ezVBlQysqdqwYwM9OvQIOoy4UaG6JJtqqhqXKmOd7F9lJWzZAps3f/1z69bwZcby8q+f\nb9/+zce2bbBz59ePHTvCCVpuLrRrF06w2rcPP699tG379c82bb7+WfvIzQ0/op/n5obvjox+XvvI\nzm74Z+0jK2vf59E/G3qY7fvc7JvP63uvsUdrpEJ1EUk4JVWN01iXmdyhoiK8nM+uXeHH7t1fb4v+\nuWdP+FH7vKIinODVPvbsCf+sqvrm9urq8LboR3X119ujf9bUfP1e7evabbU/3b/eXneb+9evo7fV\n/mxoW91HXfUlW409r7ut7vaGttX3Xt04on/+7Gdw661191FSJSIJpqSqcRrrRPbVUMLV2PO62+pu\nb2hbfe/VjaXue23b7tvKQ3f/iYiISMppzZcBm0MdP0RERETiQEmViIiISBwoqRIRERGJAyVVIiIi\nInGgpEpEREQkDpRUiYiIiMSBkioRERGROFBSJSIiIhIHMSVVZnaMmS0wsw8iP4+OV2AiIolkZvea\n2Toz+zhq23QzW2xmH5rZ02bWKeq9681sSeT9U4KJWkRSWawzVdOB37n7UcBk4ObYQ0q8kpKSoEMA\nUicOUCz1SZU4ILViSSP3A+PqbHsZGObuI4ElwPUAZjYUOAcYAnwPuNssfftDt/a/b609fmj936G1\nx99SsSZVa4DOkeddgFUxHi8pUuUPO1XiAMVSn1SJA1IrlnTh7m8Am+tse8XdayIv3wb6Rp6fCTzu\n7lXuvpRwwjUqWbEmW2v/+9ba44fW/x1ae/wtFevaf9cBb5rZrYABJ8QekohISrgQeCzyvA/wj6j3\nVkW2iYjstd+kyszmAAdEbwIc+B1wGXCZu//dzH4I3AecnIhARUSSxcx+C1S6+2P73VlEJMLcveUf\nNit39+hCzq3u3rmBfVt+IhFJKe6eFvVEZtYPeM7dR0RtmwhcBJzk7hWRbdcB7u7TIq9fBCa7+zv1\nHFNjnUiaaO5YF+vlvyVm9h13f83MxgL/jFdgIiJJYJFH+IXZqcA1wJjahCriWeARM5tB+LLfQGBB\nfQfUWCeSuWJNqn4B/MnM2gC7gf+IPSQRkcQzs0eBIqC7mS0nfAfzb4A2wJzIzX1vu/vF7r7IzJ4E\nFgGVwMUeyzS/iKSlmC7/iYiIiEhY0juqm9llkeZ5C83spmSfv04svzazGjPrFmAMDTYbTNL5TzWz\nT83sn2ZWnMxz14mjr5m9amahyN+Ny4OKJRJPlpmVmtmzAcfR2cz+Gvk7EjKzYwOM5fpIDB+b2SOR\nGWqJkiq/T83RQBPUrmb2spl9ZmYvmVm9tbKpoKGxo7V8BzNra2bvRJpoh8zsvyPbW0X8teqOma0w\n/qVm9lFtM/PItmZ/h6QmVWZWBJwBHOHuRwC3JPP8dWLpS/hOxWVBxRBRb7PBZDCzLOAuwg0QhwE/\nMrPByTp/HVXAVe4+DDgeuCTAWACuIHypJ2i3A8+7+xDgSGBxEEFECrovAo6KFHXnAOcGEUuqSrHf\np+aorwnqdcAr7n448CpJHJdaoKGxo1V8h0jt3r9EmmiPAE4ys9G0kvij1B0zW1v8NUCRux/l7rU9\n6Jr9HZI9U/VL4CZ3rwJw97Iknz/aDMIFqYFqpNlgMowClrj7MnevBB4Hzkri+fdy97Xu/mHk+XbC\nyUMgfYAiCfdpwP8Gcf6oODoB33b3+wEijSfLAwqnHNgDdDCzHCAPWB1QLKkqZX6fmqO+JqiE434w\n8vxB4PtJDaoZGhg7+tK6vsPOyNO2hP+/vJlWFH8DY2ariT/C2DcnavZ3SHZSNQgYY2Zvm9k8C2it\nQDM7E1jh7guDOH8jLgReSOL5+gArol6vJAUaGprZIcBIYJ/b1ZOkNuEOuuCwP1BmZvdHptXvMbP2\nQQTi7puBW4HlhBtfbnH3V4KIJYWl5O9TC/V093UQTlqAngHH0yRRY8fbwAGt5TtELp19AKwFStx9\nEa0ofuofM1tT/BCOfY6ZvWtm/x7Z1uzvEOvdf/uwxpuF5gBd3f04MzsGeBIYEO8YmhDHb/hmk9KE\n3gLdSCy/dffnIvvUNht8NJGxpDoz6wg8BVwR+Vdnss9/OrDO3T+MXK4O8vb4HKAQuMTd3zOzPxKe\njp6c7EDMbABwJdAP2Ao8ZWY/zvS/rxkk6H9g7FfdscP27ReWst8hcrXiqMjs9EuRsadVxF/PmNmQ\nlIw/ymh3X2NmPYCXzewzWvBnEPekyt0b7KhuZpOAZyL7vRspEu/u7huTFYeZDQcOAT4yMyM8Tfy+\nmY1y9/XxjqOxWKJimkh46vSkRJy/EauAg6Ne9yXA9Rsjl5WeAh5291kBhTEaONPMTgPaA/lm9pC7\nnx9ALCsJz6i+F3n9FBBU8fPRwJvuvgnAzJ4hvCyVkqqvpdTvU4zWmdkB7r7OzHoBCRkb46WBsaNV\nfQcAdy83s+cJ/761lvjrGzMfBta2kvgBcPc1kZ8bzOzvhC/nN/vPINmX//5OJHEws0FAbiISqsa4\n+yfu3svdB7h7f8L/4zoqUQnV/tjXzQbPrNNsMBneBQaaWb/InVznEm5yGJT7gEXufntQAbj7b9z9\nYHcfQPi/x6sBJVREpp1XRH5XAMYSXPH8Z8BxZtYu8o+RsQRUNJ/CUu33qTm+0QSVcNwTI89/BgT1\nj5ymqm/saBXfwcwKau8qi1zePxn4gFYSfwNj5k+B52gF8QOYWV5kphMz6wCcAiykBX8GcZ+p2o/7\ngfvMbCFQAQTyP6s6nGAv8dxJPc0Gk3Fid682s0sJ34GYBdzr7kHdXTYaOA9YGKktcOA37v5iEPGk\nkMsJd/LOBb4ELggiCHf/yMweAt4HqgkP+vcEEUuqSqXfp+aw+pug3gT81cwuJHyH9DnBRdi4hsYO\nYMjnsCMAAAPrSURBVBrwZCv4Dr2BByP/WMkiPNs2N/JdWkP8DbmJ1hP/AcDfIpeMc4BH3P1lM3uP\nZn4HNf8UERERiYOkN/8UERERSUdKqkRERETiQEmViIiISBwoqRIRERGJAyVVIiIiInGgpEpEREQk\nDpRUiYhIyjOzbmb2QWQdzDVmtjLqdZN6LprZvWZ22H72udjMfhSfqOs9/g+iGvpKmlGfKhERaVXM\n7L+A7e5+Wz3vmafw/9giS7g8FeBSXJJAmqkSEZHWZu8qGGZ2qJmFzOwvZvYJ0MvM/mxmC8xsoZn9\nLmrf181shJllm9lmM5tqZh+a2ZtmVhDZ50Yzuzxq/6lm9o6ZLTaz4yLb88zsKTP7xMz+ambvmtmI\nfYI0uzkS24eR45xIeJ3X2yIzbAeb2UAzezFyjBIzGxj57MNmdreZvWdmn0aWNMPMhke+W2nkuIck\n7L+yNFuyl6kRERGJt8OBn7j7BwBmVuzuW8wsG5hnZk+5+6d1PtP5/7d3PyE2hWEcx79PjSjTzE6p\nKUmjkAZNkSzsNAs1CyJ2xEJRZmehrMmGFHYypUwmicnInw1WxmKKKRqymGah5H8NM34W9xmuM/di\n6tTMrd+nTr333Pd9z1k+Pc/beYAHko5FxGlgH3Cy1uaSNkbEdiotfLqAw8C4pB0ZTA0V10TEEqBL\n0pr83VLVMLlP0o28fx/YL+l1RGwGzgHbcps2SZ1ZLrwbESuAQ8ApSX3Zvmou26xZgYMqMzNrdKPT\nAVXam/3amqj01lsNFIOqr5Lu5HgI2FJn7/6qOctyvIVKbzskDUfEsxrr3gFTEXERGABuFidkI+VN\nwLXs/Qd/VpCu5jNeZF/GduAxcDwzVP2SRuu8t80Bl//MzKzRfZkeZPnsCLBVUgcwCCyqseZb1XiK\n+kmGif+YMyNbJGkS6ASuA93ArTrr3kraIGl9Xh3V2xTmSlJv7jcB3M6Sos0TDqrMzKzRVQc1LcBH\n4HNELOV3Ke1va2brEbALICLWAqtmbB7RDLRKGgB6gHX516d8RyS9B8YjojvXROFs1s68vxJoA15G\nxHJJrySdoZL9mnGWy+aOy39mZtbofmV0JD2NiBFgBHgDPKw1rzD+574FZ4FLeTD+eV4fCnNagf6I\nWEglgDua968AFyKih0rGaTdwPiJOAAuAXmA4545FxBNgMXBA0mRE7MlPPnwHxqic87J5wp9UMDMz\nm4U8AN8kaSLLjYNAu6QfJT7jMlUH2q0xOFNlZmY2O83AvaqPjh4sM6BKzng0IGeqzMzMzErgg+pm\nZmZmJXBQZWZmZlYCB1VmZmZmJXBQZWZmZlYCB1VmZmZmJXBQZWZmZlaCn3k/n05X32zbAAAAAElF\nTkSuQmCC\n", 665 | "text/plain": [ 666 | "" 667 | ] 668 | } 669 | } 670 | ], 671 | "execution_count": 0 672 | }, 673 | { 674 | "cell_type": "markdown", 675 | "metadata": { 676 | "id": "lSWT9YsLP1de", 677 | "colab_type": "text" 678 | }, 679 | "source": [ 680 | "This version of the code has a lot more comments at each step. Read through the code and the comments.\n", 681 | "\n", 682 | "The core piece is the loop, which contains a single `run` call. `run` executes the operations necessary for the `GradientDescentOptimizer` operation. That includes several other operations, all of which are also executed each time through the loop. The `GradientDescentOptimizer` execution has a side effect of assigning to weights, so the variable weights changes each time in the loop.\n", 683 | "\n", 684 | "The result is that, in each iteration of the loop, the code processes the entire input data set, generates all the estimates $\\hat{y}$ for each $x$ given the current weights $w_i$, finds all the errors and L2 losses $(\\hat{y} - y)^2$, and then changes the weights $w_i$ by a small amount in the direction of that will reduce the L2 loss.\n", 685 | "\n", 686 | "After many iterations of the loop, the amount we are changing the weights gets smaller and smaller, and the loss gets smaller and smaller, as we narrow in on near optimal values for the weights. By the end of the loop, we should be near the lowest possible values for the L2 loss, and near the best possible weights we could have." 687 | ] 688 | }, 689 | { 690 | "cell_type": "markdown", 691 | "metadata": { 692 | "id": "dFOk7ERATLk2", 693 | "colab_type": "text" 694 | }, 695 | "source": [ 696 | "## The details\n", 697 | "\n", 698 | "This code works, but there are still a few black boxes that are worth diving into here. `l2_loss`? `GradientDescentOptimizer`? What exactly are those doing?\n", 699 | "\n", 700 | "One way to understand exactly what those are doing is to do the same thing without using those functions. Here is equivalent code that calculates the gradients (derivatives), L2 loss (sum squared error), and `GradientDescentOptimizer` from scratch without using those functions." 701 | ] 702 | }, 703 | { 704 | "cell_type": "code", 705 | "metadata": { 706 | "id": "_geHN4sPTeRk", 707 | "colab_type": "code", 708 | "colab": { 709 | "autoexec": { 710 | "startup": false, 711 | "wait_interval": 0 712 | }, 713 | "output_extras": [ 714 | { 715 | "item_id": 1 716 | } 717 | ] 718 | }, 719 | "cellView": "both", 720 | "executionInfo": { 721 | "elapsed": 657, 722 | "status": "ok", 723 | "timestamp": 1446499870301, 724 | "user": { 725 | "color": "", 726 | "displayName": "", 727 | "isAnonymous": false, 728 | "isMe": true, 729 | "permissionId": "", 730 | "photoUrl": "", 731 | "sessionId": "0", 732 | "userId": "" 733 | }, 734 | "user_tz": 480 735 | }, 736 | "outputId": "85c49bf6-8d07-401a-ae08-79c6933adff5" 737 | }, 738 | "source": [ 739 | "#@test {\"output\": \"ignore\"}\n", 740 | "\n", 741 | "# Use the same input data and parameters as the examples above.\n", 742 | "# We're going to build up a list of the errors over time as we train to display later.\n", 743 | "losses = []\n", 744 | "\n", 745 | "with tf.Session() as sess:\n", 746 | " # Set up all the tensors.\n", 747 | " # The input is the x values with the bias appended on to each x.\n", 748 | " input = tf.constant(x_with_bias)\n", 749 | " # We're trying to find the best fit for the target y values.\n", 750 | " target = tf.constant(np.transpose([y]).astype(np.float32))\n", 751 | " # Let's set up the weights randomly\n", 752 | " weights = tf.Variable(tf.random_normal([2, 1], 0, 0.1))\n", 753 | "\n", 754 | " tf.initialize_all_variables().run()\n", 755 | " \n", 756 | " # learning_rate is the step size, so how much we jump from the current spot\n", 757 | " learning_rate = 0.002\n", 758 | " \n", 759 | " # The operations in the operation graph.\n", 760 | " # Compute the predicted y values given our current weights\n", 761 | " yhat = tf.matmul(input, weights)\n", 762 | " # How much does this differ from the actual y?\n", 763 | " yerror = tf.sub(yhat, target)\n", 764 | " # Change the weights by subtracting derivative with respect to that weight\n", 765 | " loss = 0.5 * tf.reduce_sum(tf.mul(yerror, yerror))\n", 766 | " gradient = tf.reduce_sum(tf.transpose(tf.mul(input, yerror)), 1, keep_dims=True)\n", 767 | " update_weights = tf.assign_sub(weights, learning_rate * gradient)\n", 768 | " \n", 769 | " # Repeatedly run the operation graph over the training data and weights.\n", 770 | " for _ in range(training_steps):\n", 771 | " sess.run(update_weights)\n", 772 | "\n", 773 | " # Here, we're keeping a history of the losses to plot later\n", 774 | " # so we can see the change in loss as training progresses.\n", 775 | " losses.append(loss.eval())\n", 776 | "\n", 777 | " # Training is done, compute final values for the graph.\n", 778 | " betas = weights.eval()\n", 779 | " yhat = yhat.eval()\n", 780 | "\n", 781 | "# Show the results.\n", 782 | "fig, (ax1, ax2) = plt.subplots(1, 2)\n", 783 | "plt.subplots_adjust(wspace=.3)\n", 784 | "fig.set_size_inches(10, 4)\n", 785 | "ax1.scatter(x, y, alpha=.7)\n", 786 | "ax1.scatter(x, np.transpose(yhat)[0], c=\"g\", alpha=.6)\n", 787 | "line_x_range = (-4, 6)\n", 788 | "ax1.plot(line_x_range, [betas[0] + a * betas[1] for a in line_x_range], \"g\", alpha=0.6)\n", 789 | "ax2.plot(range(0, training_steps), losses)\n", 790 | "ax2.set_ylabel(\"Loss\")\n", 791 | "ax2.set_xlabel(\"Training steps\")\n", 792 | "plt.show()" 793 | ], 794 | "outputs": [ 795 | { 796 | "output_type": "display_data", 797 | "metadata": {}, 798 | "data": { 799 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlUAAAEPCAYAAABr+zG+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl8VPXZ///XlYUlLEEIIIsCCqKAqCmgFkvTUkUrLnft\nXay2va12obZK9RZTWr8/oO19K2pLXWqxLWqxWnGrWEpVVCK3K2pcBxSs7MgSdgiELNfvj5ngGJKQ\nZCZzZnk/H4/zyJkzZ865RuDjlc/5fK6PuTsiIiIiEpusoAMQERERSQdKqkRERETiQEmViIiISBwo\nqRIRERGJAyVVIiIiInGgpEpEREQkDuKSVJnZFDMLmdm7ZvaAmbWJx3VFRFqLmc02s01m9m7UseFm\n9rKZvWNm88ysY9R7U8xshZktM7OzgolaRJJZzEmVmfUDvg+c4u7DgRzg4livKyLSyu4FxtU59mfg\nenc/Cfg7cD2AmQ0BvgGcAJwD3GVmlsBYRSQFxKOnahdwAOhgZjlAHrAhDtcVEWk17v4isL3O4UGR\n4wDPAhdF9s8HHnL3KndfBawARiUkUBFJGTEnVe6+HfgNsAZYD+xw92djva6ISABCZnZ+ZP8bQN/I\nfh9gbdR56yPHREQOisfjv2OAa4B+QG+go5ldEut1RUQCcDnwYzN7HehAuBdeRKRJcuJwjRHAS+6+\nDcDMHgc+DzwYfZKZaZFBkTTh7mk5nsjdlxMZZ2Vmg4BzI2+tB46KOrVv5Ngh1NaJpI/mtnXxGFP1\nIXCambWLDNwcCyxrILik2KZOnRp4DMkUh2JJ7jiSLZY0Y5Et/MKse+RnFnADMCvy1pPAxWbWxswG\nAAOBJQ1dNOg/o3T6+5aJ8afDd0j1+N1b1tbF3FPl7u+Y2RzgTaAaeAv4Y6zXFRFpTWb2IFAEdDOz\nNcBUoJOZ/Rhw4HF3vw/A3Zea2cPAUqASuNJb2uqKSNqKx+M/3P0W4JZ4XEtEJBHcvaGxn7c3cP6N\nwI2tF5GIpLqMrKheVFQUdAhA8sQBiqU+yRIHJFcskv5S/e9bqscPqf8dUj3+lrJE9WCbmXrLRdKA\nmeFpOlA9HtTWiaSHlrR1GdlTJSIiIhJvSqpERERE4kBJlUiSq6yubPH0XhERSRwlVSJJrKqmijuX\n3MmS9Q2WRBIRkSShpEokSbk7c96ZQ7ucdozsMzLocERE5DCUVIkkqSc+eIIte7fwvcLvkWX6pyoi\nkuzUUoskoZJVJZR+UsqPR/2Y3OzcoMMREZEmUFIlkmTe2fgOC1YsYNJpk+jYpmPQ4YiISBMpqRJJ\nIh9v/5j7372fH4/8MQV5BUGHIyIizaCkSiRJbN67mT+8/gcuO/ky+nXpF3Q4IiLSTEqqRJLA7ord\n3PbqbZw/+HyG9RgWdDgiItICSqpEAlZRVcGdS+7k1L6n8oV+Xwg6HBERaSElVSIBqvEa/lz6Z3p3\n6s15x50XdDgSJzU1QUcgIkFQUiUSEHfnwfcepKqmim8N/xZmzVoMXZLY7t1BRyAiQVBSJRKQf330\nL1btWMUPR/yQ7KzsoMORONqxI+gIRCQISqpEAvDK2ld4cc2LXDXqKtrltAs6HIkzJVUimUlJlUiC\nLduyjMeWPcZVo64iv11+0OFIK1BSJZKZ4pJUmVm+mT1iZsvMLGRmp8bjuiLpZu3Otcx+azYTR0yk\nV6deQYcjrURJlUhmyonTdW4DFrj7f5pZDpAXp+uKHKK0tJS5c58GYMKEcRQWFgYcUdNsLd/KnUvu\n5JITL2Fg14FBhyOtaOfOoCMQkSDEnFSZWWfgC+5+GYC7VwG7Yr2uSH1KS0uZOHEm2dmXAbBo0Uxm\nzbom6ROr8spy7lhyB2cdexaFvZI7VomdeqpEMlM8Hv8NAMrM7F4zKzWzP5pZ+zhcV+QQc+c+TXb2\nZRQUjKWgYCzZ2Zcd7LVKVpXVldz1+l0M7T6UsceMDTocSQAlVSKZKR5JVQ5QCPze3QuBcuBncbiu\nSMpzd+57+z7y2+bz9SFfDzociWJms81sk5m9G3VspJktMbO3Ij9HRL03xcxWRMaOntXYtZVUiWSm\neIypWgesdfc3Iq8fBYrrO3HatGkH94uKiigqKorD7SWTTJgwjkWLZlJWFn5dXX0fEyZcE2xQdYRC\nIeYvnA9A7gm51HSqYdKpk1K2uGdJSQklJSVBh9Ea7gXuAOZEHbsZuMHdnzGzc4BbgC+Z2RDgG8AJ\nQF/gWTMb5O5e34WVVIlkppiTKnffZGZrzew4d18OjAWW1ndudFIl0hKFhYVMmnQ2t99+GwCTJk2I\n+3iqWAbCh0Ihrr/lenKH5rKlegub/7WZB7/7ILnZuXGNMZHq/gI0ffr04IKJI3d/0cz61Tn8CVBb\n56ILsD6yfz7wUGTM6CozWwGMAl6r79pKqkQyU7xm/10NPGBmucDHwHfjdF2RzygtLeW2254iO3sS\nALfddh8nnHDCwcQn1pmBsQ6En79wPrlDc8k+Opu92/cyKGsQzy96npEnjWxWHBKYnwEvmdlvAAM+\nHzneB3gl6rz1kWP10uw/kcwUl6TK3d8B9H8NaXXRA9UBysrCxwoLC+MyM7Cx6zfVnpo9bNm2hWE9\nhrF3994mf06SwmzgKnd/wsy+DtwDnNnciyxdOo3ajnkNdRBJDfEY6hCvniqRwMUjIYrVqWecyt0P\n3k1/+rN3z14qQ5WMnzw+YfeXmJ3q7mcCuPujZvbnyPH1wFFR5/Xl00eDh8jL+zSpEpHUEI+hDkqq\nJKW09kD1WK6/c/9Ontn5DL+68FesK10HwPjJ4xk6dGjc4pO4s8hWa4WZfdHdXzCzscCKyPEnCQ9x\nmEn4sd9AYElDF9WYKpHMZA1MXon/jcwamigj0iwNjZuq+/ivuvq+FhUGbcm4rP1V+7n15Vsp7FXI\nVwd9tVn3SzVmhrun5lTGKGb2IFAEdAM2AVOBd4G7gDbAfuBKd38rcv4U4AqgEpjk7s80cF3PznYq\nKyFFJ3yKCC1r65RUSVoJYgmb6ppqfv/67+naviuXnnhpypZOaKp0Sapai5l5hw7Oxo3QsWPQ0YhI\nSympEmlF9SVs7s6cd+aw+8Burhx5JVmWlbJrEzaVkqrGmZn36eO8+ir07Rt0NCLSUi1p6zSmSqQJ\nGppZuL7Dejbs3sC1p197MKFKxbUJJb66dAmPq1JSJZJZlFSJNEF9Mwt/88jdDDizO8Wji2mb07bB\n8xI9A1GCl5+vweoimSgea/+JZJzyjitZkb2Mq0+9mk5tOwUdjiSZ2p4qEcks6qkSaYIJE8axYMFU\n1m99iKoOuzkw5HXuGTeTHh16HHJesq9NKK1PSZVIZlJSJVKP6MHmI0YM5JU3XmF33nvUnPwe+wp2\n03/j0Rzb9dhDPldYWMisWddEDVTXeKpM1KWLlqoRyURKqkTqiB5sXl6+irsf/yldB7Zl3yn72N91\nP4X9Csnvnc/8hfPrLexZWFioRCrDqadKJDNpTJVIHdGDzQ9klZF1fH/KrYIDRxygbfu2lG8rDzpE\nSXJKqkQyk5IqkcNwHB/s2Hoja10Wu9fsDq/pd6bW9JP6afafSGbS4z+ROiZMGMfjj1/LsjVTqKos\np7LLKrp1PJK+Wcex8dWNnD3mbK74yRVa008apJ4qkcykpEqaJd2rhQO89NJLrDvwGtV9quFYwypg\n1KZRnDTkJMZfrgWS5fA0UF0kMympkibLhGrhoVCIX/7pl9ScUUNOtxyq9lTRoaYj27dtp/inxUGH\nJylCPVUimUlJlTRZJlQLn79wPjbYsL5GTacacmtyqfyoEnoc/rMitZRUiWQmDVQXqaNPvz5UVVbB\nbqjaXkXV6ira1xRQXHwjpaWlQYcnKUAD1UUyk5IqabIJE8ZRXX0fZWXPUVb2XKRa+Ligw4qrL3zx\nC+yq3sXg7MF0/rgzeW/l0d2H8cknl7F48SgmTpypxEoOqzapcg86EhFJJPM4/as3syzgDWCdu59f\nz/ser3tJcNJxoHooFGL+wvlUeiUb+25keNfhbH9vOwDL3/uEpUvPi3rk+RxjxixhxowpQYYcKDPD\n3S3oOJJVbVvXvj1s3Qp5eUFHJCIt0ZK2Lp5jqiYBS4HOcbymJJl0qxYeCoW4/pbryR6SzcdVH5Nb\nksvEH01k2NhhABQX3xhwhJKqamcAKqkSyRxxSarMrC/wVeB/gGvjcU2R1hLd27av5hNyhuSwo/sO\njvAj6JrTlX8++0+GDQsnVVogWVqqdrB6r15BRyIiiRKvnqqZwGQgP07XE2kVdctCrN1yHV0urqTD\nER0Y3nM4W7Zu+cz5WiBZWkozAEUyT8xJlZmdC2xy97fNrAho8PnjtGnTDu4XFRVRVFQU6+1FmmXu\n3KepqCjiQNYbAFivk1m/+h+M7nwaW3ZsCS8/M/mzy8+k2yPP5iopKaGkpCToMFKOZgCKZJ6YB6qb\n2f8C3wKqgPZAJ+Bxd/9OnfM0UF0Cd8UVP+Xxxa/S9qQjqWy3k70HPmSsj2bM6BEAjD9TFdMPRwPV\nG1fb1l18MVxwAXzzm0FHJCItEchAdXf/OfDzSABfBP67bkIlUiuW2YPxmHnYoWsNDF5HVbc89nVa\nTduPazj22F6qli5xp6VqRDKPKqpLwsSyzE08lsgJhUKsWLmCzj2y2V3wHkdZT3qdcDQc+HSWX7qU\niZDgaUyVSOaJa/FPd3+hvhpVIvDZZW4KCsaSnX3ZwZ6n1vwsfFo6YW+/vZTVlOGfVNE3qydVS6tY\n9NRKFi8epeKeGcbMZpvZJjN7N+rYQ2ZWGtlWmllp1HtTzGyFmS0zs7MOd30lVSKZRz1VkhHmL5xP\n1pAs9h+xn5N3n8yOF3fQZlcb+nc9kaUdz0vr9QylQfcCdwBzag+4+8W1+2Z2K7Ajsn8C8A3gBKAv\n8KyZDWpsoGh+Pqxa1TqBi0hyUlIlCdPSmk+lpaVs3LieVav+lz179tKxY4dm14uq8RpWV62mW243\nBg0cxObqzZyeezrbPqlp6deRFOfuL5pZv0ZO+QZQFNm/AHjI3auAVWa2AhgFvNbQh9VTJZJ5lFRJ\nwrSk5lP0WKru3YexZctNfO5zI7n66qaPp3J3bLBR/VQ1nXM6s3nL5oOlEyoqKlTcUw5hZl8ANrr7\nx5FDfYBXok5ZHznWICVVIplHSZUkVHNrPtUdS9WhwyB69lzSrGv866N/UdWxivu/dz8Ln18IwPjJ\nn5ZOUHFPqcc3gb+19MPTpk1j7Vp4/30oKVFNPpFUEI+afEqqJK29vPZlXlzzIsWji8lvl0/h8EMT\npkwv7imfZWbZwNeA6L8U64Gjol73jRyr17Rp01i2DF5+GZRPiaSGukXJp0+f3uxrxHX2n0i8TZgw\njurq+ygre46ysucij+fGNemzS7cs5fFlj3PVqKvIb6cVlKRexqGrQJwJLHP3DVHHngQuNrM2ZjYA\nGAgsaezCevwnknlirqje5Buporq0UHOKfoZCIeYvnM+Omh18cuQn3HD2DQzsOjBRoWaEdKmobmYP\nEh6I3g3YBEx193vN7F7gFXf/Y53zpwBXAJXAJHd/poHrurtTXg7dusG+fa36NUSklbSkrVNSJWmj\nthaVD3FWVK6g5+qe3D3pbi07E2fpklS1ltq2zh3atoXdu8M/RSS1tKSt0+M/SRvzF87Hhhib8jcx\nuP9gCgYVMH/h/KDDkgxlpqVqRDKNkipJG9VezaqqVXRt35U+nRud7S6SEBpXJZJZlFRJWnB3Dgw8\ngH1i5G3MY9MHm8K1qM4cH3RoksGUVIlkFpVUkLTw6NJH6ditI/f/4H6efi48qD26FpVIEPLzlVSJ\nZBIlVWmqOTPmUt2zHz9LaEuI60dfT15uHiefeHLQIYkA6qkSyTRKqtJQ9NIuAIsWzWTWrPSpFD5v\n3jxm3T8LgK9c9BU2F2ym+Ixi8nLzAo5M5LOUVIlkFiVVaSh6aReAsrLwsaYkVcnewzVv3jy+P+37\nZI/Kpiq7ihf++QK3X3A7Xdt3DTo0kUNo9p9IZtFAdTmotodr8eJRLF48iokTZ1JaWhp0WAeFQiEm\n/2oy+zrtI7d7LjX9a+iQ34HH5j4WdGgi9VJPlUhmUU9VGpowYRyLFs2krCz8Ory0yzWH/VwsPVyt\nrbaw544BO6hoV8HGDRspsAK8SgVlJXl16QIbNhz+PBFJD0qq0lBhYSGzZl0T9RgvtcZT1X0E2bZt\nW66beh2ru6/m2MHH8samN6Acdr+wm/a72zNx2sSAIxapn2b/iWQWJVVpqrCwsNmJVEt7uOKp7iD7\nBQumkj9wO9ttO9tqtvHJrk8Y0H0A29/ezhHlR3DLtFu44IILEhqjSFPp8Z9IZol5TJWZ9TWz580s\nZGbvmdnV8QhMEq+2h2vMmCWMGbMkkBmD0Y8gCwrGsrPiSDblb+OEL59A5b5KqvdXk70um1EdRvH3\nv/xdCZUkNQ1UF8ks8eipqgKudfe3zawj8KaZPePuH8Th2pJgLenhSoTd+bvpfWJvql6tor/15+bp\nN6uwpyQ99VSJZJaYkyp33whsjOzvMbNlQB9ASZU0W+0jyOXLX2LTnvlUVa6kYFsHVq1ZxcDcgdAJ\nbp6shEpSg5IqkcwS1zFVZtYfOBl4LZ7XleAlqn5VYWEhRUW9mPnwr7FCo21BGzbu2815a89jxPEj\ntPSMpBQNVBfJLHFLqiKP/h4FJrn7nvrOmTZt2sH9oqIiioqK4nV7aUWJrNAeCoW49x/3kjXayOmf\nw/7K/XTa2ok9ZXso/mlx3O8nh1dSUkJJSUnQYaSkjh1h/36orITc3KCjEZHWFpekysxyCCdU97v7\nvIbOi06qJHXEu35VY71e8xfOx3oYWR2zqG5fTa7lUrmrMubvIC1X9xeg6dOnBxdMijEL91bt3AkF\nBUFHIyKtLV4V1e8Blrr7bXG6nqSpplRtP3rI0VTurcQ/capXVuPvOBO/rVpUkpo0A1Akc8SjpMJo\n4FLgy2b2lpmVmtnZsYcmyWLChHFUV99HWdlzlJU9F6lfNa5F16pbMiE7+7KDvVYA48aOY0+bPRzT\n7Rjy38+nc2lnbrzqRpVOkJSlweoimSMes/9eArLjEIskqURVaK/xGl7e/zKXnnkpuStysQHG+DOD\nGZie7AtLS+pQUiWSOVRRXZokXvWrGqra7u7MfX8uFVUVTPnqFHKygvurmciB+ZL+NANQJHMoqZKE\naqjX6+mPnuajbR9x3eevCzShguReWFpSj3qqRDKHkipJuLq9XkvWL6FkVQnFZxTTPrd9gJGJxJ8G\nqotkDiVVknChUIj5C+cDMOTUIbyw+wWuPf1aurTrEnBkYcmwsLS0PjObDYwHNrn78KjjVwFXEl6C\n65/u/rPI8SnA5ZHjk9z9mabcRz1VIplDSZUk1Lx58yj+bTFZQ7Lo2r0rdz9yN3d84w56d+oddGgH\nJWpgvgTuXuAOYE7tATMrAs4DTnT3KjMriBw/AfgGcALQF3jWzAa5ux/uJl26wEcftUL0IpJ0lFRJ\nwoRCIYpvLmbnsJ3k9MlhXfk6BncczPuvvs+5p50bdHifkawLS0v8uPuLZtavzuEfATe5e1XknEh/\nJRcAD0WOrzKzFcAomrAklwaqi2SOeBX/FDms+Qvnk90zm9yOuexvu5/27duzd8veoMMSiXYcMMbM\nXjWzRWb2ucjxPsDaqPPWR44dVrdusGVLnKMUkaSknipJqO4DurN++3pyc3KxHUbNBzWM/8n4oMMS\nqZUDHOHup5nZSOAR4JjmXiR6Sa5jjy3iww+L4hWfiLSSeKxzak0YEhAXZtaU4QeSxt5//30umXUJ\nFR0ryF6eTc2mGmZcP0PV0lOMmeHuFnQc8RB5/PeP2oHqZrYAmOHuL0RerwBOA74P4O43RY4/BUx1\n90Me/9Vt66qqoFOncGmODh1a+xuJSLy0pK1TT5XEVWOVyJdnL+fML59Jt9XdyD4yO7Bq6SJRLLLV\negL4MvCCmR0HtHH3rWb2JPCAmf2W8GO/gcCSptwgJwcGDYIPPwQN0xNJb0qqJG4aq0ResqqEtze+\nzS/H/5IObfTrugTPzB4EioBuZrYGmEp4cfh7zew9oAL4DoC7LzWzh4GlQCVwZXO63ocMgaVLlVSJ\npDslVRI3DVUit17GghULuH709UqoJGm4+yUNvPXtBs6/EbixJfeqTapEJL0pqZK42707xKad89m7\n9yM+3l7FX98t4+pTr6YgryDo0EQCMWQI/PWvQUchIq1NSZV8RmNjog5nwoRxPP74tawr/xh6GNlH\nl/Nybhsu7fR7+nWpWw5IJHOop0okMyipSgOxJEJ1r9PQmKimaNu2LZVdV5E1YjtZedk4VfTscBQf\nLvkQPt+ikETSwsCBsGYN7N8P7doFHY2ItBYV/0xxtYnQ4sWjWLx4FBMnzqS0tLRF14oeE1VQMJbs\n7MsOJmtNMX/hfPIK8ygY2pV2x+eSV5BHxUcVLYpFpKnM7FgzaxvZLzKzq80sORaSjGjTBo45BpYv\nDzoSEWlNSqpSXKyJULwd2f1Idu3Zhe93snZkUb2pmvFnqrintKrHgGozGwj8ETgKeDDYkA41dCiE\nQkFHISKtSUmVAOEer02bNrJq1f9j9epZlJU9R3X1fUyYMK7J1zj3K+eybes2urftTvcN3enyQRdm\nXD9DtaiktdVE1uT7D+AOd58M9Ao4pkNoXJVI+tOYqhQ3YcI4Fi2aSVlk2ddwInRNkz5bOxZr48b1\nvPnmBjp1+jEFBWPZsuU2TjnlBCZNavp4KoDVbVYz5ktj6Lm2JzlH5DD+JyruKQlRaWbfBP4LOC9y\nLDfAeOo1ZAg8/HDQUYhIa4pLUmVmZwO/I9zzNdvdZ8TjunJ4hYWFzJp1TdRA9aYlQtGD0levfphd\nu86isHAE/fvn07FjB448cslhrzNv3jxm3T8LgKKvFbGz505+fd6v6dy2c8zfS6QZvgtMBP7H3Vea\n2QDg/oBjOoR6qkTSX8xJlZllAXcCY4ENwOtmNs/dP4j12tI0hYWFzZ7xFz0Wa8uWJeza1Z0NGzaT\nn5/fpM/fcccdTLlzCjbSyOmQwwsLXuCuC+9SQiUJ5+5LgasBzOwIoFMy/mJ33HGwciUcOBAeuC4i\n6SceY6pGASvcfbW7VwIPAVohN4X07j0OmEN5+ctNGksVCoX45R2/pHJEJTWDatjTcw/t8tox96G5\niQtaJMLMSsyss5l1BUqBP0XW6EsqbdtCv36wYkXQkYhIa4nH478+wNqo1+sIJ1qSxOqOxerXr5KR\nI0vp2XNDo48QQ6EQ1029jvKqctgH1dnV5JJLZXnlwXPiVTdLpIny3X2XmX0PmOPuU83s3aCDqk/t\nI0ANNRRJTwkdqD5t2rSD+0VFRRQVFSXy9hLl0LFYvz5s8hMKhbj+lutZ3X01uaflUr6vHHs/iyw3\ncktzmXjrxJgLiEryKSkpoaSkJOgwGpNjZr2AbwC/CDqYxmhclUh6i0dStR44Oup138ixQ0QnVRK8\n5ozFqu2hWt19NX2G9GHt2rVkfZKNP5uLVbWle/tBHHXUUQ0uqqykKnXV/QVo+vTpwQVTv18CTwMv\nufvrZnYMkJQP2YYMgSefDDoKEWkt8UiqXgcGmlk/4BPgYuCbcbiuJInaHqo1NWvYVrONjes30tbb\nU7m5K53bDWFIv1upqNgYaNFRyVzu/gjwSNTrj4GLgouoYUOGwE03BR2FiLSWmAequ3s18BPgGSAE\nPOTuy2K9riSP+Qvnkzs0lyFjh1C1p4qqA1XUfOS0/7gXQ/rdSqdOnw4QmTBhHNXV91FW9lyLCoiK\nNJeZ9TWzv5vZ5sj2mJn1DTqu+gweDB99BFVVQUciIq0hLmOq3P0pYHA8riXJq7xLOT1O6oG/5nTd\n25Vd7bpSUbGRioqNB4uOtrRulkgM7iW8LM1/Rl5/K3LszMAiakBeHvTuHU6sjj8+6GhEJN7M3RNz\nIzNP1L0kvkKhED+47Qds7reZQbmDsKXGzZNvpqKiotVm+WkGYfIyM9zdgo6jlpm97e4nH+5YAuNp\ntK077zz47nfha19LYFAi0mwtaeu0TI3UKxQKMX/hfABOPO1EBhUNYszGMXTJ6sL4yZ8uP9MayY5m\nEEozbTWzbwF/i7z+JrA1wHgaVTsDUEmVSPpRUiWHqB2Ynjs0l/2+n7sfvpvf/efvOP+S8xNyf80g\nlGa6HLgDmAk48DJwWZABNWbIEHhaczpE0lI8KqpLGqktnbBq1yq8nbOp8yb69ujLstdSf+5BaWkp\nxcU3Ulx8I6WlpUGHI3ESWc3hfHfv7u493P1CknT2H4QLf6pWlUh6UlIlB4VCIa6ceiVv2puszlnN\n8288T4d9Heia3TWhcbTGDMLaR4qLF49i8eJRTJw4U4lVers26AAacvzxsHw5VFcHHYmIxJse/6WZ\nWAZ4z/7rbD7I/YCc43Ko2F+Bb3I2LNhAfp98xk8e31ohH6I1ZhDqkWLGSZqB9HV17Ag9eoQXVx44\nMOhoRCSelFSlkVgGeNcOTN87YC/t27Qnv30++9fvJ2d7Djf/7uaDA9MTpTnV3kXqkdRTjWsHqyup\nEkkvevyXRqJ7YwoKxpKdfVmTqpzXDkyvPraaivIKdqzdQe6aXDqs6cD4seMbTKhSaYySipKmHzPb\nbWa76tl2A72b8PnZZrYpevFlM5tqZuvMrDSynR313hQzW2Fmy8zsrFhiHzIEQqFYriAiyUhJVYar\nHZi+sv1Kup3WjbaD2tL2w7ZUv1bN4CMGc8V3rqj3c6k2Rqn2keKYMUsYM2aJSjSkAXfv5O6d69k6\nuXtTeuHvBerLrH/r7oWR7SkAMzuB8ILNJwDnAHeZWYsfMZ5+Oixa1NJPi0iy0uO/NDJhwjgWLZpJ\nWVn4dW2V84bMmzeP4puL2bZ/G5VHVFK9pZoRx4xgc8Vm+m3px63Tb22wlyoVxyjpkaJEc/cXI2uW\n1lVfsnQB4SW4qoBVZrYCGAW81pJ7jxsXLgC6Ywd06dKSK4hIMlJPVRppTm/MvHnz+N7/+x7re6+n\nakgVu/fuJmdTDpuXbWbAvgGNJlQiae4nZva2mf3ZzPIjx/oAa6POWR851iIdO0JRESxYEEOUIpJ0\n1FOVhGLoBo+PAAAgAElEQVSZwdeU3phQKETxzcXsH76fqt5VVFBBp02dyHk7h34D+nHz9PASNMXF\nNzYYQ3N7xURSxF3AL93dzezXwG+A7zX3ItOmTTu4X1RURFFR0SHnXHghPPEEXHJJi2MVkTgqKSmh\npKQkpmto7b8kU3cGX3X1fXEf/zPjdzOYs3gO2/puY2uXrdgBo82/29BnQx8e++NjVFRUNCkGrc+X\nmZJt7b9YRB7//cPdhzf2npn9DHB3nxF57ylgqrsf8vivqW3dli0waBBs3Ajt2sX8VUQkzrT2XxpI\n1Fil7gO6s37HejrmdqRyUyXtPmzHjF/NYOjQoRQX39ikGDRGSdKAETWGysyOdPeNkZdfA96P7D8J\nPGBmMwk/9hsILInlxt27w/Dh8NxzcO65sVxJRJKFkqoMUluLauMnG1m/ez29uvfClhs1m2qY8asZ\nXHDBBUGHKJIwZvYgUAR0M7M1wFTgS2Z2MlADrAJ+CODuS83sYWApUAlcGY+u99pHgEqqRNKDHv8l\nmdZ6/FdbiypnSA7rq9ez/ePtnFdwHjVVNezdlkVBQc+Dj/AS8QhSUlc6Pf5rDc1p6z7+OFxeYcMG\nyM5u5cBEpFla0tYpqUpC9Y1VinX80ozfzeCVylc40PsAW/Zu4cgdRzJgywBefWF7vcmTxktJQ5RU\nNa65bd1JJ8Fdd8Ho0a0YlIg0m5KqFNZYEhNLz1HtI7+SF0vYfMxmao6p4aSeJ7Hjox1sX7ifA7sn\nR42deo4xY5YwY8aUuH8/SR9KqhrX3LZu6lQoL4dbbmnFoESk2VrS1qlOVRI4XHXyli4/M2/ePC66\n8iL+8uFfWD9gPcs2LCPvozx2fLSDylAlA486vvW+lIg0yYUXwt//DvqdUyT1xTRQ3cxuBs4DKoB/\nA991913xCCyTtMaMv9paVDuH7SS7Tza7yncxsNdAOq/szOl9T2f85PEHSyeo1pRIcE4+GaqqwmsB\nDhsWdDQiEotYZ/89A/zM3WvM7CZgSmSTOGpJoc3Zc2azbf82KlZXYD2NDnkdqFpfRdEZRRT/tPjg\nebNmXRP12FGD0UUSzezTWYBKqkRSW9zGVJnZhcBF7v7tBt7XmKoGNGXMVHMGjodCIS668iLKBpSx\ns/1ObLXRuUNnCjYX8Nhdj2n5GYmJxlQ1riVtXUkJXHcdvPFG68QkIs0X6EB1M3uS8IKjDzbwvpKq\nRsRjtl0oFGL2nNnMf24+5YPL2dV7FzntcjjwwQHav9OeP9/yZ9WikpgpqWpcS9q6qio48kh46y04\n6qhWCkxEmqVVKqqb2UKgZ/QhwIFfuPs/Iuf8AqhsKKGq1ZT1sDJVrNXJQ6EQP/r5j/hw+4fszdvL\n/o77aVfTjj7eh+qCas6+6OyUSqhU0iF5xGM9LGlcTg6cfz787W9w/fVBRyMiLRVzT5WZXQZ8H/iy\nu1c0cp56qlpJKBTiuqnX8ebKN8kZlkP10dVs+3gbba0tR3c6mgH7BnDz5JtT5rGfio8mN/VUNa6l\nbd2bb4bHVn38MeTmtkJgItIsCS+pYGZnA5OB8xtLqKT11FZKX919NfuO2UfZ5jIO2AG6detG3to8\n+m3pl1IJFbS8hIRIKvvc58ILLM+dG3QkItJSsc7+uwNoAyw0M4BX3f3KmKOSJpv919msbL+SnG45\n5OTkUF5VTuVLlbTp0YYTup/ArdNvrTeh0uM1keRz3XXw85/DpZeGZwWKSGqJqafK3Qe5ez93L4xs\nSqgSKBQKsWDxArbXbGdru60cyDtAj/Ie9N7Zm0v7X8pd0+9qMKFqrNho0CZMGEd19X2UlT1HWdlz\nkRIS44IOS6TVnX02VFbCc88FHYmItESsPVUSkHnz5jH5V5MpKy/DPjCqulXRfk978lfl89hfGi+b\n0BrFRuOpsLBQ9bMkI2VlwX//N9x6K3zlK0FHIyLNpaQqBc2bN4/vT/s++4buo8IqqC6vpteKXuRX\n5zPutHEpNX6qIbHOhhRJVZdeCjfcAO++C8OHBx2NiDSH1v5LQbPun0X2qGy6n9Id+kNWtywOfHyA\n/p37c8V3rjjs5/V4TSR5tW0LV10Fv/1t0JGISHPFrfjnYW+kkgpxc87Xz+Gtbm/hgxyvdsrfK6f3\nB735+1/+3uReKg1Ul5ZSSYXGxaOt27YNBg6E996DPn3iFJiINEugFdUPeyMlVXHzxBNP8J17vgMD\nIW9vHjVLavjTtD+lVHFPSV1KqhoXr7Zu0iRo1w5mzIhDUCLSbEqq0lgoFGL+wvkAdBrWiZf//TJb\nnt1Clmcx8dsTlVBJwiipaly82rqVK2HEiPDPzp3jEJiINIuSqjQ1b948in9bTNaQLPK657Fj9w4e\n+K8HOPXkU4MOTTKQkqrGxbOt+/a3YcAA+OUv43I5EWmGhFdUl9YXCoUovrmYncfvZEefHSyvWk6v\nTr20FptIBvjf/4Xf/x5Wrw46EhFpCiVVSW7+wvlk98wmu1M2+3P30ymvE1u3bA06LBFJgKOOCs8E\nLC4OOhIRaQolVSmg6zFd2bV7F23L2lKztoaapTWMP3N80GGJSAJcfz28/DK8+GLQkYjI4SipSnJf\n/tKX2Vi1kWPzj6Xrv7uS/34+M66dkRYFPkXk8PLy4Kab4Kc/hZqaoKMRkcZooHoSqp3pV+3VbO23\nlaM6HEXFsgoAxp85nqFDh6rOlAQmXQaqm9lsYDywyd2H13nvv4FbgAJ33xY5NgW4HKgCJrn7Mw1c\nN+5tnTuMHg3f/z5897txvbSINECz/9JAKBTi+luuJ2dIDqurVlP9STV/++HfGDZs2MFzSktL+c53\nfs22beEyCl27zmPOnBuUWElCpFFSdQawB5gTnVSZWV/gz8Bg4HPuvs3MTgAeBEYCfYFngUH1NWqt\n1da9/jpccAF8+CF06hT3y4tIHZr9l+JCoRDXTb2Ole1XsrnjZtr3aM+Aowbwz2f/+Znzbr/9L6xa\ndRb79n2Vffu+yqpVZ3H77X9JaKylpaUUF99IcfGNlJaWJvTeIvHg7i8C2+t5ayYwuc6xC4CH3L3K\n3VcBK4BRrRvhZ40cCWeeGZ4RKCLJSUlVkpg3bx4X/eAi3lz5Jut9PUvXLqVXTi+y7NA/omXLVgLd\nadMmvEH3yLHEKC0tZeLEmSxePIrFi0cxceJMJVaSFszsfGCtu79X560+wNqo1+sjxxLqxhvhT3+C\nDz5I9J1FpClygg5AIrWoflvMzmE7qampYe+OveRX5PPBjg8YsG8A4yd/dqbf8ccfTSg0hwMHukSO\nzOH4449OWLxz5z5NdvZlFBSMBaCsLHxMjx8llZlZe+DnwJmxXmvatGkH94uKiigqKor1kgD07g2/\n+lW4KOjLL0NublwuKyJASUlJzDUglVQlgfkL55M1JIusPll4G6fbB93wd5x+A/px8/SbD5npN2nS\n5bzxxg1s3/4wAL17VzJp0uVBhC6STo4F+gPvmJkRHjtVamajCPdMRf/m0jdyrF7RSVW8TZwI8+aF\nHwNOndpqtxHJOHV/AZo+fXqzr6GkKkl07d6VdeXr6EQnanJryG+Xz63Tb623dEJhYSH33//rqNl/\nP0xoL9GECeNYtGgmZWXh19XV9zFhwjUJu79IHFlkw93fB448+IbZSqDQ3beb2ZPAA2b2W8KP/QYC\nSwKIFzO45x445RT46lfDY61EJDlo9l8SeOWtV/j2nG/TpVMXyreUU7O0hhnXzkjqRZJV0iFzpdHs\nvweBIqAbsAmY6u73Rr3/MTCiTkmFK4BKElxSoT5z54Z7qkpLw7WsRCS+AiupUF9Nl3rOUVJVj/1V\n+7nlpVvoXtmdne/vBD6tRSWSjNIlqWotiWzrLrkECgrg9tsTcjuRjBJIUlVfTZcGzlNSVUdVTRV3\nLrmT7nndueTESwgP4xBJbkqqGpfItm7bNjjpJLj3XvjKVxJyS5GMEVSdqvpqushhuDv3v3M/uVm5\nfPPEbyqhEpFm69oVZs8OV1nftCnoaEQkpoHq0TVdlBQ0z5MfPsnGPRu59vRrD9ai0jglEWmus86C\nyy6Diy6C55+HNm2Cjkgkcx02qTKzhUDP6EOAAzdwaE2XRjOr1qrdkmoWr17MGxve4PrR19M2py3w\naUHN7OzLAFi0aCazZl0Tc2KlRE1iFY/aLdK6pk+H996Dn/wE7r47PENQRBKvxWOqzGwY4fWvygkn\nU7V1W0a5++Z6zs/YMVW1CyQDDBo5iFf2vsLk0ZPp0aHHwXOKi29k8eJRUQU1n2PMmCXMmDGlxfet\nm6hVV98Xl0RNMpvGVDUuqLZu9244/XS48srwJiKxaUlb1+LHf43VdGnpNdNR7QLJuUNzKa8p5+7H\n7uYPF//hMwlVa1Hlc5HM0alTuCjo6NEwZAhk6IMAkUDFc+0/5zCP/zLR7DmzWbVrFWs+XsNqW02/\nnv14++W3DzlvwoRxVFffR1nZc5SVPRcpqDkO0OLFItI0xx4LDzwAF18MKxO3HKiIRMStorq7HxOv\na6WLUCjEglcXsP2E7VQcUUGbN9rQ56g+cNSh5xYWFjJr1jVR45/Cj+liGWulyucimWfsWPjFL2D8\neFi8GLp1CzoikcyhZWpa0fyF8+n1xV5srNxIm/ZtyD4imw2vbmD85ePrPb+wsPCQZCmWR3gNJWoi\nkt5+8hPYsCE8M/C556BLl8N/RkRip6SqFdV4DdvabWNQz0HYTmNP1h7OHnN2Qqul15eoiUh6Mwsv\nuFxeDuecA888Ex5zJSKtK55jqiSKu5N9fDaVGyvptbcXvXN7M2DfAK741hXNuk5jY61ERBpiBr/7\nHZx4Ipx/fjjBEpHWpQWVW8lTHz3F6+tf59wjzmXh8wuBlq/pp1pTkkxUUqFxydbWVVeHi4Nu2RKe\nHdi2bdARiaSGwBZUbtKNkqyhaU2vrXuNJz54guIziunSToMZJL0oqWpcMrZ1VVXwzW/C3r3wyCPQ\noUPQEYkkPyVVSeCDsg/4c+mfufb0a+ndqTeQuJ4m9WhJIiipalyytnWVlfDDH4Yrr//zn9Cj9Uvl\niaQ0JVUBW7drHb979Xf84HM/4LhuxwGJq2qu6umSKEqqGpfMbZ17eEmbv/4V/vUvGDQo6IhEkldL\n2joNVI+T7fu2c+eSO7l42MUHEyqA2267h3Xr8tiyZQm5uUeQnX3Zwd6keIouvVBQMLbV7iMiqcsM\npk2DKVNgzBh49dWgIxJJL0qq4qC8spzbX7udsQPGMqL3iIPHS0tLeeaZZezadRZbt47i3Xdnsnfv\nigAjFRGBK66Ae+4Jzwp85JGgoxFJH0qqYlRVU8UfXv8Dxxccz1eO+cpn3ps792kKCiaRk3MMMJyq\nqgvZsmVOq5REUOkFEWmOc86Bp56C4mKYNAkqKoKOSCT1KamKgbtz39v30aFNB/5z6H9iduij144d\nOzB8+DF067aDzp33MG7cyFYZ51RbPX3MmCWMGbNE46lE5LAKC+HNN2HNGjjjDK0XKBIrDVSPwWNL\nH+Pf2//NNaddQ2527iHvBz14vO5sQECzAyVmGqjeuFRs69zhttvCVdjvvhv+4z+CjkgkeJr9l0CL\nVi5i0apFFI8upkObhou+BFXmoG5Ct2vXbzBrR6dOPwY0O1BaTklV41K5rXvtNbj4YjjzTLj5Zq0Z\nKJmtJW2d1v5rgbc+eYunPnqKyaMnN5pQQXBr79VdiHn16oeBzzNgQPMXZhaRzHDqqfD22/Czn8HQ\noXDHHfC1rwUdlUjq0JiqZvr3tn/z13f/ypUjr6QgryDocERE4io/H/7wB3joIfj5z8NJ1YYNQUcl\nkhqUVDXDpj2bmPXGLC4/5XL6dekXdDiNqjsb8Igj1tK16zzNDhSJMLPZZrbJzN6NOvZLM3vHzN42\ns2fNrG/Ue1PMbIWZLTOzs4KJOnG+8IVwr9WwYXDSSfCb38D+/UFHJZLcNKaqiXZV7GLGizP46qCv\nMvro0UGH0yQaqC6tIV3GVJnZGcAeYI67D48c6+jueyL7VwHD3f37ZjYEeAAYCfQFngUG1deopXpb\nV59ly8KPBN95B379a7jkEsjSr+SS5jRQvZVUVFXwm1d+w/Cewxl/3PigwxEJVLokVQBm1g/4R21S\nVee9nwFd3P1nkX139xmR9/4FTHP31+r5XMq2dYfzf/8HkyeHa1rNmBEe0F5PJRmRtBDIMjVmdlWk\nO/w9M7sp1uslm+qaau5+8276du7LuYPODTocEWllZvZrM1sDXAbcGDncB1gbddr6yLGM8oUvwCuv\nwA03wFVXwWmnweOPQ3V10JGJJIeYZv+ZWRFwHnCiu1eZWVqN3HZ3HnjvAQzj0hMvrbe4p4ikF3e/\nAbjBzIqB3wHfbe41pk2bdnC/qKiIoqKieIUXODO46CK48EKYNy/cYzVlClx3HXz729CuXdARirRM\nSUkJJSUlMV0jpsd/ZjYXuNvdn2/CuSnXJT5/+Xze2fgO133+OtrmtA06HJGkkEGP/44CFrj7ifU8\n/nsKmJppj//q4w4vvBCua1VaCt/9Lnzve3DssUFHJhKbIB7/HQeMMbNXzWyRmY047CdSxEtrXuKV\nta9w1alXKaESSV8W2cIvzAZGvXch8HZk/0ngYjNrY2YDgIHAkoRFmcTMoKgIFiyARYvgwAE4/XQY\nOzZclkFrCkomOWxPlZktBHpGHwIcuAH4H+B5d59kZiOBue5+TAPXSZnf3kKbQ9z39n1c9/nr6Nmx\n5+E/IJJB0qWnysweBIqAbsAmYCpwLjAYqAI+Bn7k7psj508BrgAqgUnu/kwD102Ztq61VFSEHw3+\n6U/h3qsLLoBvfCOcaOUeuqKXSFJK+Ow/M1sAzHD3FyKvPwJOdfet9ZzrU6dOPfg6WccZrNm5htte\nvY0rR17JsV3Vfy1Sd5zB9OnT0yKpai1Kqj5r3Tp49FF4+GFYvjw8FutrX4MvfQnatw86OpGGBZFU\n/QDo4+5Tzew4YKG711sVMxUamq3lW7n5pZu5eNjFnNLrlKDDEUlK6dJT1VpSoa0Lypo14QRr3jx4\n6y0YPRrOOSe8DRoUdHQinxVEUpUL3AOcDFQA/13ba1XPuUnd0Ow9sJcZL83gS/2/xJcGfCnocESS\nlpKqxiV7W5csduyAZ5+Ff/0rvLVpA2PGwBe/GP45cKBqYEmwVPyzhSqrK5n56kyOPeJYLhpyUdDh\niCQ1JVWNS+a2Llm5w4cfwuLF4ZmEL7wQrn11+ukwcmR4GzECunQJOlLJJEqqWqDGa/jjm38kJyuH\nK065QrWoRA5DSVXjkrWtSyXusGoVvPYavP56eHvrLejdO7wO4YknfroNGKAlc6R1KKlqJnfn4dDD\nrNu1jkmnTSInK6ZaqCIZQUlV45KxrUsH1dXhNQjffRfee+/TbevW8HiswYPhuOPC26BB4WSre3c9\nQpSWU1LVTAv/vZCX177M5NGTycvNCzqcFqm7aLIWSZbWpqSqccnY1qWzXbvCswqXLw8/QqzdX7ky\nXDOrf/9wgtW/P/TtC336hH/27Rvu+cpLzaZfEkBJVTO8vv51Hlv2GMWjizmi/RFBh9MipaWlTJw4\nk+zsywCorr6PWbOuUWIlrUpJVeOSra3LZDt3hh8jrlwJq1fD+vXhEg+124YN4QHyPXvCkUeGtx49\noKDgs1vXrnDEEeEtP1+PGzNFS9q6jHzetXzrcuaG5vLT036asgkVwNy5T5OdfRkFBWMBKCsLH1NS\nJSISToBOOim81cc93NO1ceOn25Yt4bb0ww/hpZfCr7dtg+3bw9uePdC5c/jatVvt644doVOnT392\n6PDplpf36c/27Q/dcnP1qDIdZFxStWH3Bv745h/5XuH36Nu5b9DhiIhIQMw+TYwGD27aZ6qqwj1g\nO3eGE7Lo/T17YPfu8M8NG8L75eWwd+9nf+7bd+hWXQ1t24a3du0+3W/T5tOfbdqEk6/an/VtOTmf\n/ozesrPDW/R+3S0rq+GfdffNDt2veyz6eFO32vNr/3yit9pj0e/Vtx/9s6nHOnSIz+zSjEqqamf6\nfX3I1zm+4Pigw4nZhAnjWLRoJmVl4dfV1fcxYcI1wQYlIpLGcnKgW7fwFk81NeHlffbvD28VFeEx\nYQcOhPcrKqCyMrzVHq99XbtVVX36M3qr/Ux1dXirqvp0v3arqWn4Z91j7uH92p8N7bsfun+4raYm\n/N+j7vHaY9Hv1bcf/bM5xy69NLwoeKwybkzVropddG7bOegw4kYD1SXRNKaqccnS1olIbDRQXURa\nnZKqxqmtE0kPLWnrNIdBREREJA6UVImIiIjEgZIqERERkThQUiUiIiISB0qqREREROJASZWIiIhI\nHCipEhEREYkDJVUiIiIicaCkSkRERCQOlFSJiIiIxEFMSZWZjTSzJWb2VuTniHgFJiLSmsxstplt\nMrN3o47dbGbLzOxtM3vMzDpHvTfFzFZE3j8rmKhFJJnF2lN1M3CDu58CTAVuiT2k1ldSUhJ0CEDy\nxAGKpT7JEgckVyxp5F5gXJ1jzwBD3f1kYAUwBcDMhgDfAE4AzgHuMrO0Xf8w1f++pXr8kPrfIdXj\nb6lYk6pPgPzIfhdgfYzXS4hk+cNOljhAsdQnWeKA5IolXbj7i8D2OseedfeayMtXgb6R/fOBh9y9\nyt1XEU64RiUq1kRL9b9vqR4/pP53SPX4Wyonxs//DHjJzH4DGPD52EMSEUkKlwN/i+z3AV6Jem99\n5JiIyEGHTarMbCHQM/oQ4MANwFXAVe7+hJl9HbgHOLM1AhURSRQz+wVQ6e5/O+zJIiIR5u4t/7DZ\nLnePHsi5093zGzi35TcSkaTi7mkxnsjM+gH/cPfhUccuA74PfNndKyLHfga4u8+IvH4KmOrur9Vz\nTbV1ImmiuW1drI//VpjZF939BTMbCyyPV2AiIglgkS38wuxsYDIwpjahingSeMDMZhJ+7DcQWFLf\nBdXWiWSuWJOqHwK/N7M2wH7gB7GHJCLS+szsQaAI6GZmawjPYP450AZYGJnc96q7X+nuS83sYWAp\nUAlc6bF084tIWorp8Z+IiIiIhCW8orqZXRUpnveemd2U6PvXieW/zazGzLoGGEODxQYTdP+zzewD\nM1tuZsWJvHedOPqa2fNmFor83bg6qFgi8WSZWamZPRlwHPlm9kjk70jIzE4NMJYpkRjeNbMHIj3U\nEiVZ/j01RwNFUI8ws2fM7EMze9rM6h0rmwwaajtS5TuYWVszey1SRDtkZv8bOZ4S8deq22amYPyr\nzOyd2mLmkWPN/g4JTarMrAg4DzjR3U8Ebk3k/evE0pfwTMXVQcUQUW+xwUQwsyzgTsIFEIcC3zSz\n4xN1/zqqgGvdfShwOvDjAGMBmET4UU/QbgMWuPsJwEnAsiCCiAzo/j5wSmRQdw5wcRCxJKsk+/fU\nHPUVQf0Z8Ky7DwaeJ4HtUgs01HakxHeIjN37UqSI9nDgy2Y2mhSJP0rdNjPV4q8Bitz9FHevrUHX\n7O+Q6J6qHwE3uXsVgLuXJfj+0WYSHpAaqEaKDSbCKGCFu69290rgIeCCBN7/IHff6O5vR/b3EE4e\nAqkDFEm4vwr8OYj7R8XRGfiCu98LECk8uSugcHYBB4AOZpYD5AEbAoolWSXNv6fmqK8IKuG4/xLZ\n/wtwYUKDaoYG2o6+pNZ3KI/stiX8/+XtpFD8DbSZKRN/hHFoTtTs75DopOo4YIyZvWpmiyygtQLN\n7Hxgrbu/F8T9G3E58K8E3q8PsDbq9TqSoKChmfUHTgYOma6eILUJd9ADDgcAZWZ2b6Rb/Y9m1j6I\nQNx9O/AbYA3hwpc73P3ZIGJJYkn576mFerj7JggnLUCPgONpkqi241WgZ6p8h8ijs7eAjUCJuy8l\nheKn/jYzleKHcOwLzex1M/te5Fizv0Oss/8OYY0XC80BjnD308xsJPAwcEy8Y2hCHD/ns0VKW3UK\ndCOx/MLd/xE5p7bY4IOtGUuyM7OOwKPApMhvnYm+/7nAJnd/O/K4Osjp8TlAIfBjd3/DzH5HuDt6\naqIDMbNjgGuAfsBO4FEzuyTT/75mkKB/wTisum2HHVovLGm/Q+RpxSmR3umnI21PSsRfT5vZkKSM\nP8pod//EzLoDz5jZh7TgzyDuSZW7N1hR3cwmAo9Hzns9Mki8m7tvTVQcZjYM6A+8Y2ZGuJv4TTMb\n5e6b4x1HY7FExXQZ4a7TL7fG/RuxHjg66nVfAly/MfJY6VHgfnefF1AYo4HzzeyrQHugk5nNcffv\nBBDLOsI9qm9EXj8KBDX4eQTwkrtvAzCzxwkvS6Wk6lNJ9e8pRpvMrKe7bzKzI4FWaRvjpYG2I6W+\nA4C77zKzBYT/vaVK/PW1mfcDG1MkfgDc/ZPIzy1m9gThx/nN/jNI9OO/J4gkDmZ2HJDbGglVY9z9\nfXc/0t2PcfcBhP/HdUprJVSHY58WGzy/TrHBRHgdGGhm/SIzuS4mXOQwKPcAS939tqACcPefu/vR\n7n4M4f8ezweUUBHpdl4b+bcCMJbgBs9/CJxmZu0iv4yMJaBB80ks2f49NcdniqASjvuyyP5/AUH9\nktNU9bUdKfEdzKygdlZZ5PH+mcBbpEj8DbSZ3wb+QQrED2BmeZGeTsysA3AW8B4t+DOIe0/VYdwL\n3GNm7wEVQCD/s6rDCfYRzx3UU2wwETd292oz+wnhGYhZwGx3D2p22WjgUuC9yNgCB37u7k8FEU8S\nuZpwJe9c4GPgu0EE4e7vmNkc4E2gmnCj/8cgYklWyfTvqTms/iKoNwGPmNnlhGdIfyO4CBvXUNsB\nzAAeToHv0Av4S+SXlSzCvW3PRb5LKsTfkJtInfh7An+PPDLOAR5w92fM7A2a+R1U/FNEREQkDhJe\n/FNEREQkHSmpEhEREYkDJVUiIiIicaCkSkRERCQOlFSJiIiIxIGSKhEREZE4UFIlIiJJz8y6mtlb\nkXUwPzGzdVGvm1Rz0cxmm9mgw5xzpZl9Mz5R13v9/4gq6CtpRnWqREQkpZjZ/wfscfff1vOeeRL/\nj/Hvu0AAAAM/SURBVC2yhMujAS7FJa1IPVUiIpJqDq6CYWbHmlnIzP5qZu8DR5rZ3Wa2xMzeM7Mb\nos79PzMbbmbZZrbdzG40s7fN7CUzK4ic8yszuzrq/BvN7DUzW2Zmp0WO55nZo2b2vpk9Ymavm9nw\nQ4I0uyUS29uR65xBeJ3X30Z62I42s4Fm9lTkGiVmNjDy2fvN7C4ze8PMPogsaYaZDYt8t9LIdfu3\n2n9labZEL1MjIiISb4OBb7n7WwBmVuzuO8wsG1hkZo+6+wd1PpMPLHL3KWb2G+By4Ob6Lu7up5rZ\neYSX8DkHuAr4xN2/Hkmm3qz7GTPrAZzj7kMjrztHLZj8iLs/GTn+PHCFu680s88DvwfGRS7T191H\nRB4XPmtmxwJXAre4+yOR5auCXGZN6lBSJSIiqe7ftQlVxKWR9dpyCK+tNwSom1SVu/szkf03gTMa\nuPbjUef0i+yfQXhtO9z9XTML1fO5bUC1mf0RWADMr3tCZCHl04DHImv/wWefID0cucfyyLqMg4CX\ngf8X6aF63N3/3UDcEgA9/hMRkVS3t3Yn8vjsaqDI3U8Cngba1fOZA1H71TTcyVDRhHMO6S1y9yr+\n/3bun7WKIArD+HNAsUgwtWCTIgEL8Q9+kDSSYJ/0ySfIZ0gaLcWAndgoWthpFywsTCEIFiFFGjU2\nQfG1uINeNtFwYcBceH6wsOzOzG75cuYwcAd4CiwBz/4y7zDJ7SS32nVjfJnB2CTZaesdAy/alqLO\nCUOVJGnajYeay8BX4FtVXeHPVtq/5kzqDbAMUFXXgWsnFq+aBeaSPAc2gJvt1VH7R5J8Bg6qaqnN\nqUFv1t32fBG4CnyoqvkkH5NsMap+nejl0v/j9p8kadr9rugkeVtVe8Ae8Al4fdq4wf2Z6w5sAw9b\nY/z7dn0ZjJkDnlTVJUYBbr09fww8qKoNRhWnFeB+VW0CF4Ed4F0bu19Vu8AMsJrkR1Xda0c+fAf2\nGfV56ZzwSAVJkibQGuAvJDlu240vgYUkPzt+4xFjDe2aDlaqJEmazCzwauzQ0bWegaqx4jGFrFRJ\nkiR1YKO6JElSB4YqSZKkDgxVkiRJHRiqJEmSOjBUSZIkdWCokiRJ6uAX5JpS92QHQFAAAAAASUVO\nRK5CYII=\n", 800 | "text/plain": [ 801 | "" 802 | ] 803 | } 804 | } 805 | ], 806 | "execution_count": 0 807 | }, 808 | { 809 | "cell_type": "markdown", 810 | "metadata": { 811 | "id": "TzIETgHwTexL", 812 | "colab_type": "text" 813 | }, 814 | "source": [ 815 | "This code looks very similar to the code above, but without using `l2_loss` or `GradientDescentOptimizer`. Let's look at exactly what it is doing instead.\n", 816 | "\n", 817 | "This code is the key difference:\n", 818 | "\n", 819 | ">`loss = 0.5 * tf.reduce_sum(tf.mul(yerror, yerror))`\n", 820 | "\n", 821 | ">`gradient = tf.reduce_sum(tf.transpose(tf.mul(input, yerror)), 1, keep_dims=True)`\n", 822 | "\n", 823 | ">`update_weights = tf.assign_sub(weights, learning_rate * gradient)`\n", 824 | "\n", 825 | "The first line calculates the L2 loss manually. It's the same as `l2_loss(yerror)`, which is half of the sum of the squared error, so $\\frac{1}{2} \\sum (\\hat{y} - y)^2$. With this code, you can see exactly what the `l2_loss` operation does. It's the total of all the squared differences between the target and our estimates. And minimizing the L2 loss will minimize how much our estimates of $y$ differ from the true values of $y$.\n", 826 | "\n", 827 | "The second line calculates $\\begin{bmatrix}\\sum{(\\hat{y} - y)*1} \\\\ \\sum{(\\hat{y} - y)*x_i}\\end{bmatrix}$. What is that? It's the partial derivatives of the L2 loss with respect to $w_1$ and $w_2$, the same thing as what `gradients(loss, weights)` does in the earlier code. Not sure about that? Let's look at it in more detail. The gradient calculation is going to get the partial derivatives of loss with respect to each of the weights so we can change those weights in the direction that will reduce the loss. L2 loss is $\\frac{1}{2} \\sum (\\hat{y} - y)^2$, where $\\hat{y} = w_2 x + w_1$. So, using the chain rule and substituting in for $\\hat{y}$ in the derivative, $\\frac{\\partial}{\\partial w_2} = \\sum{(\\hat{y} - y)\\, *x_i}$ and $\\frac{\\partial}{\\partial w_1} = \\sum{(\\hat{y} - y)\\, *1}$. `GradientDescentOptimizer` does these calculations automatically for you based on the graph structure.\n", 828 | "\n", 829 | "The third line is equivalent to `weights -= learning_rate * gradient`, so it subtracts a constant the gradient after scaling by the learning rate (to avoid jumping too far each time, which risks moving in the wrong direction). It's also the same thing that `GradientDescentOptimizer(learning_rate).minimize(loss)` does in the earlier code. Gradient descent updates its first parameter based on the values in the second after scaling by the third, so it's equivalent to the `assign_sub(weights, learning_rate * gradient)`.\n", 830 | "\n", 831 | "Hopefully, this other code gives you a better understanding of what the operations we used previously are actually doing. In practice, you'll want to use those high level operators most of the time rather than calculating things yourself. For this toy example and simple network, it's not too bad to compute and apply the gradients yourself from scratch, but things get more complicated with larger networks." 832 | ] 833 | } 834 | ] 835 | } -------------------------------------------------------------------------------- /notebooks/BUILD: -------------------------------------------------------------------------------- 1 | package(default_visibility = ["//visibility:private"]) 2 | 3 | licenses(["notice"]) # Apache 2.0 4 | 5 | exports_files(["LICENSE"]) 6 | 7 | filegroup( 8 | name = "all_files", 9 | srcs = glob( 10 | ["**/*"], 11 | exclude = [ 12 | "**/METADATA", 13 | "**/OWNERS", 14 | ], 15 | ), 16 | visibility = ["//tensorflow:__subpackages__"], 17 | ) 18 | -------------------------------------------------------------------------------- /notebooks/LICENSE: -------------------------------------------------------------------------------- 1 | Copyright 2015 The TensorFlow Authors. All rights reserved. 2 | 3 | Licensed under the Apache License, Version 2.0 (the "License"); 4 | you may not use this file except in compliance with the License. 5 | You may obtain a copy of the License at 6 | 7 | http://www.apache.org/licenses/LICENSE-2.0 8 | 9 | Unless required by applicable law or agreed to in writing, software 10 | distributed under the License is distributed on an "AS IS" BASIS, 11 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | See the License for the specific language governing permissions and 13 | limitations under the License. 14 | -------------------------------------------------------------------------------- /notebooks/tf.contrib.learn Quickstart.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 8, 6 | "metadata": { 7 | "collapsed": false, 8 | "deletable": true, 9 | "editable": true 10 | }, 11 | "outputs": [ 12 | { 13 | "name": "stdout", 14 | "output_type": "stream", 15 | "text": [ 16 | "INFO:tensorflow:Using default config.\n", 17 | "INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_num_ps_replicas': 0, '_keep_checkpoint_max': 5, '_tf_random_seed': None, '_task_type': None, '_environment': 'local', '_is_chief': True, '_cluster_spec': , '_tf_config': gpu_options {\n", 18 | " per_process_gpu_memory_fraction: 1\n", 19 | "}\n", 20 | ", '_task_id': 0, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_evaluation_master': '', '_keep_checkpoint_every_n_hours': 10000, '_master': ''}\n", 21 | "WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/head.py:1362: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n", 22 | "Instructions for updating:\n", 23 | "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n", 24 | "INFO:tensorflow:Create CheckpointSaverHook.\n", 25 | "INFO:tensorflow:Saving checkpoints for 1 into /tmp/iris_model/model.ckpt.\n", 26 | "INFO:tensorflow:loss = 1.93453, step = 1\n", 27 | "INFO:tensorflow:global_step/sec: 115.28\n", 28 | "INFO:tensorflow:loss = 1.09608, step = 101\n", 29 | "INFO:tensorflow:global_step/sec: 128.192\n", 30 | "INFO:tensorflow:loss = 1.09607, step = 201\n", 31 | "INFO:tensorflow:global_step/sec: 128.54\n", 32 | "INFO:tensorflow:loss = 1.09607, step = 301\n", 33 | "INFO:tensorflow:global_step/sec: 132.66\n", 34 | "INFO:tensorflow:loss = 1.09607, step = 401\n", 35 | "INFO:tensorflow:global_step/sec: 129.689\n", 36 | "INFO:tensorflow:loss = 1.09607, step = 501\n", 37 | "INFO:tensorflow:global_step/sec: 127.194\n", 38 | "INFO:tensorflow:loss = 1.09607, step = 601\n", 39 | "INFO:tensorflow:global_step/sec: 127.367\n", 40 | "INFO:tensorflow:loss = 1.09607, step = 701\n", 41 | "INFO:tensorflow:global_step/sec: 127.626\n", 42 | "INFO:tensorflow:loss = 1.09607, step = 801\n", 43 | "INFO:tensorflow:global_step/sec: 124.069\n", 44 | "INFO:tensorflow:loss = 1.09607, step = 901\n", 45 | "INFO:tensorflow:global_step/sec: 129.075\n", 46 | "INFO:tensorflow:loss = 1.09607, step = 1001\n", 47 | "INFO:tensorflow:global_step/sec: 127.563\n", 48 | "INFO:tensorflow:loss = 1.09607, step = 1101\n", 49 | "INFO:tensorflow:global_step/sec: 128.849\n", 50 | "INFO:tensorflow:loss = 1.09607, step = 1201\n", 51 | "INFO:tensorflow:global_step/sec: 124.76\n", 52 | "INFO:tensorflow:loss = 1.09607, step = 1301\n", 53 | "INFO:tensorflow:global_step/sec: 128.621\n", 54 | "INFO:tensorflow:loss = 1.09607, step = 1401\n", 55 | "INFO:tensorflow:global_step/sec: 131.069\n", 56 | "INFO:tensorflow:loss = 1.09607, step = 1501\n", 57 | "INFO:tensorflow:global_step/sec: 125.608\n", 58 | "INFO:tensorflow:loss = 1.09607, step = 1601\n", 59 | "INFO:tensorflow:global_step/sec: 118.155\n", 60 | "INFO:tensorflow:loss = 1.09607, step = 1701\n", 61 | "INFO:tensorflow:global_step/sec: 128.63\n", 62 | "INFO:tensorflow:loss = 1.09607, step = 1801\n", 63 | "INFO:tensorflow:global_step/sec: 129.359\n", 64 | "INFO:tensorflow:loss = 1.09607, step = 1901\n", 65 | "INFO:tensorflow:Saving checkpoints for 2000 into /tmp/iris_model/model.ckpt.\n", 66 | "INFO:tensorflow:Loss for final step: 1.09607.\n", 67 | "WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/head.py:1362: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n", 68 | "Instructions for updating:\n", 69 | "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n", 70 | "INFO:tensorflow:Starting evaluation at 2017-04-04-23:19:19\n", 71 | "INFO:tensorflow:Evaluation [1/1]\n", 72 | "INFO:tensorflow:Finished evaluation at 2017-04-04-23:19:23\n", 73 | "INFO:tensorflow:Saving dict for global step 2000: accuracy = 0.266667, auc = 0.399998, global_step = 2000, loss = 1.12176\n", 74 | "WARNING:tensorflow:Skipping summary for global_step, must be a float or np.float32.\n", 75 | "\n", 76 | "Test Accuracy: 0.266667\n", 77 | "\n", 78 | "New Samples, Class Predictions: [0, 0]\n", 79 | "\n" 80 | ] 81 | } 82 | ], 83 | "source": [ 84 | "from __future__ import absolute_import\n", 85 | "from __future__ import division\n", 86 | "from __future__ import print_function\n", 87 | "\n", 88 | "import os\n", 89 | "import urllib\n", 90 | "\n", 91 | "import numpy as np\n", 92 | "import tensorflow as tf\n", 93 | "\n", 94 | "# Data sets\n", 95 | "IRIS_TRAINING = \"iris_training.csv\"\n", 96 | "IRIS_TRAINING_URL = \"http://download.tensorflow.org/data/iris_training.csv\"\n", 97 | "\n", 98 | "IRIS_TEST = \"iris_test.csv\"\n", 99 | "IRIS_TEST_URL = \"http://download.tensorflow.org/data/iris_test.csv\"\n", 100 | "\n", 101 | "def main():\n", 102 | " # If the training and test sets aren't stored locally, download them.\n", 103 | " if not os.path.exists(IRIS_TRAINING):\n", 104 | " raw = urllib.urlopen(IRIS_TRAINING_URL).read()\n", 105 | " with open(IRIS_TRAINING, \"w\") as f:\n", 106 | " f.write(raw)\n", 107 | "\n", 108 | " if not os.path.exists(IRIS_TEST):\n", 109 | " raw = urllib.urlopen(IRIS_TEST_URL).read()\n", 110 | " with open(IRIS_TEST, \"w\") as f:\n", 111 | " f.write(raw)\n", 112 | "\n", 113 | " # Load datasets.\n", 114 | " training_set = tf.contrib.learn.datasets.base.load_csv_with_header(\n", 115 | " filename=IRIS_TRAINING,\n", 116 | " target_dtype=np.int,\n", 117 | " features_dtype=np.float32)\n", 118 | " test_set = tf.contrib.learn.datasets.base.load_csv_with_header(\n", 119 | " filename=IRIS_TEST,\n", 120 | " target_dtype=np.int,\n", 121 | " features_dtype=np.float32)\n", 122 | "\n", 123 | " # Specify that all features have real-value data\n", 124 | " feature_columns = [tf.contrib.layers.real_valued_column(\"\", dimension=4)]\n", 125 | "\n", 126 | " # Build 3 layer DNN with 10, 20, 10 units respectively.\n", 127 | " classifier = tf.contrib.learn.DNNClassifier(feature_columns=feature_columns,\n", 128 | " hidden_units=[10, 20, 10],\n", 129 | " n_classes=3,\n", 130 | " model_dir=\"/tmp/iris_model\")\n", 131 | " # Define the training inputs\n", 132 | " def get_train_inputs():\n", 133 | " x = tf.constant(training_set.data)\n", 134 | " y = tf.constant(training_set.target)\n", 135 | "\n", 136 | " return x, y\n", 137 | "\n", 138 | " # Fit model.\n", 139 | " classifier.fit(input_fn=get_train_inputs, steps=2000)\n", 140 | "\n", 141 | " # Define the test inputs\n", 142 | " def get_test_inputs():\n", 143 | " x = tf.constant(test_set.data)\n", 144 | " y = tf.constant(test_set.target)\n", 145 | "\n", 146 | " return x, y\n", 147 | "\n", 148 | " # Evaluate accuracy.\n", 149 | " accuracy_score = classifier.evaluate(input_fn=get_test_inputs,\n", 150 | " steps=1)[\"accuracy\"]\n", 151 | "\n", 152 | " print(\"\\nTest Accuracy: {0:f}\\n\".format(accuracy_score))\n", 153 | "\n", 154 | " # Classify two new flower samples.\n", 155 | " def new_samples():\n", 156 | " return np.array(\n", 157 | " [[6.4, 3.2, 4.5, 1.5],\n", 158 | " [5.8, 3.1, 5.0, 1.7]], dtype=np.float32)\n", 159 | "\n", 160 | " predictions = list(classifier.predict(input_fn=new_samples))\n", 161 | "\n", 162 | " print(\n", 163 | " \"New Samples, Class Predictions: {}\\n\"\n", 164 | " .format(predictions))\n", 165 | "\n", 166 | "if __name__ == \"__main__\":\n", 167 | " main()" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": null, 173 | "metadata": { 174 | "collapsed": true, 175 | "deletable": true, 176 | "editable": true 177 | }, 178 | "outputs": [], 179 | "source": [] 180 | } 181 | ], 182 | "metadata": { 183 | "kernelspec": { 184 | "display_name": "Python 2", 185 | "language": "python", 186 | "name": "python2" 187 | }, 188 | "language_info": { 189 | "codemirror_mode": { 190 | "name": "ipython", 191 | "version": 2 192 | }, 193 | "file_extension": ".py", 194 | "mimetype": "text/x-python", 195 | "name": "python", 196 | "nbconvert_exporter": "python", 197 | "pygments_lexer": "ipython2", 198 | "version": "2.7.9" 199 | } 200 | }, 201 | "nbformat": 4, 202 | "nbformat_minor": 2 203 | } 204 | -------------------------------------------------------------------------------- /simple_console.py: -------------------------------------------------------------------------------- 1 | # Copyright 2015 The TensorFlow Authors. All Rights Reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | # ============================================================================== 15 | 16 | """Start a simple interactive console with TensorFlow available.""" 17 | 18 | """Start a simple interactive console with TensorFlow available.""" 19 | 20 | from __future__ import absolute_import 21 | from __future__ import division 22 | from __future__ import print_function 23 | 24 | import code 25 | import sys 26 | 27 | 28 | def main(_): 29 | """Run an interactive console.""" 30 | code.interact() 31 | return 0 32 | 33 | 34 | if __name__ == '__main__': 35 | sys.exit(main(sys.argv)) 36 | --------------------------------------------------------------------------------