├── .gitignore ├── 10_Adversarial_Examples.ipynb ├── 1_Setup.ipynb ├── 2_Introduction.ipynb ├── 3_Graphs_Sessions.ipynb ├── 4_Basic_Operations.ipynb ├── 5_Variables_Gradients_Placeholders.ipynb ├── 6_Classification_Regression.ipynb ├── 7_Visualization.ipynb ├── 8_Multi_Layer_Perceptron.ipynb ├── 9_GAN.ipynb ├── 9_GAN_Harder.ipynb ├── Articles.md ├── Papers.md ├── README.md ├── Turn a GAN into a WGAN and then WGAN GP.ipynb ├── environment.yml ├── images ├── 1_setup.png ├── 2_tensorflow.png ├── 3_graph.png ├── gan1.png ├── gan1000.png ├── gan10000.png ├── gan20000.png ├── gan30000.png └── gan40000.png └── utils.py /.gitignore: -------------------------------------------------------------------------------- 1 | .ipynb_checkpoints/ 2 | .DS_Store 3 | MNIST_data/ 4 | log/ 5 | __pycache__ 6 | inception/ 7 | -------------------------------------------------------------------------------- /1_Setup.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "\n", 8 | "# 1. Setup\n", 9 | "Note: If you're already comfortable with `anaconda` and `jupyter notebook`, feel free to skip these steps and start the exercises. We'll be using `python 3.6.4` and `tensorflow 1.3.0`.\n", 10 | "\n", 11 | "## Installing Anaconda\n", 12 | "\n", 13 | "We will be using `anaconda` for this workshop. We highly recommend you do so for your own projects. `anaconda` is a \"tool box\" for ML; it comes with various libraries you need to do ML in python. Another neat thing about `anaconda` is that you can create a virtual environment, which is a separate python environment from the default one on your laptop. Virtual environments allow us to separate library dependencies across projects and make it easier for us to share our environment setups with other developers. In fact, we are going to do that right now!\n", 14 | "\n", 15 | "You can install Anaconda [here](https://www.anaconda.com/download). **Please download the `Python3.6` version.** Once you finished installing it run on your terminal:\n", 16 | "\n", 17 | "```\n", 18 | "conda -V\n", 19 | "```\n", 20 | "\n", 21 | "if you see the version of your conda (e.g. `5.0.1`), success! If not, google around and try hard to make it work...\n", 22 | "\n", 23 | "Even if you have `conda` installed already, make sure it is up-to-date. Run\n", 24 | "\n", 25 | "```\n", 26 | "conda update conda\n", 27 | "```\n", 28 | "\n", 29 | "## Setting up Jupyter\n", 30 | "\n", 31 | "First, [fork](https://help.github.com/articles/fork-a-repo/) this repository to your account.\n", 32 | "Second, open up your terminal and [`git clone`](https://help.github.com/articles/cloning-a-repository/) the forked repo to your laptop. If either of these don't work, click the link and read the github help page. Once you cloned, run the following:\n", 33 | "```\n", 34 | "conda env create environment.yml # create conda virtual environment\n", 35 | "source activate tensorflow_workshop # activate the virtual environment\n", 36 | "jupyter notebook # open jupyter on your default browser\n", 37 | "```\n", 38 | "\n", 39 | "The first line might take a while, but be patient.\n", 40 | "The last line should open the notebook on your default browser. Once you open `jupyter notebook`, make sure the kernel is set to `Python [conda env:tensorflow_workshop]`.\n", 41 | "\n", 42 | "![alt text](images/1_setup.png)\n", 43 | "\n", 44 | "Try opening some code in `3_Graphs_Sessions.ipynb`. If it successfully runs, great! Once you're done with your work, you can deactivate to get out of the virtual environment. Run on your terminal:\n", 45 | "```\n", 46 | "source deactivate # get out of the virtual environment\n", 47 | "```\n", 48 | "\n", 49 | "## You're all set!\n", 50 | "\n", 51 | "Now, you are ready to start writing some `tensorflow` code on `jupyter notebook` :)\n", 52 | "\n", 53 | "If you've never used `jupyter notebook` before, you might want to watch the following video:" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": 1, 59 | "metadata": {}, 60 | "outputs": [ 61 | { 62 | "data": { 63 | "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAUDBA0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDRANDQ0ODQ0NDRUNDhERExMTDQ0WGBYSGBASExIBBQUFCAcIDwkJDxUSEhIWFhIVFRUSFRUVFRIVFRUSFRUVFRIVFRUVFRIVEhUVEhUVFRUVFRUVFRUVFRUVFRUSFf/AABEIAWgB4AMBIgACEQEDEQH/xAAcAAACAgMBAQAAAAAAAAAAAAAAAQIHBQYIBAP/xABMEAABAwIBBQkMBwcDBQEBAAABAAIDBBESBQYHITETF0FRU2Fxc5IiNHKBkZOhsbKz09QWJDIzUoPhFCNCYoKiwcLR0kNUY3Tw8aP/xAAaAQEAAwEBAQAAAAAAAAAAAAAAAQIDBAUG/8QANhEAAgECBQIEBAUDBAMAAAAAAAECAxEEEhMhUTFBFDJxoSIzYYEFscHh8CNC0QZDUpEkNPH/2gAMAwEAAhEDEQA/AOMkIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEK294Gv5aj85N8ulvA1/LUnnJvl1roT4M9WPJUqFbW8FX8tR+cm+XRvBV/LUnnJvl00J8DVjyVKhW1vA1/LUfnJvl094Gv5aj85N8umhPgaseSpEK294Gv5aj85N8ujeBr+Wo/OTfLpoT4GrHkqRCtveBr+Wo/OTfLo3ga/lqPzk3y6aE+Bqx5KkQra3ga/lqTzk3y6e8DX8tR+cm+XTQnwNWPJUiFbe8DX8tR+cm+XRvA1/LUfnJvl00J8DVjyVIhW3vA1/LUfnJvl0t4Gv5aj85N8umhPgaseSpUK2t4Kv5ak85N8ujeBr+Wo/OTfLpoT4GrHkqVCtreCr+WpPOTfLo3ga/lqTzk3y6aE+Bqx5KlQra3gq/lqTzk3y6DoBr+VpPOTfLpoT4GrHkqVCtneDr+VpPOTfLp7wVfytJ5yb5dNCfA1Y8lSoVtbwVfy1J5yb5dLeCr+WpPOTfLpoT4GrHkqZCtreCr+WpPOTfLo3gq/lqTzk3y6aE+Bqx5KlQrZ3gq/lqTzk3y6e8FX8tR+cm+XTQnwNWPJUqFbW8FX8tR+cm+XQNAVfy1J5yb5dNCfA1Y8lSoVt7wNfy1H5yb5dI6Aq/lqPzk3y6aE+Bqx5KlQrZ3g6/laTzk3y6N4Ov5ak85N8umhPgaseSpkK2d4Ov5ak85N8uobw9dytJ5yb4CaFTgasOSqUK1t4eu5Wk7c3wEbw9dytJ5yb4Cnw9TgjWhyVShWvvD13K0nnJvgI3hq7laTzk3wE8PU4GtDkqhCtbeHruVpPOTfAQdA9dytJ5yb4CeHqcDWhyVShWtvEV3K0nnJvgI3iK7laTzk3wE8PU4J1oclUoVrbxFdytJ5yb4CN4eu5Wk85N8BPD1OBrQ5KpQrW3iK7laTzk3wE26Bq7lqTzk3y6aFTga0OSqEK2d4Kv5Wk85N8ujeDr+WpPOTfLqNCfA1Y8lTIVtbwVfy1H5yb5dG8FX8tR+cm+XTQnwNWPJUqFbW8FX8tR+cm+XRvA1/LUfnJvl00J8DVjyVKhW1vBV/LUnnJvl0bwVfy1J5yb5dNCfA1Y8lSoVtbwNfy1H5yb5dB0BV/LUnnJvl00J8DVjydLpJpL1jgBCE0AWQmkgBNJMoARdJCAYQhCASaEFABSTXhynMRZo1X/APxWjHM7FZyyq56pHgbSB0lQ/aW/ib5Vja2jLRe99dj0/wCV4ito0otXTMJV5J2aNkY8HYQeg3UlrDXEbNSzWSqkuBvtFtfHfZ49SpOnl3L06uZ2PYgpoWRsQLUFSK2XImZE88TZWOiDX4rBznB3cuLDcCMja08OxVlJRV2Sot9DWLJFbpvbVX4oO2/4SxuV8yaqIFxYHtGsmN2Kw4y2wdbntqVVVg+5Z05Lsa6Eyshm7kd9TJucbmB2EuGMkA2tcAhrjexvs2Ar35yZpzUrGvkMZa52HuHONjYnXdjbXAKs5xTtfcrldrmvgJoWwZuZozVLDJGYw0OLe7c4EkAE2sx2rWFMpKKuwk30MAohbfFo+qHOe0Pguywd3b9RcMQH3fEQfGF9BozqfxQdt/wlnrQ5LacuDT0itz3t6r8UHbf8JY/L+Zk9PGZXuiLQQLMc4uu42GosA9KlVYN2TDpyXY1oIctpyNmNUTRslY6INeCRic8O1EjWBGRtHGsBlSkMUj43WLmOLTa5Fxq1EgG3iCsppuyZVxa3Z45DqXxWQybQOnkZEzCHPJAxEhuwnWQCeDiWYy9mNUU8TppHQlrcN8Dnl3dODRYGMDaRwq2eKdm92UcW90avZfVkSjFtWx5BzTqKgBzGAMOx7zhaejUXOHOAQpnJR6iEbmA3McS+T22W/u0aVFvvIb8V3+vAtYzkyBNT/esIBPcuHdMdzBw1X5jY8ypCtGTsmXnTaV2jCFBQ5bVU5hVLYTNeItEe6Wa55fhw4tQ3MC4Gu1+BaSmo9TNRb6GqoKS2HNjNGara50ZjAa7CcbnA3sDqwsdqsUlJRV2QotuyNeCayWcuQ5KWQRSFhcWB/cEkWJcBrLWm92ng4ljSiaaug1bZgiySZUkEmvsvs1115ygOUNEpnpa7jTeoxuupOCqaDCEXQgCyEICALIITSQEioplCASaEFACEIQDukhCAEITKAQQhCAaEl5cpVOEWG0+gKYxbdkVlJRV2eyyx+U6ZziLDg4xxrGPkPGfKoF5410RpOLvc5ZVlJWse2amldqIvbnH+68lRA5tsQtfZsXzLzxlIuV1dFG0+f+xFZTN/+P8Ap/1LFFZ7Mf7w+L1PWGLqadKU+tk3/wBbmuHjmqJcnsISC253OtfyvS4DcbD6DwheDgPxiOJnpuOV9t739kerXwjpxzJ3PEro0Z95Q/m+/kVLhXRoz7yg/N99Iu/F+Rev6Mzw/m+xhMv5/vhnki3Frgx2G+MgnUDxEDatjzRzkZVNcWgscwjGwm9r3sQRtabHXYHUdXHVOfnfdR1n+Atg0MA7rNxbmL9OMW9GJZ1KMdPMutkWhUlnsPO6nbR10M7RhY843AbBrwzADiLXYulxW55/UO60szRrLW7o3pjOLV0tBHjWsaarWp+O8vktGtozGr92pYnHWQ3c39LO5N+kAHxqkr5Iz42LxtmlEo1XpmRQ7jSwtOo4Mbr8Bf3Zv0Xt4lU0ORT+2fs3AJsB8AOuXeb7pWppAr9ypZSNRcNzb0v1G3Q3EfEtcS82WK7mdFWu32NDoc/Hxvnc2NrxNKZAXEghtg1o1cTWhbZmNnc+qkex0bWBrMV2km/dAW19KqBb1oY+/l6r/W1WrUoqLdiKVSTkkbdn5nK6kERaxr90LwcRIthDdlulaHnNnu+piMTomNBLTcOJPcm/Cs9pr+zT+FJ6mKtFGHpRcVK24rTkpNF4aPO84PBd7b1UuenfVR1z/Wra0d95weC723qpc9D9aqOuf61Wh8yX87lqvkRLR+frtP4Z9hys/Sr3jN0xe+jVXaOz9dp/DPsOVoaVu8ZumL30atX+dH7fmVpfLl9/yKqzHyV+0VMcZ+zcuf4DRcj+rU2/8yufOfLLKSHGW3sQxjBZtzbU3ia0AE7NgVcaFGfWZDxQO9Mkf+yzWnB53OnHG958jW29ZSus9ZRfQUnlpuR4odJsuLuoYi3iaXB1vCJIv/St+eyKsp9euOZl9e0X2Hmex3kIXPhkKuzRQ69FFzOlA868/wCVXE0lBKUdtyaFRzdmUtlGmMb3xu+0xzmO6Wkg+kLofI4vDEDrBijuOPuAqR0jMtWVA/nB8rGuPpKu/I33MXVR+wFOLd4xYwytKSKFzryX+z1EsXA112c7Hd0zp7kgHnBVi6EfuZutHsNXm015Ku2Kpbwfu3kfhN3MPQDiF/5mr06EPuZuuHsNVqk89C5WEMtWxrmmnvtn/rs95KtKC3XTUfrbP/XZ7yVaRiXRQ+WjCt52SCEg5F1qZghF0roCbHWXpXjDl6Y3alDLRJ3QCo3TuqlySRScUroCSRKg5yEB9roKSRKAkhASQDRdJAQAgFCCgGkUroLkA1JQBTugJXWEyo7uzzWHoWZusJlX7Z8XqC2o+YwxHlPMSo3QtmzTySHjEeknh17AOLjJU4nEwoU3Un0RjRoupLKjWtzPEfIUjGeI+Qqz25PjH8I8dz60jQx/gHkXgP8A1NR/4S9v8npr8Jl/yRWLYHHY0nxFbZmdk8t7o+PptYAcdrkrYf2KP8I8i+oFtQ1DmXDjv9QKtSdOnFq+zb4OnDfhunNSk72GSsdl0dwOZw9RXvJWOy47uP6h6ivF/Df/AGYW5PQxK/pS9DDXV06M+8ofzffSKlLq6tGXeUH5vv5F9ti/IvX/ACePh/N9jzZXzChmlfK6SUF5xENLABqA1XYTwLNZDyNDSsIjGEbXvcbk24XOOqwF+IDXxlVtnhnPVR1MzGTOa1r7NaA2wFhq1tutcyjlqeUWkle8fhLjh7I7n0LNUJySu9i7qwi9luZjSPl1tRMMBvHEC1p/ESbucOYkADjDQeFZ/QzlDXNCeG0rfQx/+jyKuFmcyMo7jUwvOoYsDvBf3Jv0XDvEuidJaeVGMZ/HdlpMyF9fNRbudwGv/wApO5+7HpWsaZsoXdFCD9kGR3Se5b4wA7tKy7Ln/PLKW7VM0l7tLy1vFgZ3DSOkC/jXNhk5zu+yNq7UY25MfiW96GHfv5ep/wBbVXi37Qj9/N1P+tq68Qv6bOejL40ZbTX9mn8KT1MVaKx9OR7mn8KX1MVYCQquGX9NE138bL40d95weC723LGZV0ewyySSOklBkcXEDBYFxvquwmyyWjk/UqfwXe29V5nVnbUsqZ2Nnc1rZXtaLNsADqH2brkhGbqSyu3/ANOmUoqCzG5ZD0ewwSsmbJKXMNwHYLHURrswHhXo0rd4zdMXvo1pmZWdNTJVQsfM5zXOIc0husYXHgbfaFu2lDvKbpi96xTJTVSOZ36fmQsrg8qt1/I0nQl3xL1B95Gsrpz+xT+HJ6mLH6GB9Yl6k+8jWT03Nuyn8OT1NWsn/wCQv52M0v6L/ncqkq79EveUfhS+8cqUdGrr0Tj6lH4UvvHK+Mfwff8AyUwy+P7FY6S+/ajwm+7YrvyH91D1cfsNVIaTO/ajwm+7YrvyH91D1cfsNWOJ+XD0/RGtDzyNUzRkFbk90Lz3TQ6BxPAW64n+IYD0tK8uhSItiqGuFnNnwuHEQxoI8RWuaH8q7nUuiJ7mcEDrGXc3ytxjpIVpZMyaI5J3DZM9sluJ2ANd5S3F0uKrW+DNHndE0vitLjYqvTUPrbf/AF2e8lWkWW8aau+2f+uz3kq0dd1D5aOSt52CEIK1MxIshCAAvTHsXmXqUMvEEITVSwlFwUl85CgAqTVBgUyEB9SUgxDSpoCIKkkUkBJCEIBIQi6AEIuhAFkJpXQAQsbleEmzhwCx6OArJL5SPAI9atGVncrKOZWNeJW/ZnH915PZCwRp2H+FvkC2HNoWYR/N/gLy/wAenmwr9V+Zt+H08tUypKSS81XTB1tbhb8JsviIKLdpOy5tc953S2R6CVErwOomg23R9zsGPWV4cqx4MNnO132u4rf7ruoYKnWmqcZ7vpeL7b8/QynWlCLk49PqZpxWDyrVYjYbB6SvI6QnaSeklQX0H4f+DRw09SUsz7bWsefiMa6kcqVkNXVoy7yg/N9/IqUV16Me8oPzffyL0cX5F6/5McP5vsVhn733UdZ/gLBhZvP3vuo6z/AWCut6flXoZT8zJL4yvUpHL4OWqRlJnScx/dnwD7K5sadQXSU/3Z8A+yubGnUuLBf3fb9TpxfYkt/0IffzdT/raq/W/wChA/v5up/1tXRiPlsxo+dGT05/ZpvCl9TFVqtHTp9mm8KX1MVXXUYX5a/ncnEedl86Nu8qfwXe8eqez477qeuk9oq4dG3eVP4LvePVO58991PXP9orDDfNl9/zNa/y4/zsejRz37T+GfYcrS0rH6jN0xe+jVWaOe/afwz7DlaelfvGbpi99Glf50ft+Yo/Kl9/yNJ0LVH1p4P8UDrdIfGfVfyLOab2/uoHcAkcD0uaCPZKrnNXKpp6iObWQ090BtLHDC4dNiSOcBXnlShhrYMJOKOQBzXsOsEbHNPARsII4wRtSv8ABVU30Jo/FTce5QAlCu7RawiiivwmQjoMj7eW11rsOidmLuqhxZf7IjDXW4sZe4ePCtxyzlGKip76mtjaGRMvrcQLNYOE7NZ4BclUxFWNRKMNy1GDg25bFQaQJQ6sqCPx4ey0NPpCu7In3UPVx+w1c4VFQ5znOcbucS5x4y43J8pXR2Q/uoerj9hqnFRyxihh5XbZznBUmOQPZqcx4c087TcekLo3Jda2WNkrfsyNa8c1xe3SNh6Fza86z0lW3oVyrjhfATridib1b9dvE/F2gtMZC8c3Bnhp2lbk1vTWfrbf/XZ7yVaOCt40199s/wDXZ7yVaNZb0PloxredghBQtTMLoKEihJOLaF6CV8KfavuqstEEigpEqCwwo2CaAgG0JpIQEiU7pFCAniSKSd0A0XSQgGhIIQDQkEIB3TKihASSISundARIWXyBVAXaeE3HqssSormxeGjiKTpy2uaUqjpyzI3ElRJWsQ1jxscfX67r6nKUn4vQP9l8zL/T1e/wyj7r9GelH8Qh3TPfW0Bc/EHWGrpFuJebLcwJAHBe/jtq9C8ktY87XH1epfEL2MJ+H1YShKtJPIrRSXKtu++30OWtiINNQVs27uO6V0JL1zjGs3kzOuqhY2OOXCxt8LdzidbE4uOtzC43cSdZWDukocU+pKbXQ9GUKt8j3SPOJ7jdxsBc9DQAPEF8LouoTFSkQ2fNzlEoJSVzI2V2fVda276rW+6g2WtyXEtZCElEYqPRWJcm+rJL35Dy1NTuLoX4HOGEnCx1xe9rPa4bQNYWOQpaT2YTt0Mtl3OGoqQ0TSYwy5b3EbbXtf7DG3vYbbrFFCSJJKyDbfUz+Tc8auJjY45sLGCzW7nC6wJJ2ujLjrJ2lYeuqnSPdI84nvJc42AuTrJs0ADxAL4FIBQopO6Qcm9mz1ZOrXxPbJGcL2G7XWBsbEbHAtOonaCsplXO6rnYYpZsbHWu3c4m3wkOGtkYcLEA6isCUI4pu7QUmla5K6yWRsuz099xlcwHWQLFpPGWOBaTz2WMSsjSezCbXQ21+kSttbdWjnEUd/S0j0LXsp5SlmdjlkdI7jcb25gNjRzAALxpqI04x6Ilzk+rArZIM+61oDRPYNAAG5QGwAsNsVzqHCtaKQUyipdVchSa6MkV7ci5Wlp37pC/A4tLSbNd3JIJFnhw2gHZwLxIUtJ7MJ2PdlrK8tQ8STPxvDQ0HCxvcgkgWY1o2uOu19a8N0JIklsh1BF0JIBouokpoD704X1KhFsC+8UdxrWU5qO7NIq58lGy+5g51EwlVVWL7lnFnyQpOaorQqMIQEBANSBUUgEBMppFqQKAad1EuQgPbk6gfKSGW1C5ubL2/Rybib2lhgUw5VafYlWMx9HJuJvaR9HJuJvaWIujEotLn2JuuDL/AEcm4m9r9EfRybib2liLpFyWlz7C64Mwc25uJva/RH0cm4m9r9Fh8SMSWlz7C64Mwc3JuJvaR9HJuJvaWIui6Wlz7C64MsM25uJvl/RH0cm4m9pYm6LpaXPsLrgy30cm4m9pP6OTcTe0sPdF0tLn2F1wZj6NzcTe0l9HJuJva/RYcuSJS0ufYXXBmfo3NxN7X6I+jk3E3tfosOCkXJaXPsLrgzBzcm4m9pfKTNubib2v0WLLivPiVkpc+xWTXHuZn6Nz8Te1+iPozNxN7Swoci6taXPt+5S8ePczX0Zm4m9r9EvozPxN7Sw4ckSlpc+37i8ePczP0Zn4m9r9EDNmfib2v0WGuldLS59v3F48e5mjmzNxN7SPozNxN7X6LC4k7paXPt+4vHj3Mz9GZ+Jva/RH0Zn4m9r9FhgUXS0ufb9xePHuZg5sz8Te1+iPozPxN7X6LClyCSlpc+37i8ePczJzan4m9r9FL6NT8Te0sJiSLulLS59v3F48e5m/ozPxN7SPo1PxN7X6LCgoS0ufb9xePHuZv6MzcTe0kc2J+Jva/RYUFDnFRaXPsLx49zNfRmfib2v0R9GJ+Jva/RYMuRiS0ufb9ybx49zOfRmfib2v0R9GZ+Jva/RYNpUrpaXPsLx49zNfRmfib2v0Qc2J+Jva/RYW6V+dLS59hePHuZr6MT8Te1+iBmxPxN7Swt04tqWlz7E7ce5sQzbm4m9pQrcnviwh9rkX1G6xbF6yVyYhvZM2hYE0iUXXKaErL5SQ8S+gKTirRm4vYhpM8hCCvvMy6811306mdXMJRsSKGNUHFTDloQSJULp3SCALJkpXTKAYQkE0AFMFRTQDugJFCACUXSsghAO6aimgHdbBmBkiOon3OTFh3Nzu5NjcYba9fGtfW4aIO+/ypPW1Z1XaDaL01eSNjzjzDpYoJpG7piZG9zbvuLtaSLi2sKrFfeenelT1MnslUFdY4WTkndmleKTVh3UUyouC6jAaiUJXQDcdS+IU5TqXzAVkUkMlBKi4r5OcVJU+10L4MkC+t0BJIlCSAd1mc0MgOq5TE14YQwvu4EiwLRaw8JYWy3jQn327qH+3Gs6snGDaL00nJJk8uaOJIYpJTMxwjaXEBrgTbgFytFJXQWkDvOp6p3+Fz2ssNUlOLcjSvBRaSEFK6iUiukwJ3CRIUQU7oCSLpXRdABKRcmVBABKMaRSQDD1PEoNKkhI7ouldAKAAV9qcbV8gV94hYKGSj0U+1ei6+NMNvkX1K8+u7yOmC2JXSJSCV1gXJXTco3TugAFeadtivQV8qgagtqErS9Ss1sfNwQhC7zAaRCV0XQDui6SEBJNRCLoBoSui6AaQKEkBJCSEAyhJF0A7rcdD/ff5UnratNW46Hu+/wAqT1tWVbyMvT8yLNz070qepk9kqgir9z070qepk9krn951HoWOD8rNcR1RuY0cVf8A4vOH/itYyvQuhkdE+2Jhs6xuL2vqPjXRMWwdA9SojSF35UdZ/palCtKcrMirTUVdEs3c0J6lhkiwYQ4s7p2E3AB2YTqs4L45zZsTUoYZcFnkgYHYtbbE31DjVjaGO9X9e/2I14dN/wBin8OT1NRVpauXsHTWTMaBm9m/LVOcyLDdrcRxuwi1wNWo67le3LuY1TTxOmk3PAy18LyTrcGiwwjhIWe0JH9/N1I9tqsjOOphjhc+cXiZhc4YcVyHtLO54Tjw7dXHqU1MRKM8qEaMZRuymciZg1c7Q8NbG12tplcW4hxhrWufY8ZAuvPnJmTVUzS97GuYNr4zia3wgQHAc5FudbszSxHjAMDhHfW/GC8DjwBtv6Q5WMC1zeBzXDpDmuHpBBVZ4irB/EthGjTkvhZzDRUxc9rQQC9zWi+oXcQASeAAnWVuo0YVn/h84f8AgsBnfkr9nqZYhqDX3ZzMdZ7NfM0gX4wugMi1m6xRSj/qRsf43NBI8R1LWvWlFJx7lKNJSbUuxzdURFjnNO1ri0jnabH0hZjNfNearDzFg/d4Q4vdh+1itawN/sm/SFPSJSbnWVA45MY/MAf63FWRoUpMNK5/KSuI6GhrB6Q5Xq1XGnmXexSnTvPKyus5Mz56WMSS7nhLgwYXkkuIJ2YRwNJW7aMM0Z6ebdpNzwOhIGFxJu4scLjCOAHhXl061lzBCOJ8hHT3DPU9bHmRniyocIGxvaWRXLiWkHBhadmvXe656lScqV+b39DaEIKpbgzmdVC6anmiZbE9haLmwueM8CpvL+ZM9PGZZBHgBAOF1zdxsNWEK68tV4hiklILhG0uIG024BdVpnlnoyqgMQiewlzXXcWkdyb7AsaFSUfS5rWjF9epXjgFsGQsyKmoaHsYGsOx8hwAjjAsXEc4Fl7NHuQGz1DQ/umMBkeOB1rBrTzFxFxwgEK5MqV7IY3SSHCxg1+oADjJsAFvVxVtomVOjfeRT9Touq2i4ML/AOVrzf8AvYwelafXUD43Fj2lr2mxa4WI8Xpvwq+82c7YalxYzG14GLC8AXaDYkYXEarjVt19KxGlzIglgMwH7yGxvwujJs5p8G+McVncarSxUs1pEzoRteJTuS6J0j2Rttikc1jb6hdxsLngC2waMKz/AMPnD/wWJzLB/a6bV/14/aC6CBV6+IlBrKVpUYyW5zhkDN6epcWwsxYftOJsxvhOOrxC5PEvvnbmtLR7mJXRndA4t3NznWwYb4sTG2+0Nl+FXVDVUlCxkJkjjsL2J7txO17gNd3HhIt5FoGmqujm/ZXRSMkbaYEscHWN4tRtsPMVNPESnNK2xEqKjH6lbFIr6OjKgWldhz2E1TuoNTKEEsSSGhfVkXGlybCiavQFFfWnbwrKc8quy8Y9j0MFgEwUroBXmt3dzpQ8SChK6gEkEqOJO6AYUKjYUyozbCrw8yIfQ+F07qCd16RzjKAUrqN0BO6RKV0ICQKYKhdAQE0KN0ICSFG6LoCSFG6CUBJCjdAQEluWh7vv8qT1tWl3W56Hu+/ypPW1ZVvIy9PzIs7PXvSp6mT2Sufn7Cugc9e9KnqZPZK59k2FY4Pys1xHVHS8WwdA9SofSEfrlR1n+lqviLYOgepUPpC78qOs/wBLVlhPM/QviPKiwtC/er+vf7Ea8OnI9xT+HJ6mL3aFe9X9e/2Il4NOX2Kfw5PZai+f9w/lGN0I/fzdSPbatx0rd4z/AJfvWLTdB/383VD22rctKveM/wCX71imp89eqIh8p/coW66E0ezF1FTE8kG9glnqaue1f2jA/UafwX+8etsb5F6mWF8z9CudNkdqsH8UDD4w6RvqAW8aHa7HRtbfXE98fivjb6H28S03Tn3zF1A95IvXoIrrPnhP8TGyNHgHC722+RVnHNQX0Ji7VmeLThS4apj+CSFvaY5wP9pYrJzApdzo6duy8TXnpk/eH2lq2nHJ5eymcNu6mLxygEelnpVgEtjZxNjZ5Gsb/sFjUnelFfzY2hG1STKQ0n1u6Vsx4GYYm/0NGL+8vWR0Ln627qH+3GtNq6gve952vc556XEuPpK3bQwy1S7nhf7Ua6ato07fSxhDedyx8+O9Kjqz/hUYVeefHelR1Z/wqMXDDob1epY+haLVUO542+2T6wsnpeJ/ZmgXN5m3tr1Brz67HxLxaF/u5+sb7K2jOrL7aVjZHNc4OeGWba9yHOvrI1dyqvzF4+Uq7RyHNrIdRAJeDqOwxv8A82VuZdhxwzN/FFIPKwrTzpPh5KXys/5L51GkyEtI3GXWCNrOEW/EjTbEWkrXNGzNP1qn66P2gr6aqCzKP1qm66P2gr9akyKXQ50yq8ySPkcSXPcXE85PqGwDgAC8Zp16X7T0lRxLojVku5g4pnldGRweRQcvakVqsS+6K6Z4Cn4l6jGFHcBzrRYiJXTZ8LouvuIAptYBwKHiI9gqbPkyK+1fdK6V1zTqOb3NVFImEXUSUErMkaMSjdF0BIFMqCkgHdRm2FO6jUHUrwV5IiXQ85KAVC3OkvSOc+l0iV83HZ/9wKQcgJApgqCd0BJCjdNASSukkgJ3SuooQEkiVEoQErpXSuolAfW63TQ0Prf5MnrZ/usJmlmtLWbpuToxueDFujnD7eO1sLHX+wb3twK09H2Zn7Jie94fK8Ye5BDWtuCQL6ySQCSQNg1cfPXqxUWu5tSg20zL57n6pU9TJ7JXPUh1FXlpXygI6ORt+6lLY2jjuQXeRgPjI41Rj9h6FTCL4W/qTiHudORbB0D1KhdIZ+u1HWf6Wq+ad12tI2EAjyKr888w6mWqkkjwFkhDsTnYcJsAQ4WvtH8IOpYYaSjJ3Na8W47Ga0K96v69/sRLwadPsU/hyey1bbmXkEUkO5B+M4i97tgxuDdQG0CwG3Xw8NlqWnT7FP4cnssUwkpVroSVqdmYzQd9/N1I9tq3LSt3jP8Al+9YtN0HffzdUPbarKzlyS2phfC5xaHgd021wWuDht1HWBq4uJKztWu/oRTV6dvU5tuugtG0WGiph/IXeJz3OHoK0XejkxAftDMF9bsDg+3M25bf+pWrSQNjY1jdTWNDW8zWiw18wG1aYqtGaSiUw9KUW2ym9N816to/DAwHpL5HeohYnRjX7lWwG+p7jEfzAWj+/CfEvFnrlUVFVNKDdrn2ZzsYAxh8bWh3jWJhmLSHN1OaQ5vS03HpC64Q/p5Xwc8pfHm+p0ll/JombGPwTwyj8t4J/tusbpLrdzo5jwvaIx+YQ139hcfEs9R1AkYx7fsva146HAEegqu9OddZkEX4nOkP9ADW+XG7yLzKScppHfUdotlWxjWt/wBD/fTupf7Ua0GlOtbromqQ2sAP8cb2Dp1Pt5GFdGKd3Y56PUsvPrvSo6t3+FRV10LleiEsUkRNhIxzL8WIWv4jrVR5y5kSU0TpXSMcA5oAaHXOI2vr1D0rlgzepFvc2HQpLqqG8Rjd5Q8f4Xv0yN+rMPFM32JFqOijKYjqQ0mzZmmPmx3xM8pBaOdytLOfJDamF8TjhxWLXWvhc03abar69RFxcEqHtImO8bFJ5s5OE88cRJaHkgkAEizS7Yehb3VaNI2tc7d39y0n7LeAXXrzKzFNPLu0kjXlocGBoNgXCxcSf5SRYDh26lms/wDKQhpZTfuntMbBwlzxbV0Nu7xI5b7ERhZblQ5md9U3XR+0FfrVQWZx+tU3XR+0FfrUmTT6HN8h29JUAUSHWek+tQurmJ9UiVDEi6AZSugpEoAupBQCYQDuhRJQCgJXSJSKiG86AldF0kBASBUlBMlATXwqjsX0BXmqHa1vh1eRSb2EUk1EldxiIG9lIL5MGpfRpQDJspXUCi6AkndRui6AkCndRBQgJXSQi6ACohMlRBQEihJNAbVo9zsFHu14zJuu57HBuHBunG03vj9C2ap0sau4ptfG6W4HiDBfyhVhdF1lKhCTu0XjVklZGVzky9LVPxyuvYWa0CzGDhDRr28JJJOrXqCxRRdF1okkrIo3csHNHSQYYmxTRmQMGFj2kB2EbGuB1Gw1B1xqA1HafrlzSm9zS2CIRk/xvIe4dDbYQeclw5lXN0lloQvexpqyta5vWZuf37NG9skb5nvldIXmSxJc1g13aST3O268mf2eDaxsQERj3MuNy8OviAHA0WtZahdBKsqMVLNbcjUlaxYeg0/v5upHttW8aTZS2jmc0lrmmItcDYgiaMggjYQVo2gv7+bqR7bVu2lTvGf8v3rFx1fnL7HRD5b+5puQ9KrmtDaiLdCP+pGQ0u8JhGG/OCBzLHZ5aSZKhhiiZuLHCz3F15HNO1uoAMB2G1ydlwL30myg5i61Qpp3sczqzta580i5Scwq/dH+R4BS00m4xCQxMcZNzZjLrbS617891atWVNXIp0nN2PfmNA9lJTteCHCJtwdo1XAI4CBYW5lVOl+ux1jm8ETGRjptjd6X28SsnOzPanpmu7tsktu5iY4ON+DGRcMbx3122Aqia2qc97pHm7nuc9x43ONz4rnYuXDQbk5tHRWkrKKPpSN2leqGZzHNc0lrmkOaRtBGsEdC89PsCniWdV3kysehZOS9KFmgTQkuG10bgAefA7Z4j5Fj89c+mVMJibE9t3NOJzm/wm+wcfStGuo3WWVF87JtP/wW+ZB0lSMaGzs3W2rGHYXnwhYtcefUeO+1aDdIlS1chNroWlU6U47dxA8n+Z7Wj0BxWi5y5wy1Tw6Qiw+wxuprAdthtJPC43PiACw90kypFnJs9+Ra3cpopSMW5va+17XwkG19dlYbdKjP+3d5wf8ABVekjVwpNdCbnetRJUHICFRpgpBO6AajdCRQDCAohPEgGUJJgoBgpXRdIlAK6YSTsgC6d1EBSAQkCvISvRKbBeVdeGXVmNRkikUFIhdRmNqd1EJhANxTBUCU7oADtdubb/hNRCaAkEXSCd0AwFN0LhrLXDnIIXszZ74p+vh941dFZUpGyxvif9mRrmHocLXHONo5wsK1bTaVjWnSzo5mKQvsGtfbKFI6N7436nMc5julpINuZZbR+frlN1rf8rVysrmaW9jDiB/4Hdk/7L5rqNjtYXLUZ1DoCyo1tS+3Q0qU8liaLJFyMS3MhpJlJACYXsyDGHTwtcLtdLG1w4w57QR4wr1GZVF/28f93/JY1ayp9TSFNz6HP1kl6cqMAkkAFgJHgDiAcQB5F5lsjMyubecEtK5z4sILm4TibiFrg+sLIZbz6qp43RSGPA+17MsdTg4WN+MBa0kquEW72LZnawwUiUKJKsVPsIHfhd5D/slK52w4ug39RXSGbTvq9P1EPu2qltLvf83RF7li56VfPK1jadLKr3NTsOJBCLqUe0LduyuYpHsa1NRDk15h0DKSLoJQCKSLouhIAoSTUAd02NJ1C/rKjZbRor79i6JPdPQlGtuhd+F3ZP8AsvkCujcvu/cT9TL7DlzkAoTuWlGxIFASui6koNJCAEAKJ2qYCRKAV0ITQBZBCaRQCTQkUA0XSCHusFKVwfGpfwL43RdNejCOVWOeTuwui6LJFXIIudbgU8ST07IBXTBUgkgAJKSEArp2QmgMhm13xT9fD7xq6RXNubXfFP18PvGrpJcGM6o6sP0ZTmmzJGCds4HczNs7rGWH9zMPZctczA78putarj0hZKFTSSNbrc0brHbXdzNdh4TcTf6lTej/AL8putataU81JrgpUjaa+p0QzaFyzGdQ6AupmbQuWoxqHQqYPv8Ab9S2J7Ek1smYuaL6xx14ImEY5LX1nXgYNV3W18QBBO0A2ZS6NaJosWPeeN0jgfIwtHoW868YOzMo0pS3RRwTVyZY0XU7mncXPifwXO6M8Yd3XjDtXEVUuWMnvgkdFIML2Gx4jwgg8IIsQeIqadaM+hE6bj1Ptmz3xT9fD7xq6SCq/RdmnTywx1Dw7dGykgh5AvG4Fvc7OBWeuLFTUpWXY6aEWl6nNGWj++l62T2yvLdXblTR7SESSYZMRD3/AHhtiN3bOngVIgrtpVVNbdjmqQceowkQrGzJ0cbqxstQ5zGuALI22Di06w57iDhB24QL24RsW3HRxQ2tubundZL+1b0KksTCLsWjRk1cooIK33P7R+adpmhc58Q+211scY/FcABzeAmwI1bRcjQnBbQmpq6KSi4uzOk82u96fqIfdtVL6XO/puiL3TFdGbfe9P1EPu2rRsu5kPq6+aR5McA3IYgO6kIiZdrL6rDhebgHVYm9vPoSUZtv+bnVVi3FJFSBfWl2q73aNqHDbA8H8Qlfi6bElv8Aaq7z0zRdRuFnY4nk4HEWII/gfbVe2sEbRfULELeWIjKLSMtGUXdmu3RdSst6zKzAMzBLM5zGO1sY22Nw/ESQQ1p4NRJGvVqvyt2LqLZoSLK7W6PqK33bjz7rJf0OA9C17OnRsA0vpnOJFyYnWOLmY4AG/E03vxhRmRbTaKzsiykQsrmrkCSqk3NmoAXe83sxuy/OTsDeHXsAJElTEEICuWg0b0jR3YklPCXPLR4gy1h0kpZQ0bUrh3G6RHgIcXjxh9yR0EdKrmRfTZSU7ta2jRLIf26Hol909eHOjN99NKY5BwYmvF7PbwOHFxEcB8ROS0UxWroTfgk909dbqwcLfQxUWpfcubOD7ifqZfYcub2zhdIZwfcT9TL7ty5p3I6tRWWHhGSdzSu2rWPU14PCpAK082tGUG4sNQHmZwxODXlobfYyw2lo2njvwWWj6Q6Omhm3GmDv3YtK4vLu7P8AAOLCNvOSP4VKhGUrRZR3irswqCFt2YeYslS0SyuMcJ+zYAvktqJbfU1v8xvfi4Vv0OjyjAsWPdzmR9/7S0ehZztF2Lxi5K5SaRCtPOPRozCXUznBw17m83a7ma7UWnwrjo2qr5G2JadRBIIOogg2II2gg6rFQncNWItTARZBCFRWQAgJoSRIRZSKi4qQC80r7qUsl+hfJdlGlbdmM532QiFEk8XkP+9lOyF0GYkAKQCEArKQSQUA0IQgAphNJABSCkiyA9+bXfNP18PvGrpILm3Nrvin6+H3jV0kFwYzqjqw/Rmo6NMr7oyaInuoJ5G/lue5zD4jib0NC0U5H/Z8rxsAsx0zZI/AeSQBzNdiZ/SvlmLlfccouBNmTSyRO6XPOA9sNHQSrHzuyRjmo5wNcM7Wu6t5tf8ApfbtuR/05vhoL44r6M2Zm0LlqPYOgLqVm0Lltg1DoVsH3+36kYnsdB6O6ARUcAA1uYJXc7pBj19AIb4gtS0nZ6zwz7hA4MwNaXuwtc4ucMVu7BaGhpbwXuSrAyEP3EPVR+w1UppW7/n/ACvcRLOglOo7+peq3GCsWlo3zgdVQY5AN0Y8scQLB2prg63BcOsRxg7FqWnagF4JgNZxRuPHazmet/oWQ0F/cz9cPYClp0H1eHrx7t6mKy1rIh70tzQ8zcv1EckMLJXNiMzAWANsQ97Q7aCdfSr+C5uzY74p+vh941dIhMWkmrDDvZlC5YzwrBJKwTvw45G2sz7OIi32eLUsXmfk4T1MERF2ueMQ42NBc8eNrSF58sj97N1sntuWxaH4710fMyU/2Ef5XW7Rg2uDnV5SSZd1XOGNc8/ZY1zj0NBJ9AVOZP0l1O7Bzy3cS4YogxoDWE68L7Y8TRruSQSNitvL1IZYZY2kB0kb2Am9gXNLQTYE218AVUHRPU8rT+WT4S4qGnZ5zpq57rKXDNEHAtcAWuBa4HYWkWI6CFzTlak3KSSPk3vZ04HFt/Ha66XjFgBxALn7SKy1bUj/AMl+00OPrV8G92iMQtky882+96fqIvdtWgaR8+qiCd8EOBgYGd2W4nkuY138V2gDFa1jsVgZt970/URe7aqY0tj69N0Re6Yq4eKlUdyasmoKxtmi3PSeeYwTuDyWlzH4WtN22u0hga0gtub2uLcN9WzaSqISUcvGwCRvMWHX/ZiHjVV6Jj9fg6JfcyK5852XpqgccEw//m5RiIqM9hSblDcojItHus0UfA+RjD0OcAfRddDsaBYDUBqAGwAbB4lQ+Ydv2un6weoq+HLGZemU/lHSJVbq50bmiMOOGMsaQWg6sTiMdyNuFw5lbOTKsSRxyDUJGNeBxYgDbxXsucmFX9mV3pTdTH7ISSIg2yotItCIquZo1NcRIB4YDj/cXKwtEFCGUuP+KV7nE8zCWNHQLOP9RWnaYB9b/Kj9blYWjcfUoPBd7x6PoIr4jEaUM6Zabc44SGveC5zi0OIaDYBocC25N7kg7OdfXRhnPJUtkZNYvjwkOADcTXXGsDubgjgA2jiWr6a++IuoHvJF99CP3s/Vt9pLbE3eYzmmWhDqdslu6ikGv+WTuSO0GHxLSNFvfsPRJ7p6sfSp3lN0xe9Yq40W9+xdEnunouhEvMXFl77ibqZfYcqz0R5t7o/9pkHcRn92D/FIP4uhntW/CVadfBjY9l7Y2OZfixNIv4rqOTqNsTGRsFmMaGtHMOPjJ2k8JJVU9i7V2YnPrL4pYS4feP7mIfzW1uI4mDX04RwqhXxYjrJu46yTckk6ySeHhut20xbp+1DEbs3NpiHABsd48YJJ4sPEtLIWtNuPQyqbvc6SpqcMa1jRZrGhrRxBosB5Aqnz2z7qmVEkcLhGyJxZ9hji4t1EuL2nab2DbarLesxc421UQ1gTMaBK3huNWMDha7bzE26cfnzmO2oJliIZNw3+xJYWGL8LrADEPGOEVhLK91f1LzV1serR3nR+1xnEA2aMgSNF7EH7L2g7AbEEcBB4CFXOmGniFTjjexxkbeRrHBxbI3US4A9yXNw6jwhxWtZVhkhe6J4fG61nt1i422NtTmnUQdYPAvAAu2jSs8679jmqVLrKxtcRwqYmKgQkuhwi+qMlJo++783kS3cc6+IQWrPQgW1GfWSbiXxLiU3BRwq8acY9EVcm+oi5NRwC+tTsrkBZFkwhAACEJOCAlZKyYSKACEwhNANJCQKAdkXQgBAZDNrvin6+H3jV0iFzdm33xT9fD7xq6RC4MZ1R1YfozmnLBtNLbV+9k18Rxmy6AzSyr+0U8UvC5ox8z29y/wDuBI5rLn7LP30vWye25WJoOytrlpydv71nSLNkHkwG3M5a4mF4X4KUZWlbktNm0Lltg1eJdSM2hcuR7B0KmD7/AG/Utiex0tkP7mHqo/YaqT0r9/z/AJXuIlc+a82Kmp3DhgiP9jb+Q6lTmlyEiulJ/jbE4c4EbWethHiVMN8x/wA7lq/kRuGgofuZ+uHsBfTTn3vF1493IjQbERTyu4HTWHPhYy/pNvEV89Oko3GBvCZXOHQ1hB9sJ/v/AHH+0VrmyPrFP18PvGro8Lm/Nk/WKfr4feNXSAU4zqiMP0ZzTln76XrZPbK2bQ6frreeOQf23/wtay4LTTA7RLIPI9wWY0ZVGCtgJ2Oc5nbY5o/uIXVPem/Qwh5l6l15z1DmU872nC5kMjmnUbODCQderUeNUmM+a7/uHdiL/grvzgpjJBMwbXxSNHS5hA9K5uYwmwG06gOG51AdN9S5sLGLTujeu2mrGw/Tuu/7h3Yi/wCCwmUKx8r3SSOxPdrc42FzYDgAGwAaguiosjwgAbjFcAD7tnAPBVE58lv7XUYAA0SOaAAABhs02A1DWCtaFSMnsrGdWDit3cvbNvveDqIvdtVK6Xj9em6IvcsV1Zt97wdRF7tqpbS4369N0Re6YsMN8x/zua1/Ij56JD9fg/O9xKruzhP7ifqZfduVP6GKMurA7gjje4nncNzHlxHyFWln9ViOjqHHhiczxyfux6XKMVvUSQobQZUGYPflP1g9RV9OVAZkzBtXTk7N1YO0cP8AldAELCZpT6HM8QGpdAZld6U3Ux+yFQUsZaS06i0lpHEWmxv4wugc0oi2mp2kWIhjuOI4RqUzIp9SsNMHff5UfrerC0b95QeC73j1XGlqa9Y4fhjjaem2L1OCsPRjLeih5sbT4pHf4sVD6Ex8zNK01D6xF1A95IvtoS+9n6tvtKOmyI7tC7gMRaOlr3E+h48q+2hGI453cAYxt+clxt5Ap/tI/vNp0p95TdMXvmKt9FvfsXRJ7t6sPSzLaikH4nxNHbDvU0qvNFvfsXRJ7p6iPQS8yLvWu5Jzrjmq5aZljubL476nPa60jRxht26+Eh3AAVlcvOIgmINiIZSCNRBDHWI51QGZOUtwqoZNjQ8Nf4D+4efECT0ha0aWeLYqVMrRZ+mrJ2KnZMBcwvs7wJLNPkeGeUqpo3XC6MyvQiaOSJ2yRjmHmxC1/EdfiXN5icx7mOFnNJa4cTmmx8hBU00pQfK/IpV2l6nqoql8bg9jix7TcOabEfpxg6irWzLz+ZLaOotHJsD9kbzz/gceLYeC2oLFZP0XksJfOMRaS0Ri7cRHc3e6xLb2vZo1cKruphcwua8FrmkhwO0EaiCs9pE7xL7zrzciq2YZBZwBwSAd2w8x4W8bDqPMbEUHl3JclPK+GQWc07RscD9lzeZw1+jaCrs0V1b5KKIvJJBewE7S1jiG9Nh3N/5Vp2niACSnf/E5kjT0Mc0t9ty3w8nGeRla0U45iuEkk13nKIKTVG6YKACkU0kBCSIHjUgUyogICSHIuliQEgEOCLqNtfMNn/3/ANwoAjZbUFOyCEIBJpICAEXQVEoCSV0JIDI5s98U/Xw+8aukwuXqeUtIc0kOaQ5pG0EG4I5wdazf0wrP+5l7S5q9F1GrG1KoodTH5a++l62T23L7Zt5TNPPFMP4HguHGw6njxsJC8EjySSTckkk8ZOsnxqJXRbazMr73OoIHg2INwbEEbCDrB8YXL0Y1DoWbps6atjWtbUSta0ANAdqAGoAcwCwjXLChRdO9zSrUU7FvaHc42uiFK9wEkZO531Y2El1hxuYSdX4bW2G245ayDBUW3aJry3YTcEDixNINua9lzi51vEs1RZ41rBYVEtuDEQ+3jeHFUqYZuWaLsXhWVrSRf1JTRxMDWNbHGwHULNa0bST6SSeclUlpPzhFTP3BvFECxh/ESbveOYkADjDQeFYXKmcFRMLSzSPH4S7ueyLN8dl4MStRw+R5n1K1KuZWQUspY9rxta5rh0tII9IXSeR8osnjZLGbteLjjB4Wnic06iFzSF7sk5XmgJMUr477cLiAelv2T4wVavR1FsRSqZC8su5qUr90ldC0yFrnF13C7sJ1kBwaTzkKhKWUtLXNNnNIcDxOBuD5Qs7PnlWOFjUPsRYgBg1HwWgrX7JRpygmpO4qTUuh0Tmrl6OqibIwjFYboy+tj+EEbbX2HhCjHmxSiXdhAwS4sWLX9q98QbfAHX14gL31rn+iq3xuxxvcxw/iY4tPRcHZzLNfTWttb9ofboZfy4b+lYvCyT+Fmirq3xIubO7OBlLEZHEYiCI2cL38Attwg6y7gHPZc9TSFxJcblxJceMk3J8Z1qdZVPkdie9z3Ha57i53lJJXxBW9GiqaMqlTOzpHNvveDqIvdtWOrqWiqZpIpI43zRhuIOFpC1zWuBa4Wc5oDgNR1Hi1XpmHOqraA1tRKGtAaAHagALAdACx9TlGV8m6ukeZdX7zEQ8WFgQ4WIIAAuFgsK7t3NXXVrWOhsi5FhpwRDG2MON3WuSbbLucS424BfVc8arrTfnC0htKw3IdjmtwEfYZ06y4jgs1ajJnhWFuE1MltmogO7YAd6VgXC/Pxq1PDtSzSdys6yasgpKgtIINrG4I2gg6iOgrojM/OBlVE17SMYAEjOFjuHVtwnaDxc4IXOjmr60VW+NwdG9zHD+Jji0+UEFXrYdT3XUrTq5ToeqzYpXybq6Bjnk3JN7E8bm3wuPOQbr2ZYylHBG6SR2FrfK48DWjhceAKjIc+qy1v2h/kYfSW3WOyhlCSU4pJHyHgLnF1uYXOocwsuKVKUep0Kqn0PtleudNLJK77Uji63FxN6ALDxLd9EecDYy6nkcGh7sUZJ1YyAHMJ4MVgRzgjaQq6TL0aKp2dzovK+Sop24ZWNe0G4BvcHjBBBB6CnkrJsULcETGsbe9hwnjJNyTYAXJOxUXQZz1UYwsnkDRsBOIDoDwQB0akV+cdTKLPnkLeFt8LSOIhtgfGqZWaaiNl0t5wtlc2CNwcyMlz3A3BktYAHhwC+scLjxLFaLe/YeiT3b1rRC+9DVvjcHxuLHC9nDaLix184JCtbaxnm3uX/nB9xP1MvsOXNNls8udFU4FpqJCCCCC7aCLEeMLDFg4VtQqad7lavx9C/Mxsp7vSwyHW4swv8NncOJ6S3F4wqv0r5J3Krc8fZnaHjwh3Lx03Ad/WsHk3LU8LcEU0jG3xYWusLm1z6AoZTyvNNbdZHSYb4cRva9r26bDyLPo3boy7d0ky0tF+czZY2wPdaWMYW3/AOowfZtxuaNRG2wB47bDlfNumndilha934tbSbbMRaRits13VAA21+O+zxhZVueVYwYW1Elh+Kzz5Xgn0qI0nJ/CS6iS3L5Y1kTLDDHGxvM1jGjyAAKjtJecLaqe7LmKNuBh/Fru59toDjqHMBxrC5Uy1PN97NJINoa5xLb8zfsg9AXguuyjh8ju+phUq5lZEXISQAukxJNKYSsm1AOyWFSui6AiAlZTCiQgIkIDVIIBQEbptC51GnGu5Kl7E3x0xpyruSpOxN8dc/ioG2hI6LQudN/Ou5Kk7E3x09/Ou5Kk7E3x1HioDQkdFXSK5138q7kqTsTfHRv5V3JUvYm+Op8VAaEjohF1zvv5V3JUvYm+OjfyruSpOxN8dPFQGhI6HCa5338q7kqTsTfHRv513JUnYm+OnioDQkdEEJrnbfyruSpOxN8dG/lXclS9ib46jxUBoSOiUwVzrv5V3JUvYm+OmNOddyVL2Jvjp4qA0JHRaRC5139K7kqTsTfHRv6V3JUnYm+Op8VAaEjolNc67+ldyVJ2Jvjo39K7kqTsTfHTxUBoSOiwEiFzrv6V3JUnYm+Ojf0ruSpOxN8dR4qA0JHRdkrLnXf0ruSpPNzfHT39K7kqTsTfHU+KgNCR0WkSudN/Su5Kk7E3x0b+ldyVJ2Jvjp4qA0JHRIKZC513867kqTsTfHRv6V3JUnm5vjp4qA0JHRITXOu/pXclSdib46N/Ou5Kk7E3x1HioDQkdEhBK523867kqXsTfHS38q7kqTsTfHU+KgNCR0WE1zoNOldyVJ2JvjoOnOu5Kk7E3x08VAaEjohwXzIXPW/nXclS9ib46e/pXclSdib46eKgNCR0GmCVz1v5V3JUnYm+Olv413JUvYm+Oo8VTGhI6ObUcetNsoXOG/jW8lS9ib46N/Gu5Kl7Evx1lKdFllTmdJDXsPqX1jbZc07+VdyVJ2JvjqQ07V3JUvYm+Osm4dn7F1CXB0sEWXNjdPNfyVJ5uX46e/1X8lSebm+OqOSJyM6RISsub9/qv5Kk83N8dG/1X8lSebm+OozInIzpByRXN5081/JUnm5vjr5nTpXclS9ib46snHuyHGXY6Pkm4vKvkVzrv513JUvYm+Ogac67kqXsTfHXVCvSitjKVKbOiSE3LnU6dK7kqTsTfHQdOldyVJ2Jvjq/ioEaEjoiydlzrv5V3JUvYm+Ogac67kqTsTfHTxUBoSOiyENXO2/pXclSebm+Olv6V3JUnm5vjp4qA0JHRQCd1zodOldyVJ2Jvjo3867kqXsTfHTxUBoSOjAFFc7b+ldyVJ2Jvjpb+ddyVJ2Jvjp4qA0JHRWFKy523867kqTsTfHRv6V3JUvYm+Oo8VAaEirEIQvNO0EIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEIAQhCAEIQgBCEID/9k=\n", 64 | "text/html": [ 65 | "\n", 66 | " \n", 73 | " " 74 | ], 75 | "text/plain": [ 76 | "" 77 | ] 78 | }, 79 | "execution_count": 1, 80 | "metadata": {}, 81 | "output_type": "execute_result" 82 | } 83 | ], 84 | "source": [ 85 | "from IPython.lib.display import YouTubeVideo\n", 86 | "YouTubeVideo('HW29067qVWk')" 87 | ] 88 | }, 89 | { 90 | "cell_type": "markdown", 91 | "metadata": {}, 92 | "source": [ 93 | "## Whenever you want to sync your forked repo to ours...\n", 94 | "Run on your terminal:\n", 95 | "```\n", 96 | "# Add a separate remote repository from your forked one\n", 97 | "git remote add original git@github.com:kojino/CS282R_Deep_Learning_Workshop.git\n", 98 | "\n", 99 | "git pull original master\n", 100 | "```" 101 | ] 102 | }, 103 | { 104 | "cell_type": "code", 105 | "execution_count": null, 106 | "metadata": {}, 107 | "outputs": [], 108 | "source": [] 109 | } 110 | ], 111 | "metadata": { 112 | "kernelspec": { 113 | "display_name": "Python [conda env:tensorflow_workshop]", 114 | "language": "python", 115 | "name": "conda-env-tensorflow_workshop-py" 116 | }, 117 | "language_info": { 118 | "codemirror_mode": { 119 | "name": "ipython", 120 | "version": 3 121 | }, 122 | "file_extension": ".py", 123 | "mimetype": "text/x-python", 124 | "name": "python", 125 | "nbconvert_exporter": "python", 126 | "pygments_lexer": "ipython3", 127 | "version": "3.6.4" 128 | } 129 | }, 130 | "nbformat": 4, 131 | "nbformat_minor": 2 132 | } 133 | -------------------------------------------------------------------------------- /2_Introduction.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "\n", 8 | "# 2. Introduction to TensorFlow\n", 9 | "\n", 10 | "## What is TensorFlow?\n", 11 | "TensorFlow is an open source machine learning library developed by Google. It was originally being used internally, then it was open sourced in 2015. TensorFlow APIs are provided in `Python` and `C++`. We'll use the Python API, but note that most operations are implemented in `C++`under the hood for fast numerical computation. If interested, definitely check out the [source code](https://github.com/tensorflow/tensorflow), as some of the sample codes can be useful for your project as well.\n", 12 | "\n", 13 | "TensorFlow supports runs on multiple CPUs and GPUs, making it scalable in \"big data\" and \"deep learning\" settings. Another notable feature is TensorBoard, which is a visualization tool for your model runs (covered in Section ).\n", 14 | "\n", 15 | "## Why TensorFlow?\n", 16 | "TensorFlow is the go-to framework for modern machine learning. As you can see below, TensorFlow is by far the most popular deep learning framework on GitHub. This also means that there are tons of sample code, StackOverflow questions and blog posts online. It's very powerful.\n", 17 | "\n", 18 | "![](images/2_tensorflow.png)\n", 19 | "\n", 20 | "Although we'll be using TensorFlow, other libraries are also very nice, and there are many others not on this graph that are unique and interesting too.\n", 21 | "\n", 22 | "Few notes on some of the other libraries listed above:\n", 23 | "- Keras: Think of Keras as a \"wrapper\" for other libraries like TensorFlow and Theano. The APIs are much higher level, and so you can run reasonably sophisticated neural network models very quickly. For example, check out that [LSTM](https://keras.io/layers/recurrent/#lstm) is just a one liner! Citing the [Keras documentation](https://keras.io/), \"It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.\"\n", 24 | "- PyTorch: Primarily developed by Facebook people. Sasha likes it. The main difference between Tensorflow is that it is \"imperative\". What does it mean? Citing the [PyTorch documentation](http://pytorch.org/about/), \"PyTorch is designed to be intuitive, linear in thought and easy to use. When you execute a line of code, it gets executed. There isn’t an asynchronous view of the world.\"\n", 25 | "- Theano: Unfortunately, the library development and support ended in 2017. So I don't recommend using it if you're new. I know a couple of people who have been using it for their reseach, but are trying to convert their code to TensorFlow.\n", 26 | "\n", 27 | "## Exercise\n", 28 | "Google or search on GitHub and find three interesting open source projects on machine learning and deep learning.\n", 29 | "\n", 30 | "- Pyro\n", 31 | "- Colaboratory\n", 32 | "- PaintsChainer \n", 33 | "\n", 34 | "## Reference\n", 35 | "- [Stanford CS224d, Deep Learning for NLP](http://web.stanford.edu/class/cs224n/syllabus.html)" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": null, 41 | "metadata": {}, 42 | "outputs": [], 43 | "source": [] 44 | } 45 | ], 46 | "metadata": { 47 | "kernelspec": { 48 | "display_name": "Python [conda env:tensorflow_workshop]", 49 | "language": "python", 50 | "name": "conda-env-tensorflow_workshop-py" 51 | }, 52 | "language_info": { 53 | "codemirror_mode": { 54 | "name": "ipython", 55 | "version": 3 56 | }, 57 | "file_extension": ".py", 58 | "mimetype": "text/x-python", 59 | "name": "python", 60 | "nbconvert_exporter": "python", 61 | "pygments_lexer": "ipython3", 62 | "version": "3.6.4" 63 | } 64 | }, 65 | "nbformat": 4, 66 | "nbformat_minor": 2 67 | } 68 | -------------------------------------------------------------------------------- /4_Basic_Operations.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 4. Basic Math, Matrix Operations\n", 8 | "\n", 9 | "In this section, we'll cover some basic operations in TensorFlow. There are many more cool operations not covered here. For a more holistic overview, see [the official API documentation](https://www.tensorflow.org/api_docs/python/tf#functions)." 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "## Scalar Operations" 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": {}, 22 | "source": [ 23 | "### Arithmetic" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": 1, 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "import numpy as np\n", 33 | "import tensorflow as tf" 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": 2, 39 | "metadata": {}, 40 | "outputs": [ 41 | { 42 | "name": "stdout", 43 | "output_type": "stream", 44 | "text": [ 45 | "a: 5\n", 46 | "b: 8\n", 47 | "c: 30\n", 48 | "d: 15\n", 49 | "e: 1024\n" 50 | ] 51 | } 52 | ], 53 | "source": [ 54 | "a = tf.add(2,3) # 2 + 3\n", 55 | "b = tf.subtract(10,2) # 10 - 2\n", 56 | "c = tf.multiply(3,10) # 3 * 10\n", 57 | "d = tf.div(30,2) # 30 / 2\n", 58 | "e = tf.pow(2,10) # 2 ** 10\n", 59 | "\n", 60 | "with tf.Session() as sess: # Evaluate the result\n", 61 | " print('a:',sess.run(a))\n", 62 | " print('b:',sess.run(b))\n", 63 | " print('c:',sess.run(c))\n", 64 | " print('d:',sess.run(d))\n", 65 | " print('e:',sess.run(e))" 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "metadata": {}, 71 | "source": [ 72 | "You can also use `tf.constant` to do the following:" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 3, 78 | "metadata": {}, 79 | "outputs": [ 80 | { 81 | "name": "stdout", 82 | "output_type": "stream", 83 | "text": [ 84 | "a: 8\n", 85 | "b: 9\n", 86 | "c: 17\n", 87 | "d: -1\n", 88 | "e: 98\n" 89 | ] 90 | } 91 | ], 92 | "source": [ 93 | "a = tf.constant(8)\n", 94 | "b = tf.constant(9)\n", 95 | "c = a + b\n", 96 | "d = a - b\n", 97 | "e = 10 * b + a\n", 98 | "\n", 99 | "with tf.Session() as sess:\n", 100 | " print('a:',sess.run(a))\n", 101 | " print('b:',sess.run(b))\n", 102 | " print('c:',sess.run(c))\n", 103 | " print('d:',sess.run(d))\n", 104 | " print('e:',sess.run(e))" 105 | ] 106 | }, 107 | { 108 | "cell_type": "markdown", 109 | "metadata": {}, 110 | "source": [ 111 | "### Data Types" 112 | ] 113 | }, 114 | { 115 | "cell_type": "markdown", 116 | "metadata": {}, 117 | "source": [ 118 | "What's wrong with the following?" 119 | ] 120 | }, 121 | { 122 | "cell_type": "code", 123 | "execution_count": 4, 124 | "metadata": {}, 125 | "outputs": [ 126 | { 127 | "ename": "ValueError", 128 | "evalue": "Tensor conversion requested dtype int32 for Tensor with dtype float32: 'Tensor(\"Const_3:0\", shape=(), dtype=float32)'", 129 | "output_type": "error", 130 | "traceback": [ 131 | "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", 132 | "\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)", 133 | "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0mg\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconstant\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m3.0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;31m## This throws an error ##\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mh\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mg\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", 134 | "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py\u001b[0m in \u001b[0;36mbinary_op_wrapper\u001b[0;34m(x, y)\u001b[0m\n\u001b[1;32m 854\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msparse_tensor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSparseTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 855\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 856\u001b[0;31m \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconvert_to_tensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbase_dtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"y\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 857\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 858\u001b[0m \u001b[0;31m# If the RHS is not a tensor, it might be a tensor aware object\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 135 | "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\u001b[0m in \u001b[0;36mconvert_to_tensor\u001b[0;34m(value, dtype, name, preferred_dtype)\u001b[0m\n\u001b[1;32m 609\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 610\u001b[0m \u001b[0mpreferred_dtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mpreferred_dtype\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 611\u001b[0;31m as_ref=False)\n\u001b[0m\u001b[1;32m 612\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 613\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", 136 | "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\u001b[0m in \u001b[0;36minternal_convert_to_tensor\u001b[0;34m(value, dtype, name, as_ref, preferred_dtype)\u001b[0m\n\u001b[1;32m 674\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 675\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mret\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 676\u001b[0;31m \u001b[0mret\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconversion_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mas_ref\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mas_ref\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 677\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 678\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mret\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mNotImplemented\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 137 | "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\u001b[0m in \u001b[0;36m_TensorTensorConversionFunction\u001b[0;34m(t, dtype, name, as_ref)\u001b[0m\n\u001b[1;32m 547\u001b[0m raise ValueError(\n\u001b[1;32m 548\u001b[0m \u001b[0;34m\"Tensor conversion requested dtype %s for Tensor with dtype %s: %r\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 549\u001b[0;31m % (dtype.name, t.dtype.name, str(t)))\n\u001b[0m\u001b[1;32m 550\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 551\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", 138 | "\u001b[0;31mValueError\u001b[0m: Tensor conversion requested dtype int32 for Tensor with dtype float32: 'Tensor(\"Const_3:0\", shape=(), dtype=float32)'" 139 | ] 140 | } 141 | ], 142 | "source": [ 143 | "f = tf.constant(2)\n", 144 | "g = tf.constant(3.0)\n", 145 | "## This throws an error ##\n", 146 | "h = f * g" 147 | ] 148 | }, 149 | { 150 | "cell_type": "markdown", 151 | "metadata": {}, 152 | "source": [ 153 | "The data type should match for operations to be valid. There are data types like `int32`, `float32`, and `float64`. **In general, use `float32`!**" 154 | ] 155 | }, 156 | { 157 | "cell_type": "markdown", 158 | "metadata": {}, 159 | "source": [ 160 | "### Exercise\n", 161 | "1) Implement the [sigmoid function](https://en.wikipedia.org/wiki/Sigmoid_function). \n", 162 | "\n", 163 | "2) Evalaute the result for some values.\n", 164 | "\n", 165 | "3) (optional) Plot the sigmoid function using the TensorFlow outputs.\n", 166 | "\n", 167 | "4) Repeat the above for [ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": 5, 173 | "metadata": {}, 174 | "outputs": [], 175 | "source": [ 176 | "# #1\n", 177 | "# def sigmoid(x):\n", 178 | "# return 1 / (1 + tf.exp(-x))\n", 179 | "\n", 180 | "# def relu(x):\n", 181 | "# return tf.maximum(0.0,x)" 182 | ] 183 | }, 184 | { 185 | "cell_type": "code", 186 | "execution_count": 6, 187 | "metadata": {}, 188 | "outputs": [], 189 | "source": [ 190 | "# #2\n", 191 | "# with tf.Session() as sess:\n", 192 | "# for x in [0.0, 1.0, 2.0]:\n", 193 | "# print(sess.run(sigmoid(x)))\n", 194 | "# print(sess.run(relu(x)))" 195 | ] 196 | }, 197 | { 198 | "cell_type": "code", 199 | "execution_count": 7, 200 | "metadata": {}, 201 | "outputs": [], 202 | "source": [ 203 | "# #3\n", 204 | "# with tf.Session() as sess:\n", 205 | "# def sigmoid_run(i):\n", 206 | "# return sess.run(sigmoid(i))\n", 207 | "# def relu_run(i):\n", 208 | "# return sess.run(relu(i))\n", 209 | "\n", 210 | "# nums = np.arange(-6.0,6.0,1)\n", 211 | "# sigmoid_runs = np.vectorize(sigmoid_run)\n", 212 | "# sigmoid_nums = sigmoid_runs(nums)\n", 213 | "\n", 214 | "# relu_runs = np.vectorize(relu_run)\n", 215 | "# relu_nums = relu_runs(nums)\n", 216 | "\n", 217 | "# import matplotlib.pyplot as plt\n", 218 | "# plt.plot(sigmoid_nums)" 219 | ] 220 | }, 221 | { 222 | "cell_type": "code", 223 | "execution_count": 8, 224 | "metadata": {}, 225 | "outputs": [], 226 | "source": [ 227 | "# plt.plot(relu_nums)" 228 | ] 229 | }, 230 | { 231 | "cell_type": "markdown", 232 | "metadata": {}, 233 | "source": [ 234 | "## Matrix Operations" 235 | ] 236 | }, 237 | { 238 | "cell_type": "markdown", 239 | "metadata": {}, 240 | "source": [ 241 | "### Products" 242 | ] 243 | }, 244 | { 245 | "cell_type": "code", 246 | "execution_count": 9, 247 | "metadata": {}, 248 | "outputs": [], 249 | "source": [ 250 | "mat_a = tf.constant([[1,2],[3,4],[5,6]]) # this is how you initialize a matrix\n", 251 | "mat_b = tf.constant([[1,2],[2,3]]) # you can also pass in numpy arrays\n", 252 | "mat_c = tf.constant([[2,1],[3,2]])\n", 253 | "\n", 254 | "mat_dot1 = tf.matmul(mat_a, mat_b) # matrix dot product\n", 255 | "mat_dot2 = mat_a @ mat_b # convenint notation for matrix dot product\n", 256 | "\n", 257 | "mat_el1 = tf.multiply(mat_b, mat_c) # element-wise product\n", 258 | "mat_el2 = mat_b * mat_c # convenint notation for element-wise product" 259 | ] 260 | }, 261 | { 262 | "cell_type": "code", 263 | "execution_count": 10, 264 | "metadata": {}, 265 | "outputs": [ 266 | { 267 | "name": "stdout", 268 | "output_type": "stream", 269 | "text": [ 270 | "Tensor(\"Const_4:0\", shape=(3, 2), dtype=int32)\n" 271 | ] 272 | } 273 | ], 274 | "source": [ 275 | "print(mat_a) # you can't know the values, but you can know the shapes" 276 | ] 277 | }, 278 | { 279 | "cell_type": "code", 280 | "execution_count": 11, 281 | "metadata": {}, 282 | "outputs": [ 283 | { 284 | "name": "stdout", 285 | "output_type": "stream", 286 | "text": [ 287 | "dot1:\n", 288 | " [[ 5 8]\n", 289 | " [11 18]\n", 290 | " [17 28]]\n", 291 | "dot2:\n", 292 | " [[ 5 8]\n", 293 | " [11 18]\n", 294 | " [17 28]]\n", 295 | "el1:\n", 296 | " [[2 2]\n", 297 | " [6 6]]\n", 298 | "el2:\n", 299 | " [[2 2]\n", 300 | " [6 6]]\n" 301 | ] 302 | } 303 | ], 304 | "source": [ 305 | "with tf.Session() as sess:\n", 306 | " print('dot1:\\n',sess.run(mat_dot1))\n", 307 | " print('dot2:\\n',sess.run(mat_dot2))\n", 308 | " print('el1:\\n',sess.run(mat_el1))\n", 309 | " print('el2:\\n',sess.run(mat_el2))" 310 | ] 311 | }, 312 | { 313 | "cell_type": "markdown", 314 | "metadata": {}, 315 | "source": [ 316 | "### Broadcasting" 317 | ] 318 | }, 319 | { 320 | "cell_type": "code", 321 | "execution_count": 22, 322 | "metadata": {}, 323 | "outputs": [], 324 | "source": [ 325 | "mat_e = tf.constant([[2,1],[3,2]])\n", 326 | "mat_f = tf.constant([3,4])\n", 327 | "\n", 328 | "mat_broad1 = 2 * mat_e # each element is multiplied by 2\n", 329 | "mat_broad2 = mat_e * mat_f # each row in mat_e is multiplied by mat_f" 330 | ] 331 | }, 332 | { 333 | "cell_type": "code", 334 | "execution_count": 23, 335 | "metadata": {}, 336 | "outputs": [ 337 | { 338 | "name": "stdout", 339 | "output_type": "stream", 340 | "text": [ 341 | "broad1:\n", 342 | " [[4 2]\n", 343 | " [6 4]]\n", 344 | "broad2:\n", 345 | " [[6 4]\n", 346 | " [9 8]]\n" 347 | ] 348 | } 349 | ], 350 | "source": [ 351 | "with tf.Session() as sess:\n", 352 | " print('broad1:\\n',sess.run(mat_broad1))\n", 353 | " print('broad2:\\n',sess.run(mat_broad2))" 354 | ] 355 | }, 356 | { 357 | "cell_type": "markdown", 358 | "metadata": {}, 359 | "source": [ 360 | "#### Exercise\n", 361 | "How do you multiply each COLUMN in mat_e by mat_f?" 362 | ] 363 | }, 364 | { 365 | "cell_type": "code", 366 | "execution_count": 19, 367 | "metadata": {}, 368 | "outputs": [ 369 | { 370 | "name": "stdout", 371 | "output_type": "stream", 372 | "text": [ 373 | "broad4:\n", 374 | " [[ 6 3]\n", 375 | " [12 8]]\n" 376 | ] 377 | } 378 | ], 379 | "source": [ 380 | "mat_broad4 = mat_e * tf.transpose(mat_f) # each row in mat_e is multiplied by mat_f\n", 381 | "with tf.Session() as sess:\n", 382 | " print('broad4:\\n',sess.run(mat_broad4))" 383 | ] 384 | }, 385 | { 386 | "cell_type": "code", 387 | "execution_count": 20, 388 | "metadata": {}, 389 | "outputs": [ 390 | { 391 | "data": { 392 | "text/plain": [ 393 | "" 394 | ] 395 | }, 396 | "execution_count": 20, 397 | "metadata": {}, 398 | "output_type": "execute_result" 399 | } 400 | ], 401 | "source": [ 402 | "mat_f" 403 | ] 404 | }, 405 | { 406 | "cell_type": "code", 407 | "execution_count": 21, 408 | "metadata": {}, 409 | "outputs": [ 410 | { 411 | "data": { 412 | "text/plain": [ 413 | "" 414 | ] 415 | }, 416 | "execution_count": 21, 417 | "metadata": {}, 418 | "output_type": "execute_result" 419 | } 420 | ], 421 | "source": [ 422 | "tf.transpose(mat_f)" 423 | ] 424 | }, 425 | { 426 | "cell_type": "markdown", 427 | "metadata": {}, 428 | "source": [ 429 | "### Reduce" 430 | ] 431 | }, 432 | { 433 | "cell_type": "code", 434 | "execution_count": 14, 435 | "metadata": {}, 436 | "outputs": [], 437 | "source": [ 438 | "mat_g = tf.constant([[1,2],[1,3]])\n", 439 | "\n", 440 | "red_sum_all = tf.reduce_sum(mat_g) # reduce matrix to a scalar\n", 441 | "red_sum_row = tf.reduce_sum(mat_g, axis = 0) # reduce the rows\n", 442 | "red_sum_col1 = tf.reduce_sum(mat_g, axis = 1) # reduce the columns\n", 443 | "red_sum_col2 = tf.reduce_sum(mat_g, axis = 1, keep_dims=True) # keep the dimension" 444 | ] 445 | }, 446 | { 447 | "cell_type": "code", 448 | "execution_count": 15, 449 | "metadata": {}, 450 | "outputs": [ 451 | { 452 | "name": "stdout", 453 | "output_type": "stream", 454 | "text": [ 455 | "all:\n", 456 | " 7\n", 457 | "row:\n", 458 | " [2 5]\n", 459 | "col:\n", 460 | " [3 4]\n", 461 | "col:\n", 462 | " [[3]\n", 463 | " [4]]\n" 464 | ] 465 | } 466 | ], 467 | "source": [ 468 | "with tf.Session() as sess:\n", 469 | " print('all:\\n',sess.run(red_sum_all))\n", 470 | " print('row:\\n',sess.run(red_sum_row))\n", 471 | " print('col:\\n',sess.run(red_sum_col1))\n", 472 | " print('col:\\n',sess.run(red_sum_col2))" 473 | ] 474 | }, 475 | { 476 | "cell_type": "markdown", 477 | "metadata": {}, 478 | "source": [ 479 | "**Always, always set keep_dims=True!** It keeps column vectors as column vectors. If you don't do that it might cause some really annoying bugs with broadcastng. Also check out `tf.reduce_mean`, `tf.reduce_max`, etc that are also often used." 480 | ] 481 | }, 482 | { 483 | "cell_type": "markdown", 484 | "metadata": {}, 485 | "source": [ 486 | "#### Exercise\n", 487 | "1) Say you are given a feature matrix as below (20 samples, 10 features each). Normalize the features (i.e. subtract the mean, divide by the std.dev)" 488 | ] 489 | }, 490 | { 491 | "cell_type": "code", 492 | "execution_count": 25, 493 | "metadata": {}, 494 | "outputs": [], 495 | "source": [ 496 | "features = tf.constant(np.random.uniform(1,10,[20,10]))" 497 | ] 498 | }, 499 | { 500 | "cell_type": "markdown", 501 | "metadata": {}, 502 | "source": [ 503 | "2) Implement the [softmax function](https://en.wikipedia.org/wiki/Softmax_function).\n", 504 | "\n", 505 | "3) Implement ReLU for matrices." 506 | ] 507 | }, 508 | { 509 | "cell_type": "code", 510 | "execution_count": 26, 511 | "metadata": {}, 512 | "outputs": [ 513 | { 514 | "name": "stdout", 515 | "output_type": "stream", 516 | "text": [ 517 | "[ 6.77236045e-16 -1.72951931e-16 2.77555756e-16 9.99200722e-17\n", 518 | " -3.33066907e-16 -3.44169138e-16 -3.10862447e-16 1.11022302e-17\n", 519 | " -1.55431223e-16 -2.22044605e-16]\n", 520 | "[ 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]\n" 521 | ] 522 | } 523 | ], 524 | "source": [ 525 | "#1\n", 526 | "mean, var = tf.nn.moments(features, axes=[0], keep_dims=True)\n", 527 | "features_normalized = (features - mean) / var ** (1/2)\n", 528 | "with tf.Session() as sess:\n", 529 | " features_normalized_np = sess.run(features_normalized)\n", 530 | "print(np.mean(features_normalized_np,axis=0))\n", 531 | "print(np.std(features_normalized_np,axis=0))" 532 | ] 533 | }, 534 | { 535 | "cell_type": "code", 536 | "execution_count": 27, 537 | "metadata": {}, 538 | "outputs": [], 539 | "source": [ 540 | "def softmax(x):\n", 541 | " exp_x = tf.exp(x)\n", 542 | " sum_exp_x = tf.reduce_sum(exp_x)\n", 543 | " return exp_x / sum_exp_x" 544 | ] 545 | }, 546 | { 547 | "cell_type": "code", 548 | "execution_count": 28, 549 | "metadata": {}, 550 | "outputs": [ 551 | { 552 | "data": { 553 | "text/plain": [ 554 | "array([ 0.00214401, 0.0158422 , 0.11705892, 0.86495489], dtype=float32)" 555 | ] 556 | }, 557 | "execution_count": 28, 558 | "metadata": {}, 559 | "output_type": "execute_result" 560 | } 561 | ], 562 | "source": [ 563 | "x = tf.constant([2, 4, 6, 8],dtype=tf.float32)\n", 564 | "with tf.Session() as sess:\n", 565 | " softmax_x_np = sess.run(softmax(x))\n", 566 | "softmax_x_np" 567 | ] 568 | }, 569 | { 570 | "cell_type": "code", 571 | "execution_count": 96, 572 | "metadata": {}, 573 | "outputs": [], 574 | "source": [ 575 | "x = tf.constant([-2, 4, -6, 8],dtype=tf.float32)\n", 576 | "with tf.Session() as sess:\n", 577 | " relu_x_np = sess.run(relu(x))\n", 578 | "softmax_x_np" 579 | ] 580 | }, 581 | { 582 | "cell_type": "markdown", 583 | "metadata": {}, 584 | "source": [ 585 | "### Others\n", 586 | "Here are other operations I use often:\n", 587 | "- `tf.transpose`: matrix transpose\n", 588 | "- `tf.equal`: compare matrix element-wise\n", 589 | "- `tf.argmax`: get the index of max element\n", 590 | "- `tf.gather, tf.slice`: often used for indexing, see the [official guide](https://www.tensorflow.org/api_guides/python/array_ops#Slicing_and_Joining) for details\n", 591 | "- `tf.while_loop`: when building a computation graph, don't use python for loop, use this\n", 592 | "- `tf.zeros, tf.ones`: same as `np.zeros, np.ones`\n", 593 | "- `tf.random_normal, tf.random_uniform`: randomly sample from distributions" 594 | ] 595 | }, 596 | { 597 | "cell_type": "markdown", 598 | "metadata": {}, 599 | "source": [ 600 | "## Tips\n", 601 | "- Don't use numpy APIs in between when building TensorFlow graphs because it will not be part of your computation graph.\n", 602 | "- If there is a numpy way to do it, there is usually a tf way to do it.\n", 603 | "- If you think an operation is too complicated, code it up in numpy first, then convert it to tensorflow. Check if the two outputs match." 604 | ] 605 | } 606 | ], 607 | "metadata": { 608 | "kernelspec": { 609 | "display_name": "Python [conda env:tensorflow_workshop]", 610 | "language": "python", 611 | "name": "conda-env-tensorflow_workshop-py" 612 | }, 613 | "language_info": { 614 | "codemirror_mode": { 615 | "name": "ipython", 616 | "version": 3 617 | }, 618 | "file_extension": ".py", 619 | "mimetype": "text/x-python", 620 | "name": "python", 621 | "nbconvert_exporter": "python", 622 | "pygments_lexer": "ipython3", 623 | "version": "3.6.4" 624 | } 625 | }, 626 | "nbformat": 4, 627 | "nbformat_minor": 2 628 | } 629 | -------------------------------------------------------------------------------- /7_Visualization.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 7. Visualization Using TensorBoard\n", 8 | "Computation graphs are hard to imagine. The progress of training is also hard to imagine. Let's visualize them. Fortunately, TensorFlow makes this very very easy with TensorBoard. In any TensorFlow sessions, you can save the computation graph and the summary statistics (e.g. loss, accuracy) you would like to track. These will be saved in a log folder. Tensorboard loads the files in the log folder and visualize them nicely. " 9 | ] 10 | }, 11 | { 12 | "cell_type": "code", 13 | "execution_count": 17, 14 | "metadata": {}, 15 | "outputs": [], 16 | "source": [ 17 | "import tensorflow as tf" 18 | ] 19 | }, 20 | { 21 | "cell_type": "markdown", 22 | "metadata": {}, 23 | "source": [ 24 | "In the section 4, we had the following code:" 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": 18, 30 | "metadata": {}, 31 | "outputs": [ 32 | { 33 | "name": "stdout", 34 | "output_type": "stream", 35 | "text": [ 36 | "a: 5\n" 37 | ] 38 | } 39 | ], 40 | "source": [ 41 | "a = tf.add(2,3) \n", 42 | "\n", 43 | "with tf.Session() as sess:\n", 44 | " print('a:',sess.run(a))" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": {}, 50 | "source": [ 51 | "Let's rewrite this code using more TensorFlow:" 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": 19, 57 | "metadata": {}, 58 | "outputs": [ 59 | { 60 | "name": "stdout", 61 | "output_type": "stream", 62 | "text": [ 63 | "sum_: 5\n" 64 | ] 65 | } 66 | ], 67 | "source": [ 68 | "a = tf.constant(2, name = 'a')\n", 69 | "b = tf.constant(3, name = 'b')\n", 70 | "sum_ = tf.add(a,b, name = 'sum_') \n", 71 | "\n", 72 | "with tf.Session() as sess:\n", 73 | " print('sum_:',sess.run(sum_))" 74 | ] 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "metadata": {}, 79 | "source": [ 80 | "## Basic Visualization" 81 | ] 82 | }, 83 | { 84 | "cell_type": "markdown", 85 | "metadata": {}, 86 | "source": [ 87 | "How does the computation graph of this code look like? Let's visualize it using TensorBoard. To do this, you just need to add one line to the code:" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": 20, 93 | "metadata": {}, 94 | "outputs": [ 95 | { 96 | "name": "stdout", 97 | "output_type": "stream", 98 | "text": [ 99 | "sum_: 5\n" 100 | ] 101 | } 102 | ], 103 | "source": [ 104 | "tf.reset_default_graph() # Clearing all tensors before this\n", 105 | "a = tf.constant(2, name = 'a')\n", 106 | "b = tf.constant(3, name = 'b')\n", 107 | "sum_ = tf.add(a,b, name = 'sum_') \n", 108 | "\n", 109 | "with tf.Session() as sess:\n", 110 | " print('sum_:',sess.run(sum_))\n", 111 | " writer = tf.summary.FileWriter('log/simple', sess.graph) # NEW!\n", 112 | " writer.close()" 113 | ] 114 | }, 115 | { 116 | "cell_type": "markdown", 117 | "metadata": {}, 118 | "source": [ 119 | "This should create a new directory called `log` to where this Jupyter Notebook is located. Now, run the following on your local terminal, \n", 120 | "\n", 121 | "`tensorboard --logdir=log/simple`\n", 122 | "\n", 123 | "Then, head to `http://localhost:6006` and see for yourself that the graph is being visualized!" 124 | ] 125 | }, 126 | { 127 | "cell_type": "markdown", 128 | "metadata": {}, 129 | "source": [ 130 | "## Logistic Regression Visualization" 131 | ] 132 | }, 133 | { 134 | "cell_type": "markdown", 135 | "metadata": {}, 136 | "source": [ 137 | "### Simple Visualization" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 21, 143 | "metadata": {}, 144 | "outputs": [], 145 | "source": [ 146 | "tf.reset_default_graph() # Clearing all tensors before this" 147 | ] 148 | }, 149 | { 150 | "cell_type": "code", 151 | "execution_count": 22, 152 | "metadata": {}, 153 | "outputs": [ 154 | { 155 | "name": "stdout", 156 | "output_type": "stream", 157 | "text": [ 158 | "Extracting MNIST_data/train-images-idx3-ubyte.gz\n", 159 | "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n", 160 | "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n", 161 | "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n" 162 | ] 163 | } 164 | ], 165 | "source": [ 166 | "\n", 167 | "\n", 168 | "# Import MINST data\n", 169 | "from tensorflow.examples.tutorials.mnist import input_data\n", 170 | "mnist = input_data.read_data_sets('MNIST_data', one_hot=True)\n", 171 | "\n", 172 | "# Variables\n", 173 | "M = 784\n", 174 | "K = 10\n", 175 | "X = tf.placeholder(tf.float32, shape=[None, M])\n", 176 | "Y = tf.placeholder(tf.float32, shape=[None, K])\n", 177 | "W = tf.Variable(tf.zeros([M, K]))\n", 178 | "b = tf.Variable(tf.zeros([K]))\n", 179 | "\n", 180 | "# Model\n", 181 | "Yhat = tf.nn.softmax(tf.matmul(X, W) + b)\n", 182 | "\n", 183 | "# Loss\n", 184 | "loss = tf.reduce_mean(-tf.reduce_sum(Y*tf.log(Yhat), axis=1))\n", 185 | "\n", 186 | "# Optimizer\n", 187 | "learning_rate = 0.01\n", 188 | "optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)" 189 | ] 190 | }, 191 | { 192 | "cell_type": "code", 193 | "execution_count": 23, 194 | "metadata": {}, 195 | "outputs": [], 196 | "source": [ 197 | "# Train\n", 198 | "num_epochs = 25\n", 199 | "batch_size = 100\n", 200 | "\n", 201 | "with tf.Session() as sess:\n", 202 | " writer = tf.summary.FileWriter('log/logistic1', sess.graph) # NEW!\n", 203 | " writer.close()" 204 | ] 205 | }, 206 | { 207 | "cell_type": "markdown", 208 | "metadata": {}, 209 | "source": [ 210 | "### Better: Name the variables" 211 | ] 212 | }, 213 | { 214 | "cell_type": "code", 215 | "execution_count": 24, 216 | "metadata": {}, 217 | "outputs": [], 218 | "source": [ 219 | "tf.reset_default_graph() # Clearing all tensors before this" 220 | ] 221 | }, 222 | { 223 | "cell_type": "code", 224 | "execution_count": 25, 225 | "metadata": {}, 226 | "outputs": [ 227 | { 228 | "name": "stdout", 229 | "output_type": "stream", 230 | "text": [ 231 | "Extracting MNIST_data/train-images-idx3-ubyte.gz\n", 232 | "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n", 233 | "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n", 234 | "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n" 235 | ] 236 | } 237 | ], 238 | "source": [ 239 | "\n", 240 | "\n", 241 | "# Import MINST data\n", 242 | "from tensorflow.examples.tutorials.mnist import input_data\n", 243 | "mnist = input_data.read_data_sets('MNIST_data', one_hot=True)\n", 244 | "\n", 245 | "# Variables\n", 246 | "M = 784\n", 247 | "K = 10\n", 248 | "X = tf.placeholder(tf.float32, shape=[None, M], name='Input-Images')\n", 249 | "Y = tf.placeholder(tf.float32, shape=[None, K], name='Output-Labels')\n", 250 | "W = tf.get_variable('Weights', initializer=tf.zeros([M, K]))\n", 251 | "b = tf.get_variable('Bias', initializer=tf.zeros([K]))\n", 252 | "\n", 253 | "# Model\n", 254 | "Yhat = tf.nn.softmax(tf.matmul(X, W) + b, name='Prediction-Labels')\n", 255 | "\n", 256 | "# Loss\n", 257 | "loss = tf.reduce_mean(-tf.reduce_sum(Y*tf.log(Yhat), axis=1), name=\"Cross-Entropy-Loss\")\n", 258 | "\n", 259 | "# Optimizer\n", 260 | "learning_rate = 0.01\n", 261 | "optimizer = tf.train.GradientDescentOptimizer(learning_rate, name=\"SGD-Optimizer\")\n", 262 | "update = optimizer.minimize(loss)" 263 | ] 264 | }, 265 | { 266 | "cell_type": "code", 267 | "execution_count": 26, 268 | "metadata": {}, 269 | "outputs": [], 270 | "source": [ 271 | "# Train\n", 272 | "num_epochs = 25\n", 273 | "batch_size = 100\n", 274 | "\n", 275 | "with tf.Session() as sess:\n", 276 | " writer = tf.summary.FileWriter('log/logistic2', sess.graph) # NEW!\n", 277 | " writer.close()" 278 | ] 279 | }, 280 | { 281 | "cell_type": "markdown", 282 | "metadata": {}, 283 | "source": [ 284 | "### Even Better: Group Variables Together" 285 | ] 286 | }, 287 | { 288 | "cell_type": "code", 289 | "execution_count": 28, 290 | "metadata": {}, 291 | "outputs": [], 292 | "source": [ 293 | "tf.reset_default_graph() # Clearing all tensors before this" 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": 29, 299 | "metadata": {}, 300 | "outputs": [ 301 | { 302 | "name": "stdout", 303 | "output_type": "stream", 304 | "text": [ 305 | "Extracting MNIST_data/train-images-idx3-ubyte.gz\n", 306 | "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n", 307 | "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n", 308 | "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n" 309 | ] 310 | } 311 | ], 312 | "source": [ 313 | "# Import MINST data\n", 314 | "from tensorflow.examples.tutorials.mnist import input_data\n", 315 | "mnist = input_data.read_data_sets('MNIST_data', one_hot=True)\n", 316 | "\n", 317 | "M = 784\n", 318 | "K = 10\n", 319 | "with tf.name_scope('data'):\n", 320 | " X = tf.placeholder(tf.float32, shape=[None, M], name='Input-Images')\n", 321 | " Y = tf.placeholder(tf.float32, shape=[None, K], name='Output-Labels')\n", 322 | " W = tf.get_variable('Weights', initializer=tf.zeros([M, K]))\n", 323 | " b = tf.get_variable('Bias', initializer=tf.zeros([K]))\n", 324 | "\n", 325 | "with tf.name_scope('model'):\n", 326 | " Yhat = tf.nn.softmax(tf.matmul(X, W) + b, name='Prediction-Labels')\n", 327 | "\n", 328 | "with tf.name_scope('loss'):\n", 329 | " loss = tf.reduce_mean(-tf.reduce_sum(Y*tf.log(Yhat), axis=1), name=\"Cross-Entropy-Loss\")\n", 330 | "\n", 331 | "with tf.name_scope('optimizer'):\n", 332 | " learning_rate = 0.01\n", 333 | " optimizer = tf.train.GradientDescentOptimizer(learning_rate, name=\"SGD-Optimizer\")\n", 334 | " update = optimizer.minimize(loss)\n" 335 | ] 336 | }, 337 | { 338 | "cell_type": "code", 339 | "execution_count": 30, 340 | "metadata": {}, 341 | "outputs": [], 342 | "source": [ 343 | "# Train\n", 344 | "num_epochs = 25\n", 345 | "batch_size = 100\n", 346 | "\n", 347 | "with tf.Session() as sess:\n", 348 | " writer = tf.summary.FileWriter('log/logistic3', sess.graph) # NEW!\n", 349 | " writer.close()" 350 | ] 351 | }, 352 | { 353 | "cell_type": "markdown", 354 | "metadata": {}, 355 | "source": [ 356 | "### Even Even Better: Create Summaries to Track Training" 357 | ] 358 | }, 359 | { 360 | "cell_type": "code", 361 | "execution_count": 31, 362 | "metadata": {}, 363 | "outputs": [], 364 | "source": [ 365 | "tf.reset_default_graph() # Clearing all tensors before this" 366 | ] 367 | }, 368 | { 369 | "cell_type": "code", 370 | "execution_count": 32, 371 | "metadata": {}, 372 | "outputs": [ 373 | { 374 | "name": "stdout", 375 | "output_type": "stream", 376 | "text": [ 377 | "Extracting MNIST_data/train-images-idx3-ubyte.gz\n", 378 | "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n", 379 | "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n", 380 | "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n" 381 | ] 382 | } 383 | ], 384 | "source": [ 385 | "# Import MINST data\n", 386 | "from tensorflow.examples.tutorials.mnist import input_data\n", 387 | "mnist = input_data.read_data_sets('MNIST_data', one_hot=True)\n", 388 | "\n", 389 | "M = 784\n", 390 | "K = 10\n", 391 | "with tf.name_scope('data'):\n", 392 | " X = tf.placeholder(tf.float32, shape=[None, M], name='Input-Images')\n", 393 | " Y = tf.placeholder(tf.float32, shape=[None, K], name='Output-Labels')\n", 394 | " W = tf.get_variable('Weights', initializer=tf.zeros([M, K]))\n", 395 | " b = tf.get_variable('Bias', initializer=tf.zeros([K]))\n", 396 | "\n", 397 | "with tf.name_scope('model'):\n", 398 | " Yhat = tf.nn.softmax(tf.matmul(X, W) + b, name='Prediction-Labels')\n", 399 | "\n", 400 | "with tf.name_scope('loss'):\n", 401 | " loss = tf.reduce_mean(-tf.reduce_sum(Y*tf.log(Yhat), axis=1), name=\"Cross-Entropy-Loss\")\n", 402 | "\n", 403 | "with tf.name_scope('optimizer'):\n", 404 | " learning_rate = 0.01\n", 405 | " optimizer = tf.train.GradientDescentOptimizer(learning_rate, name=\"SGD-Optimizer\")\n", 406 | " update = optimizer.minimize(loss)\n", 407 | "\n", 408 | "with tf.name_scope('summaries'): # NEW!\n", 409 | " tf.summary.scalar('loss', loss)\n", 410 | " tf.summary.histogram('histogram-loss', loss)\n", 411 | " summary_op = tf.summary.merge_all()" 412 | ] 413 | }, 414 | { 415 | "cell_type": "code", 416 | "execution_count": 33, 417 | "metadata": {}, 418 | "outputs": [ 419 | { 420 | "name": "stdout", 421 | "output_type": "stream", 422 | "text": [ 423 | "Epoch: 0 Cost: 1.18415000103\n", 424 | "Epoch: 1 Cost: 0.665413012992\n", 425 | "Epoch: 2 Cost: 0.552853881121\n", 426 | "Epoch: 3 Cost: 0.498638092279\n", 427 | "Epoch: 4 Cost: 0.46549880976\n", 428 | "Epoch: 5 Cost: 0.442564203848\n", 429 | "Epoch: 6 Cost: 0.425497464917\n", 430 | "Epoch: 7 Cost: 0.412214421332\n", 431 | "Epoch: 8 Cost: 0.40141326273\n", 432 | "Epoch: 9 Cost: 0.392426092245\n", 433 | "Epoch: 10 Cost: 0.384764256748\n", 434 | "Epoch: 11 Cost: 0.378108848442\n", 435 | "Epoch: 12 Cost: 0.3723855522\n", 436 | "Epoch: 13 Cost: 0.367273085605\n", 437 | "Epoch: 14 Cost: 0.362724641453\n", 438 | "Epoch: 15 Cost: 0.358594706519\n", 439 | "Epoch: 16 Cost: 0.354899908684\n", 440 | "Epoch: 17 Cost: 0.351458825957\n", 441 | "Epoch: 18 Cost: 0.348327035796\n", 442 | "Epoch: 19 Cost: 0.345387803154\n", 443 | "Epoch: 20 Cost: 0.342760479911\n", 444 | "Epoch: 21 Cost: 0.340282042704\n", 445 | "Epoch: 22 Cost: 0.337932195609\n", 446 | "Epoch: 23 Cost: 0.335765975389\n", 447 | "Epoch: 24 Cost: 0.333698157641\n" 448 | ] 449 | } 450 | ], 451 | "source": [ 452 | "# Train\n", 453 | "num_epochs = 25\n", 454 | "batch_size = 100\n", 455 | "\n", 456 | "with tf.Session() as sess:\n", 457 | " writer = tf.summary.FileWriter('log/logistic4', sess.graph)\n", 458 | " \n", 459 | " sess.run(tf.global_variables_initializer())\n", 460 | " total_batch = int(mnist.train.num_examples/batch_size)\n", 461 | " for epoch in range(num_epochs):\n", 462 | " average_cost = 0\n", 463 | " for batch in range(total_batch):\n", 464 | " batch_X, batch_Y = mnist.train.next_batch(batch_size)\n", 465 | " _, c = sess.run([update, loss], feed_dict={X: batch_X,\n", 466 | " Y: batch_Y})\n", 467 | " average_cost += c / total_batch\n", 468 | " summary = sess.run(summary_op, feed_dict={X: batch_X,\n", 469 | " Y: batch_Y})\n", 470 | " global_step = epoch*total_batch + batch\n", 471 | " writer.add_summary(summary, global_step=global_step)\n", 472 | " print(\"Epoch:\",epoch,\"Cost:\",average_cost)\n", 473 | " \n", 474 | " \n", 475 | " \n", 476 | " writer.close()" 477 | ] 478 | }, 479 | { 480 | "cell_type": "markdown", 481 | "metadata": {}, 482 | "source": [ 483 | "That's it! A model like logistic regression might be too simple to need a visualization. But imagine you're training a new neural network model architecture you came up with. How would you know if your model is doing well, or that there aren't any bugs in the code in the first place? TensorBoard is a great way to intuitively understand your model performance." 484 | ] 485 | }, 486 | { 487 | "cell_type": "markdown", 488 | "metadata": {}, 489 | "source": [ 490 | "## Exercise\n", 491 | "Nicely visualize the linear regression you implemented in the previous exercise.\n" 492 | ] 493 | }, 494 | { 495 | "cell_type": "code", 496 | "execution_count": 26, 497 | "metadata": {}, 498 | "outputs": [], 499 | "source": [ 500 | "tf.reset_default_graph() # Clearing all tensors before this" 501 | ] 502 | }, 503 | { 504 | "cell_type": "code", 505 | "execution_count": 27, 506 | "metadata": {}, 507 | "outputs": [], 508 | "source": [ 509 | "from sklearn.datasets import load_boston\n", 510 | "import pandas as pd\n", 511 | "import numpy as np\n", 512 | "boston = load_boston()\n", 513 | "df = pd.DataFrame(boston.data)\n", 514 | "df.columns = boston.feature_names\n", 515 | "df['PRICE'] = boston.target\n", 516 | "\n", 517 | "M = df.shape[1]\n", 518 | "N = df.shape[0]\n", 519 | "with tf.name_scope('data'):\n", 520 | " X = tf.placeholder(tf.float32, shape=[None, M],name=\"House-Features\")\n", 521 | " Y = tf.placeholder(tf.float32, shape=[None, 1],name=\"Housing-Price\")\n", 522 | " W = tf.Variable(tf.ones([M, 1]),name=\"Weights\")\n", 523 | "with tf.name_scope('model'):\n", 524 | " Yhat = tf.matmul(X,W,name=\"Price-Predictions\")\n", 525 | "with tf.name_scope('loss'):\n", 526 | " loss = tf.reduce_mean((Y-Yhat)**2,name=\"Loss\")\n", 527 | "with tf.name_scope('optimizer'):\n", 528 | " optimizer = tf.train.GradientDescentOptimizer(0.01)\n", 529 | " update = optimizer.minimize(loss, name=\"SGD-Optimizer\")\n", 530 | "\n", 531 | "with tf.name_scope('summaries'):\n", 532 | " tf.summary.scalar('loss', loss)\n", 533 | " tf.summary.histogram('histogram-loss', loss)\n", 534 | " summary_op = tf.summary.merge_all()\n", 535 | " \n", 536 | "X_np = df.iloc[:,:-1].as_matrix()\n", 537 | "X_np = (X_np - np.mean(X_np,axis=0)) / np.std(X_np,axis=0)\n", 538 | "X_np = np.hstack((X_np,np.ones([X_np.shape[0],1])))\n", 539 | "y_np = df.iloc[:,-1].as_matrix()[:,np.newaxis]\n", 540 | "np.random.seed(1)\n", 541 | "train_indices = np.random.rand(X_np.shape[0]) < 0.80\n", 542 | "X_train = X_np[train_indices]\n", 543 | "X_test = X_np[~train_indices]\n", 544 | "y_train = y_np[train_indices]\n", 545 | "y_test = y_np[~train_indices]" 546 | ] 547 | }, 548 | { 549 | "cell_type": "code", 550 | "execution_count": 28, 551 | "metadata": {}, 552 | "outputs": [ 553 | { 554 | "name": "stdout", 555 | "output_type": "stream", 556 | "text": [ 557 | "Epoch: 0 Cost: 604.989\n", 558 | "Epoch: 1 Cost: 566.648\n", 559 | "Epoch: 2 Cost: 533.349\n", 560 | "Epoch: 3 Cost: 504.086\n", 561 | "Epoch: 4 Cost: 478.079\n", 562 | "Epoch: 5 Cost: 454.72\n", 563 | "Epoch: 6 Cost: 433.534\n", 564 | "Epoch: 7 Cost: 414.152\n", 565 | "Epoch: 8 Cost: 396.281\n", 566 | "Epoch: 9 Cost: 379.692\n", 567 | "Epoch: 10 Cost: 364.203\n", 568 | "Epoch: 11 Cost: 349.67\n", 569 | "Epoch: 12 Cost: 335.978\n", 570 | "Epoch: 13 Cost: 323.032\n", 571 | "Epoch: 14 Cost: 310.756\n", 572 | "Epoch: 15 Cost: 299.089\n", 573 | "Epoch: 16 Cost: 287.978\n", 574 | "Epoch: 17 Cost: 277.381\n", 575 | "Epoch: 18 Cost: 267.259\n", 576 | "Epoch: 19 Cost: 257.581\n", 577 | "Epoch: 20 Cost: 248.321\n", 578 | "Epoch: 21 Cost: 239.453\n", 579 | "Epoch: 22 Cost: 230.957\n", 580 | "Epoch: 23 Cost: 222.812\n", 581 | "Epoch: 24 Cost: 215.002\n", 582 | "Epoch: 25 Cost: 207.51\n", 583 | "Epoch: 26 Cost: 200.322\n", 584 | "Epoch: 27 Cost: 193.424\n", 585 | "Epoch: 28 Cost: 186.802\n", 586 | "Epoch: 29 Cost: 180.446\n", 587 | "Epoch: 30 Cost: 174.343\n", 588 | "Epoch: 31 Cost: 168.483\n", 589 | "Epoch: 32 Cost: 162.856\n", 590 | "Epoch: 33 Cost: 157.452\n", 591 | "Epoch: 34 Cost: 152.263\n", 592 | "Epoch: 35 Cost: 147.278\n", 593 | "Epoch: 36 Cost: 142.491\n", 594 | "Epoch: 37 Cost: 137.893\n", 595 | "Epoch: 38 Cost: 133.476\n", 596 | "Epoch: 39 Cost: 129.233\n", 597 | "Epoch: 40 Cost: 125.157\n", 598 | "Epoch: 41 Cost: 121.242\n", 599 | "Epoch: 42 Cost: 117.48\n", 600 | "Epoch: 43 Cost: 113.867\n", 601 | "Epoch: 44 Cost: 110.395\n", 602 | "Epoch: 45 Cost: 107.06\n", 603 | "Epoch: 46 Cost: 103.855\n", 604 | "Epoch: 47 Cost: 100.777\n", 605 | "Epoch: 48 Cost: 97.8185\n", 606 | "Epoch: 49 Cost: 94.9762\n", 607 | "Epoch: 50 Cost: 92.2452\n", 608 | "Epoch: 51 Cost: 89.6211\n", 609 | "Epoch: 52 Cost: 87.0995\n", 610 | "Epoch: 53 Cost: 84.6765\n", 611 | "Epoch: 54 Cost: 82.3482\n", 612 | "Epoch: 55 Cost: 80.1107\n", 613 | "Epoch: 56 Cost: 77.9605\n", 614 | "Epoch: 57 Cost: 75.8941\n", 615 | "Epoch: 58 Cost: 73.9082\n", 616 | "Epoch: 59 Cost: 71.9997\n", 617 | "Epoch: 60 Cost: 70.1654\n", 618 | "Epoch: 61 Cost: 68.4025\n", 619 | "Epoch: 62 Cost: 66.7081\n", 620 | "Epoch: 63 Cost: 65.0794\n", 621 | "Epoch: 64 Cost: 63.514\n", 622 | "Epoch: 65 Cost: 62.0093\n", 623 | "Epoch: 66 Cost: 60.5629\n", 624 | "Epoch: 67 Cost: 59.1725\n", 625 | "Epoch: 68 Cost: 57.836\n", 626 | "Epoch: 69 Cost: 56.5511\n", 627 | "Epoch: 70 Cost: 55.3159\n", 628 | "Epoch: 71 Cost: 54.1284\n", 629 | "Epoch: 72 Cost: 52.9866\n", 630 | "Epoch: 73 Cost: 51.889\n", 631 | "Epoch: 74 Cost: 50.8335\n", 632 | "Epoch: 75 Cost: 49.8187\n", 633 | "Epoch: 76 Cost: 48.843\n", 634 | "Epoch: 77 Cost: 47.9047\n", 635 | "Epoch: 78 Cost: 47.0024\n", 636 | "Epoch: 79 Cost: 46.1348\n", 637 | "Epoch: 80 Cost: 45.3004\n", 638 | "Epoch: 81 Cost: 44.4979\n", 639 | "Epoch: 82 Cost: 43.7261\n", 640 | "Epoch: 83 Cost: 42.9838\n", 641 | "Epoch: 84 Cost: 42.2698\n", 642 | "Epoch: 85 Cost: 41.5831\n", 643 | "Epoch: 86 Cost: 40.9225\n", 644 | "Epoch: 87 Cost: 40.287\n", 645 | "Epoch: 88 Cost: 39.6757\n", 646 | "Epoch: 89 Cost: 39.0875\n", 647 | "Epoch: 90 Cost: 38.5217\n", 648 | "Epoch: 91 Cost: 37.9773\n", 649 | "Epoch: 92 Cost: 37.4534\n", 650 | "Epoch: 93 Cost: 36.9494\n", 651 | "Epoch: 94 Cost: 36.4643\n", 652 | "Epoch: 95 Cost: 35.9976\n", 653 | "Epoch: 96 Cost: 35.5484\n", 654 | "Epoch: 97 Cost: 35.116\n", 655 | "Epoch: 98 Cost: 34.6999\n", 656 | "Epoch: 99 Cost: 34.2993\n", 657 | "Epoch: 100 Cost: 33.9138\n", 658 | "Epoch: 101 Cost: 33.5426\n", 659 | "Epoch: 102 Cost: 33.1853\n", 660 | "Epoch: 103 Cost: 32.8412\n", 661 | "Epoch: 104 Cost: 32.51\n", 662 | "Epoch: 105 Cost: 32.191\n", 663 | "Epoch: 106 Cost: 31.8838\n", 664 | "Epoch: 107 Cost: 31.588\n", 665 | "Epoch: 108 Cost: 31.3031\n", 666 | "Epoch: 109 Cost: 31.0287\n", 667 | "Epoch: 110 Cost: 30.7643\n", 668 | "Epoch: 111 Cost: 30.5097\n", 669 | "Epoch: 112 Cost: 30.2643\n", 670 | "Epoch: 113 Cost: 30.0279\n", 671 | "Epoch: 114 Cost: 29.8002\n", 672 | "Epoch: 115 Cost: 29.5807\n", 673 | "Epoch: 116 Cost: 29.3692\n", 674 | "Epoch: 117 Cost: 29.1653\n", 675 | "Epoch: 118 Cost: 28.9688\n", 676 | "Epoch: 119 Cost: 28.7793\n", 677 | "Epoch: 120 Cost: 28.5967\n", 678 | "Epoch: 121 Cost: 28.4206\n", 679 | "Epoch: 122 Cost: 28.2508\n", 680 | "Epoch: 123 Cost: 28.087\n", 681 | "Epoch: 124 Cost: 27.9291\n", 682 | "Epoch: 125 Cost: 27.7768\n", 683 | "Epoch: 126 Cost: 27.6298\n", 684 | "Epoch: 127 Cost: 27.488\n", 685 | "Epoch: 128 Cost: 27.3512\n", 686 | "Epoch: 129 Cost: 27.2192\n", 687 | "Epoch: 130 Cost: 27.0917\n", 688 | "Epoch: 131 Cost: 26.9688\n", 689 | "Epoch: 132 Cost: 26.85\n", 690 | "Epoch: 133 Cost: 26.7354\n", 691 | "Epoch: 134 Cost: 26.6247\n", 692 | "Epoch: 135 Cost: 26.5178\n", 693 | "Epoch: 136 Cost: 26.4145\n", 694 | "Epoch: 137 Cost: 26.3148\n", 695 | "Epoch: 138 Cost: 26.2184\n", 696 | "Epoch: 139 Cost: 26.1253\n", 697 | "Epoch: 140 Cost: 26.0353\n", 698 | "Epoch: 141 Cost: 25.9484\n", 699 | "Epoch: 142 Cost: 25.8643\n", 700 | "Epoch: 143 Cost: 25.783\n", 701 | "Epoch: 144 Cost: 25.7044\n", 702 | "Epoch: 145 Cost: 25.6283\n", 703 | "Epoch: 146 Cost: 25.5548\n", 704 | "Epoch: 147 Cost: 25.4836\n", 705 | "Epoch: 148 Cost: 25.4148\n", 706 | "Epoch: 149 Cost: 25.3482\n", 707 | "Epoch: 150 Cost: 25.2837\n", 708 | "Epoch: 151 Cost: 25.2212\n", 709 | "Epoch: 152 Cost: 25.1608\n", 710 | "Epoch: 153 Cost: 25.1022\n", 711 | "Epoch: 154 Cost: 25.0455\n", 712 | "Epoch: 155 Cost: 24.9905\n", 713 | "Epoch: 156 Cost: 24.9373\n", 714 | "Epoch: 157 Cost: 24.8856\n", 715 | "Epoch: 158 Cost: 24.8356\n", 716 | "Epoch: 159 Cost: 24.7871\n", 717 | "Epoch: 160 Cost: 24.74\n", 718 | "Epoch: 161 Cost: 24.6944\n", 719 | "Epoch: 162 Cost: 24.6501\n", 720 | "Epoch: 163 Cost: 24.6071\n", 721 | "Epoch: 164 Cost: 24.5654\n", 722 | "Epoch: 165 Cost: 24.5249\n", 723 | "Epoch: 166 Cost: 24.4856\n", 724 | "Epoch: 167 Cost: 24.4474\n", 725 | "Epoch: 168 Cost: 24.4103\n", 726 | "Epoch: 169 Cost: 24.3743\n", 727 | "Epoch: 170 Cost: 24.3392\n", 728 | "Epoch: 171 Cost: 24.3052\n", 729 | "Epoch: 172 Cost: 24.2721\n", 730 | "Epoch: 173 Cost: 24.2399\n", 731 | "Epoch: 174 Cost: 24.2085\n", 732 | "Epoch: 175 Cost: 24.178\n", 733 | "Epoch: 176 Cost: 24.1484\n", 734 | "Epoch: 177 Cost: 24.1195\n", 735 | "Epoch: 178 Cost: 24.0914\n", 736 | "Epoch: 179 Cost: 24.064\n", 737 | "Epoch: 180 Cost: 24.0373\n", 738 | "Epoch: 181 Cost: 24.0113\n", 739 | "Epoch: 182 Cost: 23.9859\n", 740 | "Epoch: 183 Cost: 23.9612\n", 741 | "Epoch: 184 Cost: 23.9371\n", 742 | "Epoch: 185 Cost: 23.9136\n", 743 | "Epoch: 186 Cost: 23.8907\n", 744 | "Epoch: 187 Cost: 23.8683\n", 745 | "Epoch: 188 Cost: 23.8465\n", 746 | "Epoch: 189 Cost: 23.8252\n", 747 | "Epoch: 190 Cost: 23.8043\n", 748 | "Epoch: 191 Cost: 23.784\n", 749 | "Epoch: 192 Cost: 23.7641\n", 750 | "Epoch: 193 Cost: 23.7447\n", 751 | "Epoch: 194 Cost: 23.7257\n", 752 | "Epoch: 195 Cost: 23.7071\n", 753 | "Epoch: 196 Cost: 23.689\n", 754 | "Epoch: 197 Cost: 23.6712\n", 755 | "Epoch: 198 Cost: 23.6538\n", 756 | "Epoch: 199 Cost: 23.6368\n", 757 | "Epoch: 200 Cost: 23.6202\n", 758 | "Epoch: 201 Cost: 23.6038\n", 759 | "Epoch: 202 Cost: 23.5879\n", 760 | "Epoch: 203 Cost: 23.5722\n", 761 | "Epoch: 204 Cost: 23.5569\n", 762 | "Epoch: 205 Cost: 23.5418\n", 763 | "Epoch: 206 Cost: 23.5271\n", 764 | "Epoch: 207 Cost: 23.5126\n", 765 | "Epoch: 208 Cost: 23.4985\n", 766 | "Epoch: 209 Cost: 23.4845\n", 767 | "Epoch: 210 Cost: 23.4709\n", 768 | "Epoch: 211 Cost: 23.4575\n", 769 | "Epoch: 212 Cost: 23.4443\n", 770 | "Epoch: 213 Cost: 23.4314\n", 771 | "Epoch: 214 Cost: 23.4187\n", 772 | "Epoch: 215 Cost: 23.4063\n", 773 | "Epoch: 216 Cost: 23.394\n", 774 | "Epoch: 217 Cost: 23.382\n", 775 | "Epoch: 218 Cost: 23.3701\n", 776 | "Epoch: 219 Cost: 23.3585\n", 777 | "Epoch: 220 Cost: 23.3471\n", 778 | "Epoch: 221 Cost: 23.3358\n", 779 | "Epoch: 222 Cost: 23.3247\n", 780 | "Epoch: 223 Cost: 23.3138\n", 781 | "Epoch: 224 Cost: 23.3031\n", 782 | "Epoch: 225 Cost: 23.2926\n", 783 | "Epoch: 226 Cost: 23.2822\n", 784 | "Epoch: 227 Cost: 23.2719\n", 785 | "Epoch: 228 Cost: 23.2619\n", 786 | "Epoch: 229 Cost: 23.2519\n", 787 | "Epoch: 230 Cost: 23.2421\n", 788 | "Epoch: 231 Cost: 23.2325\n", 789 | "Epoch: 232 Cost: 23.223\n", 790 | "Epoch: 233 Cost: 23.2136\n", 791 | "Epoch: 234 Cost: 23.2044\n", 792 | "Epoch: 235 Cost: 23.1953\n", 793 | "Epoch: 236 Cost: 23.1863\n", 794 | "Epoch: 237 Cost: 23.1774\n", 795 | "Epoch: 238 Cost: 23.1687\n", 796 | "Epoch: 239 Cost: 23.16\n", 797 | "Epoch: 240 Cost: 23.1515\n", 798 | "Epoch: 241 Cost: 23.1431\n", 799 | "Epoch: 242 Cost: 23.1348\n", 800 | "Epoch: 243 Cost: 23.1266\n", 801 | "Epoch: 244 Cost: 23.1185\n", 802 | "Epoch: 245 Cost: 23.1105\n", 803 | "Epoch: 246 Cost: 23.1027\n", 804 | "Epoch: 247 Cost: 23.0949\n", 805 | "Epoch: 248 Cost: 23.0872\n", 806 | "Epoch: 249 Cost: 23.0795\n", 807 | "Epoch: 250 Cost: 23.072\n", 808 | "Epoch: 251 Cost: 23.0646\n", 809 | "Epoch: 252 Cost: 23.0572\n", 810 | "Epoch: 253 Cost: 23.05\n", 811 | "Epoch: 254 Cost: 23.0428\n", 812 | "Epoch: 255 Cost: 23.0357\n", 813 | "Epoch: 256 Cost: 23.0287\n", 814 | "Epoch: 257 Cost: 23.0217\n", 815 | "Epoch: 258 Cost: 23.0149\n", 816 | "Epoch: 259 Cost: 23.0081\n", 817 | "Epoch: 260 Cost: 23.0014\n", 818 | "Epoch: 261 Cost: 22.9947\n", 819 | "Epoch: 262 Cost: 22.9881\n", 820 | "Epoch: 263 Cost: 22.9816\n", 821 | "Epoch: 264 Cost: 22.9752\n", 822 | "Epoch: 265 Cost: 22.9688\n", 823 | "Epoch: 266 Cost: 22.9625\n", 824 | "Epoch: 267 Cost: 22.9562\n", 825 | "Epoch: 268 Cost: 22.95\n", 826 | "Epoch: 269 Cost: 22.9439\n", 827 | "Epoch: 270 Cost: 22.9379\n", 828 | "Epoch: 271 Cost: 22.9319\n", 829 | "Epoch: 272 Cost: 22.9259\n", 830 | "Epoch: 273 Cost: 22.92\n", 831 | "Epoch: 274 Cost: 22.9142\n", 832 | "Epoch: 275 Cost: 22.9084\n", 833 | "Epoch: 276 Cost: 22.9027\n", 834 | "Epoch: 277 Cost: 22.897\n", 835 | "Epoch: 278 Cost: 22.8914\n", 836 | "Epoch: 279 Cost: 22.8858\n", 837 | "Epoch: 280 Cost: 22.8803\n", 838 | "Epoch: 281 Cost: 22.8749\n", 839 | "Epoch: 282 Cost: 22.8694\n", 840 | "Epoch: 283 Cost: 22.8641\n", 841 | "Epoch: 284 Cost: 22.8588\n", 842 | "Epoch: 285 Cost: 22.8535\n", 843 | "Epoch: 286 Cost: 22.8483\n", 844 | "Epoch: 287 Cost: 22.8431\n", 845 | "Epoch: 288 Cost: 22.838\n", 846 | "Epoch: 289 Cost: 22.8329\n", 847 | "Epoch: 290 Cost: 22.8278\n", 848 | "Epoch: 291 Cost: 22.8228\n", 849 | "Epoch: 292 Cost: 22.8179\n", 850 | "Epoch: 293 Cost: 22.813\n", 851 | "Epoch: 294 Cost: 22.8081\n", 852 | "Epoch: 295 Cost: 22.8032\n", 853 | "Epoch: 296 Cost: 22.7985\n", 854 | "Epoch: 297 Cost: 22.7937\n", 855 | "Epoch: 298 Cost: 22.789\n", 856 | "Epoch: 299 Cost: 22.7843\n", 857 | "Epoch: 300 Cost: 22.7797\n", 858 | "Epoch: 301 Cost: 22.7751\n", 859 | "Epoch: 302 Cost: 22.7705\n", 860 | "Epoch: 303 Cost: 22.766\n", 861 | "Epoch: 304 Cost: 22.7615\n", 862 | "Epoch: 305 Cost: 22.7571\n", 863 | "Epoch: 306 Cost: 22.7527\n", 864 | "Epoch: 307 Cost: 22.7483\n", 865 | "Epoch: 308 Cost: 22.7439\n", 866 | "Epoch: 309 Cost: 22.7396\n", 867 | "Epoch: 310 Cost: 22.7353\n", 868 | "Epoch: 311 Cost: 22.7311\n", 869 | "Epoch: 312 Cost: 22.7269\n", 870 | "Epoch: 313 Cost: 22.7227\n", 871 | "Epoch: 314 Cost: 22.7186\n", 872 | "Epoch: 315 Cost: 22.7145\n", 873 | "Epoch: 316 Cost: 22.7104\n", 874 | "Epoch: 317 Cost: 22.7063\n", 875 | "Epoch: 318 Cost: 22.7023\n", 876 | "Epoch: 319 Cost: 22.6983\n", 877 | "Epoch: 320 Cost: 22.6944\n", 878 | "Epoch: 321 Cost: 22.6904\n", 879 | "Epoch: 322 Cost: 22.6865\n", 880 | "Epoch: 323 Cost: 22.6827\n", 881 | "Epoch: 324 Cost: 22.6788\n", 882 | "Epoch: 325 Cost: 22.675\n", 883 | "Epoch: 326 Cost: 22.6712\n", 884 | "Epoch: 327 Cost: 22.6675\n", 885 | "Epoch: 328 Cost: 22.6638\n", 886 | "Epoch: 329 Cost: 22.6601\n", 887 | "Epoch: 330 Cost: 22.6564\n", 888 | "Epoch: 331 Cost: 22.6527\n", 889 | "Epoch: 332 Cost: 22.6491\n", 890 | "Epoch: 333 Cost: 22.6455\n", 891 | "Epoch: 334 Cost: 22.642\n", 892 | "Epoch: 335 Cost: 22.6384\n", 893 | "Epoch: 336 Cost: 22.6349\n", 894 | "Epoch: 337 Cost: 22.6314\n", 895 | "Epoch: 338 Cost: 22.628\n", 896 | "Epoch: 339 Cost: 22.6245\n", 897 | "Epoch: 340 Cost: 22.6211\n", 898 | "Epoch: 341 Cost: 22.6177\n", 899 | "Epoch: 342 Cost: 22.6144\n", 900 | "Epoch: 343 Cost: 22.611\n", 901 | "Epoch: 344 Cost: 22.6077\n", 902 | "Epoch: 345 Cost: 22.6044\n", 903 | "Epoch: 346 Cost: 22.6012\n", 904 | "Epoch: 347 Cost: 22.5979\n", 905 | "Epoch: 348 Cost: 22.5947\n", 906 | "Epoch: 349 Cost: 22.5915\n", 907 | "Epoch: 350 Cost: 22.5883\n", 908 | "Epoch: 351 Cost: 22.5852\n", 909 | "Epoch: 352 Cost: 22.582\n", 910 | "Epoch: 353 Cost: 22.5789\n", 911 | "Epoch: 354 Cost: 22.5758\n", 912 | "Epoch: 355 Cost: 22.5728\n", 913 | "Epoch: 356 Cost: 22.5697\n", 914 | "Epoch: 357 Cost: 22.5667\n", 915 | "Epoch: 358 Cost: 22.5637\n", 916 | "Epoch: 359 Cost: 22.5607\n", 917 | "Epoch: 360 Cost: 22.5578\n", 918 | "Epoch: 361 Cost: 22.5548\n", 919 | "Epoch: 362 Cost: 22.5519\n", 920 | "Epoch: 363 Cost: 22.549\n", 921 | "Epoch: 364 Cost: 22.5461\n", 922 | "Epoch: 365 Cost: 22.5433\n", 923 | "Epoch: 366 Cost: 22.5404\n", 924 | "Epoch: 367 Cost: 22.5376\n", 925 | "Epoch: 368 Cost: 22.5348\n", 926 | "Epoch: 369 Cost: 22.532\n", 927 | "Epoch: 370 Cost: 22.5292\n", 928 | "Epoch: 371 Cost: 22.5265\n", 929 | "Epoch: 372 Cost: 22.5238\n", 930 | "Epoch: 373 Cost: 22.5211\n", 931 | "Epoch: 374 Cost: 22.5184\n", 932 | "Epoch: 375 Cost: 22.5157\n", 933 | "Epoch: 376 Cost: 22.513\n", 934 | "Epoch: 377 Cost: 22.5104\n", 935 | "Epoch: 378 Cost: 22.5078\n", 936 | "Epoch: 379 Cost: 22.5052\n", 937 | "Epoch: 380 Cost: 22.5026\n", 938 | "Epoch: 381 Cost: 22.5\n", 939 | "Epoch: 382 Cost: 22.4975\n", 940 | "Epoch: 383 Cost: 22.495\n", 941 | "Epoch: 384 Cost: 22.4924\n", 942 | "Epoch: 385 Cost: 22.4899\n", 943 | "Epoch: 386 Cost: 22.4875\n", 944 | "Epoch: 387 Cost: 22.485\n", 945 | "Epoch: 388 Cost: 22.4825\n", 946 | "Epoch: 389 Cost: 22.4801\n", 947 | "Epoch: 390 Cost: 22.4777\n", 948 | "Epoch: 391 Cost: 22.4753\n", 949 | "Epoch: 392 Cost: 22.4729\n", 950 | "Epoch: 393 Cost: 22.4705\n", 951 | "Epoch: 394 Cost: 22.4682\n", 952 | "Epoch: 395 Cost: 22.4659\n", 953 | "Epoch: 396 Cost: 22.4635\n", 954 | "Epoch: 397 Cost: 22.4612\n", 955 | "Epoch: 398 Cost: 22.4589\n", 956 | "Epoch: 399 Cost: 22.4567\n", 957 | "Epoch: 400 Cost: 22.4544\n", 958 | "Epoch: 401 Cost: 22.4521\n", 959 | "Epoch: 402 Cost: 22.4499\n", 960 | "Epoch: 403 Cost: 22.4477\n", 961 | "Epoch: 404 Cost: 22.4455\n", 962 | "Epoch: 405 Cost: 22.4433\n", 963 | "Epoch: 406 Cost: 22.4411\n", 964 | "Epoch: 407 Cost: 22.439\n", 965 | "Epoch: 408 Cost: 22.4368\n", 966 | "Epoch: 409 Cost: 22.4347\n", 967 | "Epoch: 410 Cost: 22.4326\n", 968 | "Epoch: 411 Cost: 22.4305\n", 969 | "Epoch: 412 Cost: 22.4284\n", 970 | "Epoch: 413 Cost: 22.4263\n" 971 | ] 972 | }, 973 | { 974 | "name": "stdout", 975 | "output_type": "stream", 976 | "text": [ 977 | "Epoch: 414 Cost: 22.4242\n", 978 | "Epoch: 415 Cost: 22.4222\n", 979 | "Epoch: 416 Cost: 22.4201\n", 980 | "Epoch: 417 Cost: 22.4181\n", 981 | "Epoch: 418 Cost: 22.4161\n", 982 | "Epoch: 419 Cost: 22.4141\n", 983 | "Epoch: 420 Cost: 22.4121\n", 984 | "Epoch: 421 Cost: 22.4101\n", 985 | "Epoch: 422 Cost: 22.4082\n", 986 | "Epoch: 423 Cost: 22.4062\n", 987 | "Epoch: 424 Cost: 22.4043\n", 988 | "Epoch: 425 Cost: 22.4023\n", 989 | "Epoch: 426 Cost: 22.4004\n", 990 | "Epoch: 427 Cost: 22.3985\n", 991 | "Epoch: 428 Cost: 22.3966\n", 992 | "Epoch: 429 Cost: 22.3948\n", 993 | "Epoch: 430 Cost: 22.3929\n", 994 | "Epoch: 431 Cost: 22.391\n", 995 | "Epoch: 432 Cost: 22.3892\n", 996 | "Epoch: 433 Cost: 22.3874\n", 997 | "Epoch: 434 Cost: 22.3856\n", 998 | "Epoch: 435 Cost: 22.3837\n", 999 | "Epoch: 436 Cost: 22.3819\n", 1000 | "Epoch: 437 Cost: 22.3802\n", 1001 | "Epoch: 438 Cost: 22.3784\n", 1002 | "Epoch: 439 Cost: 22.3766\n", 1003 | "Epoch: 440 Cost: 22.3749\n", 1004 | "Epoch: 441 Cost: 22.3731\n", 1005 | "Epoch: 442 Cost: 22.3714\n", 1006 | "Epoch: 443 Cost: 22.3697\n", 1007 | "Epoch: 444 Cost: 22.368\n", 1008 | "Epoch: 445 Cost: 22.3663\n", 1009 | "Epoch: 446 Cost: 22.3646\n", 1010 | "Epoch: 447 Cost: 22.3629\n", 1011 | "Epoch: 448 Cost: 22.3612\n", 1012 | "Epoch: 449 Cost: 22.3596\n", 1013 | "Epoch: 450 Cost: 22.3579\n", 1014 | "Epoch: 451 Cost: 22.3563\n", 1015 | "Epoch: 452 Cost: 22.3547\n", 1016 | "Epoch: 453 Cost: 22.3531\n", 1017 | "Epoch: 454 Cost: 22.3515\n", 1018 | "Epoch: 455 Cost: 22.3499\n", 1019 | "Epoch: 456 Cost: 22.3483\n", 1020 | "Epoch: 457 Cost: 22.3467\n", 1021 | "Epoch: 458 Cost: 22.3451\n", 1022 | "Epoch: 459 Cost: 22.3436\n", 1023 | "Epoch: 460 Cost: 22.342\n", 1024 | "Epoch: 461 Cost: 22.3405\n", 1025 | "Epoch: 462 Cost: 22.339\n", 1026 | "Epoch: 463 Cost: 22.3374\n", 1027 | "Epoch: 464 Cost: 22.3359\n", 1028 | "Epoch: 465 Cost: 22.3344\n", 1029 | "Epoch: 466 Cost: 22.3329\n", 1030 | "Epoch: 467 Cost: 22.3314\n", 1031 | "Epoch: 468 Cost: 22.33\n", 1032 | "Epoch: 469 Cost: 22.3285\n", 1033 | "Epoch: 470 Cost: 22.3271\n", 1034 | "Epoch: 471 Cost: 22.3256\n", 1035 | "Epoch: 472 Cost: 22.3242\n", 1036 | "Epoch: 473 Cost: 22.3227\n", 1037 | "Epoch: 474 Cost: 22.3213\n", 1038 | "Epoch: 475 Cost: 22.3199\n", 1039 | "Epoch: 476 Cost: 22.3185\n", 1040 | "Epoch: 477 Cost: 22.3171\n", 1041 | "Epoch: 478 Cost: 22.3157\n", 1042 | "Epoch: 479 Cost: 22.3143\n", 1043 | "Epoch: 480 Cost: 22.3129\n", 1044 | "Epoch: 481 Cost: 22.3116\n", 1045 | "Epoch: 482 Cost: 22.3102\n", 1046 | "Epoch: 483 Cost: 22.3089\n", 1047 | "Epoch: 484 Cost: 22.3075\n", 1048 | "Epoch: 485 Cost: 22.3062\n", 1049 | "Epoch: 486 Cost: 22.3049\n", 1050 | "Epoch: 487 Cost: 22.3036\n", 1051 | "Epoch: 488 Cost: 22.3023\n", 1052 | "Epoch: 489 Cost: 22.301\n", 1053 | "Epoch: 490 Cost: 22.2997\n", 1054 | "Epoch: 491 Cost: 22.2984\n", 1055 | "Epoch: 492 Cost: 22.2971\n", 1056 | "Epoch: 493 Cost: 22.2958\n", 1057 | "Epoch: 494 Cost: 22.2946\n", 1058 | "Epoch: 495 Cost: 22.2933\n", 1059 | "Epoch: 496 Cost: 22.2921\n", 1060 | "Epoch: 497 Cost: 22.2908\n", 1061 | "Epoch: 498 Cost: 22.2896\n", 1062 | "Epoch: 499 Cost: 22.2884\n", 1063 | "Epoch: 500 Cost: 22.2872\n", 1064 | "Epoch: 501 Cost: 22.2859\n", 1065 | "Epoch: 502 Cost: 22.2847\n", 1066 | "Epoch: 503 Cost: 22.2835\n", 1067 | "Epoch: 504 Cost: 22.2824\n", 1068 | "Epoch: 505 Cost: 22.2812\n", 1069 | "Epoch: 506 Cost: 22.28\n", 1070 | "Epoch: 507 Cost: 22.2788\n", 1071 | "Epoch: 508 Cost: 22.2777\n", 1072 | "Epoch: 509 Cost: 22.2765\n", 1073 | "Epoch: 510 Cost: 22.2754\n", 1074 | "Epoch: 511 Cost: 22.2742\n", 1075 | "Epoch: 512 Cost: 22.2731\n", 1076 | "Epoch: 513 Cost: 22.2719\n", 1077 | "Epoch: 514 Cost: 22.2708\n", 1078 | "Epoch: 515 Cost: 22.2697\n", 1079 | "Epoch: 516 Cost: 22.2686\n", 1080 | "Epoch: 517 Cost: 22.2675\n", 1081 | "Epoch: 518 Cost: 22.2664\n", 1082 | "Epoch: 519 Cost: 22.2653\n", 1083 | "Epoch: 520 Cost: 22.2642\n", 1084 | "Epoch: 521 Cost: 22.2631\n", 1085 | "Epoch: 522 Cost: 22.2621\n", 1086 | "Epoch: 523 Cost: 22.261\n", 1087 | "Epoch: 524 Cost: 22.2599\n", 1088 | "Epoch: 525 Cost: 22.2589\n", 1089 | "Epoch: 526 Cost: 22.2578\n", 1090 | "Epoch: 527 Cost: 22.2568\n", 1091 | "Epoch: 528 Cost: 22.2557\n", 1092 | "Epoch: 529 Cost: 22.2547\n", 1093 | "Epoch: 530 Cost: 22.2537\n", 1094 | "Epoch: 531 Cost: 22.2527\n", 1095 | "Epoch: 532 Cost: 22.2517\n", 1096 | "Epoch: 533 Cost: 22.2507\n", 1097 | "Epoch: 534 Cost: 22.2496\n", 1098 | "Epoch: 535 Cost: 22.2487\n", 1099 | "Epoch: 536 Cost: 22.2477\n", 1100 | "Epoch: 537 Cost: 22.2467\n", 1101 | "Epoch: 538 Cost: 22.2457\n", 1102 | "Epoch: 539 Cost: 22.2447\n", 1103 | "Epoch: 540 Cost: 22.2438\n", 1104 | "Epoch: 541 Cost: 22.2428\n", 1105 | "Epoch: 542 Cost: 22.2418\n", 1106 | "Epoch: 543 Cost: 22.2409\n", 1107 | "Epoch: 544 Cost: 22.2399\n", 1108 | "Epoch: 545 Cost: 22.239\n", 1109 | "Epoch: 546 Cost: 22.2381\n", 1110 | "Epoch: 547 Cost: 22.2371\n", 1111 | "Epoch: 548 Cost: 22.2362\n", 1112 | "Epoch: 549 Cost: 22.2353\n", 1113 | "Epoch: 550 Cost: 22.2344\n", 1114 | "Epoch: 551 Cost: 22.2334\n", 1115 | "Epoch: 552 Cost: 22.2325\n", 1116 | "Epoch: 553 Cost: 22.2316\n", 1117 | "Epoch: 554 Cost: 22.2307\n", 1118 | "Epoch: 555 Cost: 22.2298\n", 1119 | "Epoch: 556 Cost: 22.229\n", 1120 | "Epoch: 557 Cost: 22.2281\n", 1121 | "Epoch: 558 Cost: 22.2272\n", 1122 | "Epoch: 559 Cost: 22.2263\n", 1123 | "Epoch: 560 Cost: 22.2255\n", 1124 | "Epoch: 561 Cost: 22.2246\n", 1125 | "Epoch: 562 Cost: 22.2237\n", 1126 | "Epoch: 563 Cost: 22.2229\n", 1127 | "Epoch: 564 Cost: 22.222\n", 1128 | "Epoch: 565 Cost: 22.2212\n", 1129 | "Epoch: 566 Cost: 22.2204\n", 1130 | "Epoch: 567 Cost: 22.2195\n", 1131 | "Epoch: 568 Cost: 22.2187\n", 1132 | "Epoch: 569 Cost: 22.2179\n", 1133 | "Epoch: 570 Cost: 22.217\n", 1134 | "Epoch: 571 Cost: 22.2162\n", 1135 | "Epoch: 572 Cost: 22.2154\n", 1136 | "Epoch: 573 Cost: 22.2146\n", 1137 | "Epoch: 574 Cost: 22.2138\n", 1138 | "Epoch: 575 Cost: 22.213\n", 1139 | "Epoch: 576 Cost: 22.2122\n", 1140 | "Epoch: 577 Cost: 22.2114\n", 1141 | "Epoch: 578 Cost: 22.2106\n", 1142 | "Epoch: 579 Cost: 22.2099\n", 1143 | "Epoch: 580 Cost: 22.2091\n", 1144 | "Epoch: 581 Cost: 22.2083\n", 1145 | "Epoch: 582 Cost: 22.2075\n", 1146 | "Epoch: 583 Cost: 22.2068\n", 1147 | "Epoch: 584 Cost: 22.206\n", 1148 | "Epoch: 585 Cost: 22.2052\n", 1149 | "Epoch: 586 Cost: 22.2045\n", 1150 | "Epoch: 587 Cost: 22.2037\n", 1151 | "Epoch: 588 Cost: 22.203\n", 1152 | "Epoch: 589 Cost: 22.2023\n", 1153 | "Epoch: 590 Cost: 22.2015\n", 1154 | "Epoch: 591 Cost: 22.2008\n", 1155 | "Epoch: 592 Cost: 22.2001\n", 1156 | "Epoch: 593 Cost: 22.1993\n", 1157 | "Epoch: 594 Cost: 22.1986\n", 1158 | "Epoch: 595 Cost: 22.1979\n", 1159 | "Epoch: 596 Cost: 22.1972\n", 1160 | "Epoch: 597 Cost: 22.1965\n", 1161 | "Epoch: 598 Cost: 22.1958\n", 1162 | "Epoch: 599 Cost: 22.1951\n", 1163 | "Epoch: 600 Cost: 22.1944\n", 1164 | "Epoch: 601 Cost: 22.1937\n", 1165 | "Epoch: 602 Cost: 22.193\n", 1166 | "Epoch: 603 Cost: 22.1923\n", 1167 | "Epoch: 604 Cost: 22.1916\n", 1168 | "Epoch: 605 Cost: 22.1909\n", 1169 | "Epoch: 606 Cost: 22.1902\n", 1170 | "Epoch: 607 Cost: 22.1896\n", 1171 | "Epoch: 608 Cost: 22.1889\n", 1172 | "Epoch: 609 Cost: 22.1882\n", 1173 | "Epoch: 610 Cost: 22.1876\n", 1174 | "Epoch: 611 Cost: 22.1869\n", 1175 | "Epoch: 612 Cost: 22.1862\n", 1176 | "Epoch: 613 Cost: 22.1856\n", 1177 | "Epoch: 614 Cost: 22.1849\n", 1178 | "Epoch: 615 Cost: 22.1843\n", 1179 | "Epoch: 616 Cost: 22.1836\n", 1180 | "Epoch: 617 Cost: 22.183\n", 1181 | "Epoch: 618 Cost: 22.1824\n", 1182 | "Epoch: 619 Cost: 22.1817\n", 1183 | "Epoch: 620 Cost: 22.1811\n", 1184 | "Epoch: 621 Cost: 22.1805\n", 1185 | "Epoch: 622 Cost: 22.1799\n", 1186 | "Epoch: 623 Cost: 22.1792\n", 1187 | "Epoch: 624 Cost: 22.1786\n", 1188 | "Epoch: 625 Cost: 22.178\n", 1189 | "Epoch: 626 Cost: 22.1774\n", 1190 | "Epoch: 627 Cost: 22.1768\n", 1191 | "Epoch: 628 Cost: 22.1762\n", 1192 | "Epoch: 629 Cost: 22.1756\n", 1193 | "Epoch: 630 Cost: 22.175\n", 1194 | "Epoch: 631 Cost: 22.1744\n", 1195 | "Epoch: 632 Cost: 22.1738\n", 1196 | "Epoch: 633 Cost: 22.1732\n", 1197 | "Epoch: 634 Cost: 22.1726\n", 1198 | "Epoch: 635 Cost: 22.172\n", 1199 | "Epoch: 636 Cost: 22.1714\n", 1200 | "Epoch: 637 Cost: 22.1709\n", 1201 | "Epoch: 638 Cost: 22.1703\n", 1202 | "Epoch: 639 Cost: 22.1697\n", 1203 | "Epoch: 640 Cost: 22.1691\n", 1204 | "Epoch: 641 Cost: 22.1686\n", 1205 | "Epoch: 642 Cost: 22.168\n", 1206 | "Epoch: 643 Cost: 22.1674\n", 1207 | "Epoch: 644 Cost: 22.1669\n", 1208 | "Epoch: 645 Cost: 22.1663\n", 1209 | "Epoch: 646 Cost: 22.1658\n", 1210 | "Epoch: 647 Cost: 22.1652\n", 1211 | "Epoch: 648 Cost: 22.1647\n", 1212 | "Epoch: 649 Cost: 22.1641\n", 1213 | "Epoch: 650 Cost: 22.1636\n", 1214 | "Epoch: 651 Cost: 22.1631\n", 1215 | "Epoch: 652 Cost: 22.1625\n", 1216 | "Epoch: 653 Cost: 22.162\n", 1217 | "Epoch: 654 Cost: 22.1615\n", 1218 | "Epoch: 655 Cost: 22.1609\n", 1219 | "Epoch: 656 Cost: 22.1604\n", 1220 | "Epoch: 657 Cost: 22.1599\n", 1221 | "Epoch: 658 Cost: 22.1594\n", 1222 | "Epoch: 659 Cost: 22.1588\n", 1223 | "Epoch: 660 Cost: 22.1583\n", 1224 | "Epoch: 661 Cost: 22.1578\n", 1225 | "Epoch: 662 Cost: 22.1573\n", 1226 | "Epoch: 663 Cost: 22.1568\n", 1227 | "Epoch: 664 Cost: 22.1563\n", 1228 | "Epoch: 665 Cost: 22.1558\n", 1229 | "Epoch: 666 Cost: 22.1553\n", 1230 | "Epoch: 667 Cost: 22.1548\n", 1231 | "Epoch: 668 Cost: 22.1543\n", 1232 | "Epoch: 669 Cost: 22.1538\n", 1233 | "Epoch: 670 Cost: 22.1533\n", 1234 | "Epoch: 671 Cost: 22.1528\n", 1235 | "Epoch: 672 Cost: 22.1523\n", 1236 | "Epoch: 673 Cost: 22.1518\n", 1237 | "Epoch: 674 Cost: 22.1514\n", 1238 | "Epoch: 675 Cost: 22.1509\n", 1239 | "Epoch: 676 Cost: 22.1504\n", 1240 | "Epoch: 677 Cost: 22.1499\n", 1241 | "Epoch: 678 Cost: 22.1495\n", 1242 | "Epoch: 679 Cost: 22.149\n", 1243 | "Epoch: 680 Cost: 22.1485\n", 1244 | "Epoch: 681 Cost: 22.148\n", 1245 | "Epoch: 682 Cost: 22.1476\n", 1246 | "Epoch: 683 Cost: 22.1471\n", 1247 | "Epoch: 684 Cost: 22.1467\n", 1248 | "Epoch: 685 Cost: 22.1462\n", 1249 | "Epoch: 686 Cost: 22.1458\n", 1250 | "Epoch: 687 Cost: 22.1453\n", 1251 | "Epoch: 688 Cost: 22.1448\n", 1252 | "Epoch: 689 Cost: 22.1444\n", 1253 | "Epoch: 690 Cost: 22.1439\n", 1254 | "Epoch: 691 Cost: 22.1435\n", 1255 | "Epoch: 692 Cost: 22.1431\n", 1256 | "Epoch: 693 Cost: 22.1426\n", 1257 | "Epoch: 694 Cost: 22.1422\n", 1258 | "Epoch: 695 Cost: 22.1417\n", 1259 | "Epoch: 696 Cost: 22.1413\n", 1260 | "Epoch: 697 Cost: 22.1409\n", 1261 | "Epoch: 698 Cost: 22.1405\n", 1262 | "Epoch: 699 Cost: 22.14\n", 1263 | "Epoch: 700 Cost: 22.1396\n", 1264 | "Epoch: 701 Cost: 22.1392\n", 1265 | "Epoch: 702 Cost: 22.1387\n", 1266 | "Epoch: 703 Cost: 22.1383\n", 1267 | "Epoch: 704 Cost: 22.1379\n", 1268 | "Epoch: 705 Cost: 22.1375\n", 1269 | "Epoch: 706 Cost: 22.1371\n", 1270 | "Epoch: 707 Cost: 22.1367\n", 1271 | "Epoch: 708 Cost: 22.1363\n", 1272 | "Epoch: 709 Cost: 22.1359\n", 1273 | "Epoch: 710 Cost: 22.1354\n", 1274 | "Epoch: 711 Cost: 22.135\n", 1275 | "Epoch: 712 Cost: 22.1346\n", 1276 | "Epoch: 713 Cost: 22.1342\n", 1277 | "Epoch: 714 Cost: 22.1338\n", 1278 | "Epoch: 715 Cost: 22.1334\n", 1279 | "Epoch: 716 Cost: 22.133\n", 1280 | "Epoch: 717 Cost: 22.1326\n", 1281 | "Epoch: 718 Cost: 22.1323\n", 1282 | "Epoch: 719 Cost: 22.1319\n", 1283 | "Epoch: 720 Cost: 22.1315\n", 1284 | "Epoch: 721 Cost: 22.1311\n", 1285 | "Epoch: 722 Cost: 22.1307\n", 1286 | "Epoch: 723 Cost: 22.1303\n", 1287 | "Epoch: 724 Cost: 22.1299\n", 1288 | "Epoch: 725 Cost: 22.1296\n", 1289 | "Epoch: 726 Cost: 22.1292\n", 1290 | "Epoch: 727 Cost: 22.1288\n", 1291 | "Epoch: 728 Cost: 22.1284\n", 1292 | "Epoch: 729 Cost: 22.1281\n", 1293 | "Epoch: 730 Cost: 22.1277\n", 1294 | "Epoch: 731 Cost: 22.1273\n", 1295 | "Epoch: 732 Cost: 22.1269\n", 1296 | "Epoch: 733 Cost: 22.1266\n", 1297 | "Epoch: 734 Cost: 22.1262\n", 1298 | "Epoch: 735 Cost: 22.1259\n", 1299 | "Epoch: 736 Cost: 22.1255\n", 1300 | "Epoch: 737 Cost: 22.1251\n", 1301 | "Epoch: 738 Cost: 22.1248\n", 1302 | "Epoch: 739 Cost: 22.1244\n", 1303 | "Epoch: 740 Cost: 22.1241\n", 1304 | "Epoch: 741 Cost: 22.1237\n", 1305 | "Epoch: 742 Cost: 22.1234\n", 1306 | "Epoch: 743 Cost: 22.123\n", 1307 | "Epoch: 744 Cost: 22.1227\n", 1308 | "Epoch: 745 Cost: 22.1223\n", 1309 | "Epoch: 746 Cost: 22.122\n", 1310 | "Epoch: 747 Cost: 22.1216\n", 1311 | "Epoch: 748 Cost: 22.1213\n", 1312 | "Epoch: 749 Cost: 22.1209\n", 1313 | "Epoch: 750 Cost: 22.1206\n", 1314 | "Epoch: 751 Cost: 22.1203\n", 1315 | "Epoch: 752 Cost: 22.1199\n", 1316 | "Epoch: 753 Cost: 22.1196\n", 1317 | "Epoch: 754 Cost: 22.1193\n", 1318 | "Epoch: 755 Cost: 22.1189\n", 1319 | "Epoch: 756 Cost: 22.1186\n", 1320 | "Epoch: 757 Cost: 22.1183\n", 1321 | "Epoch: 758 Cost: 22.1179\n", 1322 | "Epoch: 759 Cost: 22.1176\n", 1323 | "Epoch: 760 Cost: 22.1173\n", 1324 | "Epoch: 761 Cost: 22.117\n", 1325 | "Epoch: 762 Cost: 22.1166\n", 1326 | "Epoch: 763 Cost: 22.1163\n", 1327 | "Epoch: 764 Cost: 22.116\n", 1328 | "Epoch: 765 Cost: 22.1157\n", 1329 | "Epoch: 766 Cost: 22.1154\n", 1330 | "Epoch: 767 Cost: 22.1151\n", 1331 | "Epoch: 768 Cost: 22.1147\n", 1332 | "Epoch: 769 Cost: 22.1144\n", 1333 | "Epoch: 770 Cost: 22.1141\n", 1334 | "Epoch: 771 Cost: 22.1138\n", 1335 | "Epoch: 772 Cost: 22.1135\n", 1336 | "Epoch: 773 Cost: 22.1132\n", 1337 | "Epoch: 774 Cost: 22.1129\n", 1338 | "Epoch: 775 Cost: 22.1126\n", 1339 | "Epoch: 776 Cost: 22.1123\n", 1340 | "Epoch: 777 Cost: 22.112\n", 1341 | "Epoch: 778 Cost: 22.1117\n", 1342 | "Epoch: 779 Cost: 22.1114\n", 1343 | "Epoch: 780 Cost: 22.1111\n", 1344 | "Epoch: 781 Cost: 22.1108\n", 1345 | "Epoch: 782 Cost: 22.1105\n", 1346 | "Epoch: 783 Cost: 22.1102\n", 1347 | "Epoch: 784 Cost: 22.1099\n", 1348 | "Epoch: 785 Cost: 22.1096\n", 1349 | "Epoch: 786 Cost: 22.1093\n", 1350 | "Epoch: 787 Cost: 22.109\n", 1351 | "Epoch: 788 Cost: 22.1087\n" 1352 | ] 1353 | }, 1354 | { 1355 | "name": "stdout", 1356 | "output_type": "stream", 1357 | "text": [ 1358 | "Epoch: 789 Cost: 22.1084\n", 1359 | "Epoch: 790 Cost: 22.1082\n", 1360 | "Epoch: 791 Cost: 22.1079\n", 1361 | "Epoch: 792 Cost: 22.1076\n", 1362 | "Epoch: 793 Cost: 22.1073\n", 1363 | "Epoch: 794 Cost: 22.107\n", 1364 | "Epoch: 795 Cost: 22.1067\n", 1365 | "Epoch: 796 Cost: 22.1065\n", 1366 | "Epoch: 797 Cost: 22.1062\n", 1367 | "Epoch: 798 Cost: 22.1059\n", 1368 | "Epoch: 799 Cost: 22.1056\n", 1369 | "Epoch: 800 Cost: 22.1054\n", 1370 | "Epoch: 801 Cost: 22.1051\n", 1371 | "Epoch: 802 Cost: 22.1048\n", 1372 | "Epoch: 803 Cost: 22.1045\n", 1373 | "Epoch: 804 Cost: 22.1043\n", 1374 | "Epoch: 805 Cost: 22.104\n", 1375 | "Epoch: 806 Cost: 22.1037\n", 1376 | "Epoch: 807 Cost: 22.1035\n", 1377 | "Epoch: 808 Cost: 22.1032\n", 1378 | "Epoch: 809 Cost: 22.1029\n", 1379 | "Epoch: 810 Cost: 22.1027\n", 1380 | "Epoch: 811 Cost: 22.1024\n", 1381 | "Epoch: 812 Cost: 22.1022\n", 1382 | "Epoch: 813 Cost: 22.1019\n", 1383 | "Epoch: 814 Cost: 22.1016\n", 1384 | "Epoch: 815 Cost: 22.1014\n", 1385 | "Epoch: 816 Cost: 22.1011\n", 1386 | "Epoch: 817 Cost: 22.1009\n", 1387 | "Epoch: 818 Cost: 22.1006\n", 1388 | "Epoch: 819 Cost: 22.1004\n", 1389 | "Epoch: 820 Cost: 22.1001\n", 1390 | "Epoch: 821 Cost: 22.0999\n", 1391 | "Epoch: 822 Cost: 22.0996\n", 1392 | "Epoch: 823 Cost: 22.0994\n", 1393 | "Epoch: 824 Cost: 22.0991\n", 1394 | "Epoch: 825 Cost: 22.0989\n", 1395 | "Epoch: 826 Cost: 22.0986\n", 1396 | "Epoch: 827 Cost: 22.0984\n", 1397 | "Epoch: 828 Cost: 22.0981\n", 1398 | "Epoch: 829 Cost: 22.0979\n", 1399 | "Epoch: 830 Cost: 22.0976\n", 1400 | "Epoch: 831 Cost: 22.0974\n", 1401 | "Epoch: 832 Cost: 22.0972\n", 1402 | "Epoch: 833 Cost: 22.0969\n", 1403 | "Epoch: 834 Cost: 22.0967\n", 1404 | "Epoch: 835 Cost: 22.0964\n", 1405 | "Epoch: 836 Cost: 22.0962\n", 1406 | "Epoch: 837 Cost: 22.096\n", 1407 | "Epoch: 838 Cost: 22.0957\n", 1408 | "Epoch: 839 Cost: 22.0955\n", 1409 | "Epoch: 840 Cost: 22.0953\n", 1410 | "Epoch: 841 Cost: 22.095\n", 1411 | "Epoch: 842 Cost: 22.0948\n", 1412 | "Epoch: 843 Cost: 22.0946\n", 1413 | "Epoch: 844 Cost: 22.0943\n", 1414 | "Epoch: 845 Cost: 22.0941\n", 1415 | "Epoch: 846 Cost: 22.0939\n", 1416 | "Epoch: 847 Cost: 22.0937\n", 1417 | "Epoch: 848 Cost: 22.0934\n", 1418 | "Epoch: 849 Cost: 22.0932\n", 1419 | "Epoch: 850 Cost: 22.093\n", 1420 | "Epoch: 851 Cost: 22.0928\n", 1421 | "Epoch: 852 Cost: 22.0925\n", 1422 | "Epoch: 853 Cost: 22.0923\n", 1423 | "Epoch: 854 Cost: 22.0921\n", 1424 | "Epoch: 855 Cost: 22.0919\n", 1425 | "Epoch: 856 Cost: 22.0917\n", 1426 | "Epoch: 857 Cost: 22.0914\n", 1427 | "Epoch: 858 Cost: 22.0912\n", 1428 | "Epoch: 859 Cost: 22.091\n", 1429 | "Epoch: 860 Cost: 22.0908\n", 1430 | "Epoch: 861 Cost: 22.0906\n", 1431 | "Epoch: 862 Cost: 22.0904\n", 1432 | "Epoch: 863 Cost: 22.0901\n", 1433 | "Epoch: 864 Cost: 22.0899\n", 1434 | "Epoch: 865 Cost: 22.0897\n", 1435 | "Epoch: 866 Cost: 22.0895\n", 1436 | "Epoch: 867 Cost: 22.0893\n", 1437 | "Epoch: 868 Cost: 22.0891\n", 1438 | "Epoch: 869 Cost: 22.0889\n", 1439 | "Epoch: 870 Cost: 22.0887\n", 1440 | "Epoch: 871 Cost: 22.0885\n", 1441 | "Epoch: 872 Cost: 22.0883\n", 1442 | "Epoch: 873 Cost: 22.0881\n", 1443 | "Epoch: 874 Cost: 22.0879\n", 1444 | "Epoch: 875 Cost: 22.0877\n", 1445 | "Epoch: 876 Cost: 22.0874\n", 1446 | "Epoch: 877 Cost: 22.0872\n", 1447 | "Epoch: 878 Cost: 22.087\n", 1448 | "Epoch: 879 Cost: 22.0868\n", 1449 | "Epoch: 880 Cost: 22.0866\n", 1450 | "Epoch: 881 Cost: 22.0864\n", 1451 | "Epoch: 882 Cost: 22.0862\n", 1452 | "Epoch: 883 Cost: 22.0861\n", 1453 | "Epoch: 884 Cost: 22.0859\n", 1454 | "Epoch: 885 Cost: 22.0857\n", 1455 | "Epoch: 886 Cost: 22.0855\n", 1456 | "Epoch: 887 Cost: 22.0853\n", 1457 | "Epoch: 888 Cost: 22.0851\n", 1458 | "Epoch: 889 Cost: 22.0849\n", 1459 | "Epoch: 890 Cost: 22.0847\n", 1460 | "Epoch: 891 Cost: 22.0845\n", 1461 | "Epoch: 892 Cost: 22.0843\n", 1462 | "Epoch: 893 Cost: 22.0841\n", 1463 | "Epoch: 894 Cost: 22.0839\n", 1464 | "Epoch: 895 Cost: 22.0837\n", 1465 | "Epoch: 896 Cost: 22.0835\n", 1466 | "Epoch: 897 Cost: 22.0834\n", 1467 | "Epoch: 898 Cost: 22.0832\n", 1468 | "Epoch: 899 Cost: 22.083\n", 1469 | "Epoch: 900 Cost: 22.0828\n", 1470 | "Epoch: 901 Cost: 22.0826\n", 1471 | "Epoch: 902 Cost: 22.0824\n", 1472 | "Epoch: 903 Cost: 22.0823\n", 1473 | "Epoch: 904 Cost: 22.0821\n", 1474 | "Epoch: 905 Cost: 22.0819\n", 1475 | "Epoch: 906 Cost: 22.0817\n", 1476 | "Epoch: 907 Cost: 22.0815\n", 1477 | "Epoch: 908 Cost: 22.0813\n", 1478 | "Epoch: 909 Cost: 22.0812\n", 1479 | "Epoch: 910 Cost: 22.081\n", 1480 | "Epoch: 911 Cost: 22.0808\n", 1481 | "Epoch: 912 Cost: 22.0806\n", 1482 | "Epoch: 913 Cost: 22.0805\n", 1483 | "Epoch: 914 Cost: 22.0803\n", 1484 | "Epoch: 915 Cost: 22.0801\n", 1485 | "Epoch: 916 Cost: 22.0799\n", 1486 | "Epoch: 917 Cost: 22.0798\n", 1487 | "Epoch: 918 Cost: 22.0796\n", 1488 | "Epoch: 919 Cost: 22.0794\n", 1489 | "Epoch: 920 Cost: 22.0792\n", 1490 | "Epoch: 921 Cost: 22.0791\n", 1491 | "Epoch: 922 Cost: 22.0789\n", 1492 | "Epoch: 923 Cost: 22.0787\n", 1493 | "Epoch: 924 Cost: 22.0786\n", 1494 | "Epoch: 925 Cost: 22.0784\n", 1495 | "Epoch: 926 Cost: 22.0782\n", 1496 | "Epoch: 927 Cost: 22.078\n", 1497 | "Epoch: 928 Cost: 22.0779\n", 1498 | "Epoch: 929 Cost: 22.0777\n", 1499 | "Epoch: 930 Cost: 22.0775\n", 1500 | "Epoch: 931 Cost: 22.0774\n", 1501 | "Epoch: 932 Cost: 22.0772\n", 1502 | "Epoch: 933 Cost: 22.077\n", 1503 | "Epoch: 934 Cost: 22.0769\n", 1504 | "Epoch: 935 Cost: 22.0767\n", 1505 | "Epoch: 936 Cost: 22.0766\n", 1506 | "Epoch: 937 Cost: 22.0764\n", 1507 | "Epoch: 938 Cost: 22.0762\n", 1508 | "Epoch: 939 Cost: 22.0761\n", 1509 | "Epoch: 940 Cost: 22.0759\n", 1510 | "Epoch: 941 Cost: 22.0758\n", 1511 | "Epoch: 942 Cost: 22.0756\n", 1512 | "Epoch: 943 Cost: 22.0754\n", 1513 | "Epoch: 944 Cost: 22.0753\n", 1514 | "Epoch: 945 Cost: 22.0751\n", 1515 | "Epoch: 946 Cost: 22.075\n", 1516 | "Epoch: 947 Cost: 22.0748\n", 1517 | "Epoch: 948 Cost: 22.0747\n", 1518 | "Epoch: 949 Cost: 22.0745\n", 1519 | "Epoch: 950 Cost: 22.0743\n", 1520 | "Epoch: 951 Cost: 22.0742\n", 1521 | "Epoch: 952 Cost: 22.074\n", 1522 | "Epoch: 953 Cost: 22.0739\n", 1523 | "Epoch: 954 Cost: 22.0737\n", 1524 | "Epoch: 955 Cost: 22.0736\n", 1525 | "Epoch: 956 Cost: 22.0734\n", 1526 | "Epoch: 957 Cost: 22.0733\n", 1527 | "Epoch: 958 Cost: 22.0731\n", 1528 | "Epoch: 959 Cost: 22.073\n", 1529 | "Epoch: 960 Cost: 22.0728\n", 1530 | "Epoch: 961 Cost: 22.0727\n", 1531 | "Epoch: 962 Cost: 22.0725\n", 1532 | "Epoch: 963 Cost: 22.0724\n", 1533 | "Epoch: 964 Cost: 22.0722\n", 1534 | "Epoch: 965 Cost: 22.0721\n", 1535 | "Epoch: 966 Cost: 22.0719\n", 1536 | "Epoch: 967 Cost: 22.0718\n", 1537 | "Epoch: 968 Cost: 22.0717\n", 1538 | "Epoch: 969 Cost: 22.0715\n", 1539 | "Epoch: 970 Cost: 22.0714\n", 1540 | "Epoch: 971 Cost: 22.0712\n", 1541 | "Epoch: 972 Cost: 22.0711\n", 1542 | "Epoch: 973 Cost: 22.0709\n", 1543 | "Epoch: 974 Cost: 22.0708\n", 1544 | "Epoch: 975 Cost: 22.0707\n", 1545 | "Epoch: 976 Cost: 22.0705\n", 1546 | "Epoch: 977 Cost: 22.0704\n", 1547 | "Epoch: 978 Cost: 22.0702\n", 1548 | "Epoch: 979 Cost: 22.0701\n", 1549 | "Epoch: 980 Cost: 22.0699\n", 1550 | "Epoch: 981 Cost: 22.0698\n", 1551 | "Epoch: 982 Cost: 22.0697\n", 1552 | "Epoch: 983 Cost: 22.0695\n", 1553 | "Epoch: 984 Cost: 22.0694\n", 1554 | "Epoch: 985 Cost: 22.0693\n", 1555 | "Epoch: 986 Cost: 22.0691\n", 1556 | "Epoch: 987 Cost: 22.069\n", 1557 | "Epoch: 988 Cost: 22.0688\n", 1558 | "Epoch: 989 Cost: 22.0687\n", 1559 | "Epoch: 990 Cost: 22.0686\n", 1560 | "Epoch: 991 Cost: 22.0684\n", 1561 | "Epoch: 992 Cost: 22.0683\n", 1562 | "Epoch: 993 Cost: 22.0682\n", 1563 | "Epoch: 994 Cost: 22.068\n", 1564 | "Epoch: 995 Cost: 22.0679\n", 1565 | "Epoch: 996 Cost: 22.0678\n", 1566 | "Epoch: 997 Cost: 22.0676\n", 1567 | "Epoch: 998 Cost: 22.0675\n", 1568 | "Epoch: 999 Cost: 22.0674\n", 1569 | "Test Loss: 22.3642\n" 1570 | ] 1571 | } 1572 | ], 1573 | "source": [ 1574 | "num_epochs = 1000\n", 1575 | "losses = []\n", 1576 | "with tf.Session() as sess:\n", 1577 | " writer = tf.summary.FileWriter('log/linear', sess.graph)\n", 1578 | " sess.run(tf.global_variables_initializer())\n", 1579 | " for epoch in range(num_epochs):\n", 1580 | " _, c, summary = sess.run([update, loss, summary_op], feed_dict={X: X_train, \n", 1581 | " Y: y_train})\n", 1582 | " losses.append(c)\n", 1583 | " print(\"Epoch:\",epoch,\"Cost:\",c)\n", 1584 | " writer.add_summary(summary, global_step=epoch)\n", 1585 | " test_loss = sess.run(loss, feed_dict={X: X_test, \n", 1586 | " Y: y_test})\n", 1587 | " print(\"Test Loss:\",test_loss)" 1588 | ] 1589 | }, 1590 | { 1591 | "cell_type": "code", 1592 | "execution_count": null, 1593 | "metadata": {}, 1594 | "outputs": [], 1595 | "source": [] 1596 | } 1597 | ], 1598 | "metadata": { 1599 | "kernelspec": { 1600 | "display_name": "Python [conda env:tensorflow_workshop]", 1601 | "language": "python", 1602 | "name": "conda-env-tensorflow_workshop-py" 1603 | }, 1604 | "language_info": { 1605 | "codemirror_mode": { 1606 | "name": "ipython", 1607 | "version": 3 1608 | }, 1609 | "file_extension": ".py", 1610 | "mimetype": "text/x-python", 1611 | "name": "python", 1612 | "nbconvert_exporter": "python", 1613 | "pygments_lexer": "ipython3", 1614 | "version": "3.6.4" 1615 | } 1616 | }, 1617 | "nbformat": 4, 1618 | "nbformat_minor": 2 1619 | } 1620 | -------------------------------------------------------------------------------- /8_Multi_Layer_Perceptron.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 8. Multi-Layer Perceptron\n", 8 | "We introduce a multi-layer perceptron, aka a 3 layer fully connected neural network. We do this using the MNIST data once again. We first write the model as we were doing before. Next, we will clean it up a bit using more TensorFlow API. Finally, we will show how we can cleanly organize the functions using `tf.Estimaor`.\n", 9 | "\n", 10 | "## Preparation" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 12, 16 | "metadata": {}, 17 | "outputs": [ 18 | { 19 | "name": "stdout", 20 | "output_type": "stream", 21 | "text": [ 22 | "Extracting MNIST_data/train-images-idx3-ubyte.gz\n", 23 | "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n", 24 | "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n", 25 | "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n" 26 | ] 27 | } 28 | ], 29 | "source": [ 30 | "import tensorflow as tf\n", 31 | "import matplotlib.pyplot as plt\n", 32 | "import numpy as np\n", 33 | "\n", 34 | "# Import MINST data\n", 35 | "from tensorflow.examples.tutorials.mnist import input_data\n", 36 | "mnist = input_data.read_data_sets('MNIST_data', one_hot=True)" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 13, 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "num_features = mnist.train.images.shape[1]\n", 46 | "num_classes = mnist.train.labels.shape[1]\n", 47 | "num_hidden_1 = 256\n", 48 | "num_hidden_2 = 256" 49 | ] 50 | }, 51 | { 52 | "cell_type": "markdown", 53 | "metadata": {}, 54 | "source": [ 55 | "## Version 1" 56 | ] 57 | }, 58 | { 59 | "cell_type": "markdown", 60 | "metadata": {}, 61 | "source": [ 62 | "Let's build a three layer fully connected neural network. It's actually very straightforward.\n", 63 | "We have two hidden layers and one output layer. Two hidden layers have a activation function (chosen to be ReLU). " 64 | ] 65 | }, 66 | { 67 | "cell_type": "code", 68 | "execution_count": 14, 69 | "metadata": {}, 70 | "outputs": [], 71 | "source": [ 72 | "tf.reset_default_graph() # Clearing all tensors before this" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 15, 78 | "metadata": {}, 79 | "outputs": [], 80 | "source": [ 81 | "with tf.name_scope('data'):\n", 82 | " X = tf.placeholder(tf.float32, shape=[None, num_features], name='Input-Images')\n", 83 | " Y = tf.placeholder(tf.float32, shape=[None, num_classes], name='Output-Labels')" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": 16, 89 | "metadata": {}, 90 | "outputs": [], 91 | "source": [ 92 | "with tf.name_scope('fc1'): # first hidden layer variables\n", 93 | " W1 = tf.Variable(tf.random_normal([num_features, num_hidden_1]),name='weights')\n", 94 | " b1 = tf.Variable(tf.random_normal([num_hidden_1]),name='bias')\n", 95 | "\n", 96 | "with tf.name_scope('fc2'): # second hidden layer variables\n", 97 | " W2 = tf.Variable(tf.random_normal([num_hidden_1, num_hidden_2]),name='weights')\n", 98 | " b2 = tf.Variable(tf.random_normal([num_hidden_2]),name='bias')\n", 99 | "\n", 100 | "with tf.name_scope('out'): # output layer variables\n", 101 | " Wout = tf.Variable(tf.random_normal([num_hidden_2, num_classes]),name='weights')\n", 102 | " bout = tf.Variable(tf.random_normal([num_classes]),name='bias')" 103 | ] 104 | }, 105 | { 106 | "cell_type": "code", 107 | "execution_count": 18, 108 | "metadata": {}, 109 | "outputs": [], 110 | "source": [ 111 | "with tf.name_scope('multilayer_perceptron'): # 3 layer fully connected network\n", 112 | " H1 = tf.nn.relu(X @ W1 + b1, name='H1')\n", 113 | " H2 = tf.nn.relu(H1 @ W2 + b2, name='H2')\n", 114 | " logits = tf.add(H2 @ Wout, bout, name='out')" 115 | ] 116 | }, 117 | { 118 | "cell_type": "markdown", 119 | "metadata": {}, 120 | "source": [ 121 | "The final layer outputs a `logit`, which is just the output of the neural network before `softmax` is applied to make it a probability. This is because TensorFlow has a convenient API `tf.nn.softmax_cross_entropy_with_logits` that applies `softmax` to `logit`, and then computes a `cross_entopy` on each element. So, all we have to do is to sum up those cross entropies to reduce it to a single loss." 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": 19, 127 | "metadata": {}, 128 | "outputs": [], 129 | "source": [ 130 | "with tf.name_scope('loss'):\n", 131 | " loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n", 132 | " logits=logits, labels=Y),name='loss')\n", 133 | " correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(Y, 1))\n", 134 | " accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) \n", 135 | " \n", 136 | "with tf.name_scope('optimizer'):\n", 137 | " learning_rate = 0.01\n", 138 | " optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\n", 139 | " update = optimizer.minimize(loss)\n", 140 | "\n", 141 | "with tf.name_scope('summaries'):\n", 142 | " tf.summary.scalar('loss', loss)\n", 143 | " tf.summary.histogram('histogram-loss', loss)\n", 144 | " summary_op = tf.summary.merge_all()" 145 | ] 146 | }, 147 | { 148 | "cell_type": "markdown", 149 | "metadata": {}, 150 | "source": [ 151 | "We can train the model just like we did before. The accuracy doesn't seem to improve much (with more epochs, it should)." 152 | ] 153 | }, 154 | { 155 | "cell_type": "code", 156 | "execution_count": 20, 157 | "metadata": {}, 158 | "outputs": [ 159 | { 160 | "name": "stdout", 161 | "output_type": "stream", 162 | "text": [ 163 | "Epoch: 0 Cost: 62.5973903366\n", 164 | "Epoch: 1 Cost: 14.0762353808\n", 165 | "Epoch: 2 Cost: 8.52743232933\n", 166 | "Epoch: 3 Cost: 6.08494931553\n", 167 | "Epoch: 4 Cost: 4.57107012645\n", 168 | "Epoch: 5 Cost: 3.62662561966\n", 169 | "Epoch: 6 Cost: 2.90314452117\n", 170 | "Epoch: 7 Cost: 2.3972983414\n", 171 | "Epoch: 8 Cost: 1.99353022361\n", 172 | "Epoch: 9 Cost: 1.68832156922\n", 173 | "Epoch: 10 Cost: 1.41917506241\n", 174 | "Epoch: 11 Cost: 1.22195894537\n", 175 | "Epoch: 12 Cost: 1.05779955423\n", 176 | "Epoch: 13 Cost: 0.908503072511\n", 177 | "Epoch: 14 Cost: 0.793050222413\n", 178 | "Epoch: 15 Cost: 0.678756705873\n", 179 | "Epoch: 16 Cost: 0.602186917308\n", 180 | "Epoch: 17 Cost: 0.502267919275\n", 181 | "Epoch: 18 Cost: 0.456717148443\n", 182 | "Epoch: 19 Cost: 0.400008908998\n", 183 | "Epoch: 20 Cost: 0.355178469353\n", 184 | "Epoch: 21 Cost: 0.32593929646\n", 185 | "Epoch: 22 Cost: 0.273985019122\n", 186 | "Epoch: 23 Cost: 0.239091789515\n", 187 | "Epoch: 24 Cost: 0.217190507096\n", 188 | "Test Accuracy: 0.9306\n" 189 | ] 190 | } 191 | ], 192 | "source": [ 193 | "# Train\n", 194 | "num_epochs = 25\n", 195 | "batch_size = 100\n", 196 | "\n", 197 | "with tf.Session() as sess:\n", 198 | " writer = tf.summary.FileWriter('log/multilayer_perceptron1', sess.graph)\n", 199 | " \n", 200 | " sess.run(tf.global_variables_initializer())\n", 201 | " total_batch = int(mnist.train.num_examples/batch_size)\n", 202 | " for epoch in range(num_epochs):\n", 203 | " average_cost = 0\n", 204 | " for batch in range(total_batch):\n", 205 | " batch_X, batch_Y = mnist.train.next_batch(batch_size)\n", 206 | " _, c = sess.run([update, loss], feed_dict={X: batch_X,\n", 207 | " Y: batch_Y})\n", 208 | " average_cost += c / total_batch\n", 209 | " summary = sess.run(summary_op, feed_dict={X: batch_X,\n", 210 | " Y: batch_Y})\n", 211 | " global_step = epoch*total_batch + batch\n", 212 | " writer.add_summary(summary, global_step=global_step)\n", 213 | " print(\"Epoch:\",epoch,\"Cost:\",average_cost)\n", 214 | " \n", 215 | " print(\"Test Accuracy:\", accuracy.eval({X: mnist.test.images, Y: mnist.test.labels})) \n", 216 | " writer.close()" 217 | ] 218 | }, 219 | { 220 | "cell_type": "markdown", 221 | "metadata": {}, 222 | "source": [ 223 | "## Version 2" 224 | ] 225 | }, 226 | { 227 | "cell_type": "markdown", 228 | "metadata": {}, 229 | "source": [ 230 | "The only differense in this version is that each layer is replaced with a single API `tf.layers.dense`. The convenient thing about this is that 1) it's clean and 2) you don't need to initialize weight and bias variables yourself. The rest is the same as before." 231 | ] 232 | }, 233 | { 234 | "cell_type": "code", 235 | "execution_count": 92, 236 | "metadata": {}, 237 | "outputs": [], 238 | "source": [ 239 | "tf.reset_default_graph() # Clearing all tensors before this" 240 | ] 241 | }, 242 | { 243 | "cell_type": "code", 244 | "execution_count": 93, 245 | "metadata": {}, 246 | "outputs": [], 247 | "source": [ 248 | "with tf.name_scope('data'):\n", 249 | " X = tf.placeholder(tf.float32, shape=[None, num_features], name='Input-Images')\n", 250 | " Y = tf.placeholder(tf.float32, shape=[None, num_classes], name='Output-Labels')" 251 | ] 252 | }, 253 | { 254 | "cell_type": "code", 255 | "execution_count": 94, 256 | "metadata": {}, 257 | "outputs": [], 258 | "source": [ 259 | "with tf.name_scope('multilayer_perceptron'): # Now, it is only three lines\n", 260 | " # Hidden fully connected layer with 256 neurons\n", 261 | " layer_1 = tf.layers.dense(X, num_hidden_1, tf.nn.relu, name='fc1')\n", 262 | " # Hidden fully connected layer with 256 neurons\n", 263 | " layer_2 = tf.layers.dense(layer_1, num_hidden_2, tf.nn.relu, name='fc2')\n", 264 | " # Output fully connected layer with a neuron for each class\n", 265 | " logits = tf.layers.dense(layer_2, num_classes, name='out')" 266 | ] 267 | }, 268 | { 269 | "cell_type": "code", 270 | "execution_count": 95, 271 | "metadata": {}, 272 | "outputs": [], 273 | "source": [ 274 | "with tf.name_scope('loss'):\n", 275 | " loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n", 276 | " logits=logits, labels=Y),name='loss')\n", 277 | " correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(Y, 1))\n", 278 | " accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) \n", 279 | " \n", 280 | "with tf.name_scope('optimizer'):\n", 281 | " learning_rate = 0.01\n", 282 | " optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\n", 283 | " update = optimizer.minimize(loss)\n", 284 | "\n", 285 | "with tf.name_scope('summaries'):\n", 286 | " tf.summary.scalar('loss', loss)\n", 287 | " tf.summary.histogram('histogram-loss', loss)\n", 288 | " summary_op = tf.summary.merge_all()" 289 | ] 290 | }, 291 | { 292 | "cell_type": "code", 293 | "execution_count": 72, 294 | "metadata": {}, 295 | "outputs": [ 296 | { 297 | "name": "stdout", 298 | "output_type": "stream", 299 | "text": [ 300 | "Epoch: 0 Cost: 1.1429620449651363\n", 301 | "Epoch: 1 Cost: 0.44573915083299925\n", 302 | "Epoch: 2 Cost: 0.35664164621721617\n", 303 | "Epoch: 3 Cost: 0.31739128164269714\n", 304 | "Epoch: 4 Cost: 0.2910888133265754\n", 305 | "Epoch: 5 Cost: 0.2713912793858482\n", 306 | "Epoch: 6 Cost: 0.25498610958456974\n", 307 | "Epoch: 7 Cost: 0.2407954240658066\n", 308 | "Epoch: 8 Cost: 0.22825071882117884\n", 309 | "Epoch: 9 Cost: 0.21688424812121837\n", 310 | "Epoch: 10 Cost: 0.20677164849909882\n", 311 | "Epoch: 11 Cost: 0.19733745005320408\n", 312 | "Epoch: 12 Cost: 0.1889122671769422\n", 313 | "Epoch: 13 Cost: 0.18079517665234468\n", 314 | "Epoch: 14 Cost: 0.17358795196495266\n", 315 | "Epoch: 15 Cost: 0.16686928619037983\n", 316 | "Epoch: 16 Cost: 0.1604892960935832\n", 317 | "Epoch: 17 Cost: 0.15470593801953564\n", 318 | "Epoch: 18 Cost: 0.14925623104653593\n", 319 | "Epoch: 19 Cost: 0.14406697281382297\n", 320 | "Epoch: 20 Cost: 0.1391177260740237\n", 321 | "Epoch: 21 Cost: 0.1345416298576378\n", 322 | "Epoch: 22 Cost: 0.13013896662741917\n", 323 | "Epoch: 23 Cost: 0.12604238837618725\n", 324 | "Epoch: 24 Cost: 0.12211634944108385\n", 325 | "Test Accuracy: 0.9623\n" 326 | ] 327 | } 328 | ], 329 | "source": [ 330 | "# Train\n", 331 | "num_epochs = 25\n", 332 | "batch_size = 100\n", 333 | "\n", 334 | "with tf.Session() as sess:\n", 335 | " writer = tf.summary.FileWriter('log/multilayer_perceptron2', sess.graph)\n", 336 | " \n", 337 | " sess.run(tf.global_variables_initializer())\n", 338 | " total_batch = int(mnist.train.num_examples/batch_size)\n", 339 | " for epoch in range(num_epochs):\n", 340 | " average_cost = 0\n", 341 | " for batch in range(total_batch):\n", 342 | " batch_X, batch_Y = mnist.train.next_batch(batch_size)\n", 343 | " _, c = sess.run([update, loss], feed_dict={X: batch_X,\n", 344 | " Y: batch_Y})\n", 345 | " average_cost += c / total_batch\n", 346 | " summary = sess.run(summary_op, feed_dict={X: batch_X,\n", 347 | " Y: batch_Y})\n", 348 | " global_step = epoch*total_batch + batch\n", 349 | " writer.add_summary(summary, global_step=global_step)\n", 350 | " print(\"Epoch:\",epoch,\"Cost:\",average_cost)\n", 351 | " \n", 352 | " print(\"Test Accuracy:\", accuracy.eval({X: mnist.test.images, Y: mnist.test.labels})) \n", 353 | " writer.close()" 354 | ] 355 | }, 356 | { 357 | "cell_type": "markdown", 358 | "metadata": {}, 359 | "source": [ 360 | "The performance is much better now! Why? Probably because the weight initialization method is different. When we manually initialize the network variables, we use the standard normal distribution. Probably Tensorflow uses random normal but with smaller variance." 361 | ] 362 | }, 363 | { 364 | "cell_type": "markdown", 365 | "metadata": {}, 366 | "source": [ 367 | "## Version 3" 368 | ] 369 | }, 370 | { 371 | "cell_type": "markdown", 372 | "metadata": {}, 373 | "source": [ 374 | "The final version introduces the newly introduced `tf.estimator`. This is essentially a wrapper for the models you create. If we use `tf.estimator` we no longer have to code up `for` loops for the mini batch training. All we need to do is to write the model, and tell the estimator to run it for a certain number of steps. `tf.estimator` is also cool because it is very easy to make a prediction, and evaluate it." 375 | ] 376 | }, 377 | { 378 | "cell_type": "code", 379 | "execution_count": 9, 380 | "metadata": {}, 381 | "outputs": [], 382 | "source": [ 383 | "tf.reset_default_graph() # Clearing all tensors before this" 384 | ] 385 | }, 386 | { 387 | "cell_type": "markdown", 388 | "metadata": {}, 389 | "source": [ 390 | "First, create a multi-layer perceptron just like we did before, but this time, let's make it a function." 391 | ] 392 | }, 393 | { 394 | "cell_type": "code", 395 | "execution_count": 10, 396 | "metadata": {}, 397 | "outputs": [], 398 | "source": [ 399 | "def multilayer_perceptron(X_dict):\n", 400 | " # Estimator input is a dict, in case of multiple inputs\n", 401 | " X = X_dict['images']\n", 402 | " layer_1 = tf.layers.dense(X, num_hidden_1, tf.nn.relu, name='fc1')\n", 403 | " layer_2 = tf.layers.dense(layer_1, num_hidden_2, tf.nn.relu, name='fc2')\n", 404 | " logits = tf.layers.dense(layer_2, num_classes, name='out')\n", 405 | " return logits" 406 | ] 407 | }, 408 | { 409 | "cell_type": "markdown", 410 | "metadata": {}, 411 | "source": [ 412 | "Next, we create the input to the estimator. We need to specify the entire model, including the loss, optimizer, etc. We build a function `model_fn` with a pre-specified signature.\n", 413 | "\n", 414 | "`model_fn` should take in features, labels and mode. Mode tells the function whether you're running it in `TRAIN`, `EVAL` or `PREDICTION`. You can code different behaviors for each mode." 415 | ] 416 | }, 417 | { 418 | "cell_type": "code", 419 | "execution_count": 5, 420 | "metadata": {}, 421 | "outputs": [], 422 | "source": [ 423 | "def model_fn(features, labels, mode):\n", 424 | " # get logit prediction from NN\n", 425 | " logits = multilayer_perceptron(features)\n", 426 | "\n", 427 | " pred_classes = tf.argmax(logits, axis=1)\n", 428 | "\n", 429 | " # if prediction mode, early return\n", 430 | " if mode == tf.estimator.ModeKeys.PREDICT:\n", 431 | " return tf.estimator.EstimatorSpec(mode, predictions=pred_classes)\n", 432 | " \n", 433 | " # these are all the same code as before\n", 434 | " with tf.name_scope('loss'):\n", 435 | " loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n", 436 | " logits=logits, labels=labels),name='loss')\n", 437 | " accuracy = tf.metrics.accuracy(tf.argmax(logits, 1), tf.argmax(labels, 1))\n", 438 | "\n", 439 | " with tf.name_scope('optimizer'):\n", 440 | " learning_rate = 0.01\n", 441 | " optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\n", 442 | " update = optimizer.minimize(loss,\n", 443 | " global_step=tf.train.get_global_step())\n", 444 | "\n", 445 | " with tf.name_scope('summaries'):\n", 446 | " tf.summary.scalar('loss', loss)\n", 447 | " tf.summary.histogram('histogram-loss', loss)\n", 448 | " summary_op = tf.summary.merge_all()\n", 449 | "\n", 450 | " # Estimator requires to return a EstimatorSpec, that specify\n", 451 | " # the different ops for training, evaluating, ...\n", 452 | " estim_specs = tf.estimator.EstimatorSpec(\n", 453 | " mode=mode,\n", 454 | " predictions=pred_classes,\n", 455 | " loss=loss,\n", 456 | " train_op=update,\n", 457 | " eval_metric_ops={'accuracy': accuracy})\n", 458 | "\n", 459 | " return estim_specs" 460 | ] 461 | }, 462 | { 463 | "cell_type": "markdown", 464 | "metadata": {}, 465 | "source": [ 466 | "Now, you can build the model with a single line." 467 | ] 468 | }, 469 | { 470 | "cell_type": "code", 471 | "execution_count": null, 472 | "metadata": {}, 473 | "outputs": [], 474 | "source": [ 475 | "model = tf.estimator.Estimator(model_fn) # build the Estimator" 476 | ] 477 | }, 478 | { 479 | "cell_type": "markdown", 480 | "metadata": {}, 481 | "source": [ 482 | "Training then becomes extremely simple. First, define an input using `tf.estimator.inputs`." 483 | ] 484 | }, 485 | { 486 | "cell_type": "code", 487 | "execution_count": null, 488 | "metadata": {}, 489 | "outputs": [], 490 | "source": [ 491 | "# define the input function for training\n", 492 | "batch_size = 128\n", 493 | "input_fn = tf.estimator.inputs.numpy_input_fn(\n", 494 | " x={'images': mnist.train.images}, y=mnist.train.labels,\n", 495 | " batch_size=batch_size, num_epochs=None, shuffle=True)" 496 | ] 497 | }, 498 | { 499 | "cell_type": "markdown", 500 | "metadata": {}, 501 | "source": [ 502 | "Then, training is a one liner!" 503 | ] 504 | }, 505 | { 506 | "cell_type": "code", 507 | "execution_count": 8, 508 | "metadata": {}, 509 | "outputs": [ 510 | { 511 | "name": "stdout", 512 | "output_type": "stream", 513 | "text": [ 514 | "INFO:tensorflow:Using default config.\n", 515 | "WARNING:tensorflow:Using temporary folder as model directory: /var/folders/lj/p0jqksf54pldc98grzy8m6p00000gn/T/tmpv7e4ojg3\n", 516 | "INFO:tensorflow:Using config: {'_model_dir': '/var/folders/lj/p0jqksf54pldc98grzy8m6p00000gn/T/tmpv7e4ojg3', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n", 517 | "INFO:tensorflow:Create CheckpointSaverHook.\n", 518 | "INFO:tensorflow:Saving checkpoints for 1 into /var/folders/lj/p0jqksf54pldc98grzy8m6p00000gn/T/tmpv7e4ojg3/model.ckpt.\n", 519 | "INFO:tensorflow:loss = 2.3240983, step = 1\n", 520 | "INFO:tensorflow:global_step/sec: 239.482\n", 521 | "INFO:tensorflow:loss = 1.7728766, step = 101 (0.419 sec)\n", 522 | "INFO:tensorflow:global_step/sec: 243.212\n", 523 | "INFO:tensorflow:loss = 1.2416974, step = 201 (0.411 sec)\n", 524 | "INFO:tensorflow:global_step/sec: 228.785\n", 525 | "INFO:tensorflow:loss = 0.90739834, step = 301 (0.437 sec)\n", 526 | "INFO:tensorflow:global_step/sec: 202.656\n", 527 | "INFO:tensorflow:loss = 0.633636, step = 401 (0.494 sec)\n", 528 | "INFO:tensorflow:global_step/sec: 188.722\n", 529 | "INFO:tensorflow:loss = 0.6099938, step = 501 (0.530 sec)\n", 530 | "INFO:tensorflow:global_step/sec: 216.691\n", 531 | "INFO:tensorflow:loss = 0.6928723, step = 601 (0.461 sec)\n", 532 | "INFO:tensorflow:global_step/sec: 193.455\n", 533 | "INFO:tensorflow:loss = 0.47528398, step = 701 (0.517 sec)\n", 534 | "INFO:tensorflow:global_step/sec: 208.692\n", 535 | "INFO:tensorflow:loss = 0.32653236, step = 801 (0.479 sec)\n", 536 | "INFO:tensorflow:global_step/sec: 214.847\n", 537 | "INFO:tensorflow:loss = 0.4268006, step = 901 (0.466 sec)\n", 538 | "INFO:tensorflow:global_step/sec: 222.776\n", 539 | "INFO:tensorflow:loss = 0.2887102, step = 1001 (0.448 sec)\n", 540 | "INFO:tensorflow:global_step/sec: 225.474\n", 541 | "INFO:tensorflow:loss = 0.36707115, step = 1101 (0.444 sec)\n", 542 | "INFO:tensorflow:global_step/sec: 223.104\n", 543 | "INFO:tensorflow:loss = 0.37811956, step = 1201 (0.448 sec)\n", 544 | "INFO:tensorflow:global_step/sec: 223.205\n", 545 | "INFO:tensorflow:loss = 0.24650377, step = 1301 (0.448 sec)\n", 546 | "INFO:tensorflow:global_step/sec: 229.923\n", 547 | "INFO:tensorflow:loss = 0.32238364, step = 1401 (0.435 sec)\n", 548 | "INFO:tensorflow:global_step/sec: 197.548\n", 549 | "INFO:tensorflow:loss = 0.37275034, step = 1501 (0.507 sec)\n", 550 | "INFO:tensorflow:global_step/sec: 128.376\n", 551 | "INFO:tensorflow:loss = 0.3773331, step = 1601 (0.779 sec)\n", 552 | "INFO:tensorflow:global_step/sec: 206.329\n", 553 | "INFO:tensorflow:loss = 0.3100102, step = 1701 (0.485 sec)\n", 554 | "INFO:tensorflow:global_step/sec: 179.334\n", 555 | "INFO:tensorflow:loss = 0.2926942, step = 1801 (0.560 sec)\n", 556 | "INFO:tensorflow:global_step/sec: 160.309\n", 557 | "INFO:tensorflow:loss = 0.22825822, step = 1901 (0.623 sec)\n", 558 | "INFO:tensorflow:global_step/sec: 208.936\n", 559 | "INFO:tensorflow:loss = 0.38949537, step = 2001 (0.477 sec)\n", 560 | "INFO:tensorflow:global_step/sec: 214.187\n", 561 | "INFO:tensorflow:loss = 0.11541493, step = 2101 (0.467 sec)\n", 562 | "INFO:tensorflow:global_step/sec: 181.656\n", 563 | "INFO:tensorflow:loss = 0.35640004, step = 2201 (0.550 sec)\n", 564 | "INFO:tensorflow:global_step/sec: 205.717\n", 565 | "INFO:tensorflow:loss = 0.25524706, step = 2301 (0.486 sec)\n", 566 | "INFO:tensorflow:global_step/sec: 203.046\n", 567 | "INFO:tensorflow:loss = 0.2732802, step = 2401 (0.496 sec)\n", 568 | "INFO:tensorflow:global_step/sec: 185.345\n", 569 | "INFO:tensorflow:loss = 0.23652542, step = 2501 (0.536 sec)\n", 570 | "INFO:tensorflow:global_step/sec: 202.786\n", 571 | "INFO:tensorflow:loss = 0.15657504, step = 2601 (0.493 sec)\n", 572 | "INFO:tensorflow:global_step/sec: 220.827\n", 573 | "INFO:tensorflow:loss = 0.17799367, step = 2701 (0.453 sec)\n", 574 | "INFO:tensorflow:global_step/sec: 208.917\n", 575 | "INFO:tensorflow:loss = 0.25340196, step = 2801 (0.482 sec)\n", 576 | "INFO:tensorflow:global_step/sec: 182.818\n", 577 | "INFO:tensorflow:loss = 0.18117118, step = 2901 (0.546 sec)\n", 578 | "INFO:tensorflow:global_step/sec: 171.274\n", 579 | "INFO:tensorflow:loss = 0.40909755, step = 3001 (0.584 sec)\n", 580 | "INFO:tensorflow:global_step/sec: 203.199\n", 581 | "INFO:tensorflow:loss = 0.31386548, step = 3101 (0.490 sec)\n", 582 | "INFO:tensorflow:global_step/sec: 180.449\n", 583 | "INFO:tensorflow:loss = 0.19484442, step = 3201 (0.558 sec)\n", 584 | "INFO:tensorflow:global_step/sec: 205.776\n", 585 | "INFO:tensorflow:loss = 0.2726854, step = 3301 (0.482 sec)\n", 586 | "INFO:tensorflow:global_step/sec: 198.91\n", 587 | "INFO:tensorflow:loss = 0.19425893, step = 3401 (0.506 sec)\n", 588 | "INFO:tensorflow:global_step/sec: 195.069\n", 589 | "INFO:tensorflow:loss = 0.3121909, step = 3501 (0.514 sec)\n", 590 | "INFO:tensorflow:global_step/sec: 125.823\n", 591 | "INFO:tensorflow:loss = 0.2365881, step = 3601 (0.791 sec)\n", 592 | "INFO:tensorflow:global_step/sec: 125.806\n", 593 | "INFO:tensorflow:loss = 0.31625783, step = 3701 (0.794 sec)\n", 594 | "INFO:tensorflow:global_step/sec: 124.013\n", 595 | "INFO:tensorflow:loss = 0.11293273, step = 3801 (0.806 sec)\n", 596 | "INFO:tensorflow:global_step/sec: 113.159\n", 597 | "INFO:tensorflow:loss = 0.26701343, step = 3901 (0.884 sec)\n", 598 | "INFO:tensorflow:global_step/sec: 116.711\n", 599 | "INFO:tensorflow:loss = 0.24090382, step = 4001 (0.857 sec)\n", 600 | "INFO:tensorflow:global_step/sec: 217.682\n", 601 | "INFO:tensorflow:loss = 0.17926148, step = 4101 (0.458 sec)\n", 602 | "INFO:tensorflow:global_step/sec: 212.883\n", 603 | "INFO:tensorflow:loss = 0.2478786, step = 4201 (0.470 sec)\n", 604 | "INFO:tensorflow:global_step/sec: 114.365\n", 605 | "INFO:tensorflow:loss = 0.14586964, step = 4301 (0.876 sec)\n", 606 | "INFO:tensorflow:global_step/sec: 136.918\n", 607 | "INFO:tensorflow:loss = 0.35951608, step = 4401 (0.729 sec)\n", 608 | "INFO:tensorflow:global_step/sec: 174.771\n", 609 | "INFO:tensorflow:loss = 0.15912426, step = 4501 (0.573 sec)\n", 610 | "INFO:tensorflow:global_step/sec: 169.611\n", 611 | "INFO:tensorflow:loss = 0.32917944, step = 4601 (0.592 sec)\n", 612 | "INFO:tensorflow:global_step/sec: 164.258\n", 613 | "INFO:tensorflow:loss = 0.18930726, step = 4701 (0.609 sec)\n", 614 | "INFO:tensorflow:global_step/sec: 171.941\n", 615 | "INFO:tensorflow:loss = 0.24800149, step = 4801 (0.578 sec)\n", 616 | "INFO:tensorflow:global_step/sec: 192.536\n", 617 | "INFO:tensorflow:loss = 0.1476735, step = 4901 (0.521 sec)\n", 618 | "INFO:tensorflow:Saving checkpoints for 5000 into /var/folders/lj/p0jqksf54pldc98grzy8m6p00000gn/T/tmpv7e4ojg3/model.ckpt.\n", 619 | "INFO:tensorflow:Loss for final step: 0.17756143.\n", 620 | "INFO:tensorflow:Starting evaluation at 2018-02-09-00:16:52\n", 621 | "INFO:tensorflow:Restoring parameters from /var/folders/lj/p0jqksf54pldc98grzy8m6p00000gn/T/tmpv7e4ojg3/model.ckpt-5000\n", 622 | "INFO:tensorflow:Finished evaluation at 2018-02-09-00:16:52\n", 623 | "INFO:tensorflow:Saving dict for global step 5000: accuracy = 0.9386, global_step = 5000, loss = 0.2187681\n", 624 | "Testing Accuracy: 0.9386\n" 625 | ] 626 | } 627 | ], 628 | "source": [ 629 | "num_steps = 5000\n", 630 | "model.train(input_fn, steps=num_steps) # train the Model" 631 | ] 632 | }, 633 | { 634 | "cell_type": "markdown", 635 | "metadata": {}, 636 | "source": [ 637 | "To evaluate the model, you need to specify the test input and run `.evaluate` instead of `.train`. Then, the mode will be set to `PREDICTION`." 638 | ] 639 | }, 640 | { 641 | "cell_type": "code", 642 | "execution_count": null, 643 | "metadata": {}, 644 | "outputs": [], 645 | "source": [ 646 | "\n", 647 | "# define the input function for evaluating\n", 648 | "input_fn = tf.estimator.inputs.numpy_input_fn(\n", 649 | " x={'images': mnist.test.images}, y=mnist.test.labels,\n", 650 | " batch_size=batch_size, shuffle=False)\n", 651 | "# use the Estimator 'evaluate' method\n", 652 | "e = model.evaluate(input_fn)\n", 653 | "\n", 654 | "print(\"Testing Accuracy:\", e['accuracy'])" 655 | ] 656 | }, 657 | { 658 | "cell_type": "markdown", 659 | "metadata": {}, 660 | "source": [ 661 | "That's it! You can choose whichever version you prefer when constructing your model. Although we didn't introduce here, it is also a good strategy to create a python `Class` for each of your model and try to be object oriented. In the sense that it tries to \"package\" a model, `Class` is similar to `tf.Estimator`." 662 | ] 663 | }, 664 | { 665 | "cell_type": "markdown", 666 | "metadata": {}, 667 | "source": [ 668 | "## Exercise\n", 669 | "Improve the above network by even a bit.\n", 670 | "For example, you can try CNN, dropout, different activation function, tune the learning rate, etc." 671 | ] 672 | } 673 | ], 674 | "metadata": { 675 | "kernelspec": { 676 | "display_name": "Python [conda env:tensorflow_workshop]", 677 | "language": "python", 678 | "name": "conda-env-tensorflow_workshop-py" 679 | }, 680 | "language_info": { 681 | "codemirror_mode": { 682 | "name": "ipython", 683 | "version": 3 684 | }, 685 | "file_extension": ".py", 686 | "mimetype": "text/x-python", 687 | "name": "python", 688 | "nbconvert_exporter": "python", 689 | "pygments_lexer": "ipython3", 690 | "version": "3.6.4" 691 | } 692 | }, 693 | "nbformat": 4, 694 | "nbformat_minor": 2 695 | } 696 | -------------------------------------------------------------------------------- /9_GAN.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 9. GAN\n", 8 | "This section will be an exercise. Surprisingly, you can build GAN fairly easily just by using the concepts we learned so far.\n", 9 | "\n", 10 | "## Preparation\n", 11 | "Read section 1, 2, 3 of the original [GAN paper](https://arxiv.org/pdf/1406.2661.pdf). Then, follow the next 7 steps to implement GAN.\n", 12 | "\n", 13 | "To summarie, we have the following problem setup:\n", 14 | "- $x$: data with distribution $p_{data}$\n", 15 | "- $p_g$: distribution trained by the generator\n", 16 | "- $z$: prior input noise variables\n", 17 | "- $p_z$: prior of $z$\n", 18 | "- $G(z;\\theta_G)$: generator neural network with parameter $\\theta_G$\n", 19 | "- $D(z;\\theta_D)$: discriminator neural network with parameter $\\theta_D$\n", 20 | "\n", 21 | "The goal for $D,G$ are the following:\n", 22 | "- $D$: $max_D V(D) = E_{x\\sim p_{data}}(x)[logD(x)] + E_{z\\sim p_z(z)}[log(1-D(G(z))]$\n", 23 | "- $G$: $min_G V(G) = E_{z\\sim p_z(z)}[log(1-D(G(z))]$" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": 1, 29 | "metadata": {}, 30 | "outputs": [ 31 | { 32 | "name": "stdout", 33 | "output_type": "stream", 34 | "text": [ 35 | "Extracting MNIST_data/train-images-idx3-ubyte.gz\n", 36 | "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n", 37 | "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n", 38 | "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n" 39 | ] 40 | } 41 | ], 42 | "source": [ 43 | "import matplotlib\n", 44 | "matplotlib.use('TkAgg')\n", 45 | "import matplotlib.pyplot as plt\n", 46 | "import numpy as np\n", 47 | "import tensorflow as tf\n", 48 | "\n", 49 | "# Import MNIST data\n", 50 | "from tensorflow.examples.tutorials.mnist import input_data\n", 51 | "mnist = input_data.read_data_sets('MNIST_data', one_hot=True)" 52 | ] 53 | }, 54 | { 55 | "cell_type": "markdown", 56 | "metadata": {}, 57 | "source": [ 58 | "We pre-selected the hyper parameters for you this time. You usually need to tune this yourself." 59 | ] 60 | }, 61 | { 62 | "cell_type": "code", 63 | "execution_count": 2, 64 | "metadata": {}, 65 | "outputs": [], 66 | "source": [ 67 | "# Training Params\n", 68 | "num_steps = 500000\n", 69 | "batch_size = 128\n", 70 | "learning_rate = 0.0002" 71 | ] 72 | }, 73 | { 74 | "cell_type": "markdown", 75 | "metadata": {}, 76 | "source": [ 77 | "The hidden dimensions of the generator and the discriminator are also prespecified." 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": 3, 83 | "metadata": {}, 84 | "outputs": [], 85 | "source": [ 86 | "# Network Params\n", 87 | "image_dim = 784 # 28*28 pixels\n", 88 | "gen_hidden_dim = 256\n", 89 | "disc_hidden_dim = 256\n", 90 | "noise_dim = 100 # Noise data points" 91 | ] 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": 4, 96 | "metadata": {}, 97 | "outputs": [], 98 | "source": [ 99 | "tf.reset_default_graph() # Clearing all tensors before this" 100 | ] 101 | }, 102 | { 103 | "cell_type": "markdown", 104 | "metadata": {}, 105 | "source": [ 106 | "### 1. Implement generator and discriminator.\n", 107 | "\n", 108 | "Let both G and D be a 1-hidden layer fully connected neural network. Use ReLU for activation function for the hidden layer. For out layer, you should know what to use :)\n", 109 | "Since G and D are both 1-hidden layer fully connected NN, we need \n", 110 | "- weights for hidden layer and out layer\n", 111 | "- bias for hidden layer and out layer\n", 112 | "\n", 113 | "for each. Use `tf.layers.dense`.\n", 114 | "\n", 115 | "Becareful about the dimensions of the layers:\n", 116 | "- $G$ takes in noise and generates an image.\n", 117 | "- $D$ takes in an image and outputs a probability of the image being real." 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": 5, 123 | "metadata": {}, 124 | "outputs": [], 125 | "source": [ 126 | "# Generator\n", 127 | "def generator(noises, reuse=False):\n", 128 | " with tf.variable_scope('generator') as scope:\n", 129 | " if (reuse):\n", 130 | " tf.get_variable_scope().reuse_variables()\n", 131 | " # hidden layer with name \"g_hidden\"\n", 132 | " hidden = tf.layers.dense(noises, gen_hidden_dim, tf.nn.relu, name='g_hidden')\n", 133 | " # out layer with name \"g_out\"\n", 134 | " out_images = tf.layers.dense(hidden, image_dim, tf.nn.sigmoid, name='g_out')\n", 135 | " return out_images\n", 136 | "\n", 137 | "# Discriminator\n", 138 | "def discriminator(images, reuse=False):\n", 139 | " with tf.variable_scope('discriminator') as scope:\n", 140 | " if (reuse):\n", 141 | " tf.get_variable_scope().reuse_variables() \n", 142 | " # hidden layer with name \"d_hidden\"\n", 143 | " hidden = tf.layers.dense(images, disc_hidden_dim, tf.nn.relu, name='d_hidden')\n", 144 | " # out layer with name \"d_out\"\n", 145 | " out_prob = tf.layers.dense(hidden, 1, tf.nn.sigmoid, name='d_out')\n", 146 | " return out_prob" 147 | ] 148 | }, 149 | { 150 | "cell_type": "markdown", 151 | "metadata": {}, 152 | "source": [ 153 | "### 2. Define the inputs to generator and discriminator.\n", 154 | "- Input to G: (batch size) $\\times$ ??\n", 155 | "- Input to D: (batch size) $\\times$ ??\n", 156 | "\n", 157 | "Think about what ?? should be." 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": 6, 163 | "metadata": {}, 164 | "outputs": [], 165 | "source": [ 166 | "gen_input = tf.placeholder(tf.float32, shape=[None, noise_dim], name='input_noise')\n", 167 | "disc_input = tf.placeholder(tf.float32, shape=[None, image_dim], name='disc_input')" 168 | ] 169 | }, 170 | { 171 | "cell_type": "markdown", 172 | "metadata": {}, 173 | "source": [ 174 | "### 3. Input noise to G and generate images.\n", 175 | "This should be a one linear." 176 | ] 177 | }, 178 | { 179 | "cell_type": "code", 180 | "execution_count": 7, 181 | "metadata": {}, 182 | "outputs": [], 183 | "source": [ 184 | "gen_sample = generator(gen_input)" 185 | ] 186 | }, 187 | { 188 | "cell_type": "markdown", 189 | "metadata": {}, 190 | "source": [ 191 | "### 3. Input real and fake images to D and get predictions.\n", 192 | "For D, you should have two inputs: real data and fake data. The latter is the output of $G$. For the latter, set `reuse=True`. I won't go into detail about it, but basically, you are reusing the samve variables in the above `discriminator` function and so you want to make them reusable." 193 | ] 194 | }, 195 | { 196 | "cell_type": "code", 197 | "execution_count": 8, 198 | "metadata": {}, 199 | "outputs": [], 200 | "source": [ 201 | "disc_real = discriminator(disc_input)\n", 202 | "disc_fake = discriminator(gen_sample, reuse=True)" 203 | ] 204 | }, 205 | { 206 | "cell_type": "code", 207 | "execution_count": 12, 208 | "metadata": {}, 209 | "outputs": [], 210 | "source": [ 211 | "### 4. Define the objective.\n", 212 | "Expectation should be approximated using the sample mean. As a reminder, they are:\n", 213 | "- $D$: $max_D V(D) = E_{x\\sim p_{data}}(x)[logD(x)] + E_{z\\sim p_z(z)}[log(1-D(G(z))]$\n", 214 | "- $G$: $min_G V(G) = E_{z\\sim p_z(z)}[log(1-D(G(z))]$" 215 | ] 216 | }, 217 | { 218 | "cell_type": "code", 219 | "execution_count": 9, 220 | "metadata": {}, 221 | "outputs": [], 222 | "source": [ 223 | "gen_loss = -tf.reduce_mean(tf.log(disc_fake))\n", 224 | "disc_loss = -tf.reduce_mean(tf.log(disc_real) + tf.log(1. - disc_fake))" 225 | ] 226 | }, 227 | { 228 | "cell_type": "markdown", 229 | "metadata": {}, 230 | "source": [ 231 | "### 5. Minimize (or maximize) the objective.\n", 232 | "Adam optimizer is recommended. We should have two optimizers for D and G. Be careful to only take the gradient with respect to the variables to optimize. Namely\n", 233 | "- $V(D)$: weights and biases of D\n", 234 | "- $V(G)$: weights and biases of G\n", 235 | "\n", 236 | "We provided the code for extracting these variables from the computation graph." 237 | ] 238 | }, 239 | { 240 | "cell_type": "code", 241 | "execution_count": 10, 242 | "metadata": {}, 243 | "outputs": [], 244 | "source": [ 245 | "tvars = tf.trainable_variables()\n", 246 | "disc_vars = [var for var in tvars if 'd_' in var.name]\n", 247 | "gen_vars = [var for var in tvars if 'g_' in var.name]" 248 | ] 249 | }, 250 | { 251 | "cell_type": "code", 252 | "execution_count": 11, 253 | "metadata": {}, 254 | "outputs": [], 255 | "source": [ 256 | "optimizer_gen = tf.train.AdamOptimizer(learning_rate=learning_rate)\n", 257 | "optimizer_disc = tf.train.AdamOptimizer(learning_rate=learning_rate)" 258 | ] 259 | }, 260 | { 261 | "cell_type": "code", 262 | "execution_count": 12, 263 | "metadata": {}, 264 | "outputs": [], 265 | "source": [ 266 | "train_gen = optimizer_gen.minimize(gen_loss, var_list=gen_vars)\n", 267 | "train_disc = optimizer_disc.minimize(disc_loss, var_list=disc_vars)" 268 | ] 269 | }, 270 | { 271 | "cell_type": "markdown", 272 | "metadata": {}, 273 | "source": [ 274 | "### 6. Train the model.\n", 275 | "For each iteration, take some batch of MNIST. Generate a prior noise $z$ by `np.random.uniform(-1., 1., size=[batch_size, noise_dim])`. Feed the batch data and prior noise to the model to update the objective.\n", 276 | "\n", 277 | "After some epochs of training, for each noise generated, get the output $x$ by the generator and plot it using matplotlib. This time we prepared the code for you but read through it to understand it. Then, change the variable names if they are different from yours." 278 | ] 279 | }, 280 | { 281 | "cell_type": "code", 282 | "execution_count": 13, 283 | "metadata": {}, 284 | "outputs": [ 285 | { 286 | "name": "stdout", 287 | "output_type": "stream", 288 | "text": [ 289 | "Step 1: Generator Loss: 1.104980, Discriminator Loss: 1.427862\n", 290 | "gan1.png\n" 291 | ] 292 | }, 293 | { 294 | "ename": "KeyboardInterrupt", 295 | "evalue": "", 296 | "output_type": "error", 297 | "traceback": [ 298 | "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", 299 | "\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)", 300 | "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0mfeed_dict\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0mdisc_input\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbatch_x\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgen_input\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mz\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 13\u001b[0m _, _, gl, dl = sess.run([train_gen, train_disc, gen_loss, disc_loss],\n\u001b[0;32m---> 14\u001b[0;31m feed_dict=feed_dict)\n\u001b[0m\u001b[1;32m 15\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 16\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mstep\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;36m1000\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mstep\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 301 | "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m 893\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 894\u001b[0m result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 895\u001b[0;31m run_metadata_ptr)\n\u001b[0m\u001b[1;32m 896\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 897\u001b[0m \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 302 | "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m 1122\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mfeed_dict_tensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1123\u001b[0m results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m-> 1124\u001b[0;31m feed_dict_tensor, options, run_metadata)\n\u001b[0m\u001b[1;32m 1125\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1126\u001b[0m \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 303 | "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m 1319\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1320\u001b[0m return self._do_call(_run_fn, self._session, feeds, fetches, targets,\n\u001b[0;32m-> 1321\u001b[0;31m options, run_metadata)\n\u001b[0m\u001b[1;32m 1322\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1323\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_prun_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeeds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetches\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 304 | "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m 1325\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1326\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1327\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1328\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1329\u001b[0m \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 305 | "\u001b[0;32m~/anaconda/envs/tensorflow_workshop/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m 1304\u001b[0m return tf_session.TF_Run(session, options,\n\u001b[1;32m 1305\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1306\u001b[0;31m status, run_metadata)\n\u001b[0m\u001b[1;32m 1307\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1308\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 306 | "\u001b[0;31mKeyboardInterrupt\u001b[0m: " 307 | ] 308 | } 309 | ], 310 | "source": [ 311 | "with tf.Session() as sess:\n", 312 | "\n", 313 | " # Run the initializer\n", 314 | " sess.run(tf.global_variables_initializer())\n", 315 | " \n", 316 | " for step in range(1, num_steps+1):\n", 317 | "\n", 318 | " batch_x, _ = mnist.train.next_batch(batch_size)\n", 319 | " # Generate noise to feed to the generator\n", 320 | " z = np.random.uniform(-1., 1., size=[batch_size, noise_dim])\n", 321 | " # Train\n", 322 | " feed_dict = {disc_input: batch_x, gen_input: z}\n", 323 | " _, _, gl, dl = sess.run([train_gen, train_disc, gen_loss, disc_loss],\n", 324 | " feed_dict=feed_dict)\n", 325 | " \n", 326 | " if step % 1000 == 0 or step == 1:\n", 327 | " print('Step %i: Generator Loss: %f, Discriminator Loss: %f' % (step, gl, dl))\n", 328 | " \n", 329 | " # Generate images from noise, using the generator network.\n", 330 | " if step % 10000 == 0 or step == 1:\n", 331 | " f, a = plt.subplots(4, 10, figsize=(10, 4))\n", 332 | " for i in range(10):\n", 333 | " # Noise input.\n", 334 | " z = np.random.uniform(-1., 1., size=[4, noise_dim])\n", 335 | " g = sess.run([gen_sample], feed_dict={gen_input: z})\n", 336 | " g = np.reshape(g, newshape=(4, 28, 28, 1))\n", 337 | " # Reverse colours for better display\n", 338 | " g = -1 * (g - 1)\n", 339 | " for j in range(4):\n", 340 | " # Generate image from noise. Extend to 3 channels for matplot figure.\n", 341 | " img = np.reshape(np.repeat(g[j][:, :, np.newaxis], 3, axis=2),\n", 342 | " newshape=(28, 28, 3))\n", 343 | " a[j][i].imshow(img)\n", 344 | "\n", 345 | " plt.draw()\n", 346 | " print('gan'+str(step)+'.png')\n", 347 | " plt.savefig('gan'+str(step)+'.png')" 348 | ] 349 | }, 350 | { 351 | "cell_type": "markdown", 352 | "metadata": {}, 353 | "source": [ 354 | "### 7. [Optional] use TensorBoard to check the computation graph and loss.\n", 355 | "You might want to read about [variable sharing](https://www.tensorflow.org/versions/r1.1/programmers_guide/variable_scope) and [variable scope](https://stackoverflow.com/questions/35919020/whats-the-difference-of-name-scope-and-a-variable-scope-in-tensorflow).\n", 356 | "\n", 357 | "This might be a bit more involved than the previous steps..." 358 | ] 359 | }, 360 | { 361 | "cell_type": "code", 362 | "execution_count": null, 363 | "metadata": {}, 364 | "outputs": [], 365 | "source": [] 366 | }, 367 | { 368 | "cell_type": "markdown", 369 | "metadata": {}, 370 | "source": [ 371 | "Ok, this was probably the hardest section so far since there's less hand holding. But if you could complete this exercise, this means that you can build reasonably sophisticated neural network models in TensorFlow! Look back and see how far you got :)\n", 372 | "\n", 373 | "Check [this](https://github.com/tensorflow/models/blob/master/research/gan/tutorial.ipynb) out if you're more interested in GANs.\n", 374 | "\n", 375 | "Thanks for completing this workshop. If you liked it, please `star` this repo, so that more and more people can learn about TensorFlow! Feedback is always welcome!\n" 376 | ] 377 | }, 378 | { 379 | "cell_type": "code", 380 | "execution_count": null, 381 | "metadata": {}, 382 | "outputs": [], 383 | "source": [] 384 | } 385 | ], 386 | "metadata": { 387 | "kernelspec": { 388 | "display_name": "Python [conda env:tensorflow_workshop]", 389 | "language": "python", 390 | "name": "conda-env-tensorflow_workshop-py" 391 | }, 392 | "language_info": { 393 | "codemirror_mode": { 394 | "name": "ipython", 395 | "version": 3 396 | }, 397 | "file_extension": ".py", 398 | "mimetype": "text/x-python", 399 | "name": "python", 400 | "nbconvert_exporter": "python", 401 | "pygments_lexer": "ipython3", 402 | "version": "3.6.4" 403 | } 404 | }, 405 | "nbformat": 4, 406 | "nbformat_minor": 2 407 | } 408 | -------------------------------------------------------------------------------- /9_GAN_Harder.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 9. GAN\n", 8 | "This section will be an exercise. Surprisingly, you can build GAN fairly easily just by using the concepts we learned so far.\n", 9 | "\n", 10 | "## Preparation\n", 11 | "Read section 1, 2, 3 of the original [GAN paper](https://arxiv.org/pdf/1406.2661.pdf). Then, follow the next 7 steps to implement GAN.\n", 12 | "\n", 13 | "To summarie, we have the following problem setup:\n", 14 | "- $x$: data with distribution $p_{data}$\n", 15 | "- $p_g$: distribution trained by the generator\n", 16 | "- $z$: prior input noise variables\n", 17 | "- $p_z$: prior of $z$\n", 18 | "- $G(z;\\theta_G)$: generator neural network with parameter $\\theta_G$\n", 19 | "- $D(z;\\theta_D)$: discriminator neural network with parameter $\\theta_D$\n", 20 | "\n", 21 | "The goal for $D,G$ are the following:\n", 22 | "- $D$: $max_D V(D) = E_{x\\sim p_{data}}(x)[logD(x)] + E_{z\\sim p_z(z)}[log(1-D(G(z))]$\n", 23 | "- $G$: $min_G V(G) = E_{z\\sim p_z(z)}[log(1-D(G(z))]$" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": 3, 29 | "metadata": {}, 30 | "outputs": [ 31 | { 32 | "name": "stdout", 33 | "output_type": "stream", 34 | "text": [ 35 | "Extracting MNIST_data/train-images-idx3-ubyte.gz\n", 36 | "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n", 37 | "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n", 38 | "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n" 39 | ] 40 | } 41 | ], 42 | "source": [ 43 | "import matplotlib\n", 44 | "matplotlib.use('TkAgg')\n", 45 | "import matplotlib.pyplot as plt\n", 46 | "import numpy as np\n", 47 | "import tensorflow as tf\n", 48 | "\n", 49 | "# Import MNIST data\n", 50 | "from tensorflow.examples.tutorials.mnist import input_data\n", 51 | "mnist = input_data.read_data_sets('MNIST_data', one_hot=True)" 52 | ] 53 | }, 54 | { 55 | "cell_type": "markdown", 56 | "metadata": {}, 57 | "source": [ 58 | "We pre-selected the hyper parameters for you this time. You usually need to tune this yourself." 59 | ] 60 | }, 61 | { 62 | "cell_type": "code", 63 | "execution_count": 4, 64 | "metadata": {}, 65 | "outputs": [], 66 | "source": [ 67 | "# Training Params\n", 68 | "num_steps = 500000\n", 69 | "batch_size = 128\n", 70 | "learning_rate = 0.0002" 71 | ] 72 | }, 73 | { 74 | "cell_type": "markdown", 75 | "metadata": {}, 76 | "source": [ 77 | "The hidden dimensions of the generator and the discriminator are also prespecified." 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": 5, 83 | "metadata": {}, 84 | "outputs": [], 85 | "source": [ 86 | "# Network Params\n", 87 | "image_dim = 784 # 28*28 pixels\n", 88 | "gen_hidden_dim = 256\n", 89 | "disc_hidden_dim = 256\n", 90 | "noise_dim = 100 # Noise data points" 91 | ] 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": 6, 96 | "metadata": {}, 97 | "outputs": [], 98 | "source": [ 99 | "tf.reset_default_graph() # Clearing all tensors before this" 100 | ] 101 | }, 102 | { 103 | "cell_type": "markdown", 104 | "metadata": {}, 105 | "source": [ 106 | "### 1. Implement generator and discriminator.\n", 107 | "\n", 108 | "Let both G and D be a 1-hidden layer fully connected neural network. Use ReLU for activation function for the hidden layer. For out layer, you should know what to use :)\n", 109 | "Since G and D are both 1-hidden layer fully connected NN, we need \n", 110 | "- weights for hidden layer and out layer\n", 111 | "- bias for hidden layer and out layer\n", 112 | "\n", 113 | "for each. Use `tf.layers.dense`.\n", 114 | "\n", 115 | "Becareful about the dimensions of the layers:\n", 116 | "- $G$ takes in noise and generates an image.\n", 117 | "- $D$ takes in an image and outputs a probability of the image being real." 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": 5, 123 | "metadata": {}, 124 | "outputs": [], 125 | "source": [] 126 | }, 127 | { 128 | "cell_type": "markdown", 129 | "metadata": {}, 130 | "source": [ 131 | "### 2. Define the inputs to generator and discriminator.\n", 132 | "- Input to G: (batch size) $\\times$ ??\n", 133 | "- Input to D: (batch size) $\\times$ ??\n", 134 | "\n", 135 | "Think about what ?? should be." 136 | ] 137 | }, 138 | { 139 | "cell_type": "code", 140 | "execution_count": 6, 141 | "metadata": {}, 142 | "outputs": [], 143 | "source": [] 144 | }, 145 | { 146 | "cell_type": "markdown", 147 | "metadata": {}, 148 | "source": [ 149 | "### 3. Input noise to G and generate images.\n", 150 | "This should be a one linear." 151 | ] 152 | }, 153 | { 154 | "cell_type": "code", 155 | "execution_count": 7, 156 | "metadata": {}, 157 | "outputs": [], 158 | "source": [] 159 | }, 160 | { 161 | "cell_type": "markdown", 162 | "metadata": {}, 163 | "source": [ 164 | "### 3. Input real and fake images to D and get predictions.\n", 165 | "For D, you should have two inputs: real data and fake data. The latter is the output of $G$. For the latter, set `reuse=True`. I won't go into detail about it, but basically, you are reusing the samve variables in the above `discriminator` function and so you want to make them reusable." 166 | ] 167 | }, 168 | { 169 | "cell_type": "code", 170 | "execution_count": 8, 171 | "metadata": {}, 172 | "outputs": [], 173 | "source": [] 174 | }, 175 | { 176 | "cell_type": "markdown", 177 | "metadata": {}, 178 | "source": [ 179 | "### 4. Define the objective.\n", 180 | "Expectation should be approximated using the sample mean. As a reminder, they are:\n", 181 | "- $D$: $max_D V(D) = E_{x\\sim p_{data}}(x)[logD(x)] + E_{z\\sim p_z(z)}[log(1-D(G(z))]$\n", 182 | "- $G$: $min_G V(G) = E_{z\\sim p_z(z)}[log(1-D(G(z))]$" 183 | ] 184 | }, 185 | { 186 | "cell_type": "code", 187 | "execution_count": 9, 188 | "metadata": {}, 189 | "outputs": [], 190 | "source": [] 191 | }, 192 | { 193 | "cell_type": "markdown", 194 | "metadata": {}, 195 | "source": [ 196 | "### 5. Minimize (or maximize) the objective.\n", 197 | "Adam optimizer is recommended. We should have two optimizers for D and G. Be careful to only take the gradient with respect to the variables to optimize. Namely\n", 198 | "- $V(D)$: weights and biases of D\n", 199 | "- $V(G)$: weights and biases of G\n", 200 | "\n", 201 | "We provided the code for extracting these variables from the computation graph." 202 | ] 203 | }, 204 | { 205 | "cell_type": "code", 206 | "execution_count": 12, 207 | "metadata": {}, 208 | "outputs": [], 209 | "source": [] 210 | }, 211 | { 212 | "cell_type": "markdown", 213 | "metadata": {}, 214 | "source": [ 215 | "### 6. Train the model.\n", 216 | "For each iteration, take some batch of MNIST. Generate a prior noise $z$ by `np.random.uniform(-1., 1., size=[batch_size, noise_dim])`. Feed the batch data and prior noise to the model to update the objective.\n", 217 | "\n", 218 | "After some epochs of training, for each noise generated, get the output $x$ by the generator and plot it using matplotlib. This time we prepared the code for you but read through it to understand it. Then, change the variable names if they are different from yours." 219 | ] 220 | }, 221 | { 222 | "cell_type": "code", 223 | "execution_count": null, 224 | "metadata": {}, 225 | "outputs": [], 226 | "source": [] 227 | }, 228 | { 229 | "cell_type": "markdown", 230 | "metadata": {}, 231 | "source": [ 232 | "### 7. [Optional] use TensorBoard to check the computation graph and loss.\n", 233 | "You might want to read about [variable sharing](https://www.tensorflow.org/versions/r1.1/programmers_guide/variable_scope) and [variable scope](https://stackoverflow.com/questions/35919020/whats-the-difference-of-name-scope-and-a-variable-scope-in-tensorflow).\n", 234 | "\n", 235 | "This might be a bit more involved than the previous steps..." 236 | ] 237 | }, 238 | { 239 | "cell_type": "code", 240 | "execution_count": null, 241 | "metadata": {}, 242 | "outputs": [], 243 | "source": [] 244 | }, 245 | { 246 | "cell_type": "markdown", 247 | "metadata": {}, 248 | "source": [ 249 | "Ok, this was probably the hardest section so far since there's less hand holding. But if you could complete this exercise, this means that you can build reasonably sophisticated neural network models in TensorFlow! Look back and see how far you got :)\n", 250 | "\n", 251 | "Check [this](https://github.com/tensorflow/models/blob/master/research/gan/tutorial.ipynb) out if you're more interested in GANs.\n", 252 | "\n", 253 | "Thanks for completing this workshop. If you liked it, please `star` this repo, so that more and more people can learn about TensorFlow! Feedback is always welcome!\n" 254 | ] 255 | }, 256 | { 257 | "cell_type": "code", 258 | "execution_count": null, 259 | "metadata": {}, 260 | "outputs": [], 261 | "source": [] 262 | } 263 | ], 264 | "metadata": { 265 | "kernelspec": { 266 | "display_name": "Python [conda env:tensorflow_workshop]", 267 | "language": "python", 268 | "name": "conda-env-tensorflow_workshop-py" 269 | }, 270 | "language_info": { 271 | "codemirror_mode": { 272 | "name": "ipython", 273 | "version": 3 274 | }, 275 | "file_extension": ".py", 276 | "mimetype": "text/x-python", 277 | "name": "python", 278 | "nbconvert_exporter": "python", 279 | "pygments_lexer": "ipython3", 280 | "version": "3.6.4" 281 | } 282 | }, 283 | "nbformat": 4, 284 | "nbformat_minor": 2 285 | } 286 | -------------------------------------------------------------------------------- /Articles.md: -------------------------------------------------------------------------------- 1 | 2 | ## Articles on Robust Machine Learning 3 | 4 | ### Government 5 | 6 | - [China to build giant facial recognition database to identify any citizen within seconds](http://www.scmp.com/news/china/society/article/2115094/china-build-giant-facial-recognition-database-identify-any) 7 | 8 | - [Warnings of abuse of Australian surveillance tech in police state Bahrain](http://www.smh.com.au/world/warnings-of-abuse-of-australian-surveillance-tech-in-police-state-bahrain-20160623-gpqb8i.html) 9 | 10 | ### Technology 11 | 12 | - [Even a mask won’t hide you from the latest face recognition tech](https://www.newscientist.com/article/2146703-even-a-mask-wont-hide-you-from-the-latest-face-recognition-tech/) 13 | 14 | ### Fake News 15 | 16 | - [He Predicted The 2016 Fake News Crisis. Now He's Worried About An Information Apocalypse.](https://www.buzzfeed.com/charliewarzel/the-terrifying-future-of-fake-news) 17 | -------------------------------------------------------------------------------- /Papers.md: -------------------------------------------------------------------------------- 1 | 2 | ## Papers on Robust Machine Learning 3 | 4 | ### Adversarial Examples 5 | * [Szegedy et al., 2014: Intriguing properties of neural networks](https://arxiv.org/pdf/1312.6199.pdf), 6 | the first paper to introduce adversarial examples 7 | * [Moosavi-Dezfooli et al., 2016: DeepFool: a simple and accurate method to fool deep neural netwoks](https://arxiv.org/pdf/1511.04599.pdf), develops gradient descent like and Newton's method like ways of finding adversarial examples 8 | * [Carlini et al., 2017: Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods](https://arxiv.org/abs/1705.07263) 9 | 10 | #### Transferability 11 | * [Tramer et al., 2017: The Space of Transferable Adversarial Examples](https://arxiv.org/abs/1704.03453) 12 | * [Mohsen et al., 2016: Universal adversarial perturbations](https://arxiv.org/abs/1610.08401), aggregate minimal perturbations from multiple classifiers, quite empirical 13 | * [Fawzi et al., 2018: Adversarial vulnerability for any classifier](https://arxiv.org/abs/1802.08686) 14 | 15 | ### GAN 16 | * [Goodfellow et al., 2014: Generative Adversarial Nets](https://arxiv.org/pdf/1406.2661.pdf), the first paper to introduce GAN 17 | 18 | #### WGAN 19 | * [Arjovsky et al., 2017: Wasserstein GAN](https://arxiv.org/abs/1701.07875), use EM distance instead of JS divergence for more stable GAN training 20 | 21 | #### Convergence 22 | * [Daskalakis et al., 2017: Training GANs with Optimism](https://arxiv.org/abs/1711.00141) 23 | * [Grnarova et al., 2017: An Online Learning Approach to Generative Adversarial Networks](https://arxiv.org/abs/1706.03269) 24 | 25 | ### Differential Privacy 26 | * [Abadi et al., 2016: Deep Learning with Differential Privacy](https://arxiv.org/abs/1607.00133) 27 | * [Papernot et al., 2017: Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data](https://arxiv.org/abs/1610.05755) 28 | 29 | ### Understanding Deep Learning 30 | * [Koh et al., 2017: Understanding Black-box Predictions via Influence Functions](https://arxiv.org/abs/1703.04730) 31 | * [Zhang et al., 2018: Visual Interpretability for Deep Learning: a Survey](https://arxiv.org/abs/1802.00614) 32 | * [Narayanan et al., 2018: How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation](https://arxiv.org/abs/1802.00682?utm_campaign=Revue%20newsletter&utm_medium=Newsletter&utm_source=piqcy) 33 | 34 | ### Generalization 35 | * [Zhang et al., 2016: Understanding deep learning requires rethinking generalization](https://arxiv.org/abs/1611.03530) 36 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # CS282R: Robust Machine Learning Workshop 2 | Coding tutorial for robust machine learning algorithms for CS282R "Robust Machine Learning" taught at Harvard University in Spring 2018. 3 | 4 | ## Overview 5 | This workshop covers the fundamentals of TensorFlow. By the end of this workshop, you will be well equipped to start building your own neural network models in TensorFlow. 6 | The workshop will start by introducing concepts unique to TensorFlow, e.g. sessions, variables, optimizers. 7 | Later, we'll put these concepts together to build logistic regression, 3-layer NN, and finally, GAN. 8 | 9 | The side theme of this workshop is writing clean, structured and well-documented code. ML code tends to get messy. This is especially the case in academic settings where we don't have a manager to review your code. We are also free from the pressure that our code will be used by millions of people. BUT, this never means that you can write sloppy code. The code should be nicely written so that you, three months from now, should be able to come back to the code and tell exactly what each function is doing. 10 | 11 | That said, my code is not the best thing in the world. If you find any improvements, or errors in the code, please let me know. Pull requests are always welcome. 12 | 13 | ## Prerequisite 14 | - Comfortability with Python 15 | - Understanding of basic deep learning concepts. 16 | - Fully charged laptop (so tat we don't run out of outlets). 17 | - Install `TensorFlow` and other packages by following section 1. Run code in section 3 and make sure things run smoothly before class. 18 | 19 | ## Index 20 | 1. [Setup](1_Setup.ipynb) 21 | 2. [Introduction](2_Introduction.ipynb) 22 | 3. [Graphs, Sessions](3_Graphs_Sessions.ipynb) 23 | 4. [Basic Operations](4_Basic_Operations.ipynb) 24 | 5. [Variables, Gradients, Placeholders](5_Variables_Gradients_Placeholders.ipynb) 25 | 6. [Classification, Regression](6_Classification_Regression.ipynb) 26 | 7. [Visualization](7_Visualization.ipynb) 27 | 8. [Multi-layer Perceptron](8_Multi_Layer_Perceptron.ipynb) 28 | 9. [GAN](9_GAN.ipynb) 29 | 10. [Adversarial Examples](10_Adversarial_Examples.ipynb) 30 | 31 | Appendix: [Articles](Articles.md), [Papers](Papers.md) on robust machine learning 32 | 33 | ## Workshop Format 34 | When learning a new framework, it is better to move hands than to listen. So, we prepared quite a bit of programming exercises. 35 | You will work with your project partner on these exercises. [Pair programming](https://en.wikipedia.org/wiki/Pair_programming) is extremely effective for being a better coder whether it be software or ML! 36 | - Introduction to TensorFlow (section 1~6, 30 min) 37 | - Pair Programming Exercises on Basics (section 1~6, 45 min) 38 | - Break (10 min) 39 | - Introduction to Deep Learning with TensorFlow (section 7~9, 30 min) 40 | - Pair Programming Exercises on Deep Learning (section 7~9, 45 min) 41 | 42 | ## Things we will not cover 43 | Here are the list of things we will not cover but you might want to self-learn: 44 | - Other Deep Learning models (e.g. RNN, CNN, VAE) 45 | - [Using GPUs](https://www.tensorflow.org/programmers_guide/using_gpu) 46 | - [Saving variables and graphs](https://www.tensorflow.org/programmers_guide/saved_model) 47 | - [Testing your code](https://github.com/nfmcclure/tensorflow_cookbook/tree/master/10_Taking_TensorFlow_to_Production) 48 | 49 | ## Reference 50 | - [Official Tutorial on handwritten digits (MNIST) classification](https://www.tensorflow.org/tutorials/layers) 51 | - [Stanford CS20, TensorFlow for Deep Learning Research](https://web.stanford.edu/class/cs20si/syllabus.html) 52 | - [Stanford CS224d, Deep Learning for NLP](http://web.stanford.edu/class/cs224n/syllabus.html) 53 | - [Most popular TF code samples on GitHub](https://github.com/aymericdamien/TensorFlow-Examples) 54 | 55 | ## How to contribute 56 | 57 | We want this repository to be a source of information related to robust machine learning. To do so, we need your help. In particular, you can **send a pull request** to this repository. For details, see [this](https://help.github.com/articles/creating-a-pull-request/). Some types of contribution can be: 58 | 59 | - Adding more models from our assigned papers to the tutorial. 60 | - Adding papers/articles you found interesting to [Papers.md](papers.md), [Articles.md](articles.md). 61 | - Fixing typos (always a good start if you're new to OSS). 62 | 63 | Finally, don't forget to add your name below to give yourself a credit for the work you put in :) 64 | 65 | Material created by: [Kojin Oshiba](http://kojinoshiba.com/), Jerry Anunrojwong. 66 |
67 | Course taught by Professor [Yaron Singer](https://people.seas.harvard.edu/~yaron/). 68 | -------------------------------------------------------------------------------- /Turn a GAN into a WGAN and then WGAN GP.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# How to turn a vanilla GAN into a vanilla WGAN\n", 8 | "This notebook will help to enhance on the understanding of WGAN algorithm and WGAN_GP by changing a vanilla GAN into a WGAN, and then a WGAN_GP
\n", 9 | "The GAN code is based on Kojin's GAN workshop" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "## Different Loss Functions \n", 17 | "### GAN\n", 18 | "- $D$: $max_D V(D) = E_{x\\sim p_{data}}(x)[logD(x)] + E_{z\\sim p_z(z)}[log(1-D(G(z))]$\n", 19 | "- $G$: $min_G V(G) = E_{z\\sim p_z(z)}[log(1-D(G(z))]$\n", 20 | "\n", 21 | "### WGAN\n", 22 | "- $D$: $max_D V(D) = E_{x\\sim p_{data}}[D(x)] - E_{z\\sim p_z(z)}[D(G(z)]$\n", 23 | "- $G$: $max_G V(G) = E_{z\\sim p_z(z)}[D(G(z)]$\n", 24 | "\n", 25 | "### WGAN-GP\n", 26 | "- $D$: $max_D V(D) = E_{x\\sim p_{data}}[D(x)] - E_{z\\sim p_z(z)}[D(G(z)] + \\lambda(\\lVert \\nabla D(\\hat{x}) \\rVert_{2}-1)^2$\n", 27 | "- $G$: $max_G V(G) = E_{z\\sim p_z(z)}[D(G(z)]$" 28 | ] 29 | }, 30 | { 31 | "cell_type": "markdown", 32 | "metadata": {}, 33 | "source": [ 34 | "|Features | GAN | WGAN |WGAN-GP\n", 35 | "| --------------------------- |:------:|:----: |:-----\n", 36 | "|output layer of Discriminator |Sigmoid | Linear |Linear\n", 37 | "|optimizer | Adam | RMS | Adam\n", 38 | "|weight clipping | False | True | False\n", 39 | "|Batch Normalization | False | True | False" 40 | ] 41 | }, 42 | { 43 | "cell_type": "markdown", 44 | "metadata": {}, 45 | "source": [ 46 | "##### * Batch Normalization (not covered here)\n", 47 | "batch normalization normalize the output of each activation layer by subtracting the batch mean and dividing by the batch standard deviation to stablize training.
" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": {}, 53 | "source": [ 54 | "## Preperation" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": {}, 61 | "outputs": [], 62 | "source": [ 63 | "import matplotlib\n", 64 | "matplotlib.use('TkAgg')\n", 65 | "import matplotlib.pyplot as plt\n", 66 | "import numpy as np\n", 67 | "import tensorflow as tf" 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": null, 73 | "metadata": {}, 74 | "outputs": [], 75 | "source": [ 76 | "# Import MNIST data\n", 77 | "from tensorflow.examples.tutorials.mnist import input_data\n", 78 | "mnist = input_data.read_data_sets('MNIST_data', one_hot=True)" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": null, 84 | "metadata": {}, 85 | "outputs": [], 86 | "source": [ 87 | "# Network Params\n", 88 | "image_dim = 784 # 28*28 pixels\n", 89 | "gen_hidden_dim = 256\n", 90 | "disc_hidden_dim = 256\n", 91 | "noise_dim = 128 # Noise data points" 92 | ] 93 | }, 94 | { 95 | "cell_type": "code", 96 | "execution_count": null, 97 | "metadata": {}, 98 | "outputs": [], 99 | "source": [ 100 | "tf.reset_default_graph() # Clearing all tensors before this" 101 | ] 102 | }, 103 | { 104 | "cell_type": "markdown", 105 | "metadata": {}, 106 | "source": [ 107 | "## GAN to WGAN " 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": null, 113 | "metadata": {}, 114 | "outputs": [], 115 | "source": [ 116 | "# add hyperparameter Critic_Iters and c for WGAN" 117 | ] 118 | }, 119 | { 120 | "cell_type": "code", 121 | "execution_count": null, 122 | "metadata": {}, 123 | "outputs": [], 124 | "source": [ 125 | "# trainning param\n", 126 | "Batch_Size = 50\n", 127 | "Critic_Iters = 5 # for WGAN and WGAN-GP, number of critic iters per gen iter\n", 128 | "c = 0.01 # threshold for weight cliping (-c,c)\n", 129 | "Iters = 20001 # number of generator iterations to train for" 130 | ] 131 | }, 132 | { 133 | "cell_type": "code", 134 | "execution_count": null, 135 | "metadata": {}, 136 | "outputs": [], 137 | "source": [ 138 | "# Generator\n", 139 | "def generator(noises, reuse=False):\n", 140 | " with tf.variable_scope('generator') as scope:\n", 141 | " if (reuse):\n", 142 | " tf.get_variable_scope().reuse_variables()\n", 143 | " # hidden layer with name \"g_hidden\"\n", 144 | " hidden = tf.layers.dense(noises, gen_hidden_dim, tf.nn.relu, name='g_hidden')\n", 145 | " # out layer with name \"g_out\"\n", 146 | " out_images = tf.layers.dense(hidden, image_dim, tf.nn.sigmoid, name='g_out')\n", 147 | " return out_images\n", 148 | "\n", 149 | "# Discriminator\n", 150 | "def discriminator(images, reuse=False):\n", 151 | " with tf.variable_scope('discriminator') as scope:\n", 152 | " if (reuse):\n", 153 | " tf.get_variable_scope().reuse_variables() \n", 154 | " # hidden layer with name \"d_hidden\"\n", 155 | " hidden = tf.layers.dense(images, disc_hidden_dim, tf.nn.relu, name='d_hidden') \n", 156 | " # out layer with name \"d_out\"\n", 157 | " out = tf.layers.dense(hidden, 1, None, name='d_out') # ReLU output into linear activation\n", 158 | " return out" 159 | ] 160 | }, 161 | { 162 | "cell_type": "code", 163 | "execution_count": null, 164 | "metadata": {}, 165 | "outputs": [], 166 | "source": [ 167 | "gen_input = tf.placeholder(tf.float32, shape=[None, noise_dim], name='input_noise')" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": null, 173 | "metadata": {}, 174 | "outputs": [], 175 | "source": [ 176 | "fake_data = generator(gen_input)\n", 177 | "real_data = tf.placeholder(tf.float32, shape=[None, image_dim], name='real_data')" 178 | ] 179 | }, 180 | { 181 | "cell_type": "code", 182 | "execution_count": null, 183 | "metadata": {}, 184 | "outputs": [], 185 | "source": [ 186 | "disc_real = discriminator(real_data)\n", 187 | "disc_fake = discriminator(fake_data, reuse=True)" 188 | ] 189 | }, 190 | { 191 | "cell_type": "code", 192 | "execution_count": null, 193 | "metadata": {}, 194 | "outputs": [], 195 | "source": [ 196 | "# cost function" 197 | ] 198 | }, 199 | { 200 | "cell_type": "code", 201 | "execution_count": null, 202 | "metadata": {}, 203 | "outputs": [], 204 | "source": [ 205 | "gen_cost = -tf.reduce_mean(disc_fake)\n", 206 | "disc_cost = tf.reduce_mean(disc_fake) - tf.reduce_mean(disc_real)" 207 | ] 208 | }, 209 | { 210 | "cell_type": "code", 211 | "execution_count": null, 212 | "metadata": {}, 213 | "outputs": [], 214 | "source": [ 215 | "tvars = tf.trainable_variables()\n", 216 | "disc_vars = [var for var in tvars if 'd_' in var.name]\n", 217 | "gen_vars = [var for var in tvars if 'g_' in var.name]" 218 | ] 219 | }, 220 | { 221 | "cell_type": "code", 222 | "execution_count": null, 223 | "metadata": {}, 224 | "outputs": [], 225 | "source": [ 226 | "# optimizer to RMS" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": null, 232 | "metadata": {}, 233 | "outputs": [], 234 | "source": [ 235 | "train_gen = tf.train.RMSPropOptimizer(\n", 236 | " learning_rate=5e-5, \n", 237 | " ).minimize(gen_cost, var_list=gen_vars)\n", 238 | "\n", 239 | "train_disc = tf.train.RMSPropOptimizer(\n", 240 | " learning_rate=5e-5, \n", 241 | " ).minimize(disc_cost, var_list=disc_vars)" 242 | ] 243 | }, 244 | { 245 | "cell_type": "code", 246 | "execution_count": null, 247 | "metadata": {}, 248 | "outputs": [], 249 | "source": [ 250 | "# add weight clipping" 251 | ] 252 | }, 253 | { 254 | "cell_type": "code", 255 | "execution_count": null, 256 | "metadata": {}, 257 | "outputs": [], 258 | "source": [ 259 | "clip_D = [p.assign(tf.clip_by_value(p,-c,c)) for p in disc_vars]" 260 | ] 261 | }, 262 | { 263 | "cell_type": "code", 264 | "execution_count": null, 265 | "metadata": {}, 266 | "outputs": [], 267 | "source": [ 268 | "with tf.Session() as sess:\n", 269 | " sess.run(tf.global_variables_initializer())\n", 270 | " for step in range(Iters):\n", 271 | "\n", 272 | " batch_x, _ = mnist.train.next_batch(Batch_Size)\n", 273 | " # Generate noise to feed to the generator\n", 274 | " z = np.random.uniform(-1., 1., size=[Batch_Size, noise_dim])\n", 275 | " \n", 276 | " # train discriminator\n", 277 | " for i in range(Critic_Iters): #have inner loop for discriminator\n", 278 | " _,dl = sess.run([train_disc,disc_cost],\n", 279 | " feed_dict={real_data:batch_x,gen_input:z})\n", 280 | " _ = sess.run(clip_D) #put weight_clipping here\n", 281 | " \n", 282 | " # train generator\n", 283 | " _,gl=sess.run([train_gen,gen_cost],\n", 284 | " feed_dict={gen_input:z})\n", 285 | " \n", 286 | " if step % 1000 == 0 or step == 1:\n", 287 | " print('Step %i: Generator Loss: %f, Discriminator Loss: %f' % (step, gl, dl))\n", 288 | " \n", 289 | " # Generate images from noise, using the generator network.\n", 290 | " if step % 10000 == 0 or step == 1:\n", 291 | " f, a = plt.subplots(4, 10, figsize=(10, 4))\n", 292 | " for i in range(10):\n", 293 | " # Noise input.\n", 294 | " z = np.random.uniform(-1., 1., size=[4, noise_dim])\n", 295 | " g = sess.run([fake_data], feed_dict={gen_input: z})\n", 296 | " g = np.reshape(g, newshape=(4, 28, 28, 1))\n", 297 | " # Reverse colours for better display\n", 298 | " g = -1 * (g - 1)\n", 299 | " img_inventory[\"WGAN_\"+str(step)] = g\n", 300 | " \n", 301 | " for j in range(4):\n", 302 | " # Generate image from noise. Extend to 3 channels for matplot figure.\n", 303 | " img = np.reshape(np.repeat(g[j][:, :, np.newaxis], 3, axis=2),\n", 304 | " newshape=(28, 28, 3))\n", 305 | " a[j][i].imshow(img)\n", 306 | "\n", 307 | " plt.draw()\n", 308 | " print('wgan'+str(step)+'.png')\n", 309 | " plt.savefig('wgan'+str(step)+'.png')" 310 | ] 311 | }, 312 | { 313 | "cell_type": "markdown", 314 | "metadata": {}, 315 | "source": [ 316 | "## WGAN to WGAN-GP " 317 | ] 318 | }, 319 | { 320 | "cell_type": "markdown", 321 | "metadata": {}, 322 | "source": [ 323 | "WGAN-GP uses the similar loss function as WGAN, but WGAN-GP has extra regularization component.
\n", 324 | "Besides, they optimize in different ways:
\n", 325 | "WGAN-GP uses AdamOptimizer, WGAN uses RMSOptimizer
\n", 326 | "WGAN-GP does not require weight clipping
\n", 327 | "WGAN-GP needs to hand pick penalty coefficient" 328 | ] 329 | }, 330 | { 331 | "cell_type": "code", 332 | "execution_count": null, 333 | "metadata": {}, 334 | "outputs": [], 335 | "source": [ 336 | "tf.reset_default_graph() # Clearing all tensors before this" 337 | ] 338 | }, 339 | { 340 | "cell_type": "code", 341 | "execution_count": null, 342 | "metadata": {}, 343 | "outputs": [], 344 | "source": [ 345 | "# add hyperparameter Critic_Iters and Lambda for WGAN-GP" 346 | ] 347 | }, 348 | { 349 | "cell_type": "code", 350 | "execution_count": null, 351 | "metadata": {}, 352 | "outputs": [], 353 | "source": [ 354 | "# trainning param\n", 355 | "Batch_Size = 50\n", 356 | "Critic_Iters = 5 # for WGAN and WGAN-GP, number of critic iters per gen iter\n", 357 | "Lambda = 10 # gradient penalty lambda hyperparameter\n", 358 | "Iters = 200000 # number of generator iterations to train for" 359 | ] 360 | }, 361 | { 362 | "cell_type": "code", 363 | "execution_count": null, 364 | "metadata": {}, 365 | "outputs": [], 366 | "source": [ 367 | "# Generator\n", 368 | "def generator(noises, reuse=False):\n", 369 | " with tf.variable_scope('generator') as scope:\n", 370 | " if (reuse):\n", 371 | " tf.get_variable_scope().reuse_variables()\n", 372 | " # hidden layer with name \"g_hidden\"\n", 373 | " hidden = tf.layers.dense(noises, gen_hidden_dim, tf.nn.relu, name='g_hidden')\n", 374 | " # out layer with name \"g_out\"\n", 375 | " out_images = tf.layers.dense(hidden, image_dim, tf.nn.sigmoid, name='g_out')\n", 376 | " return out_images\n", 377 | "\n", 378 | "# Discriminator\n", 379 | "def discriminator(images, reuse=False):\n", 380 | " with tf.variable_scope('discriminator') as scope:\n", 381 | " if (reuse):\n", 382 | " tf.get_variable_scope().reuse_variables() \n", 383 | " # hidden layer with name \"d_hidden\"\n", 384 | " hidden = tf.layers.dense(images, disc_hidden_dim, tf.nn.relu, name='d_hidden')\n", 385 | " # out layer with name \"d_out\"\n", 386 | " out = tf.layers.dense(hidden, 1, None, name='d_out') # output layer turn into a linear one as WGAN does\n", 387 | " return out" 388 | ] 389 | }, 390 | { 391 | "cell_type": "code", 392 | "execution_count": null, 393 | "metadata": {}, 394 | "outputs": [], 395 | "source": [ 396 | "gen_input = tf.placeholder(tf.float32, shape=[None, noise_dim], name='input_noise')" 397 | ] 398 | }, 399 | { 400 | "cell_type": "code", 401 | "execution_count": null, 402 | "metadata": {}, 403 | "outputs": [], 404 | "source": [ 405 | "fake_data = generator(gen_input)\n", 406 | "real_data = tf.placeholder(tf.float32, shape=[None, image_dim], name='real_data')" 407 | ] 408 | }, 409 | { 410 | "cell_type": "code", 411 | "execution_count": null, 412 | "metadata": {}, 413 | "outputs": [], 414 | "source": [ 415 | "disc_real = discriminator(real_data)\n", 416 | "disc_fake = discriminator(fake_data, reuse=True)" 417 | ] 418 | }, 419 | { 420 | "cell_type": "code", 421 | "execution_count": null, 422 | "metadata": {}, 423 | "outputs": [], 424 | "source": [ 425 | "gen_cost = -tf.reduce_mean(disc_fake)\n", 426 | "disc_cost = tf.reduce_mean(disc_fake) - tf.reduce_mean(disc_real)" 427 | ] 428 | }, 429 | { 430 | "cell_type": "code", 431 | "execution_count": null, 432 | "metadata": {}, 433 | "outputs": [], 434 | "source": [ 435 | "tvars = tf.trainable_variables()\n", 436 | "disc_vars = [var for var in tvars if 'd_' in var.name]\n", 437 | "gen_vars = [var for var in tvars if 'g_' in var.name]" 438 | ] 439 | }, 440 | { 441 | "cell_type": "code", 442 | "execution_count": null, 443 | "metadata": {}, 444 | "outputs": [], 445 | "source": [ 446 | "# add regularization component" 447 | ] 448 | }, 449 | { 450 | "cell_type": "code", 451 | "execution_count": null, 452 | "metadata": {}, 453 | "outputs": [], 454 | "source": [ 455 | "alpha = tf.random_uniform(shape=[Batch_Size,1],minval=0.,maxval=1.)\n", 456 | "differences = fake_data-real_data\n", 457 | "interpolates = real_data + (alpha*differences)\n", 458 | "gradients = tf.gradients(discriminator(interpolates, reuse=True),[interpolates])[0]\n", 459 | "slopes = tf.sqrt(tf.reduce_sum(tf.square(gradients),reduction_indices=[1]))\n", 460 | "gradient_penalty = tf.reduce_mean((slopes-1.)**2)\n", 461 | "disc_cost += Lambda*gradient_penalty" 462 | ] 463 | }, 464 | { 465 | "cell_type": "code", 466 | "execution_count": null, 467 | "metadata": {}, 468 | "outputs": [], 469 | "source": [ 470 | "# keep using AdamOptimizer" 471 | ] 472 | }, 473 | { 474 | "cell_type": "code", 475 | "execution_count": null, 476 | "metadata": {}, 477 | "outputs": [], 478 | "source": [ 479 | "train_gen = tf.train.AdamOptimizer(\n", 480 | " learning_rate=1e-4, \n", 481 | " beta1=0.5,\n", 482 | " beta2=0.9\n", 483 | " ).minimize(gen_cost, var_list=gen_vars)\n", 484 | "\n", 485 | "train_disc = tf.train.AdamOptimizer(\n", 486 | " learning_rate=1e-4, \n", 487 | " beta1=0.5, \n", 488 | " beta2=0.9\n", 489 | " ).minimize(disc_cost, var_list=disc_vars)" 490 | ] 491 | }, 492 | { 493 | "cell_type": "code", 494 | "execution_count": null, 495 | "metadata": {}, 496 | "outputs": [], 497 | "source": [ 498 | "with tf.Session() as sess:\n", 499 | " sess.run(tf.global_variables_initializer())\n", 500 | "\n", 501 | " for step in range(Iters):\n", 502 | "\n", 503 | " batch_x, _ = mnist.train.next_batch(Batch_Size)\n", 504 | " # Generate noise to feed to the generator\n", 505 | " z = np.random.uniform(-1., 1., size=[Batch_Size, noise_dim])\n", 506 | " \n", 507 | " # train discriminator\n", 508 | " for i in range(Critic_Iters):\n", 509 | " _,dl = sess.run([train_disc,disc_cost],\n", 510 | " feed_dict={real_data:batch_x,gen_input:z})\n", 511 | " \n", 512 | " # train generator\n", 513 | " _,gl=sess.run([train_gen,gen_cost],\n", 514 | " feed_dict={gen_input:z})\n", 515 | " \n", 516 | " if step % 1000 == 0 or step == 1:\n", 517 | " print('Step %i: Generator Loss: %f, Discriminator Loss: %f' % (step, gl, dl))\n", 518 | " \n", 519 | " # Generate images from noise, using the generator network.\n", 520 | " if step % 10000 == 0 or step == 1:\n", 521 | " f, a = plt.subplots(4, 10, figsize=(10, 4))\n", 522 | " for i in range(10):\n", 523 | " # Noise input.\n", 524 | " z = np.random.uniform(-1., 1., size=[4, noise_dim])\n", 525 | " g = sess.run([fake_data], feed_dict={gen_input: z})\n", 526 | " g = np.reshape(g, newshape=(4, 28, 28, 1))\n", 527 | " # Reverse colours for better display\n", 528 | " g = -1 * (g - 1)\n", 529 | " for j in range(4):\n", 530 | " # Generate image from noise. Extend to 3 channels for matplot figure.\n", 531 | " img = np.reshape(np.repeat(g[j][:, :, np.newaxis], 3, axis=2),\n", 532 | " newshape=(28, 28, 3))\n", 533 | " a[j][i].imshow(img)\n", 534 | "\n", 535 | " plt.draw()\n", 536 | " print('wgan_gp'+str(step)+'.png')\n", 537 | " plt.savefig('wgan_gp'+str(step)+'.png')" 538 | ] 539 | } 540 | ], 541 | "metadata": { 542 | "kernelspec": { 543 | "display_name": "Python [conda env:tensorflow_workshop]", 544 | "language": "python", 545 | "name": "conda-env-tensorflow_workshop-py" 546 | }, 547 | "language_info": { 548 | "codemirror_mode": { 549 | "name": "ipython", 550 | "version": 3 551 | }, 552 | "file_extension": ".py", 553 | "mimetype": "text/x-python", 554 | "name": "python", 555 | "nbconvert_exporter": "python", 556 | "pygments_lexer": "ipython3", 557 | "version": "3.6.4" 558 | } 559 | }, 560 | "nbformat": 4, 561 | "nbformat_minor": 2 562 | } 563 | -------------------------------------------------------------------------------- /environment.yml: -------------------------------------------------------------------------------- 1 | name: tensorflow_workshop 2 | channels: 3 | - defaults 4 | dependencies: 5 | - _nb_ext_conf=0.4.0=py36_1 6 | - anaconda-client=1.6.9=py36_0 7 | - appnope=0.1.0=py36hf537a9a_0 8 | - asn1crypto=0.24.0=py36_0 9 | - bleach=2.1.2=py36_0 10 | - ca-certificates=2017.08.26=ha1e5d58_0 11 | - cairo=1.14.12=hab642c4_0 12 | - certifi=2018.1.18=py36_0 13 | - cffi=1.11.4=py36h342bebf_0 14 | - chardet=3.0.4=py36h96c241c_1 15 | - clyent=1.2.2=py36hae3ad88_0 16 | - cryptography=2.1.4=py36h842514c_0 17 | - cycler=0.10.0=py36hfc81398_0 18 | - decorator=4.2.1=py36_0 19 | - entrypoints=0.2.3=py36hd81d71f_2 20 | - expat=2.2.5=hb8e80ba_0 21 | - fontconfig=2.12.4=hffb9db1_2 22 | - freetype=2.8=h12048fb_1 23 | - gettext=0.19.8.1=h15daf44_3 24 | - glib=2.53.6=h33f6a65_2 25 | - graphite2=1.3.10=h337f25e_1 26 | - harfbuzz=1.7.4=h08e020e_0 27 | - html5lib=1.0.1=py36h2f9c1c0_0 28 | - icu=58.2=h4b95b61_1 29 | - idna=2.6=py36h8628d0a_1 30 | - intel-openmp=2018.0.0=h8158457_8 31 | - ipykernel=4.8.0=py36_0 32 | - ipython=6.2.1=py36h3dda519_1 33 | - ipython_genutils=0.2.0=py36h241746c_0 34 | - ipywidgets=7.1.1=py36_0 35 | - jedi=0.11.1=py36_0 36 | - jinja2=2.10=py36hd36f9c5_0 37 | - jpeg=9b=he5867d9_2 38 | - jsonschema=2.6.0=py36hb385e00_0 39 | - jupyter_client=5.2.2=py36_0 40 | - jupyter_core=4.4.0=py36h79cf704_0 41 | - libcxx=4.0.1=h579ed51_0 42 | - libcxxabi=4.0.1=hebd6815_0 43 | - libedit=3.1=hb4e282d_0 44 | - libffi=3.2.1=h475c297_4 45 | - libgfortran=3.0.1=h93005f0_2 46 | - libiconv=1.15=hdd342a3_7 47 | - libpng=1.6.34=he12f830_0 48 | - libprotobuf=3.4.1=h326466f_0 49 | - libsodium=1.0.15=hd9e47c5_0 50 | - libtiff=4.0.9=h0dac147_0 51 | - libxml2=2.9.7=hab757c2_0 52 | - markupsafe=1.0=py36h3a1e703_1 53 | - matplotlib=2.1.2=py36h6d6146d_0 54 | - mistune=0.8.3=py36_0 55 | - mkl=2018.0.1=hfbd8650_4 56 | - nb_anacondacloud=1.4.0=py36_0 57 | - nb_conda=2.2.1=py36h349edbb_0 58 | - nb_conda_kernels=2.1.0=py36_0 59 | - nbconvert=5.3.1=py36h810822e_0 60 | - nbformat=4.4.0=py36h827af21_0 61 | - nbpresent=3.0.2=py36hef4c988_1 62 | - ncurses=6.0=hd04f020_2 63 | - notebook=5.4.0=py36_0 64 | - numpy=1.12.1=py36h8871d66_1 65 | - openssl=1.0.2n=hdbc3d79_0 66 | - pandas=0.22.0=py36h0a44026_0 67 | - pandoc=1.19.2.1=ha5e8f32_1 68 | - pandocfilters=1.4.2=py36h3b0b094_1 69 | - pango=1.41.0=h3bae10e_0 70 | - parso=0.1.1=py36hc90e01c_0 71 | - pcre=8.41=hfb6ab37_1 72 | - pexpect=4.3.1=py36_0 73 | - pickleshare=0.7.4=py36hf512f8e_0 74 | - pip=9.0.1=py36h1555ced_4 75 | - pixman=0.34.0=hca0a616_3 76 | - prompt_toolkit=1.0.15=py36haeda067_0 77 | - ptyprocess=0.5.2=py36he6521c3_0 78 | - pycparser=2.18=py36h724b2fc_1 79 | - pygments=2.2.0=py36h240cd3f_0 80 | - pyopenssl=17.5.0=py36h51e4350_0 81 | - pyparsing=2.2.0=py36hb281f35_0 82 | - pysocks=1.6.7=py36hfa33cec_1 83 | - python=3.6.4=hc167b69_1 84 | - python-dateutil=2.6.1=py36h86d2abb_1 85 | - pytz=2017.3=py36hf0bf824_0 86 | - pyyaml=3.12=py36h2ba1e63_1 87 | - pyzmq=16.0.3=py36he48b5ad_0 88 | - readline=7.0=hc1231fa_4 89 | - requests=2.18.4=py36h4516966_1 90 | - scikit-learn=0.19.1=py36hffbff8c_0 91 | - scipy=1.0.0=py36h1de22e9_0 92 | - send2trash=1.4.2=py36_0 93 | - setuptools=38.4.0=py36_0 94 | - simplegeneric=0.8.1=py36he5b5b09_0 95 | - six=1.11.0=py36h0e22d5e_1 96 | - sqlite=3.22.0=h3efe00b_0 97 | - terminado=0.8.1=py36_1 98 | - testpath=0.3.1=py36h625a49b_0 99 | - tk=8.6.7=h35a86e2_3 100 | - tornado=4.5.3=py36_0 101 | - traitlets=4.3.2=py36h65bd3ce_0 102 | - urllib3=1.22=py36h68b9469_0 103 | - wcwidth=0.1.7=py36h8c6ec74_0 104 | - webencodings=0.5.1=py36h3b9701d_1 105 | - werkzeug=0.14.1=py36_0 106 | - wheel=0.30.0=py36h5eb2c71_1 107 | - widgetsnbextension=3.1.0=py36_0 108 | - xz=5.2.3=h0278029_2 109 | - yaml=0.1.7=hc338f04_2 110 | - zeromq=4.2.2=ha360ad0_2 111 | - zlib=1.2.11=hf3cbc9b_2 112 | - pip: 113 | - blessings==1.6.1 114 | - enum34==1.1.6 115 | - graphviz==0.8.2 116 | - inception==0.0.3 117 | - inquirer==2.2.0 118 | - ipython-genutils==0.2.0 119 | - jupyter-client==5.2.2 120 | - jupyter-core==4.4.0 121 | - markdown==2.6.11 122 | - nb-anacondacloud==1.4.0 123 | - nb-conda==2.2.1 124 | - nb-conda-kernels==2.1.0 125 | - pillow==5.0.0 126 | - prompt-toolkit==1.0.15 127 | - protobuf==3.5.1 128 | - readchar==0.7 129 | - tensorflow==1.3.0 130 | - tensorflow-tensorboard==0.1.8 131 | - tqdm==4.19.5 132 | -------------------------------------------------------------------------------- /images/1_setup.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kojino/Harvard-Robust-Machine-Learning/77d694794cb2b47a2172ee2d225059f8146f6590/images/1_setup.png -------------------------------------------------------------------------------- /images/2_tensorflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kojino/Harvard-Robust-Machine-Learning/77d694794cb2b47a2172ee2d225059f8146f6590/images/2_tensorflow.png -------------------------------------------------------------------------------- /images/3_graph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kojino/Harvard-Robust-Machine-Learning/77d694794cb2b47a2172ee2d225059f8146f6590/images/3_graph.png -------------------------------------------------------------------------------- /images/gan1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kojino/Harvard-Robust-Machine-Learning/77d694794cb2b47a2172ee2d225059f8146f6590/images/gan1.png -------------------------------------------------------------------------------- /images/gan1000.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kojino/Harvard-Robust-Machine-Learning/77d694794cb2b47a2172ee2d225059f8146f6590/images/gan1000.png -------------------------------------------------------------------------------- /images/gan10000.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kojino/Harvard-Robust-Machine-Learning/77d694794cb2b47a2172ee2d225059f8146f6590/images/gan10000.png -------------------------------------------------------------------------------- /images/gan20000.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kojino/Harvard-Robust-Machine-Learning/77d694794cb2b47a2172ee2d225059f8146f6590/images/gan20000.png -------------------------------------------------------------------------------- /images/gan30000.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kojino/Harvard-Robust-Machine-Learning/77d694794cb2b47a2172ee2d225059f8146f6590/images/gan30000.png -------------------------------------------------------------------------------- /images/gan40000.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kojino/Harvard-Robust-Machine-Learning/77d694794cb2b47a2172ee2d225059f8146f6590/images/gan40000.png -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- 1 | from graphviz import Digraph 2 | 3 | def tf_to_dot(graph): 4 | "Visualize TensorFlow graphs on jupyter notebook" 5 | 6 | dot = Digraph() 7 | 8 | for n in graph.as_graph_def().node: 9 | dot.node(n.name, label=n.name) 10 | 11 | for i in n.input: 12 | dot.edge(i, n.name) 13 | 14 | return dot 15 | --------------------------------------------------------------------------------