├── .gitignore ├── LICENSE ├── Lec01_Overview of Tensorflow └── Lec01_Overview of Tensorflow.ipynb ├── Lec02_Operations └── Lec02_Operations.ipynb ├── Lec03_Linear and Logistic Regression ├── How to simply use tf.data.ipynb ├── Lec03_Linear Regression with huber loss by high-level.ipynb ├── Lec03_Linear Regression with huber loss by low-level.ipynb ├── Lec03_Linear Regression with mse loss.ipynb ├── Lec03_Linear Regression with tf.data.ipynb ├── Lec03_Linear Regression with tf.data_de.ipynb ├── Lec03_Logistic Regression with ce loss.ipynb ├── Lec03_Logistic Regression with tf.data.ipynb └── Lec03_Logistic Regression with tf.data_de.ipynb ├── Lec04_Eager execution ├── Lec04_Automatic differentiation and gradient tape.ipynb ├── Lec04_Custom training basics.ipynb ├── Lec04_Custom training subclassing.ipynb ├── Lec04_Custom training walkthrough.ipynb └── Lec04_Eager execution.ipynb ├── Lec05_Variable sharing and managing experiments ├── How to use keras.ipynb ├── Lec05_Applied example with tf.data.ipynb ├── Lec05_Applied example with tf.data_kde.ipynb ├── Lec05_Applied example with tf.placeholder.ipynb ├── Lec05_Randomization.ipynb ├── Lec05_Variable sharing.ipynb └── Lec05_Word2vec_simple.ipynb ├── Lec07_ConvNet in Tensorflow ├── Lec07_ConvNet mnist by high-level.ipynb ├── Lec07_ConvNet mnist by high-level_kd.ipynb ├── Lec07_ConvNet mnist by high-level_kde.ipynb ├── Lec07_ConvNet mnist by low-level.ipynb ├── Lec07_ConvNet mnist with Weight initialization and Batch norm.ipynb ├── Lec07_ConvNet mnist with Weight initialization and Batch norm_kd.ipynb ├── Lec07_ConvNet mnist with Weight initialization and Batch norm_kde.ipynb ├── Lec07_ConvNet mnist with Weight initialization and Drop out.ipynb └── Lec07_ConvNet mnist with Weight initialization and Drop out_kde.ipynb ├── Lec11_Recurrent Neural Networks ├── Lec11_Many to Many Classification by Bi-directional GRU.ipynb ├── Lec11_Many to Many Classification by Bi-directional LSTM.ipynb ├── Lec11_Many to Many Classification by Bi-directional RNN.ipynb ├── Lec11_Many to Many Classification by GRU.ipynb ├── Lec11_Many to Many Classification by LSTM.ipynb ├── Lec11_Many to Many Classification by RNN.ipynb ├── Lec11_Many to Many Classification by Stacked Bi-directional GRU with Drop out.ipynb ├── Lec11_Many to Many Classification by Stacked Bi-directional LSTM with Drop out.ipynb ├── Lec11_Many to Many Classification by Stacked Bi-directional RNN with Drop out.ipynb ├── Lec11_Many to Many Classification by Stacked GRU with Drop out.ipynb ├── Lec11_Many to Many Classification by Stacked LSTM with Drop out.ipynb ├── Lec11_Many to Many Classification by Stacked RNN with Drop out.ipynb ├── Lec11_Many to One Classification by Bi-directional GRU.ipynb ├── Lec11_Many to One Classification by Bi-directional LSTM.ipynb ├── Lec11_Many to One Classification by Bi-directional RNN.ipynb ├── Lec11_Many to One Classification by GRU.ipynb ├── Lec11_Many to One Classification by LSTM.ipynb ├── Lec11_Many to One Classification by RNN.ipynb ├── Lec11_Many to One Classification by RNN_kde.ipynb ├── Lec11_Many to One Classification by Stacked Bi-directional GRU with Drop out.ipynb ├── Lec11_Many to One Classification by Stacked Bi-directional LSTM with Drop out.ipynb ├── Lec11_Many to One Classification by Stacked Bi-directional RNN with Drop out.ipynb ├── Lec11_Many to One Classification by Stacked GRU with Drop out.ipynb ├── Lec11_Many to One Classification by Stacked LSTM with Drop out.ipynb ├── Lec11_Many to One Classification by Stacked RNN with Drop out.ipynb └── To quickly implementing RNN.ipynb ├── Lec12_Seq2Seq with Attention ├── Lec12_Seq2Seq by Encoder Bi-directional RNN and Decoder RNN.ipynb ├── Lec12_Seq2Seq by Encoder RNN and Decoder RNN.ipynb ├── Lec12_Seq2Seq with Attention by Encoder Bi-directional RNN and Decoder RNN.ipynb └── Lec12_Seq2Seq with Attention by Encoder RNN and Decoder RNN.ipynb ├── README.md ├── data ├── lecture03 │ ├── .DS_Store │ ├── example_with_data │ │ ├── .DS_Store │ │ ├── train_dir │ │ │ ├── birth_life_2010_tr_1.txt │ │ │ ├── birth_life_2010_tr_10.txt │ │ │ ├── birth_life_2010_tr_11.txt │ │ │ ├── birth_life_2010_tr_12.txt │ │ │ ├── birth_life_2010_tr_13.txt │ │ │ ├── birth_life_2010_tr_14.txt │ │ │ ├── birth_life_2010_tr_2.txt │ │ │ ├── birth_life_2010_tr_3.txt │ │ │ ├── birth_life_2010_tr_4.txt │ │ │ ├── birth_life_2010_tr_5.txt │ │ │ ├── birth_life_2010_tr_6.txt │ │ │ ├── birth_life_2010_tr_7.txt │ │ │ ├── birth_life_2010_tr_8.txt │ │ │ └── birth_life_2010_tr_9.txt │ │ └── val_dir │ │ │ └── birth_life_2010_val.txt │ └── example_with_placeholder │ │ └── birth_life_2010.txt └── lecture04 │ ├── iris_test.csv │ └── iris_training.csv └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | # .DS_Store in mac 2 | .DS_Store 3 | 4 | # ipython notebook checkpoints 5 | .ipynb_checkpoints 6 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 BOSEOP KIM 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /Lec01_Overview of Tensorflow/Lec01_Overview of Tensorflow.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# CS 20 : TensorFlow for Deep Learning Research\n", 8 | "## Lecture 01 : Overview of Tensorflow" 9 | ] 10 | }, 11 | { 12 | "cell_type": "markdown", 13 | "metadata": {}, 14 | "source": [ 15 | "### Setup" 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": 1, 21 | "metadata": {}, 22 | "outputs": [ 23 | { 24 | "name": "stdout", 25 | "output_type": "stream", 26 | "text": [ 27 | "1.12.0\n" 28 | ] 29 | } 30 | ], 31 | "source": [ 32 | "from __future__ import absolute_import, division, print_function\n", 33 | "import numpy as np\n", 34 | "import tensorflow as tf\n", 35 | "\n", 36 | "print(tf.__version__)\n", 37 | "sess_config = tf.ConfigProto(gpu_options=tf.GPUOptions(allow_growth=True))" 38 | ] 39 | }, 40 | { 41 | "cell_type": "code", 42 | "execution_count": 2, 43 | "metadata": {}, 44 | "outputs": [ 45 | { 46 | "name": "stdout", 47 | "output_type": "stream", 48 | "text": [ 49 | "Tensor(\"Add:0\", shape=(), dtype=int32)\n" 50 | ] 51 | } 52 | ], 53 | "source": [ 54 | "a = tf.add(3, 5)\n", 55 | "print(a)" 56 | ] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "execution_count": 3, 61 | "metadata": {}, 62 | "outputs": [ 63 | { 64 | "name": "stdout", 65 | "output_type": "stream", 66 | "text": [ 67 | "8\n" 68 | ] 69 | } 70 | ], 71 | "source": [ 72 | "sess = tf.Session(config = sess_config)\n", 73 | "print(sess.run(a))\n", 74 | "sess.close()" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": 4, 80 | "metadata": {}, 81 | "outputs": [ 82 | { 83 | "name": "stdout", 84 | "output_type": "stream", 85 | "text": [ 86 | "Tensor(\"Add:0\", shape=(), dtype=int32)\n", 87 | "8\n" 88 | ] 89 | } 90 | ], 91 | "source": [ 92 | "with tf.Session(config = sess_config) as sess:\n", 93 | " print(a)\n", 94 | " print(sess.run(a))" 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": 5, 100 | "metadata": {}, 101 | "outputs": [ 102 | { 103 | "name": "stdout", 104 | "output_type": "stream", 105 | "text": [ 106 | "[, , ]\n" 107 | ] 108 | } 109 | ], 110 | "source": [ 111 | "print(tf.get_default_graph().get_operations())" 112 | ] 113 | }, 114 | { 115 | "cell_type": "code", 116 | "execution_count": 6, 117 | "metadata": {}, 118 | "outputs": [ 119 | { 120 | "name": "stdout", 121 | "output_type": "stream", 122 | "text": [ 123 | "15625 10\n" 124 | ] 125 | } 126 | ], 127 | "source": [ 128 | "tf.reset_default_graph()\n", 129 | "\n", 130 | "x, y = 2, 3\n", 131 | "add_op = tf.add(x,y)\n", 132 | "mul_op = tf.multiply(x,y)\n", 133 | "useless = tf.multiply(x, add_op)\n", 134 | "pow_op = tf.pow(add_op, mul_op)\n", 135 | "with tf.Session(config = sess_config) as sess:\n", 136 | " z, not_useless = sess.run([pow_op, useless])\n", 137 | "print(z, not_useless)" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 7, 143 | "metadata": {}, 144 | "outputs": [ 145 | { 146 | "name": "stdout", 147 | "output_type": "stream", 148 | "text": [ 149 | "[, , , , , , , , ]\n" 150 | ] 151 | } 152 | ], 153 | "source": [ 154 | "print(tf.get_default_graph().get_operations())" 155 | ] 156 | }, 157 | { 158 | "cell_type": "code", 159 | "execution_count": 8, 160 | "metadata": {}, 161 | "outputs": [ 162 | { 163 | "ename": "RuntimeError", 164 | "evalue": "The Session graph is empty. Add operations to the graph before calling run().", 165 | "output_type": "error", 166 | "traceback": [ 167 | "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", 168 | "\u001b[0;31mRuntimeError\u001b[0m Traceback (most recent call last)", 169 | "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 6\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSession\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconfig\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msess_config\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0msess\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# graph argument is none -> launch default graph\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", 170 | "\u001b[0;32m/usr/local/var/pyenv/versions/3.6.6/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m 927\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 928\u001b[0m result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 929\u001b[0;31m run_metadata_ptr)\n\u001b[0m\u001b[1;32m 930\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 931\u001b[0m \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 171 | "\u001b[0;32m/usr/local/var/pyenv/versions/3.6.6/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m 1075\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Attempted to use a closed Session.'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1076\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgraph\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mversion\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1077\u001b[0;31m raise RuntimeError('The Session graph is empty. Add operations to the '\n\u001b[0m\u001b[1;32m 1078\u001b[0m 'graph before calling run().')\n\u001b[1;32m 1079\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", 172 | "\u001b[0;31mRuntimeError\u001b[0m: The Session graph is empty. Add operations to the graph before calling run()." 173 | ] 174 | } 175 | ], 176 | "source": [ 177 | "tf.reset_default_graph()\n", 178 | "\n", 179 | "g = tf.Graph()\n", 180 | "with g.as_default():\n", 181 | " x = tf.add(3,5)\n", 182 | "with tf.Session(config = sess_config) as sess: # graph argument is none -> launch default graph\n", 183 | " print(sess.run(x))" 184 | ] 185 | }, 186 | { 187 | "cell_type": "code", 188 | "execution_count": 9, 189 | "metadata": {}, 190 | "outputs": [ 191 | { 192 | "name": "stdout", 193 | "output_type": "stream", 194 | "text": [ 195 | "[, , ]\n" 196 | ] 197 | } 198 | ], 199 | "source": [ 200 | "print(g.get_operations())" 201 | ] 202 | }, 203 | { 204 | "cell_type": "code", 205 | "execution_count": 10, 206 | "metadata": {}, 207 | "outputs": [ 208 | { 209 | "name": "stdout", 210 | "output_type": "stream", 211 | "text": [ 212 | "[]\n" 213 | ] 214 | } 215 | ], 216 | "source": [ 217 | "print(tf.get_default_graph().get_operations())" 218 | ] 219 | } 220 | ], 221 | "metadata": { 222 | "kernelspec": { 223 | "display_name": "Python 3", 224 | "language": "python", 225 | "name": "python3" 226 | }, 227 | "language_info": { 228 | "codemirror_mode": { 229 | "name": "ipython", 230 | "version": 3 231 | }, 232 | "file_extension": ".py", 233 | "mimetype": "text/x-python", 234 | "name": "python", 235 | "nbconvert_exporter": "python", 236 | "pygments_lexer": "ipython3", 237 | "version": "3.6.6" 238 | }, 239 | "varInspector": { 240 | "cols": { 241 | "lenName": 16, 242 | "lenType": 16, 243 | "lenVar": 40 244 | }, 245 | "kernels_config": { 246 | "python": { 247 | "delete_cmd_postfix": "", 248 | "delete_cmd_prefix": "del ", 249 | "library": "var_list.py", 250 | "varRefreshCmd": "print(var_dic_list())" 251 | }, 252 | "r": { 253 | "delete_cmd_postfix": ") ", 254 | "delete_cmd_prefix": "rm(", 255 | "library": "var_list.r", 256 | "varRefreshCmd": "cat(var_dic_list()) " 257 | } 258 | }, 259 | "types_to_exclude": [ 260 | "module", 261 | "function", 262 | "builtin_function_or_method", 263 | "instance", 264 | "_Feature" 265 | ], 266 | "window_display": false 267 | } 268 | }, 269 | "nbformat": 4, 270 | "nbformat_minor": 2 271 | } 272 | -------------------------------------------------------------------------------- /Lec03_Linear and Logistic Regression/How to simply use tf.data.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# How to simply use tf.data\n", 8 | "`tf.data` package를 사용하는 방법에 관한 예시, 예시 데이터는 `numpy` package를 이용하여 간단하게 data에 해당하는 `X`, target에 해당하는 `y`를 생성하여 `tf.data` package의 각종 module, function을 이용한다. epoch 마다 validation data에 대해서 validation을 하는 상황을 가정" 9 | ] 10 | }, 11 | { 12 | "cell_type": "markdown", 13 | "metadata": {}, 14 | "source": [ 15 | "## Template\n", 16 | "for문을 활용, model을 training시 data pipeline으로써 아래의 function과 method를 사용하는 방법에 대한 예시\n", 17 | "- **Dataset class**\n", 18 | " - `tf.data.Dataset.from_tensor_slices`으로 Dataset class의 instance를 생성\n", 19 | " - train data에 대한 Dataset class의 instance, tr_data\n", 20 | " - validation data에 대한 Dataset class의 instance, val_data\n", 21 | " - 아래와 같은 method를 활용하여 training 시 필요한 요소를 지정\n", 22 | " - instance의 `shuffle` method를 활용하여, shuffling\n", 23 | " - instanec의 `batch` method를 활용하여, batch size 지정\n", 24 | " - for문으로 전체 epoch를 control하므로 `repeat` method는 활용하지 않음\n", 25 | "\n", 26 | "\n", 27 | "- **Iterator class**\n", 28 | " - Dataset class의 instance에서 `make_initializable_iterator` method로 Iterator class의 instance를 생성\n", 29 | " - train data에 대한 iterator class의 instance, tr_iterator\n", 30 | " - validation data에 대한 iterator class의 instance, val_iterator\n", 31 | " - 주의사항 : `make_initializable_iterator` method로 Iterator class의 instance를 생성할 경우, random_seed를 고정 X\n", 32 | " - random_seed를 고정할 경우, 서로 다른 epoch의 step 별 mini-batch의 구성이 완전히 똑같아지기 때문\n", 33 | " - Anonymous iterator를 `tf.data.Iterator.from_string_handle`로 생성\n", 34 | " - `string_handle` argument에 `tf.placeholder`를 이용\n", 35 | " - tr_iterator를 활용할 것인지, val_iterator를 활용할 것인지 조절" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": {}, 41 | "source": [ 42 | "### Setup" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 1, 48 | "metadata": {}, 49 | "outputs": [ 50 | { 51 | "name": "stdout", 52 | "output_type": "stream", 53 | "text": [ 54 | "1.12.0\n" 55 | ] 56 | } 57 | ], 58 | "source": [ 59 | "from __future__ import absolute_import, division, print_function\n", 60 | "import numpy as np\n", 61 | "import tensorflow as tf\n", 62 | "\n", 63 | "print(tf.__version__)" 64 | ] 65 | }, 66 | { 67 | "cell_type": "code", 68 | "execution_count": 2, 69 | "metadata": {}, 70 | "outputs": [ 71 | { 72 | "name": "stdout", 73 | "output_type": "stream", 74 | "text": [ 75 | "(12, 2) (12,)\n" 76 | ] 77 | } 78 | ], 79 | "source": [ 80 | "# 전체 데이터의 개수가 12개인 임의의 데이터셋 생성\n", 81 | "X = np.c_[np.arange(12), np.arange(12)]\n", 82 | "y = np.arange(12)\n", 83 | "\n", 84 | "print(X.shape, y.shape)" 85 | ] 86 | }, 87 | { 88 | "cell_type": "code", 89 | "execution_count": 3, 90 | "metadata": {}, 91 | "outputs": [ 92 | { 93 | "name": "stdout", 94 | "output_type": "stream", 95 | "text": [ 96 | "(8, 2) (8,)\n", 97 | "(4, 2) (4,)\n" 98 | ] 99 | } 100 | ], 101 | "source": [ 102 | "# 위의 데이터를 train, validation으로 split\n", 103 | "X_tr = X[:8]\n", 104 | "y_tr = y[:8]\n", 105 | "\n", 106 | "X_val = X[8:]\n", 107 | "y_val = y[8:]\n", 108 | "\n", 109 | "print(X_tr.shape, y_tr.shape)\n", 110 | "print(X_val.shape, y_val.shape)" 111 | ] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "metadata": {}, 116 | "source": [ 117 | "### Template" 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": 4, 123 | "metadata": {}, 124 | "outputs": [ 125 | { 126 | "name": "stdout", 127 | "output_type": "stream", 128 | "text": [ 129 | "epoch : 3, batch_size : 2, total_steps : 4\n" 130 | ] 131 | } 132 | ], 133 | "source": [ 134 | "n_epoch = 3\n", 135 | "batch_size = 2\n", 136 | "total_steps = int(X_tr.shape[0] / batch_size)\n", 137 | "print('epoch : {}, batch_size : {}, total_steps : {}'.format(n_epoch, batch_size, total_steps))" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 5, 143 | "metadata": {}, 144 | "outputs": [ 145 | { 146 | "name": "stdout", 147 | "output_type": "stream", 148 | "text": [ 149 | "\n", 150 | "\n" 151 | ] 152 | } 153 | ], 154 | "source": [ 155 | "tr_data = tf.data.Dataset.from_tensor_slices((X_tr, y_tr)) # 0th dimension의 size가 같아야\n", 156 | "tr_data = tr_data.shuffle(buffer_size = 30)\n", 157 | "tr_data = tr_data.batch(batch_size = batch_size)\n", 158 | "\n", 159 | "val_data = tf.data.Dataset.from_tensor_slices((X_val, y_val))\n", 160 | "val_data = val_data.batch(batch_size = batch_size)\n", 161 | "\n", 162 | "print(tr_data)\n", 163 | "print(val_data)" 164 | ] 165 | }, 166 | { 167 | "cell_type": "code", 168 | "execution_count": 6, 169 | "metadata": {}, 170 | "outputs": [], 171 | "source": [ 172 | "tr_iterator = tr_data.make_initializable_iterator()\n", 173 | "val_iterator = val_data.make_initializable_iterator()\n", 174 | "\n", 175 | "handle = tf.placeholder(dtype = tf.string)" 176 | ] 177 | }, 178 | { 179 | "cell_type": "code", 180 | "execution_count": 7, 181 | "metadata": {}, 182 | "outputs": [], 183 | "source": [ 184 | "iterator = tf.data.Iterator.from_string_handle(string_handle = handle,\n", 185 | " output_shapes = tr_iterator.output_shapes,\n", 186 | " output_types = tr_iterator.output_types)\n", 187 | "X_mb, y_mb = iterator.get_next()" 188 | ] 189 | }, 190 | { 191 | "cell_type": "code", 192 | "execution_count": 8, 193 | "metadata": {}, 194 | "outputs": [ 195 | { 196 | "name": "stdout", 197 | "output_type": "stream", 198 | "text": [ 199 | "epoch : 1 training start\n", 200 | "step : 1\n", 201 | "[[0 0]\n", 202 | " [1 1]] [0 1]\n", 203 | "step : 2\n", 204 | "[[4 4]\n", 205 | " [3 3]] [4 3]\n", 206 | "step : 3\n", 207 | "[[5 5]\n", 208 | " [2 2]] [5 2]\n", 209 | "step : 4\n", 210 | "[[7 7]\n", 211 | " [6 6]] [7 6]\n", 212 | "epoch : 1 training finished\n", 213 | "at epoch : 1, validation start\n", 214 | "step : 1\n", 215 | "[[8 8]\n", 216 | " [9 9]] [8 9]\n", 217 | "step : 2\n", 218 | "[[10 10]\n", 219 | " [11 11]] [10 11]\n", 220 | "validation finished\n", 221 | "epoch : 2 training start\n", 222 | "step : 1\n", 223 | "[[3 3]\n", 224 | " [5 5]] [3 5]\n", 225 | "step : 2\n", 226 | "[[4 4]\n", 227 | " [0 0]] [4 0]\n", 228 | "step : 3\n", 229 | "[[1 1]\n", 230 | " [2 2]] [1 2]\n", 231 | "step : 4\n", 232 | "[[7 7]\n", 233 | " [6 6]] [7 6]\n", 234 | "epoch : 2 training finished\n", 235 | "at epoch : 2, validation start\n", 236 | "step : 1\n", 237 | "[[8 8]\n", 238 | " [9 9]] [8 9]\n", 239 | "step : 2\n", 240 | "[[10 10]\n", 241 | " [11 11]] [10 11]\n", 242 | "validation finished\n", 243 | "epoch : 3 training start\n", 244 | "step : 1\n", 245 | "[[4 4]\n", 246 | " [3 3]] [4 3]\n", 247 | "step : 2\n", 248 | "[[6 6]\n", 249 | " [1 1]] [6 1]\n", 250 | "step : 3\n", 251 | "[[0 0]\n", 252 | " [2 2]] [0 2]\n", 253 | "step : 4\n", 254 | "[[7 7]\n", 255 | " [5 5]] [7 5]\n", 256 | "epoch : 3 training finished\n", 257 | "at epoch : 3, validation start\n", 258 | "step : 1\n", 259 | "[[8 8]\n", 260 | " [9 9]] [8 9]\n", 261 | "step : 2\n", 262 | "[[10 10]\n", 263 | " [11 11]] [10 11]\n", 264 | "validation finished\n" 265 | ] 266 | } 267 | ], 268 | "source": [ 269 | "# n_tr_step, n_val_step 변수와 관련된 코드는 step 수 확인을 위해 넣어놓음\n", 270 | "sess_config = tf.ConfigProto(gpu_options=tf.GPUOptions(allow_growth=True))\n", 271 | "sess = tf.Session(config = sess_config)\n", 272 | "tr_handle, val_handle = sess.run([tr_iterator.string_handle(), val_iterator.string_handle()])\n", 273 | "\n", 274 | "for epoch in range(n_epoch):\n", 275 | " \n", 276 | " print('epoch : {} training start'.format(epoch + 1))\n", 277 | " sess.run(tr_iterator.initializer) # run tr_iterator\n", 278 | " n_tr_step = 0\n", 279 | " \n", 280 | " while True:\n", 281 | " try:\n", 282 | " n_tr_step += 1\n", 283 | " X_tmp, y_tmp = sess.run([X_mb, y_mb], feed_dict = {handle : tr_handle})\n", 284 | " print('step : {}'.format(n_tr_step))\n", 285 | " print(X_tmp, y_tmp)\n", 286 | " \n", 287 | " except:\n", 288 | " print('epoch : {} training finished'.format(epoch + 1))\n", 289 | " break\n", 290 | "\n", 291 | " print('at epoch : {}, validation start'.format(epoch + 1)) \n", 292 | " sess.run(val_iterator.initializer)\n", 293 | " n_val_step = 0\n", 294 | " while True:\n", 295 | " try:\n", 296 | " n_val_step += 1\n", 297 | " X_tmp, y_tmp = sess.run([X_mb, y_mb], feed_dict = {handle : val_handle})\n", 298 | " \n", 299 | " print('step : {}'.format(n_val_step))\n", 300 | " print(X_tmp, y_tmp)\n", 301 | " except:\n", 302 | " print('validation finished')\n", 303 | " break" 304 | ] 305 | } 306 | ], 307 | "metadata": { 308 | "kernelspec": { 309 | "display_name": "Python 3", 310 | "language": "python", 311 | "name": "python3" 312 | }, 313 | "language_info": { 314 | "codemirror_mode": { 315 | "name": "ipython", 316 | "version": 3 317 | }, 318 | "file_extension": ".py", 319 | "mimetype": "text/x-python", 320 | "name": "python", 321 | "nbconvert_exporter": "python", 322 | "pygments_lexer": "ipython3", 323 | "version": "3.6.6" 324 | } 325 | }, 326 | "nbformat": 4, 327 | "nbformat_minor": 2 328 | } 329 | -------------------------------------------------------------------------------- /Lec04_Eager execution/Lec04_Automatic differentiation and gradient tape.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# CS 20 : TensorFlow for Deep Learning Research\n", 8 | "## Lecture 04 : Eager execution\n", 9 | "### Automatic differentiation and gradient tape\n", 10 | "* Reference\n", 11 | " + https://www.tensorflow.org/tutorials/eager/automatic_differentiation?hl=ko" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "### Setup" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 1, 24 | "metadata": {}, 25 | "outputs": [ 26 | { 27 | "name": "stdout", 28 | "output_type": "stream", 29 | "text": [ 30 | "1.12.0\n" 31 | ] 32 | } 33 | ], 34 | "source": [ 35 | "from __future__ import absolute_import, division, print_function\n", 36 | "import numpy as np\n", 37 | "import tensorflow as tf\n", 38 | "\n", 39 | "tf.enable_eager_execution()\n", 40 | "\n", 41 | "print(tf.__version__)" 42 | ] 43 | }, 44 | { 45 | "cell_type": "markdown", 46 | "metadata": {}, 47 | "source": [ 48 | "### Gradient tapes\n", 49 | "\n", 50 | "TensorFlow provides the `tf.GradientTape` API for automatic differentiation - ***computing the gradient of a computation with respect to its input variables. Tensorflow \"records\" all operations executed inside the context of a `tf.GradientTape` onto a \"tape\". Tensorflow then uses that tape and the gradients associated with each recorded operation to compute the gradients of a \"recorded\" computation using reverse mode differentiation.***" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": 2, 56 | "metadata": {}, 57 | "outputs": [ 58 | { 59 | "name": "stdout", 60 | "output_type": "stream", 61 | "text": [ 62 | "tf.Tensor(8.0, shape=(), dtype=float32)\n" 63 | ] 64 | } 65 | ], 66 | "source": [ 67 | "# Trainable variables (created by `tf.Variable` or `tf.get_variable`, where\n", 68 | "# `trainable=True` is default in both cases) are automatically watched. Tensors\n", 69 | "# can be manually watched by invoking the `watch` method on this context\n", 70 | "# manager.\n", 71 | "\n", 72 | "x = tf.constant(1, dtype = tf.float32)\n", 73 | "\n", 74 | "# z = y^2, y = 2x, z = (2x)^2\n", 75 | "with tf.GradientTape() as tape:\n", 76 | " tape.watch(x)\n", 77 | " y = tf.add(x, x)\n", 78 | " z = tf.multiply(y, y)\n", 79 | " \n", 80 | "# Derivative of z with respect to the original input tensor x\n", 81 | "dz_dx = tape.gradient(target = z, sources = x)\n", 82 | "print(dz_dx)" 83 | ] 84 | }, 85 | { 86 | "cell_type": "markdown", 87 | "metadata": {}, 88 | "source": [ 89 | "You can also request gradients of the output with respect to intermediate values computed during a \"recorded\" `tf.GradientTape` context." 90 | ] 91 | }, 92 | { 93 | "cell_type": "code", 94 | "execution_count": 3, 95 | "metadata": {}, 96 | "outputs": [ 97 | { 98 | "name": "stdout", 99 | "output_type": "stream", 100 | "text": [ 101 | "tf.Tensor(4.0, shape=(), dtype=float32)\n" 102 | ] 103 | } 104 | ], 105 | "source": [ 106 | "x = tf.constant(1, dtype = tf.float32)\n", 107 | "\n", 108 | "# z = y^2, y = 2x, z = (2x)^2\n", 109 | "with tf.GradientTape() as tape:\n", 110 | " tape.watch(x)\n", 111 | " y = tf.add(x, x)\n", 112 | " z = tf.multiply(y, y)\n", 113 | " \n", 114 | "# Use the tape to compute the derivative of z with respect to the\n", 115 | "# intermediate value y.\n", 116 | "dz_dy = tape.gradient(target = z, sources = y)\n", 117 | "print(dz_dy)" 118 | ] 119 | }, 120 | { 121 | "cell_type": "markdown", 122 | "metadata": {}, 123 | "source": [ 124 | "By default, the resources held by a GradientTape are released as soon as GradientTape.gradient() method is called. To compute multiple gradients over the same computation, create a persistent gradient tape. This allows multiple calls to the `gradient()` method. as resources are released when the tape object is garbage collected. For example:" 125 | ] 126 | }, 127 | { 128 | "cell_type": "code", 129 | "execution_count": 4, 130 | "metadata": {}, 131 | "outputs": [ 132 | { 133 | "name": "stdout", 134 | "output_type": "stream", 135 | "text": [ 136 | "tf.Tensor(4.0, shape=(), dtype=float32) tf.Tensor(2.0, shape=(), dtype=float32) tf.Tensor(8.0, shape=(), dtype=float32)\n" 137 | ] 138 | } 139 | ], 140 | "source": [ 141 | "x = tf.constant(1, dtype = tf.float32)\n", 142 | "\n", 143 | "# z = y^2, y = 2x, z = (2x)^2\n", 144 | "with tf.GradientTape(persistent = True) as tape:\n", 145 | " tape.watch(x)\n", 146 | " y = tf.add(x, x)\n", 147 | " z = tf.multiply(y, y)\n", 148 | " \n", 149 | "dz_dy = tape.gradient(target = z, sources = y)\n", 150 | "dy_dx = tape.gradient(target = y, sources = x)\n", 151 | "dz_dx = tape.gradient(target = z, sources = x)\n", 152 | "\n", 153 | "print(dz_dy, dy_dx, dz_dx)" 154 | ] 155 | }, 156 | { 157 | "cell_type": "markdown", 158 | "metadata": {}, 159 | "source": [ 160 | "#### Recording control flow\n", 161 | "Because tapes record operations as they are executed, Python control flow (using `if`s and `while`s for example) is naturally handled:" 162 | ] 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": 5, 167 | "metadata": {}, 168 | "outputs": [ 169 | { 170 | "name": "stdout", 171 | "output_type": "stream", 172 | "text": [ 173 | "tf.Tensor(12.0, shape=(), dtype=float32)\n", 174 | "tf.Tensor(12.0, shape=(), dtype=float32)\n", 175 | "tf.Tensor(4.0, shape=(), dtype=float32)\n" 176 | ] 177 | } 178 | ], 179 | "source": [ 180 | "def f(x, y):\n", 181 | " output = 1.0\n", 182 | " for i in range(y):\n", 183 | " if i > 1 and i < 5:\n", 184 | " output = tf.multiply(output, x)\n", 185 | " return output\n", 186 | "\n", 187 | "def grad(x, y):\n", 188 | " with tf.GradientTape() as tape:\n", 189 | " tape.watch(x)\n", 190 | " out = f(x, y)\n", 191 | " return tape.gradient(out, x) \n", 192 | "\n", 193 | "x = tf.convert_to_tensor(2.0)\n", 194 | "\n", 195 | "print(grad(x, 6)) # out = x^3\n", 196 | "print(grad(x, 5)) # out = x^3\n", 197 | "print(grad(x, 4)) # out = x^2" 198 | ] 199 | }, 200 | { 201 | "cell_type": "markdown", 202 | "metadata": {}, 203 | "source": [ 204 | "#### Higher-order gradients" 205 | ] 206 | }, 207 | { 208 | "cell_type": "markdown", 209 | "metadata": {}, 210 | "source": [ 211 | "Operations inside of the `GradientTape` context manager are recorded for automatic differentiation. If gradients are computed in that context, then the gradient computation is recorded as well. As a result, the exact same API works for higher-order gradients as well. For example:" 212 | ] 213 | }, 214 | { 215 | "cell_type": "code", 216 | "execution_count": 6, 217 | "metadata": {}, 218 | "outputs": [ 219 | { 220 | "name": "stdout", 221 | "output_type": "stream", 222 | "text": [ 223 | "tf.Tensor(3.0, shape=(), dtype=float32)\n", 224 | "tf.Tensor(6.0, shape=(), dtype=float32)\n" 225 | ] 226 | } 227 | ], 228 | "source": [ 229 | "x = tf.Variable(1.0) # Create a Tensorflow variable initialized to 1.0\n", 230 | "\n", 231 | "with tf.GradientTape() as t:\n", 232 | " with tf.GradientTape() as t2:\n", 233 | " y = x * x * x\n", 234 | " # Compute the gradient inside the 't' context manager\n", 235 | " # which means the gradient computation is differentiable as well.\n", 236 | " dy_dx = t2.gradient(y, x)\n", 237 | "d2y_dx2 = t.gradient(dy_dx, x)\n", 238 | "\n", 239 | "print(dy_dx)\n", 240 | "print(d2y_dx2)" 241 | ] 242 | } 243 | ], 244 | "metadata": { 245 | "kernelspec": { 246 | "display_name": "Python 3", 247 | "language": "python", 248 | "name": "python3" 249 | }, 250 | "language_info": { 251 | "codemirror_mode": { 252 | "name": "ipython", 253 | "version": 3 254 | }, 255 | "file_extension": ".py", 256 | "mimetype": "text/x-python", 257 | "name": "python", 258 | "nbconvert_exporter": "python", 259 | "pygments_lexer": "ipython3", 260 | "version": "3.6.6" 261 | } 262 | }, 263 | "nbformat": 4, 264 | "nbformat_minor": 2 265 | } 266 | -------------------------------------------------------------------------------- /Lec04_Eager execution/Lec04_Custom training subclassing.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# CS 20 : TensorFlow for Deep Learning Research\n", 8 | "## Lecture 04 : Eager execution\n", 9 | "### Custon training subclassing\n", 10 | "Classifying mnist by using Tensorflow's eager execution.\n", 11 | "\n", 12 | "This guide uses these high-level TensorFlow concepts:\n", 13 | "\n", 14 | "* Enable an [eager execution](https://www.tensorflow.org/guide/eager?hl=ko) development environment,\n", 15 | "* Import data with the [Datasets API](https://www.tensorflow.org/guide/datasets?hl=ko)\n", 16 | "* Build model class by inheriting `tf.keras.Model` with TensorFlow's [Keras API](https://keras.io/getting-started/sequential-model-guide/)\n", 17 | " \n", 18 | " \n", 19 | "* Reference\n", 20 | " + https://www.tensorflow.org/tutorials/eager/custom_training_walkthrough?hl=ko" 21 | ] 22 | }, 23 | { 24 | "cell_type": "markdown", 25 | "metadata": {}, 26 | "source": [ 27 | "### Setup" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": 1, 33 | "metadata": { 34 | "scrolled": true 35 | }, 36 | "outputs": [ 37 | { 38 | "name": "stdout", 39 | "output_type": "stream", 40 | "text": [ 41 | "1.12.0\n" 42 | ] 43 | } 44 | ], 45 | "source": [ 46 | "from __future__ import absolute_import, division, print_function\n", 47 | "import numpy as np\n", 48 | "import tensorflow as tf\n", 49 | "from tensorflow import keras\n", 50 | "\n", 51 | "tf.enable_eager_execution()\n", 52 | "\n", 53 | "print(tf.__version__)" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "### Define `SoftmaxClassifier` class and `loss_fn` function" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": 2, 66 | "metadata": {}, 67 | "outputs": [], 68 | "source": [ 69 | "class SoftmaxClassifier(keras.Model):\n", 70 | " def __init__(self, num_classes):\n", 71 | " super(SoftmaxClassifier, self).__init__()\n", 72 | " self.__dense = tf.keras.layers.Dense(units = num_classes)\n", 73 | " \n", 74 | " def call(self, inputs):\n", 75 | " score = self.__dense(inputs)\n", 76 | " return score\n", 77 | "\n", 78 | "def loss_fn(model, x, y):\n", 79 | " ce_loss = tf.losses.sparse_softmax_cross_entropy(labels = y, logits = model(x))\n", 80 | " return ce_loss" 81 | ] 82 | }, 83 | { 84 | "cell_type": "markdown", 85 | "metadata": {}, 86 | "source": [ 87 | "#### Instantiate `model` instance" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": 3, 93 | "metadata": {}, 94 | "outputs": [], 95 | "source": [ 96 | "model = SoftmaxClassifier(num_classes = 10)" 97 | ] 98 | }, 99 | { 100 | "cell_type": "markdown", 101 | "metadata": {}, 102 | "source": [ 103 | "### Import dataset and split dataset into train, validation, test" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": 4, 109 | "metadata": {}, 110 | "outputs": [], 111 | "source": [ 112 | "(x_train, y_train), (x_tst, y_tst) = tf.keras.datasets.mnist.load_data()\n", 113 | "x_train = (x_train / 255).astype(np.float32).reshape(-1, 784)\n", 114 | "x_tst = (x_tst / 255).astype(np.float32).reshape(-1, 784)" 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": 5, 120 | "metadata": {}, 121 | "outputs": [ 122 | { 123 | "name": "stdout", 124 | "output_type": "stream", 125 | "text": [ 126 | "(55000, 784) (55000,)\n", 127 | "(5000, 784) (5000,)\n" 128 | ] 129 | } 130 | ], 131 | "source": [ 132 | "tr_indices = np.random.choice(range(x_train.shape[0]), size = 55000, replace = False)\n", 133 | "\n", 134 | "x_tr = x_train[tr_indices]\n", 135 | "y_tr = y_train[tr_indices].astype(np.int32)\n", 136 | "\n", 137 | "x_val = np.delete(arr = x_train, obj = tr_indices, axis = 0)\n", 138 | "y_val = np.delete(arr = y_train, obj = tr_indices, axis = 0).astype(np.int32)\n", 139 | "\n", 140 | "print(x_tr.shape, y_tr.shape)\n", 141 | "print(x_val.shape, y_val.shape)" 142 | ] 143 | }, 144 | { 145 | "cell_type": "markdown", 146 | "metadata": {}, 147 | "source": [ 148 | "### Train the model" 149 | ] 150 | }, 151 | { 152 | "cell_type": "code", 153 | "execution_count": 6, 154 | "metadata": {}, 155 | "outputs": [], 156 | "source": [ 157 | "epochs = 30\n", 158 | "batch_size = 32\n", 159 | "learning_rate = .01\n", 160 | "\n", 161 | "tr_dataset = tf.data.Dataset.from_tensor_slices((x_tr, y_tr))\n", 162 | "tr_dataset = tr_dataset.shuffle(buffer_size = 10000)\n", 163 | "tr_dataset = tr_dataset.batch(batch_size = batch_size)\n", 164 | "\n", 165 | "val_dataset = tf.data.Dataset.from_tensor_slices((x_val, y_val))\n", 166 | "val_dataset = val_dataset.batch(batch_size = batch_size)\n", 167 | "\n", 168 | "opt = tf.train.AdamOptimizer(learning_rate = learning_rate)" 169 | ] 170 | }, 171 | { 172 | "cell_type": "code", 173 | "execution_count": 7, 174 | "metadata": {}, 175 | "outputs": [ 176 | { 177 | "name": "stdout", 178 | "output_type": "stream", 179 | "text": [ 180 | "epochs : 5, tr_loss : 0.321, val_loss : 0.372\n", 181 | "epochs : 10, tr_loss : 0.316, val_loss : 0.390\n", 182 | "epochs : 15, tr_loss : 0.309, val_loss : 0.430\n", 183 | "epochs : 20, tr_loss : 0.307, val_loss : 0.445\n", 184 | "epochs : 25, tr_loss : 0.302, val_loss : 0.433\n", 185 | "epochs : 30, tr_loss : 0.303, val_loss : 0.453\n" 186 | ] 187 | } 188 | ], 189 | "source": [ 190 | "for epoch in range(epochs):\n", 191 | " avg_tr_loss = 0\n", 192 | " avg_val_loss = 0\n", 193 | " tr_step = 0\n", 194 | " val_step = 0\n", 195 | " \n", 196 | " # train\n", 197 | " for x_mb, y_mb in tr_dataset:\n", 198 | " with tf.GradientTape() as tape:\n", 199 | " tr_loss = loss_fn(model = model, x = x_mb, y = y_mb)\n", 200 | " grads = tape.gradient(target = tr_loss, sources = model.variables)\n", 201 | " opt.apply_gradients(zip(grads, model.variables))\n", 202 | " \n", 203 | " avg_tr_loss += tr_loss\n", 204 | " tr_step += 1\n", 205 | " else:\n", 206 | " avg_tr_loss /= tr_step\n", 207 | " \n", 208 | " # validation\n", 209 | " for x_mb, y_mb in val_dataset:\n", 210 | " val_loss = loss_fn(model = model, x = x_mb, y = y_mb)\n", 211 | " avg_val_loss += val_loss\n", 212 | " val_step += 1\n", 213 | " else:\n", 214 | " avg_val_loss /= val_step\n", 215 | " \n", 216 | " if (epoch + 1) % 5 == 0:\n", 217 | " print('epochs : {:2}, tr_loss : {:.3f}, val_loss : {:.3f}'.format(epoch + 1, avg_tr_loss, avg_val_loss))" 218 | ] 219 | } 220 | ], 221 | "metadata": { 222 | "kernelspec": { 223 | "display_name": "Python 3", 224 | "language": "python", 225 | "name": "python3" 226 | }, 227 | "language_info": { 228 | "codemirror_mode": { 229 | "name": "ipython", 230 | "version": 3 231 | }, 232 | "file_extension": ".py", 233 | "mimetype": "text/x-python", 234 | "name": "python", 235 | "nbconvert_exporter": "python", 236 | "pygments_lexer": "ipython3", 237 | "version": "3.6.6" 238 | } 239 | }, 240 | "nbformat": 4, 241 | "nbformat_minor": 2 242 | } 243 | -------------------------------------------------------------------------------- /Lec04_Eager execution/Lec04_Custom training walkthrough.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# CS 20 : TensorFlow for Deep Learning Research\n", 8 | "## Lecture 04 : Eager execution\n", 9 | "### Custon training walkthrough\n", 10 | "Categorizing Iris flowers by species by using Tensorflow's eager execution.\n", 11 | "\n", 12 | "This guide uses these high-level TensorFlow concepts:\n", 13 | "\n", 14 | "* Enable an [eager execution](https://www.tensorflow.org/guide/eager?hl=ko) development environment,\n", 15 | "* Import data with the [Datasets API](https://www.tensorflow.org/guide/datasets?hl=ko)\n", 16 | "* Build models and layers with TensorFlow's [Keras API](https://keras.io/getting-started/sequential-model-guide/) \n", 17 | " \n", 18 | " \n", 19 | "* Reference\n", 20 | " + https://www.tensorflow.org/tutorials/eager/custom_training_walkthrough?hl=ko" 21 | ] 22 | }, 23 | { 24 | "cell_type": "markdown", 25 | "metadata": {}, 26 | "source": [ 27 | "### Setup" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": 1, 33 | "metadata": {}, 34 | "outputs": [ 35 | { 36 | "name": "stdout", 37 | "output_type": "stream", 38 | "text": [ 39 | "1.12.0\n" 40 | ] 41 | } 42 | ], 43 | "source": [ 44 | "from __future__ import absolute_import, division, print_function\n", 45 | "import os, sys\n", 46 | "import numpy as np\n", 47 | "import tensorflow as tf\n", 48 | "import matplotlib.pyplot as plt\n", 49 | "%matplotlib inline\n", 50 | "\n", 51 | "tf.enable_eager_execution()\n", 52 | "\n", 53 | "print(tf.__version__)" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "### Create a model" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": 2, 66 | "metadata": {}, 67 | "outputs": [], 68 | "source": [ 69 | "model = tf.keras.Sequential()\n", 70 | "model.add(tf.keras.layers.Dense(10, activation = tf.nn.relu, input_shape = (4,)))\n", 71 | "model.add(tf.keras.layers.Dense(10, activation = tf.nn.relu))\n", 72 | "model.add(tf.keras.layers.Dense(3))" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 3, 78 | "metadata": {}, 79 | "outputs": [], 80 | "source": [ 81 | "def loss_fn(model, features, label):\n", 82 | " score = model(features)\n", 83 | " return tf.losses.sparse_softmax_cross_entropy(labels = label, logits = score)" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": 4, 89 | "metadata": {}, 90 | "outputs": [ 91 | { 92 | "name": "stdout", 93 | "output_type": "stream", 94 | "text": [ 95 | "[, , , , , ]\n" 136 | ] 137 | } 138 | ], 139 | "source": [ 140 | "print(model.trainable_variables) # different from tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)" 141 | ] 142 | }, 143 | { 144 | "cell_type": "markdown", 145 | "metadata": {}, 146 | "source": [ 147 | "### Import and parse the training dataset" 148 | ] 149 | }, 150 | { 151 | "cell_type": "code", 152 | "execution_count": 5, 153 | "metadata": {}, 154 | "outputs": [ 155 | { 156 | "data": { 157 | "text/plain": [ 158 | "['iris_test.csv', 'iris_training.csv']" 159 | ] 160 | }, 161 | "execution_count": 5, 162 | "metadata": {}, 163 | "output_type": "execute_result" 164 | } 165 | ], 166 | "source": [ 167 | "os.listdir('../data/lecture04/')" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": 6, 173 | "metadata": {}, 174 | "outputs": [], 175 | "source": [ 176 | "# define parsing function\n", 177 | "def parse_single_example(record):\n", 178 | " decoded = tf.decode_csv(record, [[.0],[.0],[.0],[.0],[]])\n", 179 | " features = decoded[:4]\n", 180 | " label = tf.cast(x = decoded[4], dtype = tf.int32)\n", 181 | " return features, label" 182 | ] 183 | }, 184 | { 185 | "cell_type": "code", 186 | "execution_count": 7, 187 | "metadata": {}, 188 | "outputs": [], 189 | "source": [ 190 | "epochs = 10\n", 191 | "batch_size = 8\n", 192 | "learning_rate = .03\n", 193 | "tr_dataset = tf.data.TextLineDataset(filenames = '../data/lecture04/iris_training.csv')\n", 194 | "tr_dataset = tr_dataset.map(parse_single_example)\n", 195 | "tr_dataset = tr_dataset.shuffle(200).batch(batch_size = batch_size)\n", 196 | "opt = tf.train.GradientDescentOptimizer(learning_rate = learning_rate)\n", 197 | "global_step = tf.Variable(0.)" 198 | ] 199 | }, 200 | { 201 | "cell_type": "markdown", 202 | "metadata": {}, 203 | "source": [ 204 | "### Train the model" 205 | ] 206 | }, 207 | { 208 | "cell_type": "code", 209 | "execution_count": 8, 210 | "metadata": {}, 211 | "outputs": [ 212 | { 213 | "name": "stdout", 214 | "output_type": "stream", 215 | "text": [ 216 | "epoch : 1, ce_loss : 1.154\n", 217 | "epoch : 2, ce_loss : 0.829\n", 218 | "epoch : 3, ce_loss : 0.706\n", 219 | "epoch : 4, ce_loss : 0.591\n", 220 | "epoch : 5, ce_loss : 0.553\n", 221 | "epoch : 6, ce_loss : 0.553\n", 222 | "epoch : 7, ce_loss : 0.449\n", 223 | "epoch : 8, ce_loss : 0.451\n", 224 | "epoch : 9, ce_loss : 0.514\n", 225 | "epoch : 10, ce_loss : 0.395\n" 226 | ] 227 | } 228 | ], 229 | "source": [ 230 | "for epoch in range(epochs):\n", 231 | " avg_loss = 0\n", 232 | " tr_step = 0\n", 233 | " \n", 234 | " for mb_x, mb_y in tr_dataset:\n", 235 | " with tf.GradientTape() as tape:\n", 236 | " tr_loss = loss_fn(model, mb_x, mb_y)\n", 237 | " grads = tape.gradient(tr_loss, model.variables)\n", 238 | " opt.apply_gradients(zip(grads, model.variables), global_step = global_step)\n", 239 | " \n", 240 | " avg_loss += tr_loss\n", 241 | " tr_step += 1\n", 242 | " else:\n", 243 | " avg_loss /= tr_step\n", 244 | " \n", 245 | " print('epoch : {:3}, ce_loss : {:.3f}'.format(epoch + 1, avg_loss))" 246 | ] 247 | }, 248 | { 249 | "cell_type": "markdown", 250 | "metadata": {}, 251 | "source": [ 252 | "### Evaluate the model on the test dataset" 253 | ] 254 | }, 255 | { 256 | "cell_type": "code", 257 | "execution_count": 9, 258 | "metadata": {}, 259 | "outputs": [], 260 | "source": [ 261 | "tst_dataset = tf.data.TextLineDataset(filenames = '../data/lecture04/iris_test.csv')\n", 262 | "tst_dataset = tst_dataset.map(parse_single_example)\n", 263 | "tst_dataset = tst_dataset.batch(batch_size = 30)" 264 | ] 265 | }, 266 | { 267 | "cell_type": "code", 268 | "execution_count": 10, 269 | "metadata": {}, 270 | "outputs": [], 271 | "source": [ 272 | "tst_x, tst_y = next(iter(tst_dataset))\n", 273 | "tst_yhat = tf.argmax(model(tst_x), axis = -1, output_type = tf.int32) " 274 | ] 275 | }, 276 | { 277 | "cell_type": "code", 278 | "execution_count": 11, 279 | "metadata": {}, 280 | "outputs": [ 281 | { 282 | "name": "stdout", 283 | "output_type": "stream", 284 | "text": [ 285 | "test accuracy : 96.67%\n" 286 | ] 287 | } 288 | ], 289 | "source": [ 290 | "print('test accuracy : {:.2%}'.format(np.mean(tf.equal(tst_y, tst_yhat))))" 291 | ] 292 | } 293 | ], 294 | "metadata": { 295 | "kernelspec": { 296 | "display_name": "Python 3", 297 | "language": "python", 298 | "name": "python3" 299 | }, 300 | "language_info": { 301 | "codemirror_mode": { 302 | "name": "ipython", 303 | "version": 3 304 | }, 305 | "file_extension": ".py", 306 | "mimetype": "text/x-python", 307 | "name": "python", 308 | "nbconvert_exporter": "python", 309 | "pygments_lexer": "ipython3", 310 | "version": "3.6.6" 311 | } 312 | }, 313 | "nbformat": 4, 314 | "nbformat_minor": 2 315 | } 316 | -------------------------------------------------------------------------------- /Lec04_Eager execution/Lec04_Eager execution.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# CS 20 : TensorFlow for Deep Learning Research\n", 8 | "## Lecture 04 : Eager execution\n", 9 | "### Eager execution\n", 10 | "You no longer need to worry about...\n", 11 | "\n", 12 | "1. placeholders\n", 13 | "2. sessions\n", 14 | "3. control dependencies\n", 15 | "4. \"lazy loading\"\n", 16 | "5. {name, variable, op} scopes \n", 17 | " \n", 18 | " \n", 19 | "* Reference\n", 20 | " + https://www.tensorflow.org/tutorials/eager/eager_basics?hl=ko\n", 21 | "\n", 22 | "### Notice\n", 23 | "#### Main changes in TF 2.0\n", 24 | "\n", 25 | "The API for Variables will then change in the following ways for TF 2.0:\n", 26 | "\n", 27 | "* ***tf.Variable will become an abstract base class with a well-defined interface and a scoped factory to construct instances***\n", 28 | " * ***users will be able to implement their own variable-like objects by subclassing tf.Variable and adding a scoped factory function to use those variables***\n", 29 | "* ***variable_scope and get_variable will be removed***\n", 30 | " * the tf 1.0 version of variable_scope and get_variable will be left in tf.compat.v1\n", 31 | " * ***to control variable naming users can use tf.name_scope + tf.Variable***\n", 32 | " * whether a variable is shared across sessions / processes will be controlled by a constructor argument to tf.Variable; no other type of scope reuse will be done in the framework\n", 33 | " * scoped partitioning will be implemented as a factory function at first\n", 34 | " * libraries and users are encouraged to reuse variables by reusing their objects, like Keras layers do\n", 35 | " * custom_getters will have the following API: [variable_creator_scope](https://github.com/tensorflow/tensorflow/blob/567189980f7a1c2aa09a5170bd8d01a6ec37d303/tensorflow/python/ops/variable_scope.py#L2402)\n", 36 | "* the default implementation of the tf.Variable interface will be ResourceVariable\n", 37 | " * RefVariable will be kept in tf.compat.v1 and will be the default implementation for tf.compat.v1.Variable\n", 38 | " * tf.compat.v1.Variable will have a use_resource argument to control whether a resource variable or a ref variable will be created\n", 39 | "* symbols like tf.assign* will be removed in favor of methods in tf.Variable\n", 40 | " * in tf.compat.v1 these symbols will be marked as deprecated and will call the corresponding methods in the Variable object instead\n", 41 | "https://github.com/tensorflow/community/blob/master/rfcs/20180817-variables-20.md" 42 | ] 43 | }, 44 | { 45 | "cell_type": "markdown", 46 | "metadata": {}, 47 | "source": [ 48 | "### Setup" 49 | ] 50 | }, 51 | { 52 | "cell_type": "code", 53 | "execution_count": 1, 54 | "metadata": {}, 55 | "outputs": [ 56 | { 57 | "name": "stdout", 58 | "output_type": "stream", 59 | "text": [ 60 | "1.12.0\n" 61 | ] 62 | } 63 | ], 64 | "source": [ 65 | "from __future__ import absolute_import, division, print_function\n", 66 | "import numpy as np\n", 67 | "import tensorflow as tf\n", 68 | "tf.enable_eager_execution()\n", 69 | "\n", 70 | "print(tf.__version__)" 71 | ] 72 | }, 73 | { 74 | "cell_type": "markdown", 75 | "metadata": {}, 76 | "source": [ 77 | "### Boilerplate\n", 78 | "```python\n", 79 | "x = tf.placeholder(dtype = tf.float32, shape = [1, 1])\n", 80 | "m = tf.matmul(x, x)\n", 81 | "\n", 82 | "print(m)\n", 83 | "with tf.Session() as sess:\n", 84 | " m_out = sess.run(m, feed_dict = {x : [[2.]]})\n", 85 | " print(m_out, m_out.shape)\n", 86 | "```\n", 87 | "\n", 88 | "```python\n", 89 | "Tensor(\"mul:0\", shape=(1, 1), dtype=float32)\n", 90 | "[[4.]] (1, 1)\n", 91 | "```\n", 92 | "\n", 93 | "**When using `tf.enable_eager_execution()`, Bolierplate changes as belows**" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": 2, 99 | "metadata": {}, 100 | "outputs": [ 101 | { 102 | "name": "stdout", 103 | "output_type": "stream", 104 | "text": [ 105 | "tf.Tensor([[4.]], shape=(1, 1), dtype=float32)\n", 106 | "[]\n" 107 | ] 108 | } 109 | ], 110 | "source": [ 111 | "x = [[2.]]\n", 112 | "m = tf.matmul(x, x)\n", 113 | "print(m) # No sessions()!\n", 114 | "print(tf.get_default_graph().get_operations()) # No graphs!" 115 | ] 116 | }, 117 | { 118 | "cell_type": "markdown", 119 | "metadata": {}, 120 | "source": [ 121 | "### Lazy Loading\n", 122 | "Each iteration adds nodes to the graph\n", 123 | "\n", 124 | "```python\n", 125 | "x = tf.constant(value = [[1,2],[3,4]], dtype = tf.int32)\n", 126 | "\n", 127 | "with tf.Session() as sess:\n", 128 | " for i in range(x.shape[0]):\n", 129 | " for j in range(x.shape[1]):\n", 130 | " print(sess.run(x[i, j]))\n", 131 | "```\n", 132 | "\n", 133 | "```python\n", 134 | "1\n", 135 | "2\n", 136 | "3\n", 137 | "4\n", 138 | "```\n", 139 | "\n", 140 | "**When using `tf.enable_eager_execution()`, not graph**" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": 3, 146 | "metadata": {}, 147 | "outputs": [ 148 | { 149 | "name": "stdout", 150 | "output_type": "stream", 151 | "text": [ 152 | "tf.Tensor(1, shape=(), dtype=int32)\n", 153 | "tf.Tensor(2, shape=(), dtype=int32)\n", 154 | "tf.Tensor(3, shape=(), dtype=int32)\n", 155 | "tf.Tensor(4, shape=(), dtype=int32)\n", 156 | "[]\n" 157 | ] 158 | } 159 | ], 160 | "source": [ 161 | "x = tf.constant(value = [[1,2],[3,4]], dtype = tf.int32)\n", 162 | "\n", 163 | "for i in range(x.shape[0]):\n", 164 | " for j in range(x.shape[1]):\n", 165 | " print(x[i, j])\n", 166 | " \n", 167 | "print(tf.get_default_graph().get_operations())" 168 | ] 169 | }, 170 | { 171 | "cell_type": "markdown", 172 | "metadata": {}, 173 | "source": [ 174 | "### Tensors act like numpy arrays\n", 175 | "The most obvious differences between NumPy arrays and TensorFlow Tensors are:\n", 176 | "\n", 177 | "1. Tensors can be backed by accelerator memory (like GPU, TPU)\n", 178 | "2. Tensors are **immutable**" 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": 4, 184 | "metadata": {}, 185 | "outputs": [ 186 | { 187 | "name": "stdout", 188 | "output_type": "stream", 189 | "text": [ 190 | "[[1. 4. 9.]]\n", 191 | "tf.Tensor(1.0, shape=(), dtype=float32)\n", 192 | "tf.Tensor(2.0, shape=(), dtype=float32)\n", 193 | "tf.Tensor(3.0, shape=(), dtype=float32)\n" 194 | ] 195 | } 196 | ], 197 | "source": [ 198 | "# Tensors are backed by NumPy arrays\n", 199 | "# Tensors are compatible with NumPy functions\n", 200 | "x = tf.constant(value = [[1.,2.,3]])\n", 201 | "assert type(x.numpy()) == np.ndarray\n", 202 | "squared = np.square(x)\n", 203 | "print(squared)\n", 204 | "\n", 205 | "# Tensors are iterable!\n", 206 | "for i in x[0]:\n", 207 | " print(i)" 208 | ] 209 | }, 210 | { 211 | "cell_type": "code", 212 | "execution_count": 5, 213 | "metadata": {}, 214 | "outputs": [ 215 | { 216 | "name": "stdout", 217 | "output_type": "stream", 218 | "text": [ 219 | "tf.Tensor(3, shape=(), dtype=int32)\n", 220 | "tf.Tensor([4 6], shape=(2,), dtype=int32)\n", 221 | "tf.Tensor(25, shape=(), dtype=int32)\n", 222 | "tf.Tensor(6, shape=(), dtype=int32)\n", 223 | "tf.Tensor(b'aGVsbG8gd29ybGQ', shape=(), dtype=string)\n", 224 | "tf.Tensor(13, shape=(), dtype=int32)\n" 225 | ] 226 | } 227 | ], 228 | "source": [ 229 | "print(tf.add(1, 2))\n", 230 | "print(tf.add([1, 2], [3, 4]))\n", 231 | "print(tf.square(5))\n", 232 | "print(tf.reduce_sum([1, 2, 3]))\n", 233 | "print(tf.encode_base64(\"hello world\"))\n", 234 | "\n", 235 | "# Operator overloading is also supported\n", 236 | "print(tf.square(2) + tf.square(3))" 237 | ] 238 | }, 239 | { 240 | "cell_type": "code", 241 | "execution_count": 6, 242 | "metadata": {}, 243 | "outputs": [ 244 | { 245 | "name": "stdout", 246 | "output_type": "stream", 247 | "text": [ 248 | "(1, 2)\n", 249 | "\n" 250 | ] 251 | } 252 | ], 253 | "source": [ 254 | "# Each Tensor has a shape and a datatype\n", 255 | "x = tf.matmul([[1]], [[2, 3]])\n", 256 | "print(x.shape)\n", 257 | "print(x.dtype)" 258 | ] 259 | }, 260 | { 261 | "cell_type": "markdown", 262 | "metadata": {}, 263 | "source": [ 264 | "#### Numpy Compatibility\n", 265 | "* TensorsFlow operations automatically convert NumPy ndarrays to Tensors.\n", 266 | "* NumPy operations automatically convert Tensors to NumPy ndarrays.\n", 267 | "\n", 268 | "***Tensors can be explicitly converted to NumPy ndarrays by invoking the .numpy() method on them.*** These conversions are typically cheap as the array and Tensor share the underlying memory representation if possible. ***However, sharing the underlying representation isn't always possible since the Tensor may be hosted in GPU memory while NumPy arrays are always backed by host memory, and the conversion will thus involve a copy from GPU to host memory.***" 269 | ] 270 | }, 271 | { 272 | "cell_type": "code", 273 | "execution_count": 7, 274 | "metadata": {}, 275 | "outputs": [ 276 | { 277 | "name": "stdout", 278 | "output_type": "stream", 279 | "text": [ 280 | "TensorFlow operations convert numpy arrays to Tensors automatically\n", 281 | "tf.Tensor(\n", 282 | "[[42. 42. 42.]\n", 283 | " [42. 42. 42.]\n", 284 | " [42. 42. 42.]], shape=(3, 3), dtype=float32)\n", 285 | "And NumPy operations convert Tensors to numpy arrays automatically\n", 286 | "[[43. 43. 43.]\n", 287 | " [43. 43. 43.]\n", 288 | " [43. 43. 43.]]\n", 289 | "The .numpy() method explicitly converts a Tensor to a numpy array\n", 290 | "[[42. 42. 42.]\n", 291 | " [42. 42. 42.]\n", 292 | " [42. 42. 42.]]\n" 293 | ] 294 | } 295 | ], 296 | "source": [ 297 | "ndarray = np.ones([3,3], dtype = np.float32)\n", 298 | "\n", 299 | "print(\"TensorFlow operations convert numpy arrays to Tensors automatically\")\n", 300 | "tensor = tf.multiply(ndarray, 42)\n", 301 | "print(tensor)\n", 302 | "\n", 303 | "print(\"And NumPy operations convert Tensors to numpy arrays automatically\")\n", 304 | "print(np.add(tensor, 1))\n", 305 | "\n", 306 | "print(\"The .numpy() method explicitly converts a Tensor to a numpy array\")\n", 307 | "print(tensor.numpy())" 308 | ] 309 | }, 310 | { 311 | "cell_type": "markdown", 312 | "metadata": {}, 313 | "source": [ 314 | "### GPU acceleration" 315 | ] 316 | }, 317 | { 318 | "cell_type": "code", 319 | "execution_count": 8, 320 | "metadata": {}, 321 | "outputs": [ 322 | { 323 | "name": "stdout", 324 | "output_type": "stream", 325 | "text": [ 326 | "Is there a GPU available: \n", 327 | "False\n", 328 | "Is the Tensor on GPU #0: \n", 329 | "False\n" 330 | ] 331 | } 332 | ], 333 | "source": [ 334 | "x = tf.random_uniform(shape = [3, 3])\n", 335 | "\n", 336 | "print(\"Is there a GPU available: \"),\n", 337 | "print(tf.test.is_gpu_available())\n", 338 | "\n", 339 | "print(\"Is the Tensor on GPU #0: \"),\n", 340 | "print(x.device.endswith('GPU:0'))" 341 | ] 342 | }, 343 | { 344 | "cell_type": "markdown", 345 | "metadata": {}, 346 | "source": [ 347 | "#### Device Names" 348 | ] 349 | }, 350 | { 351 | "cell_type": "code", 352 | "execution_count": 9, 353 | "metadata": {}, 354 | "outputs": [ 355 | { 356 | "data": { 357 | "text/plain": [ 358 | "'/job:localhost/replica:0/task:0/device:CPU:0'" 359 | ] 360 | }, 361 | "execution_count": 9, 362 | "metadata": {}, 363 | "output_type": "execute_result" 364 | } 365 | ], 366 | "source": [ 367 | "x.device" 368 | ] 369 | }, 370 | { 371 | "cell_type": "markdown", 372 | "metadata": {}, 373 | "source": [ 374 | "#### Explicit Device Placement" 375 | ] 376 | }, 377 | { 378 | "cell_type": "code", 379 | "execution_count": 10, 380 | "metadata": {}, 381 | "outputs": [ 382 | { 383 | "name": "stdout", 384 | "output_type": "stream", 385 | "text": [ 386 | "/job:localhost/replica:0/task:0/device:CPU:0\n" 387 | ] 388 | } 389 | ], 390 | "source": [ 391 | "with tf.device(\"CPU:0\"):\n", 392 | " y = tf.ones([1,1])\n", 393 | " print(y.device)" 394 | ] 395 | }, 396 | { 397 | "cell_type": "code", 398 | "execution_count": 11, 399 | "metadata": {}, 400 | "outputs": [ 401 | { 402 | "name": "stdout", 403 | "output_type": "stream", 404 | "text": [ 405 | "/job:localhost/replica:0/task:0/device:CPU:0\n", 406 | "/job:localhost/replica:0/task:0/device:CPU:0\n" 407 | ] 408 | } 409 | ], 410 | "source": [ 411 | "print(x.device)\n", 412 | "z = x.cpu()\n", 413 | "print(z.device)" 414 | ] 415 | }, 416 | { 417 | "cell_type": "code", 418 | "execution_count": 12, 419 | "metadata": {}, 420 | "outputs": [ 421 | { 422 | "name": "stdout", 423 | "output_type": "stream", 424 | "text": [ 425 | "tf.Tensor(\n", 426 | "[[0.5488272 0.9693705 0.47811544]\n", 427 | " [0.13793623 0.53724563 0.9553573 ]\n", 428 | " [0.9873563 0.27607608 0.21941674]], shape=(3, 3), dtype=float32) \n", 429 | " tf.Tensor(\n", 430 | "[[0.5488272 0.9693705 0.47811544]\n", 431 | " [0.13793623 0.53724563 0.9553573 ]\n", 432 | " [0.9873563 0.27607608 0.21941674]], shape=(3, 3), dtype=float32)\n" 433 | ] 434 | } 435 | ], 436 | "source": [ 437 | "print(x, '\\n',z)" 438 | ] 439 | }, 440 | { 441 | "cell_type": "code", 442 | "execution_count": 13, 443 | "metadata": {}, 444 | "outputs": [ 445 | { 446 | "data": { 447 | "text/plain": [ 448 | "" 452 | ] 453 | }, 454 | "execution_count": 13, 455 | "metadata": {}, 456 | "output_type": "execute_result" 457 | } 458 | ], 459 | "source": [ 460 | "tf.equal(x, z)" 461 | ] 462 | }, 463 | { 464 | "cell_type": "markdown", 465 | "metadata": {}, 466 | "source": [ 467 | "### Datasets\n", 468 | "We recommend using the Datasets API for building performant, complex input pipelines from simple, re-usable pieces that will feed your model's training or evaluation loops.\n", 469 | "\n", 470 | "If you're familiar with TensorFlow graphs, the API for constructing the Dataset object remains exactly the same when eager execution is enabled, but the process of iterating over elements of the dataset is slightly simpler. ***You can use Python iteration over the `tf.data.Dataset` object and do not need to explicitly create an `tf.data.Iterator` object.*** As a result, the discussion on iterators in the TensorFlow Guide is not relevant when eager execution is enabled." 471 | ] 472 | }, 473 | { 474 | "cell_type": "code", 475 | "execution_count": 14, 476 | "metadata": {}, 477 | "outputs": [], 478 | "source": [ 479 | "tensors = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4, 5, 6])\n", 480 | "tensors = tensors.map(np.square) # Numpy Compatibility magic!\n", 481 | "tensors = tensors.shuffle(2).batch(2)" 482 | ] 483 | }, 484 | { 485 | "cell_type": "code", 486 | "execution_count": 15, 487 | "metadata": {}, 488 | "outputs": [ 489 | { 490 | "name": "stdout", 491 | "output_type": "stream", 492 | "text": [ 493 | "tf.Tensor([4 1], shape=(2,), dtype=int32)\n", 494 | "tf.Tensor([16 9], shape=(2,), dtype=int32)\n", 495 | "tf.Tensor([25 36], shape=(2,), dtype=int32)\n" 496 | ] 497 | } 498 | ], 499 | "source": [ 500 | "for mb_tensor in tensors:\n", 501 | " print(mb_tensor)" 502 | ] 503 | } 504 | ], 505 | "metadata": { 506 | "kernelspec": { 507 | "display_name": "Python 3", 508 | "language": "python", 509 | "name": "python3" 510 | }, 511 | "language_info": { 512 | "codemirror_mode": { 513 | "name": "ipython", 514 | "version": 3 515 | }, 516 | "file_extension": ".py", 517 | "mimetype": "text/x-python", 518 | "name": "python", 519 | "nbconvert_exporter": "python", 520 | "pygments_lexer": "ipython3", 521 | "version": "3.6.6" 522 | } 523 | }, 524 | "nbformat": 4, 525 | "nbformat_minor": 2 526 | } 527 | -------------------------------------------------------------------------------- /Lec05_Variable sharing and managing experiments/Lec05_Randomization.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# CS 20 : TensorFlow for Deep Learning Research\n", 8 | "## Lecture 05 : Variable sharing and managing experiments\n", 9 | "### Randomization" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "### Setup" 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": 1, 22 | "metadata": {}, 23 | "outputs": [ 24 | { 25 | "name": "stdout", 26 | "output_type": "stream", 27 | "text": [ 28 | "1.12.0\n" 29 | ] 30 | } 31 | ], 32 | "source": [ 33 | "from __future__ import absolute_import, division, print_function\n", 34 | "import tensorflow as tf\n", 35 | "\n", 36 | "print(tf.__version__)" 37 | ] 38 | }, 39 | { 40 | "cell_type": "markdown", 41 | "metadata": {}, 42 | "source": [ 43 | "### Example 1 : Session keeps track of the random state" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": 2, 49 | "metadata": {}, 50 | "outputs": [ 51 | { 52 | "name": "stdout", 53 | "output_type": "stream", 54 | "text": [ 55 | "-0.85811085\n", 56 | "-0.20793143\n" 57 | ] 58 | } 59 | ], 60 | "source": [ 61 | "c = tf.random_normal(shape = [], seed = 2)\n", 62 | "\n", 63 | "with tf.Session() as sess:\n", 64 | " print(sess.run(c))\n", 65 | " print(sess.run(c))" 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "metadata": {}, 71 | "source": [ 72 | "### Example 2 : Each new session will start the random state all over again" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 3, 78 | "metadata": {}, 79 | "outputs": [ 80 | { 81 | "name": "stdout", 82 | "output_type": "stream", 83 | "text": [ 84 | "-0.85811085\n", 85 | "-0.85811085\n", 86 | "-0.20793143\n" 87 | ] 88 | } 89 | ], 90 | "source": [ 91 | "tf.reset_default_graph()\n", 92 | "\n", 93 | "c = tf.random_normal(shape = [], seed = 2)\n", 94 | "\n", 95 | "with tf.Session() as sess:\n", 96 | " print(sess.run(c))\n", 97 | " \n", 98 | "with tf.Session() as sess:\n", 99 | " print(sess.run(c))\n", 100 | " print(sess.run(c))" 101 | ] 102 | }, 103 | { 104 | "cell_type": "markdown", 105 | "metadata": {}, 106 | "source": [ 107 | "### Example 3 : With operation level random seed, each op keeps its own seed" 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": 4, 113 | "metadata": {}, 114 | "outputs": [ 115 | { 116 | "name": "stdout", 117 | "output_type": "stream", 118 | "text": [ 119 | "-0.85811085\n", 120 | "-0.85811085\n" 121 | ] 122 | } 123 | ], 124 | "source": [ 125 | "tf.reset_default_graph()\n", 126 | "\n", 127 | "c = tf.random_normal(shape = [], seed = 2)\n", 128 | "d = tf.random_normal(shape = [], seed = 2)\n", 129 | "\n", 130 | "with tf.Session() as sess:\n", 131 | " print(sess.run(c))\n", 132 | " print(sess.run(d))" 133 | ] 134 | }, 135 | { 136 | "cell_type": "markdown", 137 | "metadata": {}, 138 | "source": [ 139 | "### Example 4 : Graph level random seed" 140 | ] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": 5, 145 | "metadata": {}, 146 | "outputs": [ 147 | { 148 | "name": "stdout", 149 | "output_type": "stream", 150 | "text": [ 151 | "-1.4236197\n", 152 | "0.8052349\n" 153 | ] 154 | } 155 | ], 156 | "source": [ 157 | "tf.reset_default_graph()\n", 158 | "tf.set_random_seed(seed = 2)\n", 159 | "\n", 160 | "c = tf.random_normal(shape = [])\n", 161 | "d = tf.random_normal(shape = [])\n", 162 | "\n", 163 | "with tf.Session() as sess:\n", 164 | " print(sess.run(c))\n", 165 | " print(sess.run(d))" 166 | ] 167 | }, 168 | { 169 | "cell_type": "code", 170 | "execution_count": 6, 171 | "metadata": {}, 172 | "outputs": [ 173 | { 174 | "name": "stdout", 175 | "output_type": "stream", 176 | "text": [ 177 | "-1.4236197\n", 178 | "0.8052349\n" 179 | ] 180 | } 181 | ], 182 | "source": [ 183 | "tf.reset_default_graph()\n", 184 | "tf.set_random_seed(seed = 2)\n", 185 | "\n", 186 | "c = tf.random_normal(shape = [])\n", 187 | "d = tf.random_normal(shape = [])\n", 188 | "\n", 189 | "with tf.Session() as sess:\n", 190 | " print(sess.run(c))\n", 191 | " print(sess.run(d))" 192 | ] 193 | } 194 | ], 195 | "metadata": { 196 | "kernelspec": { 197 | "display_name": "Python 3", 198 | "language": "python", 199 | "name": "python3" 200 | }, 201 | "language_info": { 202 | "codemirror_mode": { 203 | "name": "ipython", 204 | "version": 3 205 | }, 206 | "file_extension": ".py", 207 | "mimetype": "text/x-python", 208 | "name": "python", 209 | "nbconvert_exporter": "python", 210 | "pygments_lexer": "ipython3", 211 | "version": "3.6.8" 212 | }, 213 | "varInspector": { 214 | "cols": { 215 | "lenName": 16, 216 | "lenType": 16, 217 | "lenVar": 40 218 | }, 219 | "kernels_config": { 220 | "python": { 221 | "delete_cmd_postfix": "", 222 | "delete_cmd_prefix": "del ", 223 | "library": "var_list.py", 224 | "varRefreshCmd": "print(var_dic_list())" 225 | }, 226 | "r": { 227 | "delete_cmd_postfix": ") ", 228 | "delete_cmd_prefix": "rm(", 229 | "library": "var_list.r", 230 | "varRefreshCmd": "cat(var_dic_list()) " 231 | } 232 | }, 233 | "types_to_exclude": [ 234 | "module", 235 | "function", 236 | "builtin_function_or_method", 237 | "instance", 238 | "_Feature" 239 | ], 240 | "window_display": false 241 | } 242 | }, 243 | "nbformat": 4, 244 | "nbformat_minor": 2 245 | } 246 | -------------------------------------------------------------------------------- /Lec07_ConvNet in Tensorflow/Lec07_ConvNet mnist by high-level_kd.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# CS 20 : TensorFlow for Deep Learning Research\n", 8 | "## Lecture 07 : ConvNet in TensorFlow\n", 9 | "same contents, but different style with [Lec07_ConvNet mnist by high-level.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec07_ConvNet%20in%20Tensorflow/Lec07_ConvNet%20mnist%20by%20high-level.ipynb)\n", 10 | "\n", 11 | "### ConvNet mnist by high-level\n", 12 | "- Creating the **data pipeline** with `tf.data`\n", 13 | "- Using `tf.keras`, alias `keras`\n", 14 | "- Creating the model as **Class** by subclassing `tf.keras.Model`\n", 15 | "- Training the model with **Drop out** technique by `tf.keras.layers.Dropout`\n", 16 | "- Using tensorboard" 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": {}, 22 | "source": [ 23 | "### Setup" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": 1, 29 | "metadata": {}, 30 | "outputs": [ 31 | { 32 | "name": "stdout", 33 | "output_type": "stream", 34 | "text": [ 35 | "1.12.0\n" 36 | ] 37 | } 38 | ], 39 | "source": [ 40 | "from __future__ import absolute_import, division, print_function\n", 41 | "import os, sys\n", 42 | "import numpy as np\n", 43 | "import pandas as pd\n", 44 | "import matplotlib.pyplot as plt\n", 45 | "import tensorflow as tf\n", 46 | "from tensorflow import keras\n", 47 | "%matplotlib inline\n", 48 | "\n", 49 | "print(tf.__version__)" 50 | ] 51 | }, 52 | { 53 | "cell_type": "markdown", 54 | "metadata": {}, 55 | "source": [ 56 | "### Load and Pre-process data" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": 2, 62 | "metadata": {}, 63 | "outputs": [], 64 | "source": [ 65 | "(x_train, y_train), (x_tst, y_tst) = tf.keras.datasets.mnist.load_data()\n", 66 | "x_train = x_train / 255\n", 67 | "x_train = x_train.reshape(-1, 28, 28, 1).astype(np.float32)\n", 68 | "x_tst = x_tst / 255\n", 69 | "x_tst = x_tst.reshape(-1, 28, 28, 1).astype(np.float32)\n", 70 | "y_tst = y_tst.astype(np.int32)" 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": 3, 76 | "metadata": {}, 77 | "outputs": [ 78 | { 79 | "name": "stdout", 80 | "output_type": "stream", 81 | "text": [ 82 | "(55000, 28, 28, 1) (55000,)\n", 83 | "(5000, 28, 28, 1) (5000,)\n" 84 | ] 85 | } 86 | ], 87 | "source": [ 88 | "tr_indices = np.random.choice(range(x_train.shape[0]), size = 55000, replace = False)\n", 89 | "\n", 90 | "x_tr = x_train[tr_indices]\n", 91 | "y_tr = y_train[tr_indices].astype(np.int32)\n", 92 | "\n", 93 | "x_val = np.delete(arr = x_train, obj = tr_indices, axis = 0)\n", 94 | "y_val = np.delete(arr = y_train, obj = tr_indices, axis = 0).astype(np.int32)\n", 95 | "\n", 96 | "print(x_tr.shape, y_tr.shape)\n", 97 | "print(x_val.shape, y_val.shape)" 98 | ] 99 | }, 100 | { 101 | "cell_type": "markdown", 102 | "metadata": {}, 103 | "source": [ 104 | "### Define SimpleCNN class by high-level api" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": 4, 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "class SimpleCNN(keras.Model):\n", 114 | " def __init__(self, num_classes):\n", 115 | " super(SimpleCNN, self).__init__()\n", 116 | " self.__conv1 = keras.layers.Conv2D(filters=32, kernel_size=[5,5], padding='same',\n", 117 | " kernel_initializer=keras.initializers.truncated_normal(),\n", 118 | " bias_initializer=keras.initializers.truncated_normal(),\n", 119 | " activation=tf.nn.relu)\n", 120 | " self.__conv2 = keras.layers.Conv2D(filters=64, kernel_size=[5,5], padding='same',\n", 121 | " kernel_initializer=keras.initializers.truncated_normal(),\n", 122 | " bias_initializer=keras.initializers.truncated_normal(),\n", 123 | " activation=tf.nn.relu)\n", 124 | " self.__pool = keras.layers.MaxPooling2D()\n", 125 | " self.__flatten = keras.layers.Flatten()\n", 126 | " self.__dropout = keras.layers.Dropout(rate =.5)\n", 127 | " self.__dense1 = keras.layers.Dense(units=1024, activation=tf.nn.relu, \n", 128 | " kernel_initializer=keras.initializers.truncated_normal(),\n", 129 | " bias_initializer=keras.initializers.truncated_normal())\n", 130 | " self.__dense2 = keras.layers.Dense(units=num_classes,\n", 131 | " kernel_initializer=keras.initializers.truncated_normal(),\n", 132 | " bias_initializer=keras.initializers.truncated_normal(),\n", 133 | " activation='softmax')\n", 134 | " \n", 135 | " def call(self, inputs, training=False):\n", 136 | " conv1 = self.__conv1(inputs)\n", 137 | " pool1 = self.__pool(conv1)\n", 138 | " conv2 = self.__conv2(pool1)\n", 139 | " pool2 = self.__pool(conv2)\n", 140 | " flattened = self.__flatten(pool2)\n", 141 | " fc = self.__dense1(flattened)\n", 142 | " if training:\n", 143 | " fc = self.__dropout(fc, training=training)\n", 144 | " score = self.__dense2(fc)\n", 145 | " return score" 146 | ] 147 | }, 148 | { 149 | "cell_type": "markdown", 150 | "metadata": {}, 151 | "source": [ 152 | "### Create a model of SimpleCNN" 153 | ] 154 | }, 155 | { 156 | "cell_type": "code", 157 | "execution_count": 5, 158 | "metadata": {}, 159 | "outputs": [ 160 | { 161 | "name": "stdout", 162 | "output_type": "stream", 163 | "text": [ 164 | "550\n" 165 | ] 166 | } 167 | ], 168 | "source": [ 169 | "# hyper-parameter\n", 170 | "lr = .001\n", 171 | "epochs = 10\n", 172 | "batch_size = 100\n", 173 | "total_step = int(x_tr.shape[0] / batch_size)\n", 174 | "print(total_step)" 175 | ] 176 | }, 177 | { 178 | "cell_type": "code", 179 | "execution_count": 6, 180 | "metadata": {}, 181 | "outputs": [ 182 | { 183 | "name": "stdout", 184 | "output_type": "stream", 185 | "text": [ 186 | "\n", 187 | "\n", 188 | "\n" 189 | ] 190 | } 191 | ], 192 | "source": [ 193 | "## create input pipeline with tf.data\n", 194 | "# for train\n", 195 | "tr_dataset = tf.data.Dataset.from_tensor_slices((x_tr, y_tr))\n", 196 | "tr_dataset = tr_dataset.batch(batch_size = batch_size).repeat()\n", 197 | "print(tr_dataset)\n", 198 | "\n", 199 | "# for validation\n", 200 | "val_dataset = tf.data.Dataset.from_tensor_slices((x_val,y_val))\n", 201 | "val_dataset = val_dataset.batch(batch_size = batch_size).repeat()\n", 202 | "print(val_dataset)\n", 203 | "\n", 204 | "# for test\n", 205 | "tst_dataset = tf.data.Dataset.from_tensor_slices((x_tst, y_tst))\n", 206 | "tst_dataset = tst_dataset.batch(batch_size=100)\n", 207 | "print(tst_dataset)" 208 | ] 209 | }, 210 | { 211 | "cell_type": "code", 212 | "execution_count": 7, 213 | "metadata": {}, 214 | "outputs": [], 215 | "source": [ 216 | "## create model\n", 217 | "cnn = SimpleCNN(num_classes=10)\n", 218 | "\n", 219 | "# creating callbacks for tensorboard\n", 220 | "callbacks = [keras.callbacks.TensorBoard(log_dir='../graphs/lecture07/convnet_mnist_high_kd/',\n", 221 | " write_graph=True, write_images=True)]" 222 | ] 223 | }, 224 | { 225 | "cell_type": "code", 226 | "execution_count": 8, 227 | "metadata": {}, 228 | "outputs": [], 229 | "source": [ 230 | "# complile\n", 231 | "cnn.compile(optimizer=tf.train.AdamOptimizer(learning_rate=lr),\n", 232 | " loss=keras.losses.sparse_categorical_crossentropy,\n", 233 | " callbacks=callbacks)" 234 | ] 235 | }, 236 | { 237 | "cell_type": "markdown", 238 | "metadata": {}, 239 | "source": [ 240 | "### Train a model" 241 | ] 242 | }, 243 | { 244 | "cell_type": "code", 245 | "execution_count": 9, 246 | "metadata": {}, 247 | "outputs": [ 248 | { 249 | "name": "stdout", 250 | "output_type": "stream", 251 | "text": [ 252 | "Epoch 1/10\n", 253 | "550/550 [==============================] - 4s 7ms/step - loss: 0.1362 - val_loss: 0.0472\n", 254 | "Epoch 2/10\n", 255 | "550/550 [==============================] - 3s 5ms/step - loss: 0.0381 - val_loss: 0.0508\n", 256 | "Epoch 3/10\n", 257 | "550/550 [==============================] - 3s 5ms/step - loss: 0.0246 - val_loss: 0.0233\n", 258 | "Epoch 4/10\n", 259 | "550/550 [==============================] - 3s 5ms/step - loss: 0.0177 - val_loss: 0.0200\n", 260 | "Epoch 5/10\n", 261 | "550/550 [==============================] - 3s 5ms/step - loss: 0.0149 - val_loss: 0.0160\n", 262 | "Epoch 6/10\n", 263 | "550/550 [==============================] - 3s 5ms/step - loss: 0.0108 - val_loss: 0.0206\n", 264 | "Epoch 7/10\n", 265 | "550/550 [==============================] - 3s 5ms/step - loss: 0.0087 - val_loss: 0.0112\n", 266 | "Epoch 8/10\n", 267 | "550/550 [==============================] - 3s 5ms/step - loss: 0.0082 - val_loss: 0.0133\n", 268 | "Epoch 9/10\n", 269 | "550/550 [==============================] - 3s 5ms/step - loss: 0.0087 - val_loss: 0.0075\n", 270 | "Epoch 10/10\n", 271 | "550/550 [==============================] - 3s 5ms/step - loss: 0.0060 - val_loss: 0.0054\n" 272 | ] 273 | }, 274 | { 275 | "data": { 276 | "text/plain": [ 277 | "" 278 | ] 279 | }, 280 | "execution_count": 9, 281 | "metadata": {}, 282 | "output_type": "execute_result" 283 | } 284 | ], 285 | "source": [ 286 | "cnn.fit(tr_dataset, epochs=epochs, steps_per_epoch=total_step,\n", 287 | " validation_data=val_dataset, validation_steps=5000//100)" 288 | ] 289 | }, 290 | { 291 | "cell_type": "markdown", 292 | "metadata": {}, 293 | "source": [ 294 | "### Calculate accuracy" 295 | ] 296 | }, 297 | { 298 | "cell_type": "code", 299 | "execution_count": 10, 300 | "metadata": {}, 301 | "outputs": [ 302 | { 303 | "name": "stdout", 304 | "output_type": "stream", 305 | "text": [ 306 | "Tensor(\"simple_cnn/dense_1/Softmax:0\", shape=(10000, 10), dtype=float32)\n" 307 | ] 308 | } 309 | ], 310 | "source": [ 311 | "sess = keras.backend.get_session()\n", 312 | "x_tst_tensor = tf.convert_to_tensor(x_tst)\n", 313 | "yhat = cnn(x_tst_tensor, training=False)\n", 314 | "print(yhat)" 315 | ] 316 | }, 317 | { 318 | "cell_type": "code", 319 | "execution_count": 11, 320 | "metadata": {}, 321 | "outputs": [ 322 | { 323 | "name": "stdout", 324 | "output_type": "stream", 325 | "text": [ 326 | "tst acc : 99.13%\n" 327 | ] 328 | } 329 | ], 330 | "source": [ 331 | "yhat = sess.run(yhat)\n", 332 | "print('tst acc : {:.2%}'.format(np.mean(np.argmax(yhat, axis=-1) == y_tst)))" 333 | ] 334 | } 335 | ], 336 | "metadata": { 337 | "kernelspec": { 338 | "display_name": "Python 3", 339 | "language": "python", 340 | "name": "python3" 341 | }, 342 | "language_info": { 343 | "codemirror_mode": { 344 | "name": "ipython", 345 | "version": 3 346 | }, 347 | "file_extension": ".py", 348 | "mimetype": "text/x-python", 349 | "name": "python", 350 | "nbconvert_exporter": "python", 351 | "pygments_lexer": "ipython3", 352 | "version": "3.6.8" 353 | }, 354 | "varInspector": { 355 | "cols": { 356 | "lenName": 16, 357 | "lenType": 16, 358 | "lenVar": 40 359 | }, 360 | "kernels_config": { 361 | "python": { 362 | "delete_cmd_postfix": "", 363 | "delete_cmd_prefix": "del ", 364 | "library": "var_list.py", 365 | "varRefreshCmd": "print(var_dic_list())" 366 | }, 367 | "r": { 368 | "delete_cmd_postfix": ") ", 369 | "delete_cmd_prefix": "rm(", 370 | "library": "var_list.r", 371 | "varRefreshCmd": "cat(var_dic_list()) " 372 | } 373 | }, 374 | "types_to_exclude": [ 375 | "module", 376 | "function", 377 | "builtin_function_or_method", 378 | "instance", 379 | "_Feature" 380 | ], 381 | "window_display": false 382 | } 383 | }, 384 | "nbformat": 4, 385 | "nbformat_minor": 2 386 | } 387 | -------------------------------------------------------------------------------- /Lec07_ConvNet in Tensorflow/Lec07_ConvNet mnist with Weight initialization and Batch norm_kd.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# CS 20 : TensorFlow for Deep Learning Research\n", 8 | "## Lecture 07 : ConvNet in TensorFlow\n", 9 | "same contents, but different style with [Lec07_ConvNet mnist with Weight initialization and Batch norm.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec07_ConvNet%20in%20Tensorflow/Lec07_ConvNet%20mnist%20with%20Weight%20initialization%20and%20Batch%20norm.ipynb)\n", 10 | "\n", 11 | "### ConvNet mnist with Weight initialization and Batch norm\n", 12 | "- Creating the **data pipeline** with `tf.data`\n", 13 | "- Using `tf.keras`, alias `keras`\n", 14 | "- Creating the model as **Class** by subclassing `tf.keras.Model`\n", 15 | "- Initializaing weights of model with **He initialization** by `tf.keras.initializers.he_uniform`\n", 16 | "- Training the model with **Batch Normalization** technique by `tf.keras.layers.BatchNormalization`\n", 17 | "- Using tensorboard" 18 | ] 19 | }, 20 | { 21 | "cell_type": "markdown", 22 | "metadata": {}, 23 | "source": [ 24 | "### Setup" 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": 1, 30 | "metadata": {}, 31 | "outputs": [ 32 | { 33 | "name": "stdout", 34 | "output_type": "stream", 35 | "text": [ 36 | "1.12.0\n" 37 | ] 38 | } 39 | ], 40 | "source": [ 41 | "from __future__ import absolute_import, division, print_function\n", 42 | "import os, sys\n", 43 | "import numpy as np\n", 44 | "import pandas as pd\n", 45 | "import matplotlib.pyplot as plt\n", 46 | "import tensorflow as tf\n", 47 | "from tensorflow import keras\n", 48 | "%matplotlib inline\n", 49 | "\n", 50 | "print(tf.__version__)" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": {}, 56 | "source": [ 57 | "### Load and Pre-process data" 58 | ] 59 | }, 60 | { 61 | "cell_type": "code", 62 | "execution_count": 2, 63 | "metadata": {}, 64 | "outputs": [], 65 | "source": [ 66 | "(x_train, y_train), (x_tst, y_tst) = tf.keras.datasets.mnist.load_data()\n", 67 | "x_train = x_train / 255\n", 68 | "x_train = x_train.reshape(-1, 28, 28, 1).astype(np.float32)\n", 69 | "x_tst = x_tst / 255\n", 70 | "x_tst = x_tst.reshape(-1, 28, 28, 1).astype(np.float32)" 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": 3, 76 | "metadata": {}, 77 | "outputs": [ 78 | { 79 | "name": "stdout", 80 | "output_type": "stream", 81 | "text": [ 82 | "(55000, 28, 28, 1) (55000,)\n", 83 | "(5000, 28, 28, 1) (5000,)\n" 84 | ] 85 | } 86 | ], 87 | "source": [ 88 | "tr_indices = np.random.choice(range(x_train.shape[0]), size = 55000, replace = False)\n", 89 | "\n", 90 | "x_tr = x_train[tr_indices]\n", 91 | "y_tr = y_train[tr_indices].astype(np.int32)\n", 92 | "\n", 93 | "x_val = np.delete(arr = x_train, obj = tr_indices, axis = 0)\n", 94 | "y_val = np.delete(arr = y_train, obj = tr_indices, axis = 0).astype(np.int32)\n", 95 | "\n", 96 | "print(x_tr.shape, y_tr.shape)\n", 97 | "print(x_val.shape, y_val.shape)" 98 | ] 99 | }, 100 | { 101 | "cell_type": "markdown", 102 | "metadata": {}, 103 | "source": [ 104 | "### Define SimpleCNN class by high-level api" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": 4, 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "class SimpleCNN(keras.Model):\n", 114 | " def __init__(self, num_classes):\n", 115 | " super(SimpleCNN, self).__init__()\n", 116 | " self.__conv1 = keras.layers.Conv2D(filters=32, kernel_size=[5,5], padding='same',\n", 117 | " kernel_initializer=keras.initializers.he_uniform(),\n", 118 | " bias_initializer=keras.initializers.he_uniform(),\n", 119 | " activation=tf.nn.relu)\n", 120 | " self.__conv2 = keras.layers.Conv2D(filters=64, kernel_size=[5,5], padding='same',\n", 121 | " kernel_initializer=keras.initializers.he_uniform(),\n", 122 | " bias_initializer=keras.initializers.he_uniform(),\n", 123 | " activation=tf.nn.relu)\n", 124 | " self.__pool = keras.layers.MaxPooling2D()\n", 125 | " self.__flatten = keras.layers.Flatten()\n", 126 | " self.__bnc1 = keras.layers.BatchNormalization(momentum=.9)\n", 127 | " self.__bnc2 = keras.layers.BatchNormalization(momentum=.9)\n", 128 | " self.__bnd = keras.layers.BatchNormalization(momentum=.9)\n", 129 | " self.__dense1 = keras.layers.Dense(units=1024, activation=tf.nn.relu, \n", 130 | " kernel_initializer=keras.initializers.he_uniform(),\n", 131 | " bias_initializer=keras.initializers.he_uniform())\n", 132 | " self.__dense2 = keras.layers.Dense(units=num_classes,\n", 133 | " kernel_initializer=keras.initializers.he_uniform(),\n", 134 | " bias_initializer=keras.initializers.he_uniform(),\n", 135 | " activation='softmax')\n", 136 | "\n", 137 | " def call(self, inputs, training=False):\n", 138 | " conv1 = self.__bnc1(self.__conv1(inputs), training=training)\n", 139 | " pool1 = self.__pool(conv1)\n", 140 | " conv2 = self.__bnc2(self.__conv2(pool1), training=training)\n", 141 | " pool2 = self.__pool(conv2)\n", 142 | " flattened = self.__flatten(pool2)\n", 143 | " fc = self.__bnd(self.__dense1(flattened), training=training)\n", 144 | " score = self.__dense2(fc)\n", 145 | " return score" 146 | ] 147 | }, 148 | { 149 | "cell_type": "markdown", 150 | "metadata": {}, 151 | "source": [ 152 | "### Create a model of SimpleCNN" 153 | ] 154 | }, 155 | { 156 | "cell_type": "code", 157 | "execution_count": 5, 158 | "metadata": {}, 159 | "outputs": [ 160 | { 161 | "name": "stdout", 162 | "output_type": "stream", 163 | "text": [ 164 | "550\n" 165 | ] 166 | } 167 | ], 168 | "source": [ 169 | "# hyper-parameter\n", 170 | "lr = .001\n", 171 | "epochs = 5\n", 172 | "batch_size = 100\n", 173 | "total_step = int(x_tr.shape[0] / batch_size)\n", 174 | "print(total_step)" 175 | ] 176 | }, 177 | { 178 | "cell_type": "code", 179 | "execution_count": 6, 180 | "metadata": {}, 181 | "outputs": [ 182 | { 183 | "name": "stdout", 184 | "output_type": "stream", 185 | "text": [ 186 | "\n", 187 | "\n", 188 | "\n" 189 | ] 190 | } 191 | ], 192 | "source": [ 193 | "## create input pipeline with tf.data\n", 194 | "# for train\n", 195 | "tr_dataset = tf.data.Dataset.from_tensor_slices((x_tr, y_tr))\n", 196 | "tr_dataset = tr_dataset.batch(batch_size = batch_size).repeat()\n", 197 | "print(tr_dataset)\n", 198 | "\n", 199 | "# for validation\n", 200 | "val_dataset = tf.data.Dataset.from_tensor_slices((x_val,y_val))\n", 201 | "val_dataset = val_dataset.batch(batch_size = batch_size)\n", 202 | "print(val_dataset)\n", 203 | "\n", 204 | "# for test\n", 205 | "tst_dataset = tf.data.Dataset.from_tensor_slices((x_tst, y_tst))\n", 206 | "tst_dataset = tst_dataset.batch(batch_size=100)\n", 207 | "print(tst_dataset)" 208 | ] 209 | }, 210 | { 211 | "cell_type": "code", 212 | "execution_count": 7, 213 | "metadata": {}, 214 | "outputs": [], 215 | "source": [ 216 | "## create model\n", 217 | "cnn = SimpleCNN(num_classes=10)\n", 218 | "\n", 219 | "# creating callbacks for tensorboard\n", 220 | "callbacks = [keras.callbacks.TensorBoard(log_dir='../graphs/lecture07/convnet_mnist_batch_norm_kd/',\n", 221 | " write_graph=True, write_images=True)]" 222 | ] 223 | }, 224 | { 225 | "cell_type": "code", 226 | "execution_count": 8, 227 | "metadata": {}, 228 | "outputs": [], 229 | "source": [ 230 | "# complile\n", 231 | "cnn.compile(optimizer=tf.train.AdamOptimizer(learning_rate=lr),\n", 232 | " loss=keras.losses.sparse_categorical_crossentropy,\n", 233 | " callbacks=callbacks)" 234 | ] 235 | }, 236 | { 237 | "cell_type": "markdown", 238 | "metadata": {}, 239 | "source": [ 240 | "### Train a model" 241 | ] 242 | }, 243 | { 244 | "cell_type": "code", 245 | "execution_count": 9, 246 | "metadata": {}, 247 | "outputs": [ 248 | { 249 | "name": "stdout", 250 | "output_type": "stream", 251 | "text": [ 252 | "Epoch 1/5\n", 253 | "550/550 [==============================] - 5s 9ms/step - loss: 0.1012 - val_loss: 0.0539\n", 254 | "Epoch 2/5\n", 255 | "550/550 [==============================] - 4s 7ms/step - loss: 0.0312 - val_loss: 0.0569\n", 256 | "Epoch 3/5\n", 257 | "550/550 [==============================] - 4s 7ms/step - loss: 0.0202 - val_loss: 0.0366\n", 258 | "Epoch 4/5\n", 259 | "550/550 [==============================] - 4s 7ms/step - loss: 0.0135 - val_loss: 0.0122\n", 260 | "Epoch 5/5\n", 261 | "550/550 [==============================] - 4s 7ms/step - loss: 0.0104 - val_loss: 0.0124\n" 262 | ] 263 | }, 264 | { 265 | "data": { 266 | "text/plain": [ 267 | "" 268 | ] 269 | }, 270 | "execution_count": 9, 271 | "metadata": {}, 272 | "output_type": "execute_result" 273 | } 274 | ], 275 | "source": [ 276 | "cnn.fit(tr_dataset, epochs=epochs, steps_per_epoch=total_step,\n", 277 | " validation_data=val_dataset, validation_steps=5000//100)" 278 | ] 279 | }, 280 | { 281 | "cell_type": "markdown", 282 | "metadata": {}, 283 | "source": [ 284 | "### Calculate accuracy " 285 | ] 286 | }, 287 | { 288 | "cell_type": "code", 289 | "execution_count": 10, 290 | "metadata": {}, 291 | "outputs": [], 292 | "source": [ 293 | "sess = keras.backend.get_session()" 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": 11, 299 | "metadata": {}, 300 | "outputs": [], 301 | "source": [ 302 | "x_tst_tensor = tf.convert_to_tensor(x_tst)\n", 303 | "yhat = sess.run(cnn(x_tst_tensor, training=False))" 304 | ] 305 | }, 306 | { 307 | "cell_type": "code", 308 | "execution_count": 12, 309 | "metadata": {}, 310 | "outputs": [ 311 | { 312 | "name": "stdout", 313 | "output_type": "stream", 314 | "text": [ 315 | "tst acc : 98.69%\n" 316 | ] 317 | } 318 | ], 319 | "source": [ 320 | "print('tst acc : {:.2%}'.format(np.mean(np.argmax(yhat, axis=-1) == y_tst)))" 321 | ] 322 | } 323 | ], 324 | "metadata": { 325 | "kernelspec": { 326 | "display_name": "Python 3", 327 | "language": "python", 328 | "name": "python3" 329 | }, 330 | "language_info": { 331 | "codemirror_mode": { 332 | "name": "ipython", 333 | "version": 3 334 | }, 335 | "file_extension": ".py", 336 | "mimetype": "text/x-python", 337 | "name": "python", 338 | "nbconvert_exporter": "python", 339 | "pygments_lexer": "ipython3", 340 | "version": "3.6.8" 341 | }, 342 | "varInspector": { 343 | "cols": { 344 | "lenName": 16, 345 | "lenType": 16, 346 | "lenVar": 40 347 | }, 348 | "kernels_config": { 349 | "python": { 350 | "delete_cmd_postfix": "", 351 | "delete_cmd_prefix": "del ", 352 | "library": "var_list.py", 353 | "varRefreshCmd": "print(var_dic_list())" 354 | }, 355 | "r": { 356 | "delete_cmd_postfix": ") ", 357 | "delete_cmd_prefix": "rm(", 358 | "library": "var_list.r", 359 | "varRefreshCmd": "cat(var_dic_list()) " 360 | } 361 | }, 362 | "types_to_exclude": [ 363 | "module", 364 | "function", 365 | "builtin_function_or_method", 366 | "instance", 367 | "_Feature" 368 | ], 369 | "window_display": false 370 | } 371 | }, 372 | "nbformat": 4, 373 | "nbformat_minor": 2 374 | } 375 | -------------------------------------------------------------------------------- /Lec11_Recurrent Neural Networks/Lec11_Many to Many Classification by Bi-directional GRU.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# CS 20 : TensorFlow for Deep Learning Research\n", 8 | "## Lecture 11 : Recurrent Neural Networks\n", 9 | "Simple example for Many to Many Classification (Simple pos tagger) by Bi-directional Gated Recurrent Unit. \n", 10 | "\n", 11 | "### Many to Many Classification by Bi-directional GRU\n", 12 | "- Creating the **data pipeline** with `tf.data`\n", 13 | "- Preprocessing word sequences (variable input sequence length) using `padding technique` by `user function (pad_seq)`\n", 14 | "- Using `tf.nn.embedding_lookup` for getting vector of tokens (eg. word, character)\n", 15 | "- Training **many to many classification** with `tf.contrib.seq2seq.sequence_loss`\n", 16 | "- Masking unvalid token with `tf.sequence_mask`\n", 17 | "- Creating the model as **Class**\n", 18 | "- Reference\n", 19 | " - https://github.com/aisolab/sample_code_of_Deep_learning_Basics/blob/master/DLEL/DLEL_12_2_RNN_(toy_example).ipynb" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": {}, 25 | "source": [ 26 | "### Setup" 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": 1, 32 | "metadata": {}, 33 | "outputs": [ 34 | { 35 | "name": "stdout", 36 | "output_type": "stream", 37 | "text": [ 38 | "1.8.0\n" 39 | ] 40 | } 41 | ], 42 | "source": [ 43 | "import os, sys\n", 44 | "import numpy as np\n", 45 | "import pandas as pd\n", 46 | "import matplotlib.pyplot as plt\n", 47 | "import tensorflow as tf\n", 48 | "import string\n", 49 | "%matplotlib inline\n", 50 | "\n", 51 | "slim = tf.contrib.slim\n", 52 | "print(tf.__version__)" 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "metadata": {}, 58 | "source": [ 59 | "### Prepare example data " 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 2, 65 | "metadata": {}, 66 | "outputs": [], 67 | "source": [ 68 | "sentences = [['I', 'feel', 'hungry'],\n", 69 | " ['tensorflow', 'is', 'very', 'difficult'],\n", 70 | " ['tensorflow', 'is', 'a', 'framework', 'for', 'deep', 'learning'],\n", 71 | " ['tensorflow', 'is', 'very', 'fast', 'changing']]\n", 72 | "pos = [['pronoun', 'verb', 'adjective'],\n", 73 | " ['noun', 'verb', 'adverb', 'adjective'],\n", 74 | " ['noun', 'verb', 'determiner', 'noun', 'preposition', 'adjective', 'noun'],\n", 75 | " ['noun', 'verb', 'adverb', 'adjective', 'verb']]" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": 3, 81 | "metadata": {}, 82 | "outputs": [ 83 | { 84 | "name": "stdout", 85 | "output_type": "stream", 86 | "text": [ 87 | "{'': 0, 'I': 1, 'a': 2, 'changing': 3, 'deep': 4, 'difficult': 5, 'fast': 6, 'feel': 7, 'for': 8, 'framework': 9, 'hungry': 10, 'is': 11, 'learning': 12, 'tensorflow': 13, 'very': 14}\n" 88 | ] 89 | } 90 | ], 91 | "source": [ 92 | "# word dic\n", 93 | "word_list = []\n", 94 | "for elm in sentences:\n", 95 | " word_list += elm\n", 96 | "word_list = list(set(word_list))\n", 97 | "word_list.sort()\n", 98 | "word_list = [''] + word_list\n", 99 | "\n", 100 | "word_dic = {word : idx for idx, word in enumerate(word_list)}\n", 101 | "print(word_dic)" 102 | ] 103 | }, 104 | { 105 | "cell_type": "code", 106 | "execution_count": 4, 107 | "metadata": {}, 108 | "outputs": [ 109 | { 110 | "name": "stdout", 111 | "output_type": "stream", 112 | "text": [ 113 | "['', 'adjective', 'adverb', 'determiner', 'noun', 'preposition', 'pronoun', 'verb']\n" 114 | ] 115 | }, 116 | { 117 | "data": { 118 | "text/plain": [ 119 | "{'': 0,\n", 120 | " 'adjective': 1,\n", 121 | " 'adverb': 2,\n", 122 | " 'determiner': 3,\n", 123 | " 'noun': 4,\n", 124 | " 'preposition': 5,\n", 125 | " 'pronoun': 6,\n", 126 | " 'verb': 7}" 127 | ] 128 | }, 129 | "execution_count": 4, 130 | "metadata": {}, 131 | "output_type": "execute_result" 132 | } 133 | ], 134 | "source": [ 135 | "# pos dic\n", 136 | "pos_list = []\n", 137 | "for elm in pos:\n", 138 | " pos_list += elm\n", 139 | "pos_list = list(set(pos_list))\n", 140 | "pos_list.sort()\n", 141 | "pos_list = [''] + pos_list\n", 142 | "print(pos_list)\n", 143 | "\n", 144 | "pos_dic = {pos : idx for idx, pos in enumerate(pos_list)}\n", 145 | "pos_dic" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": 5, 151 | "metadata": {}, 152 | "outputs": [ 153 | { 154 | "data": { 155 | "text/plain": [ 156 | "{0: '',\n", 157 | " 1: 'adjective',\n", 158 | " 2: 'adverb',\n", 159 | " 3: 'determiner',\n", 160 | " 4: 'noun',\n", 161 | " 5: 'preposition',\n", 162 | " 6: 'pronoun',\n", 163 | " 7: 'verb'}" 164 | ] 165 | }, 166 | "execution_count": 5, 167 | "metadata": {}, 168 | "output_type": "execute_result" 169 | } 170 | ], 171 | "source": [ 172 | "pos_idx_to_dic = {elm[1] : elm[0] for elm in pos_dic.items()}\n", 173 | "pos_idx_to_dic" 174 | ] 175 | }, 176 | { 177 | "cell_type": "markdown", 178 | "metadata": {}, 179 | "source": [ 180 | "### Create pad_seq function" 181 | ] 182 | }, 183 | { 184 | "cell_type": "code", 185 | "execution_count": 6, 186 | "metadata": {}, 187 | "outputs": [], 188 | "source": [ 189 | "def pad_seq(sequences, max_len, dic):\n", 190 | " seq_len, seq_indices = [], []\n", 191 | " for seq in sequences:\n", 192 | " seq_len.append(len(seq))\n", 193 | " seq_idx = [dic.get(char) for char in seq]\n", 194 | " seq_idx += (max_len - len(seq_idx)) * [dic.get('')] # 0 is idx of meaningless token \"\"\n", 195 | " seq_indices.append(seq_idx)\n", 196 | " return seq_len, seq_indices" 197 | ] 198 | }, 199 | { 200 | "cell_type": "markdown", 201 | "metadata": {}, 202 | "source": [ 203 | "### Pre-process data" 204 | ] 205 | }, 206 | { 207 | "cell_type": "code", 208 | "execution_count": 7, 209 | "metadata": {}, 210 | "outputs": [ 211 | { 212 | "name": "stdout", 213 | "output_type": "stream", 214 | "text": [ 215 | "[3, 4, 7, 5] (4, 10)\n" 216 | ] 217 | } 218 | ], 219 | "source": [ 220 | "max_length = 10\n", 221 | "X_length, X_indices = pad_seq(sequences = sentences, max_len = max_length, dic = word_dic)\n", 222 | "print(X_length, np.shape(X_indices))" 223 | ] 224 | }, 225 | { 226 | "cell_type": "code", 227 | "execution_count": 8, 228 | "metadata": {}, 229 | "outputs": [ 230 | { 231 | "name": "stdout", 232 | "output_type": "stream", 233 | "text": [ 234 | "(4, 10)\n" 235 | ] 236 | } 237 | ], 238 | "source": [ 239 | "y = [elm + [''] * (max_length - len(elm)) for elm in pos]\n", 240 | "y = [list(map(lambda el : pos_dic.get(el), elm)) for elm in y]\n", 241 | "print(np.shape(y))" 242 | ] 243 | }, 244 | { 245 | "cell_type": "code", 246 | "execution_count": 9, 247 | "metadata": {}, 248 | "outputs": [ 249 | { 250 | "data": { 251 | "text/plain": [ 252 | "[[6, 7, 1, 0, 0, 0, 0, 0, 0, 0],\n", 253 | " [4, 7, 2, 1, 0, 0, 0, 0, 0, 0],\n", 254 | " [4, 7, 3, 4, 5, 1, 4, 0, 0, 0],\n", 255 | " [4, 7, 2, 1, 7, 0, 0, 0, 0, 0]]" 256 | ] 257 | }, 258 | "execution_count": 9, 259 | "metadata": {}, 260 | "output_type": "execute_result" 261 | } 262 | ], 263 | "source": [ 264 | "y" 265 | ] 266 | }, 267 | { 268 | "cell_type": "markdown", 269 | "metadata": {}, 270 | "source": [ 271 | "### Define SimPosBiGRU" 272 | ] 273 | }, 274 | { 275 | "cell_type": "code", 276 | "execution_count": 10, 277 | "metadata": {}, 278 | "outputs": [], 279 | "source": [ 280 | "class SimPosBiGRU:\n", 281 | " def __init__(self, X_length, X_indices, y, n_of_classes, hidden_dim, max_len, word_dic):\n", 282 | " \n", 283 | " # Data pipeline\n", 284 | " with tf.variable_scope('input_layer'):\n", 285 | " self._X_length = X_length\n", 286 | " self._X_indices = X_indices\n", 287 | " self._y = y\n", 288 | " \n", 289 | " one_hot = tf.eye(len(word_dic), dtype = tf.float32)\n", 290 | " self._one_hot = tf.get_variable(name='one_hot_embedding', initializer = one_hot,\n", 291 | " trainable = False) # embedding vector training 안할 것이기 때문\n", 292 | " self._X_batch = tf.nn.embedding_lookup(params = self._one_hot, ids = self._X_indices)\n", 293 | " \n", 294 | " # Bi-directional GRU (many to many)\n", 295 | " with tf.variable_scope('bi-directional_gru'):\n", 296 | " gru_fw_cell = tf.contrib.rnn.GRUCell(num_units = hidden_dim,\n", 297 | " activation = tf.nn.tanh)\n", 298 | " gru_bw_cell = tf.contrib.rnn.GRUCell(num_units = hidden_dim,\n", 299 | " activation = tf.nn.tanh)\n", 300 | " outputs, _ = tf.nn.bidirectional_dynamic_rnn(cell_fw = gru_fw_cell, cell_bw = gru_bw_cell,\n", 301 | " inputs = self._X_batch, sequence_length = self._X_length,\n", 302 | " dtype = tf.float32)\n", 303 | " concatenated_outputs = tf.concat([outputs[0], outputs[1]], axis = 2)\n", 304 | " weights = tf.get_variable(name = 'weights', shape = (2 * hidden_dim, n_of_classes),\n", 305 | " initializer = tf.contrib.layers.xavier_initializer())\n", 306 | " self._score = tf.map_fn(lambda elm : tf.matmul(elm, weights), concatenated_outputs)\n", 307 | " \n", 308 | " with tf.variable_scope('seq2seq_loss'):\n", 309 | " masks = tf.sequence_mask(lengths = self._X_length, maxlen = max_len, dtype = tf.float32)\n", 310 | " self.seq2seq_loss = tf.contrib.seq2seq.sequence_loss(logits = self._score,\n", 311 | " targets = self._y, weights = masks)\n", 312 | " \n", 313 | " with tf.variable_scope('prediction'):\n", 314 | " self._prediction = tf.argmax(input = self._score,\n", 315 | " axis = 2, output_type = tf.int32)\n", 316 | " \n", 317 | " def predict(self, sess, X_length, X_indices):\n", 318 | " feed_prediction = {self._X_length : X_length, self._X_indices : X_indices}\n", 319 | " return sess.run(self._prediction, feed_dict = feed_prediction)" 320 | ] 321 | }, 322 | { 323 | "cell_type": "markdown", 324 | "metadata": {}, 325 | "source": [ 326 | "### Create a model of SimPosBiGRU" 327 | ] 328 | }, 329 | { 330 | "cell_type": "code", 331 | "execution_count": 11, 332 | "metadata": {}, 333 | "outputs": [ 334 | { 335 | "name": "stdout", 336 | "output_type": "stream", 337 | "text": [ 338 | "2\n" 339 | ] 340 | } 341 | ], 342 | "source": [ 343 | "# hyper-parameter#\n", 344 | "lr = .003\n", 345 | "epochs = 100\n", 346 | "batch_size = 2\n", 347 | "total_step = int(np.shape(X_indices)[0] / batch_size)\n", 348 | "print(total_step)" 349 | ] 350 | }, 351 | { 352 | "cell_type": "code", 353 | "execution_count": 12, 354 | "metadata": {}, 355 | "outputs": [ 356 | { 357 | "name": "stdout", 358 | "output_type": "stream", 359 | "text": [ 360 | "\n" 361 | ] 362 | } 363 | ], 364 | "source": [ 365 | "## create data pipeline with tf.data\n", 366 | "tr_dataset = tf.data.Dataset.from_tensor_slices((X_length, X_indices, y))\n", 367 | "tr_dataset = tr_dataset.shuffle(buffer_size = 20)\n", 368 | "tr_dataset = tr_dataset.batch(batch_size = batch_size)\n", 369 | "tr_iterator = tr_dataset.make_initializable_iterator()\n", 370 | "print(tr_dataset)" 371 | ] 372 | }, 373 | { 374 | "cell_type": "code", 375 | "execution_count": 13, 376 | "metadata": {}, 377 | "outputs": [], 378 | "source": [ 379 | "X_length_mb, X_indices_mb, y_mb = tr_iterator.get_next()" 380 | ] 381 | }, 382 | { 383 | "cell_type": "code", 384 | "execution_count": 14, 385 | "metadata": {}, 386 | "outputs": [], 387 | "source": [ 388 | "sim_pos_bi_gru = SimPosBiGRU(X_length = X_length_mb, X_indices = X_indices_mb, y = y_mb,\n", 389 | " n_of_classes = 8, hidden_dim = 16, max_len = max_length, word_dic = word_dic)" 390 | ] 391 | }, 392 | { 393 | "cell_type": "markdown", 394 | "metadata": {}, 395 | "source": [ 396 | "### Creat training op and train model" 397 | ] 398 | }, 399 | { 400 | "cell_type": "code", 401 | "execution_count": 15, 402 | "metadata": {}, 403 | "outputs": [], 404 | "source": [ 405 | "## create training op\n", 406 | "opt = tf.train.AdamOptimizer(learning_rate = lr)\n", 407 | "training_op = opt.minimize(loss = sim_pos_bi_gru.seq2seq_loss)" 408 | ] 409 | }, 410 | { 411 | "cell_type": "code", 412 | "execution_count": 16, 413 | "metadata": {}, 414 | "outputs": [ 415 | { 416 | "name": "stdout", 417 | "output_type": "stream", 418 | "text": [ 419 | "epoch : 10, tr_loss : 1.893\n", 420 | "epoch : 20, tr_loss : 1.587\n", 421 | "epoch : 30, tr_loss : 1.218\n", 422 | "epoch : 40, tr_loss : 0.843\n", 423 | "epoch : 50, tr_loss : 0.484\n", 424 | "epoch : 60, tr_loss : 0.249\n", 425 | "epoch : 70, tr_loss : 0.129\n", 426 | "epoch : 80, tr_loss : 0.073\n", 427 | "epoch : 90, tr_loss : 0.045\n", 428 | "epoch : 100, tr_loss : 0.031\n" 429 | ] 430 | } 431 | ], 432 | "source": [ 433 | "sess = tf.Session()\n", 434 | "sess.run(tf.global_variables_initializer())\n", 435 | "\n", 436 | "tr_loss_hist = []\n", 437 | "\n", 438 | "for epoch in range(epochs):\n", 439 | " avg_tr_loss = 0\n", 440 | " tr_step = 0\n", 441 | " \n", 442 | " sess.run(tr_iterator.initializer)\n", 443 | " try:\n", 444 | " while True:\n", 445 | " _, tr_loss = sess.run(fetches = [training_op, sim_pos_bi_gru.seq2seq_loss])\n", 446 | " avg_tr_loss += tr_loss\n", 447 | " tr_step += 1\n", 448 | " \n", 449 | " except tf.errors.OutOfRangeError:\n", 450 | " pass\n", 451 | " \n", 452 | " avg_tr_loss /= tr_step\n", 453 | " tr_loss_hist.append(avg_tr_loss)\n", 454 | " if (epoch + 1) % 10 == 0:\n", 455 | " print('epoch : {:3}, tr_loss : {:.3f}'.format(epoch + 1, avg_tr_loss))" 456 | ] 457 | }, 458 | { 459 | "cell_type": "code", 460 | "execution_count": 17, 461 | "metadata": {}, 462 | "outputs": [ 463 | { 464 | "data": { 465 | "text/plain": [ 466 | "array([[6, 7, 1, 0, 0, 0, 0, 0, 0, 0],\n", 467 | " [4, 7, 2, 1, 0, 0, 0, 0, 0, 0],\n", 468 | " [4, 7, 3, 4, 5, 1, 4, 0, 0, 0],\n", 469 | " [4, 7, 2, 1, 7, 0, 0, 0, 0, 0]], dtype=int32)" 470 | ] 471 | }, 472 | "execution_count": 17, 473 | "metadata": {}, 474 | "output_type": "execute_result" 475 | } 476 | ], 477 | "source": [ 478 | "yhat = sim_pos_bi_gru.predict(sess = sess, X_length = X_length, X_indices = X_indices)\n", 479 | "yhat" 480 | ] 481 | }, 482 | { 483 | "cell_type": "code", 484 | "execution_count": 18, 485 | "metadata": {}, 486 | "outputs": [ 487 | { 488 | "data": { 489 | "text/plain": [ 490 | "[[6, 7, 1, 0, 0, 0, 0, 0, 0, 0],\n", 491 | " [4, 7, 2, 1, 0, 0, 0, 0, 0, 0],\n", 492 | " [4, 7, 3, 4, 5, 1, 4, 0, 0, 0],\n", 493 | " [4, 7, 2, 1, 7, 0, 0, 0, 0, 0]]" 494 | ] 495 | }, 496 | "execution_count": 18, 497 | "metadata": {}, 498 | "output_type": "execute_result" 499 | } 500 | ], 501 | "source": [ 502 | "y" 503 | ] 504 | }, 505 | { 506 | "cell_type": "code", 507 | "execution_count": 19, 508 | "metadata": {}, 509 | "outputs": [ 510 | { 511 | "name": "stdout", 512 | "output_type": "stream", 513 | "text": [ 514 | "['pronoun', 'verb', 'adjective', '', '', '', '', '', '', '']\n", 515 | "['noun', 'verb', 'adverb', 'adjective', '', '', '', '', '', '']\n", 516 | "['noun', 'verb', 'determiner', 'noun', 'preposition', 'adjective', 'noun', '', '', '']\n", 517 | "['noun', 'verb', 'adverb', 'adjective', 'verb', '', '', '', '', '']\n" 518 | ] 519 | } 520 | ], 521 | "source": [ 522 | "yhat = [list(map(lambda elm : pos_idx_to_dic.get(elm), row)) for row in yhat]\n", 523 | "for elm in yhat:\n", 524 | " print(elm)" 525 | ] 526 | } 527 | ], 528 | "metadata": { 529 | "kernelspec": { 530 | "display_name": "Python 3", 531 | "language": "python", 532 | "name": "python3" 533 | }, 534 | "language_info": { 535 | "codemirror_mode": { 536 | "name": "ipython", 537 | "version": 3 538 | }, 539 | "file_extension": ".py", 540 | "mimetype": "text/x-python", 541 | "name": "python", 542 | "nbconvert_exporter": "python", 543 | "pygments_lexer": "ipython3", 544 | "version": "3.6.5" 545 | } 546 | }, 547 | "nbformat": 4, 548 | "nbformat_minor": 2 549 | } 550 | -------------------------------------------------------------------------------- /Lec11_Recurrent Neural Networks/Lec11_Many to Many Classification by Bi-directional RNN.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# CS 20 : TensorFlow for Deep Learning Research\n", 8 | "## Lecture 11 : Recurrent Neural Networks\n", 9 | "Simple example for Many to Many Classification (Simple pos tagger) by Bi-directional Recurrent Neural Networks. \n", 10 | "\n", 11 | "### Many to Many Classification by Bi-directional RNN\n", 12 | "- Creating the **data pipeline** with `tf.data`\n", 13 | "- Preprocessing word sequences (variable input sequence length) using `padding technique` by `user function (pad_seq)`\n", 14 | "- Using `tf.nn.embedding_lookup` for getting vector of tokens (eg. word, character)\n", 15 | "- Training **many to many classification** with `tf.contrib.seq2seq.sequence_loss`\n", 16 | "- Masking unvalid token with `tf.sequence_mask`\n", 17 | "- Creating the model as **Class**\n", 18 | "- Reference\n", 19 | " - https://github.com/aisolab/sample_code_of_Deep_learning_Basics/blob/master/DLEL/DLEL_12_2_RNN_(toy_example).ipynb" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": {}, 25 | "source": [ 26 | "### Setup" 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": 1, 32 | "metadata": {}, 33 | "outputs": [ 34 | { 35 | "name": "stdout", 36 | "output_type": "stream", 37 | "text": [ 38 | "1.8.0\n" 39 | ] 40 | } 41 | ], 42 | "source": [ 43 | "import os, sys\n", 44 | "import numpy as np\n", 45 | "import pandas as pd\n", 46 | "import matplotlib.pyplot as plt\n", 47 | "import tensorflow as tf\n", 48 | "import string\n", 49 | "%matplotlib inline\n", 50 | "\n", 51 | "slim = tf.contrib.slim\n", 52 | "print(tf.__version__)" 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "metadata": {}, 58 | "source": [ 59 | "### Prepare example data " 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 2, 65 | "metadata": {}, 66 | "outputs": [], 67 | "source": [ 68 | "sentences = [['I', 'feel', 'hungry'],\n", 69 | " ['tensorflow', 'is', 'very', 'difficult'],\n", 70 | " ['tensorflow', 'is', 'a', 'framework', 'for', 'deep', 'learning'],\n", 71 | " ['tensorflow', 'is', 'very', 'fast', 'changing']]\n", 72 | "pos = [['pronoun', 'verb', 'adjective'],\n", 73 | " ['noun', 'verb', 'adverb', 'adjective'],\n", 74 | " ['noun', 'verb', 'determiner', 'noun', 'preposition', 'adjective', 'noun'],\n", 75 | " ['noun', 'verb', 'adverb', 'adjective', 'verb']]" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": 3, 81 | "metadata": {}, 82 | "outputs": [ 83 | { 84 | "name": "stdout", 85 | "output_type": "stream", 86 | "text": [ 87 | "{'': 0, 'I': 1, 'a': 2, 'changing': 3, 'deep': 4, 'difficult': 5, 'fast': 6, 'feel': 7, 'for': 8, 'framework': 9, 'hungry': 10, 'is': 11, 'learning': 12, 'tensorflow': 13, 'very': 14}\n" 88 | ] 89 | } 90 | ], 91 | "source": [ 92 | "# word dic\n", 93 | "word_list = []\n", 94 | "for elm in sentences:\n", 95 | " word_list += elm\n", 96 | "word_list = list(set(word_list))\n", 97 | "word_list.sort()\n", 98 | "word_list = [''] + word_list\n", 99 | "\n", 100 | "word_dic = {word : idx for idx, word in enumerate(word_list)}\n", 101 | "print(word_dic)" 102 | ] 103 | }, 104 | { 105 | "cell_type": "code", 106 | "execution_count": 4, 107 | "metadata": {}, 108 | "outputs": [ 109 | { 110 | "name": "stdout", 111 | "output_type": "stream", 112 | "text": [ 113 | "['', 'adjective', 'adverb', 'determiner', 'noun', 'preposition', 'pronoun', 'verb']\n" 114 | ] 115 | }, 116 | { 117 | "data": { 118 | "text/plain": [ 119 | "{'': 0,\n", 120 | " 'adjective': 1,\n", 121 | " 'adverb': 2,\n", 122 | " 'determiner': 3,\n", 123 | " 'noun': 4,\n", 124 | " 'preposition': 5,\n", 125 | " 'pronoun': 6,\n", 126 | " 'verb': 7}" 127 | ] 128 | }, 129 | "execution_count": 4, 130 | "metadata": {}, 131 | "output_type": "execute_result" 132 | } 133 | ], 134 | "source": [ 135 | "# pos dic\n", 136 | "pos_list = []\n", 137 | "for elm in pos:\n", 138 | " pos_list += elm\n", 139 | "pos_list = list(set(pos_list))\n", 140 | "pos_list.sort()\n", 141 | "pos_list = [''] + pos_list\n", 142 | "print(pos_list)\n", 143 | "\n", 144 | "pos_dic = {pos : idx for idx, pos in enumerate(pos_list)}\n", 145 | "pos_dic" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": 5, 151 | "metadata": {}, 152 | "outputs": [ 153 | { 154 | "data": { 155 | "text/plain": [ 156 | "{0: '',\n", 157 | " 1: 'adjective',\n", 158 | " 2: 'adverb',\n", 159 | " 3: 'determiner',\n", 160 | " 4: 'noun',\n", 161 | " 5: 'preposition',\n", 162 | " 6: 'pronoun',\n", 163 | " 7: 'verb'}" 164 | ] 165 | }, 166 | "execution_count": 5, 167 | "metadata": {}, 168 | "output_type": "execute_result" 169 | } 170 | ], 171 | "source": [ 172 | "pos_idx_to_dic = {elm[1] : elm[0] for elm in pos_dic.items()}\n", 173 | "pos_idx_to_dic" 174 | ] 175 | }, 176 | { 177 | "cell_type": "markdown", 178 | "metadata": {}, 179 | "source": [ 180 | "### Create pad_seq function" 181 | ] 182 | }, 183 | { 184 | "cell_type": "code", 185 | "execution_count": 6, 186 | "metadata": {}, 187 | "outputs": [], 188 | "source": [ 189 | "def pad_seq(sequences, max_len, dic):\n", 190 | " seq_len, seq_indices = [], []\n", 191 | " for seq in sequences:\n", 192 | " seq_len.append(len(seq))\n", 193 | " seq_idx = [dic.get(char) for char in seq]\n", 194 | " seq_idx += (max_len - len(seq_idx)) * [dic.get('')] # 0 is idx of meaningless token \"\"\n", 195 | " seq_indices.append(seq_idx)\n", 196 | " return seq_len, seq_indices" 197 | ] 198 | }, 199 | { 200 | "cell_type": "markdown", 201 | "metadata": {}, 202 | "source": [ 203 | "### Pre-process data" 204 | ] 205 | }, 206 | { 207 | "cell_type": "code", 208 | "execution_count": 7, 209 | "metadata": {}, 210 | "outputs": [ 211 | { 212 | "name": "stdout", 213 | "output_type": "stream", 214 | "text": [ 215 | "[3, 4, 7, 5] (4, 10)\n" 216 | ] 217 | } 218 | ], 219 | "source": [ 220 | "max_length = 10\n", 221 | "X_length, X_indices = pad_seq(sequences = sentences, max_len = max_length, dic = word_dic)\n", 222 | "print(X_length, np.shape(X_indices))" 223 | ] 224 | }, 225 | { 226 | "cell_type": "code", 227 | "execution_count": 8, 228 | "metadata": {}, 229 | "outputs": [ 230 | { 231 | "name": "stdout", 232 | "output_type": "stream", 233 | "text": [ 234 | "(4, 10)\n" 235 | ] 236 | } 237 | ], 238 | "source": [ 239 | "y = [elm + [''] * (max_length - len(elm)) for elm in pos]\n", 240 | "y = [list(map(lambda el : pos_dic.get(el), elm)) for elm in y]\n", 241 | "print(np.shape(y))" 242 | ] 243 | }, 244 | { 245 | "cell_type": "code", 246 | "execution_count": 9, 247 | "metadata": {}, 248 | "outputs": [ 249 | { 250 | "data": { 251 | "text/plain": [ 252 | "[[6, 7, 1, 0, 0, 0, 0, 0, 0, 0],\n", 253 | " [4, 7, 2, 1, 0, 0, 0, 0, 0, 0],\n", 254 | " [4, 7, 3, 4, 5, 1, 4, 0, 0, 0],\n", 255 | " [4, 7, 2, 1, 7, 0, 0, 0, 0, 0]]" 256 | ] 257 | }, 258 | "execution_count": 9, 259 | "metadata": {}, 260 | "output_type": "execute_result" 261 | } 262 | ], 263 | "source": [ 264 | "y" 265 | ] 266 | }, 267 | { 268 | "cell_type": "markdown", 269 | "metadata": {}, 270 | "source": [ 271 | "### Define SimPosBiRNN" 272 | ] 273 | }, 274 | { 275 | "cell_type": "code", 276 | "execution_count": 10, 277 | "metadata": {}, 278 | "outputs": [], 279 | "source": [ 280 | "class SimPosBiRNN:\n", 281 | " def __init__(self, X_length, X_indices, y, n_of_classes, hidden_dim, max_len, word_dic):\n", 282 | " \n", 283 | " # Data pipeline\n", 284 | " with tf.variable_scope('input_layer'):\n", 285 | " self._X_length = X_length\n", 286 | " self._X_indices = X_indices\n", 287 | " self._y = y\n", 288 | " \n", 289 | " one_hot = tf.eye(len(word_dic), dtype = tf.float32)\n", 290 | " self._one_hot = tf.get_variable(name='one_hot_embedding', initializer = one_hot,\n", 291 | " trainable = False) # embedding vector training 안할 것이기 때문\n", 292 | " self._X_batch = tf.nn.embedding_lookup(params = self._one_hot, ids = self._X_indices)\n", 293 | " \n", 294 | " # Bi-directional RNN (many to many)\n", 295 | " with tf.variable_scope('bi-directional_rnn'):\n", 296 | " rnn_fw_cell = tf.contrib.rnn.BasicRNNCell(num_units = hidden_dim,\n", 297 | " activation = tf.nn.tanh)\n", 298 | " rnn_bw_cell = tf.contrib.rnn.BasicRNNCell(num_units = hidden_dim,\n", 299 | " activation = tf.nn.tanh)\n", 300 | " outputs, _ = tf.nn.bidirectional_dynamic_rnn(cell_fw = rnn_fw_cell, cell_bw = rnn_bw_cell,\n", 301 | " inputs = self._X_batch, sequence_length = self._X_length,\n", 302 | " dtype = tf.float32)\n", 303 | " concatenated_outputs = tf.concat([outputs[0], outputs[1]], axis = 2)\n", 304 | " weights = tf.get_variable(name = 'weights', shape = (2 * hidden_dim, n_of_classes),\n", 305 | " initializer = tf.contrib.layers.xavier_initializer())\n", 306 | " self._score = tf.map_fn(lambda elm : tf.matmul(elm, weights), concatenated_outputs)\n", 307 | " \n", 308 | " with tf.variable_scope('seq2seq_loss'):\n", 309 | " masks = tf.sequence_mask(lengths = self._X_length, maxlen = max_len, dtype = tf.float32)\n", 310 | " self.seq2seq_loss = tf.contrib.seq2seq.sequence_loss(logits = self._score,\n", 311 | " targets = self._y, weights = masks)\n", 312 | " \n", 313 | " with tf.variable_scope('prediction'):\n", 314 | " self._prediction = tf.argmax(input = self._score,\n", 315 | " axis = 2, output_type = tf.int32)\n", 316 | " \n", 317 | " def predict(self, sess, X_length, X_indices):\n", 318 | " feed_prediction = {self._X_length : X_length, self._X_indices : X_indices}\n", 319 | " return sess.run(self._prediction, feed_dict = feed_prediction)" 320 | ] 321 | }, 322 | { 323 | "cell_type": "markdown", 324 | "metadata": {}, 325 | "source": [ 326 | "### Create a model of SimPosBiRNN" 327 | ] 328 | }, 329 | { 330 | "cell_type": "code", 331 | "execution_count": 11, 332 | "metadata": {}, 333 | "outputs": [ 334 | { 335 | "name": "stdout", 336 | "output_type": "stream", 337 | "text": [ 338 | "2\n" 339 | ] 340 | } 341 | ], 342 | "source": [ 343 | "# hyper-parameter#\n", 344 | "lr = .003\n", 345 | "epochs = 100\n", 346 | "batch_size = 2\n", 347 | "total_step = int(np.shape(X_indices)[0] / batch_size)\n", 348 | "print(total_step)" 349 | ] 350 | }, 351 | { 352 | "cell_type": "code", 353 | "execution_count": 12, 354 | "metadata": {}, 355 | "outputs": [ 356 | { 357 | "name": "stdout", 358 | "output_type": "stream", 359 | "text": [ 360 | "\n" 361 | ] 362 | } 363 | ], 364 | "source": [ 365 | "## create data pipeline with tf.data\n", 366 | "tr_dataset = tf.data.Dataset.from_tensor_slices((X_length, X_indices, y))\n", 367 | "tr_dataset = tr_dataset.shuffle(buffer_size = 20)\n", 368 | "tr_dataset = tr_dataset.batch(batch_size = batch_size)\n", 369 | "tr_iterator = tr_dataset.make_initializable_iterator()\n", 370 | "print(tr_dataset)" 371 | ] 372 | }, 373 | { 374 | "cell_type": "code", 375 | "execution_count": 13, 376 | "metadata": {}, 377 | "outputs": [], 378 | "source": [ 379 | "X_length_mb, X_indices_mb, y_mb = tr_iterator.get_next()" 380 | ] 381 | }, 382 | { 383 | "cell_type": "code", 384 | "execution_count": 14, 385 | "metadata": {}, 386 | "outputs": [], 387 | "source": [ 388 | "sim_pos_bi_rnn = SimPosBiRNN(X_length = X_length_mb, X_indices = X_indices_mb, y = y_mb,\n", 389 | " n_of_classes = 8, hidden_dim = 16, max_len = max_length, word_dic = word_dic)" 390 | ] 391 | }, 392 | { 393 | "cell_type": "markdown", 394 | "metadata": {}, 395 | "source": [ 396 | "### Creat training op and train model" 397 | ] 398 | }, 399 | { 400 | "cell_type": "code", 401 | "execution_count": 15, 402 | "metadata": {}, 403 | "outputs": [], 404 | "source": [ 405 | "## create training op\n", 406 | "opt = tf.train.AdamOptimizer(learning_rate = lr)\n", 407 | "training_op = opt.minimize(loss = sim_pos_bi_rnn.seq2seq_loss)" 408 | ] 409 | }, 410 | { 411 | "cell_type": "code", 412 | "execution_count": 16, 413 | "metadata": {}, 414 | "outputs": [ 415 | { 416 | "name": "stdout", 417 | "output_type": "stream", 418 | "text": [ 419 | "epoch : 10, tr_loss : 1.364\n", 420 | "epoch : 20, tr_loss : 0.700\n", 421 | "epoch : 30, tr_loss : 0.297\n", 422 | "epoch : 40, tr_loss : 0.150\n", 423 | "epoch : 50, tr_loss : 0.088\n", 424 | "epoch : 60, tr_loss : 0.057\n", 425 | "epoch : 70, tr_loss : 0.042\n", 426 | "epoch : 80, tr_loss : 0.032\n", 427 | "epoch : 90, tr_loss : 0.026\n", 428 | "epoch : 100, tr_loss : 0.021\n" 429 | ] 430 | } 431 | ], 432 | "source": [ 433 | "sess = tf.Session()\n", 434 | "sess.run(tf.global_variables_initializer())\n", 435 | "\n", 436 | "tr_loss_hist = []\n", 437 | "\n", 438 | "for epoch in range(epochs):\n", 439 | " avg_tr_loss = 0\n", 440 | " tr_step = 0\n", 441 | " \n", 442 | " sess.run(tr_iterator.initializer)\n", 443 | " try:\n", 444 | " while True:\n", 445 | " _, tr_loss = sess.run(fetches = [training_op, sim_pos_bi_rnn.seq2seq_loss])\n", 446 | " avg_tr_loss += tr_loss\n", 447 | " tr_step += 1\n", 448 | " \n", 449 | " except tf.errors.OutOfRangeError:\n", 450 | " pass\n", 451 | " \n", 452 | " avg_tr_loss /= tr_step\n", 453 | " tr_loss_hist.append(avg_tr_loss)\n", 454 | " if (epoch + 1) % 10 == 0:\n", 455 | " print('epoch : {:3}, tr_loss : {:.3f}'.format(epoch + 1, avg_tr_loss))" 456 | ] 457 | }, 458 | { 459 | "cell_type": "code", 460 | "execution_count": 17, 461 | "metadata": {}, 462 | "outputs": [ 463 | { 464 | "data": { 465 | "text/plain": [ 466 | "array([[6, 7, 1, 0, 0, 0, 0, 0, 0, 0],\n", 467 | " [4, 7, 2, 1, 0, 0, 0, 0, 0, 0],\n", 468 | " [4, 7, 3, 4, 5, 1, 4, 0, 0, 0],\n", 469 | " [4, 7, 2, 1, 7, 0, 0, 0, 0, 0]], dtype=int32)" 470 | ] 471 | }, 472 | "execution_count": 17, 473 | "metadata": {}, 474 | "output_type": "execute_result" 475 | } 476 | ], 477 | "source": [ 478 | "yhat = sim_pos_bi_rnn.predict(sess = sess, X_length = X_length, X_indices = X_indices)\n", 479 | "yhat" 480 | ] 481 | }, 482 | { 483 | "cell_type": "code", 484 | "execution_count": 18, 485 | "metadata": {}, 486 | "outputs": [ 487 | { 488 | "data": { 489 | "text/plain": [ 490 | "[[6, 7, 1, 0, 0, 0, 0, 0, 0, 0],\n", 491 | " [4, 7, 2, 1, 0, 0, 0, 0, 0, 0],\n", 492 | " [4, 7, 3, 4, 5, 1, 4, 0, 0, 0],\n", 493 | " [4, 7, 2, 1, 7, 0, 0, 0, 0, 0]]" 494 | ] 495 | }, 496 | "execution_count": 18, 497 | "metadata": {}, 498 | "output_type": "execute_result" 499 | } 500 | ], 501 | "source": [ 502 | "y" 503 | ] 504 | }, 505 | { 506 | "cell_type": "code", 507 | "execution_count": 19, 508 | "metadata": {}, 509 | "outputs": [ 510 | { 511 | "name": "stdout", 512 | "output_type": "stream", 513 | "text": [ 514 | "['pronoun', 'verb', 'adjective', '', '', '', '', '', '', '']\n", 515 | "['noun', 'verb', 'adverb', 'adjective', '', '', '', '', '', '']\n", 516 | "['noun', 'verb', 'determiner', 'noun', 'preposition', 'adjective', 'noun', '', '', '']\n", 517 | "['noun', 'verb', 'adverb', 'adjective', 'verb', '', '', '', '', '']\n" 518 | ] 519 | } 520 | ], 521 | "source": [ 522 | "yhat = [list(map(lambda elm : pos_idx_to_dic.get(elm), row)) for row in yhat]\n", 523 | "for elm in yhat:\n", 524 | " print(elm)" 525 | ] 526 | } 527 | ], 528 | "metadata": { 529 | "kernelspec": { 530 | "display_name": "Python 3", 531 | "language": "python", 532 | "name": "python3" 533 | }, 534 | "language_info": { 535 | "codemirror_mode": { 536 | "name": "ipython", 537 | "version": 3 538 | }, 539 | "file_extension": ".py", 540 | "mimetype": "text/x-python", 541 | "name": "python", 542 | "nbconvert_exporter": "python", 543 | "pygments_lexer": "ipython3", 544 | "version": "3.6.5" 545 | } 546 | }, 547 | "nbformat": 4, 548 | "nbformat_minor": 2 549 | } 550 | -------------------------------------------------------------------------------- /Lec11_Recurrent Neural Networks/Lec11_Many to Many Classification by GRU.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# CS 20 : TensorFlow for Deep Learning Research\n", 8 | "## Lecture 11 : Recurrent Neural Networks\n", 9 | "Simple example for Many to Many Classification (Simple pos tagger) by Gated Recurrent Unit. \n", 10 | "\n", 11 | "### Many to Many Classification by GRU\n", 12 | "- Creating the **data pipeline** with `tf.data`\n", 13 | "- Preprocessing word sequences (variable input sequence length) using `padding technique` by `user function (pad_seq)`\n", 14 | "- Using `tf.nn.embedding_lookup` for getting vector of tokens (eg. word, character)\n", 15 | "- Training **many to many classification** with `tf.contrib.seq2seq.sequence_loss`\n", 16 | "- Masking unvalid token with `tf.sequence_mask`\n", 17 | "- Creating the model as **Class**\n", 18 | "- Reference\n", 19 | " - https://github.com/aisolab/sample_code_of_Deep_learning_Basics/blob/master/DLEL/DLEL_12_2_RNN_(toy_example).ipynb" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": {}, 25 | "source": [ 26 | "### Setup" 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": 1, 32 | "metadata": {}, 33 | "outputs": [ 34 | { 35 | "name": "stdout", 36 | "output_type": "stream", 37 | "text": [ 38 | "1.8.0\n" 39 | ] 40 | } 41 | ], 42 | "source": [ 43 | "import os, sys\n", 44 | "import numpy as np\n", 45 | "import pandas as pd\n", 46 | "import matplotlib.pyplot as plt\n", 47 | "import tensorflow as tf\n", 48 | "import string\n", 49 | "%matplotlib inline\n", 50 | "\n", 51 | "slim = tf.contrib.slim\n", 52 | "print(tf.__version__)" 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "metadata": {}, 58 | "source": [ 59 | "### Prepare example data " 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 2, 65 | "metadata": {}, 66 | "outputs": [], 67 | "source": [ 68 | "sentences = [['I', 'feel', 'hungry'],\n", 69 | " ['tensorflow', 'is', 'very', 'difficult'],\n", 70 | " ['tensorflow', 'is', 'a', 'framework', 'for', 'deep', 'learning'],\n", 71 | " ['tensorflow', 'is', 'very', 'fast', 'changing']]\n", 72 | "pos = [['pronoun', 'verb', 'adjective'],\n", 73 | " ['noun', 'verb', 'adverb', 'adjective'],\n", 74 | " ['noun', 'verb', 'determiner', 'noun', 'preposition', 'adjective', 'noun'],\n", 75 | " ['noun', 'verb', 'adverb', 'adjective', 'verb']]" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": 3, 81 | "metadata": {}, 82 | "outputs": [ 83 | { 84 | "name": "stdout", 85 | "output_type": "stream", 86 | "text": [ 87 | "{'': 0, 'I': 1, 'a': 2, 'changing': 3, 'deep': 4, 'difficult': 5, 'fast': 6, 'feel': 7, 'for': 8, 'framework': 9, 'hungry': 10, 'is': 11, 'learning': 12, 'tensorflow': 13, 'very': 14}\n" 88 | ] 89 | } 90 | ], 91 | "source": [ 92 | "# word dic\n", 93 | "word_list = []\n", 94 | "for elm in sentences:\n", 95 | " word_list += elm\n", 96 | "word_list = list(set(word_list))\n", 97 | "word_list.sort()\n", 98 | "word_list = [''] + word_list\n", 99 | "\n", 100 | "word_dic = {word : idx for idx, word in enumerate(word_list)}\n", 101 | "print(word_dic)" 102 | ] 103 | }, 104 | { 105 | "cell_type": "code", 106 | "execution_count": 4, 107 | "metadata": {}, 108 | "outputs": [ 109 | { 110 | "name": "stdout", 111 | "output_type": "stream", 112 | "text": [ 113 | "['', 'adjective', 'adverb', 'determiner', 'noun', 'preposition', 'pronoun', 'verb']\n" 114 | ] 115 | }, 116 | { 117 | "data": { 118 | "text/plain": [ 119 | "{'': 0,\n", 120 | " 'adjective': 1,\n", 121 | " 'adverb': 2,\n", 122 | " 'determiner': 3,\n", 123 | " 'noun': 4,\n", 124 | " 'preposition': 5,\n", 125 | " 'pronoun': 6,\n", 126 | " 'verb': 7}" 127 | ] 128 | }, 129 | "execution_count": 4, 130 | "metadata": {}, 131 | "output_type": "execute_result" 132 | } 133 | ], 134 | "source": [ 135 | "# pos dic\n", 136 | "pos_list = []\n", 137 | "for elm in pos:\n", 138 | " pos_list += elm\n", 139 | "pos_list = list(set(pos_list))\n", 140 | "pos_list.sort()\n", 141 | "pos_list = [''] + pos_list\n", 142 | "print(pos_list)\n", 143 | "\n", 144 | "pos_dic = {pos : idx for idx, pos in enumerate(pos_list)}\n", 145 | "pos_dic" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": 5, 151 | "metadata": {}, 152 | "outputs": [ 153 | { 154 | "data": { 155 | "text/plain": [ 156 | "{0: '',\n", 157 | " 1: 'adjective',\n", 158 | " 2: 'adverb',\n", 159 | " 3: 'determiner',\n", 160 | " 4: 'noun',\n", 161 | " 5: 'preposition',\n", 162 | " 6: 'pronoun',\n", 163 | " 7: 'verb'}" 164 | ] 165 | }, 166 | "execution_count": 5, 167 | "metadata": {}, 168 | "output_type": "execute_result" 169 | } 170 | ], 171 | "source": [ 172 | "pos_idx_to_dic = {elm[1] : elm[0] for elm in pos_dic.items()}\n", 173 | "pos_idx_to_dic" 174 | ] 175 | }, 176 | { 177 | "cell_type": "markdown", 178 | "metadata": {}, 179 | "source": [ 180 | "### Create pad_seq function" 181 | ] 182 | }, 183 | { 184 | "cell_type": "code", 185 | "execution_count": 6, 186 | "metadata": {}, 187 | "outputs": [], 188 | "source": [ 189 | "def pad_seq(sequences, max_len, dic):\n", 190 | " seq_len, seq_indices = [], []\n", 191 | " for seq in sequences:\n", 192 | " seq_len.append(len(seq))\n", 193 | " seq_idx = [dic.get(char) for char in seq]\n", 194 | " seq_idx += (max_len - len(seq_idx)) * [dic.get('')] # 0 is idx of meaningless token \"\"\n", 195 | " seq_indices.append(seq_idx)\n", 196 | " return seq_len, seq_indices" 197 | ] 198 | }, 199 | { 200 | "cell_type": "markdown", 201 | "metadata": {}, 202 | "source": [ 203 | "### Pre-process data" 204 | ] 205 | }, 206 | { 207 | "cell_type": "code", 208 | "execution_count": 7, 209 | "metadata": {}, 210 | "outputs": [ 211 | { 212 | "name": "stdout", 213 | "output_type": "stream", 214 | "text": [ 215 | "[3, 4, 7, 5] (4, 10)\n" 216 | ] 217 | } 218 | ], 219 | "source": [ 220 | "max_length = 10\n", 221 | "X_length, X_indices = pad_seq(sequences = sentences, max_len = max_length, dic = word_dic)\n", 222 | "print(X_length, np.shape(X_indices))" 223 | ] 224 | }, 225 | { 226 | "cell_type": "code", 227 | "execution_count": 8, 228 | "metadata": {}, 229 | "outputs": [ 230 | { 231 | "name": "stdout", 232 | "output_type": "stream", 233 | "text": [ 234 | "(4, 10)\n" 235 | ] 236 | } 237 | ], 238 | "source": [ 239 | "y = [elm + [''] * (max_length - len(elm)) for elm in pos]\n", 240 | "y = [list(map(lambda el : pos_dic.get(el), elm)) for elm in y]\n", 241 | "print(np.shape(y))" 242 | ] 243 | }, 244 | { 245 | "cell_type": "code", 246 | "execution_count": 9, 247 | "metadata": {}, 248 | "outputs": [ 249 | { 250 | "data": { 251 | "text/plain": [ 252 | "[[6, 7, 1, 0, 0, 0, 0, 0, 0, 0],\n", 253 | " [4, 7, 2, 1, 0, 0, 0, 0, 0, 0],\n", 254 | " [4, 7, 3, 4, 5, 1, 4, 0, 0, 0],\n", 255 | " [4, 7, 2, 1, 7, 0, 0, 0, 0, 0]]" 256 | ] 257 | }, 258 | "execution_count": 9, 259 | "metadata": {}, 260 | "output_type": "execute_result" 261 | } 262 | ], 263 | "source": [ 264 | "y" 265 | ] 266 | }, 267 | { 268 | "cell_type": "markdown", 269 | "metadata": {}, 270 | "source": [ 271 | "### Define SimPosGRU" 272 | ] 273 | }, 274 | { 275 | "cell_type": "code", 276 | "execution_count": 10, 277 | "metadata": {}, 278 | "outputs": [], 279 | "source": [ 280 | "class SimPosGRU:\n", 281 | " def __init__(self, X_length, X_indices, y, n_of_classes, hidden_dim, max_len, word_dic):\n", 282 | " \n", 283 | " # Data pipeline\n", 284 | " with tf.variable_scope('input_layer'):\n", 285 | " self._X_length = X_length\n", 286 | " self._X_indices = X_indices\n", 287 | " self._y = y\n", 288 | " \n", 289 | " one_hot = tf.eye(len(word_dic), dtype = tf.float32)\n", 290 | " self._one_hot = tf.get_variable(name='one_hot_embedding', initializer = one_hot,\n", 291 | " trainable = False) # embedding vector training 안할 것이기 때문\n", 292 | " self._X_batch = tf.nn.embedding_lookup(params = self._one_hot, ids = self._X_indices)\n", 293 | " \n", 294 | " # GRU cell (many to many)\n", 295 | " with tf.variable_scope('gru_cell'):\n", 296 | " gru_cell = tf.contrib.rnn.GRUCell(num_units = hidden_dim,\n", 297 | " activation = tf.nn.tanh)\n", 298 | " score_cell = tf.contrib.rnn.OutputProjectionWrapper(cell = gru_cell, output_size = n_of_classes)\n", 299 | " self._outputs, _ = tf.nn.dynamic_rnn(cell = score_cell, inputs = self._X_batch,\n", 300 | " sequence_length = self._X_length,\n", 301 | " dtype = tf.float32)\n", 302 | " \n", 303 | " with tf.variable_scope('seq2seq_loss'):\n", 304 | " masks = tf.sequence_mask(lengths = self._X_length, maxlen = max_len, dtype = tf.float32)\n", 305 | " self.seq2seq_loss = tf.contrib.seq2seq.sequence_loss(logits = self._outputs, targets = self._y,\n", 306 | " weights = masks)\n", 307 | " \n", 308 | " with tf.variable_scope('prediction'):\n", 309 | " self._prediction = tf.argmax(input = self._outputs,\n", 310 | " axis = 2, output_type = tf.int32)\n", 311 | " \n", 312 | " def predict(self, sess, X_length, X_indices):\n", 313 | " feed_prediction = {self._X_length : X_length, self._X_indices : X_indices}\n", 314 | " return sess.run(self._prediction, feed_dict = feed_prediction)" 315 | ] 316 | }, 317 | { 318 | "cell_type": "markdown", 319 | "metadata": {}, 320 | "source": [ 321 | "### Create a model of SimPosGRU" 322 | ] 323 | }, 324 | { 325 | "cell_type": "code", 326 | "execution_count": 11, 327 | "metadata": {}, 328 | "outputs": [ 329 | { 330 | "name": "stdout", 331 | "output_type": "stream", 332 | "text": [ 333 | "2\n" 334 | ] 335 | } 336 | ], 337 | "source": [ 338 | "# hyper-parameter#\n", 339 | "lr = .003\n", 340 | "epochs = 100\n", 341 | "batch_size = 2\n", 342 | "total_step = int(np.shape(X_indices)[0] / batch_size)\n", 343 | "print(total_step)" 344 | ] 345 | }, 346 | { 347 | "cell_type": "code", 348 | "execution_count": 12, 349 | "metadata": {}, 350 | "outputs": [ 351 | { 352 | "name": "stdout", 353 | "output_type": "stream", 354 | "text": [ 355 | "\n" 356 | ] 357 | } 358 | ], 359 | "source": [ 360 | "## create data pipeline with tf.data\n", 361 | "tr_dataset = tf.data.Dataset.from_tensor_slices((X_length, X_indices, y))\n", 362 | "tr_dataset = tr_dataset.shuffle(buffer_size = 20)\n", 363 | "tr_dataset = tr_dataset.batch(batch_size = batch_size)\n", 364 | "tr_iterator = tr_dataset.make_initializable_iterator()\n", 365 | "print(tr_dataset)" 366 | ] 367 | }, 368 | { 369 | "cell_type": "code", 370 | "execution_count": 13, 371 | "metadata": {}, 372 | "outputs": [], 373 | "source": [ 374 | "X_length_mb, X_indices_mb, y_mb = tr_iterator.get_next()" 375 | ] 376 | }, 377 | { 378 | "cell_type": "code", 379 | "execution_count": 14, 380 | "metadata": {}, 381 | "outputs": [], 382 | "source": [ 383 | "sim_pos_gru = SimPosGRU(X_length = X_length_mb, X_indices = X_indices_mb, y = y_mb,\n", 384 | " n_of_classes = 8, hidden_dim = 16, max_len = max_length, word_dic = word_dic)" 385 | ] 386 | }, 387 | { 388 | "cell_type": "markdown", 389 | "metadata": {}, 390 | "source": [ 391 | "### Creat training op and train model" 392 | ] 393 | }, 394 | { 395 | "cell_type": "code", 396 | "execution_count": 15, 397 | "metadata": {}, 398 | "outputs": [], 399 | "source": [ 400 | "## create training op\n", 401 | "opt = tf.train.AdamOptimizer(learning_rate = lr)\n", 402 | "training_op = opt.minimize(loss = sim_pos_gru.seq2seq_loss)" 403 | ] 404 | }, 405 | { 406 | "cell_type": "code", 407 | "execution_count": 16, 408 | "metadata": {}, 409 | "outputs": [ 410 | { 411 | "name": "stdout", 412 | "output_type": "stream", 413 | "text": [ 414 | "epoch : 10, tr_loss : 1.936\n", 415 | "epoch : 20, tr_loss : 1.764\n", 416 | "epoch : 30, tr_loss : 1.589\n", 417 | "epoch : 40, tr_loss : 1.373\n", 418 | "epoch : 50, tr_loss : 1.068\n", 419 | "epoch : 60, tr_loss : 0.714\n", 420 | "epoch : 70, tr_loss : 0.427\n", 421 | "epoch : 80, tr_loss : 0.272\n", 422 | "epoch : 90, tr_loss : 0.175\n", 423 | "epoch : 100, tr_loss : 0.137\n" 424 | ] 425 | } 426 | ], 427 | "source": [ 428 | "sess = tf.Session()\n", 429 | "sess.run(tf.global_variables_initializer())\n", 430 | "\n", 431 | "tr_loss_hist = []\n", 432 | "\n", 433 | "for epoch in range(epochs):\n", 434 | " avg_tr_loss = 0\n", 435 | " tr_step = 0\n", 436 | " \n", 437 | " sess.run(tr_iterator.initializer)\n", 438 | " try:\n", 439 | " while True:\n", 440 | " _, tr_loss = sess.run(fetches = [training_op, sim_pos_gru.seq2seq_loss])\n", 441 | " avg_tr_loss += tr_loss\n", 442 | " tr_step += 1\n", 443 | " \n", 444 | " except tf.errors.OutOfRangeError:\n", 445 | " pass\n", 446 | " \n", 447 | " avg_tr_loss /= tr_step\n", 448 | " tr_loss_hist.append(avg_tr_loss)\n", 449 | " if (epoch + 1) % 10 == 0:\n", 450 | " print('epoch : {:3}, tr_loss : {:.3f}'.format(epoch + 1, avg_tr_loss))" 451 | ] 452 | }, 453 | { 454 | "cell_type": "code", 455 | "execution_count": 17, 456 | "metadata": {}, 457 | "outputs": [ 458 | { 459 | "data": { 460 | "text/plain": [ 461 | "array([[6, 7, 1, 0, 0, 0, 0, 0, 0, 0],\n", 462 | " [4, 7, 2, 1, 0, 0, 0, 0, 0, 0],\n", 463 | " [4, 7, 3, 4, 5, 1, 4, 0, 0, 0],\n", 464 | " [4, 7, 2, 1, 7, 0, 0, 0, 0, 0]], dtype=int32)" 465 | ] 466 | }, 467 | "execution_count": 17, 468 | "metadata": {}, 469 | "output_type": "execute_result" 470 | } 471 | ], 472 | "source": [ 473 | "yhat = sim_pos_gru.predict(sess = sess, X_length = X_length, X_indices = X_indices)\n", 474 | "yhat" 475 | ] 476 | }, 477 | { 478 | "cell_type": "code", 479 | "execution_count": 18, 480 | "metadata": {}, 481 | "outputs": [ 482 | { 483 | "data": { 484 | "text/plain": [ 485 | "[[6, 7, 1, 0, 0, 0, 0, 0, 0, 0],\n", 486 | " [4, 7, 2, 1, 0, 0, 0, 0, 0, 0],\n", 487 | " [4, 7, 3, 4, 5, 1, 4, 0, 0, 0],\n", 488 | " [4, 7, 2, 1, 7, 0, 0, 0, 0, 0]]" 489 | ] 490 | }, 491 | "execution_count": 18, 492 | "metadata": {}, 493 | "output_type": "execute_result" 494 | } 495 | ], 496 | "source": [ 497 | "y" 498 | ] 499 | }, 500 | { 501 | "cell_type": "code", 502 | "execution_count": 19, 503 | "metadata": {}, 504 | "outputs": [ 505 | { 506 | "name": "stdout", 507 | "output_type": "stream", 508 | "text": [ 509 | "['pronoun', 'verb', 'adjective', '', '', '', '', '', '', '']\n", 510 | "['noun', 'verb', 'adverb', 'adjective', '', '', '', '', '', '']\n", 511 | "['noun', 'verb', 'determiner', 'noun', 'preposition', 'adjective', 'noun', '', '', '']\n", 512 | "['noun', 'verb', 'adverb', 'adjective', 'verb', '', '', '', '', '']\n" 513 | ] 514 | } 515 | ], 516 | "source": [ 517 | "yhat = [list(map(lambda elm : pos_idx_to_dic.get(elm), row)) for row in yhat]\n", 518 | "for elm in yhat:\n", 519 | " print(elm)" 520 | ] 521 | } 522 | ], 523 | "metadata": { 524 | "kernelspec": { 525 | "display_name": "Python 3", 526 | "language": "python", 527 | "name": "python3" 528 | }, 529 | "language_info": { 530 | "codemirror_mode": { 531 | "name": "ipython", 532 | "version": 3 533 | }, 534 | "file_extension": ".py", 535 | "mimetype": "text/x-python", 536 | "name": "python", 537 | "nbconvert_exporter": "python", 538 | "pygments_lexer": "ipython3", 539 | "version": "3.6.5" 540 | } 541 | }, 542 | "nbformat": 4, 543 | "nbformat_minor": 2 544 | } 545 | -------------------------------------------------------------------------------- /Lec11_Recurrent Neural Networks/Lec11_Many to Many Classification by LSTM.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# CS 20 : TensorFlow for Deep Learning Research\n", 8 | "## Lecture 11 : Recurrent Neural Networks\n", 9 | "Simple example for Many to Many Classification (Simple pos tagger) by LSTM. \n", 10 | "\n", 11 | "### Many to Many Classification by LSTM\n", 12 | "- Creating the **data pipeline** with `tf.data`\n", 13 | "- Preprocessing word sequences (variable input sequence length) using `padding technique` by `user function (pad_seq)`\n", 14 | "- Using `tf.nn.embedding_lookup` for getting vector of tokens (eg. word, character)\n", 15 | "- Training **many to many classification** with `tf.contrib.seq2seq.sequence_loss`\n", 16 | "- Masking unvalid token with `tf.sequence_mask`\n", 17 | "- Creating the model as **Class**\n", 18 | "- Reference\n", 19 | " - https://github.com/aisolab/sample_code_of_Deep_learning_Basics/blob/master/DLEL/DLEL_12_2_RNN_(toy_example).ipynb" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": {}, 25 | "source": [ 26 | "### Setup" 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": 1, 32 | "metadata": {}, 33 | "outputs": [ 34 | { 35 | "name": "stdout", 36 | "output_type": "stream", 37 | "text": [ 38 | "1.8.0\n" 39 | ] 40 | } 41 | ], 42 | "source": [ 43 | "import os, sys\n", 44 | "import numpy as np\n", 45 | "import pandas as pd\n", 46 | "import matplotlib.pyplot as plt\n", 47 | "import tensorflow as tf\n", 48 | "import string\n", 49 | "%matplotlib inline\n", 50 | "\n", 51 | "slim = tf.contrib.slim\n", 52 | "print(tf.__version__)" 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "metadata": {}, 58 | "source": [ 59 | "### Prepare example data " 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 2, 65 | "metadata": {}, 66 | "outputs": [], 67 | "source": [ 68 | "sentences = [['I', 'feel', 'hungry'],\n", 69 | " ['tensorflow', 'is', 'very', 'difficult'],\n", 70 | " ['tensorflow', 'is', 'a', 'framework', 'for', 'deep', 'learning'],\n", 71 | " ['tensorflow', 'is', 'very', 'fast', 'changing']]\n", 72 | "pos = [['pronoun', 'verb', 'adjective'],\n", 73 | " ['noun', 'verb', 'adverb', 'adjective'],\n", 74 | " ['noun', 'verb', 'determiner', 'noun', 'preposition', 'adjective', 'noun'],\n", 75 | " ['noun', 'verb', 'adverb', 'adjective', 'verb']]" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": 3, 81 | "metadata": {}, 82 | "outputs": [ 83 | { 84 | "name": "stdout", 85 | "output_type": "stream", 86 | "text": [ 87 | "{'': 0, 'I': 1, 'a': 2, 'changing': 3, 'deep': 4, 'difficult': 5, 'fast': 6, 'feel': 7, 'for': 8, 'framework': 9, 'hungry': 10, 'is': 11, 'learning': 12, 'tensorflow': 13, 'very': 14}\n" 88 | ] 89 | } 90 | ], 91 | "source": [ 92 | "# word dic\n", 93 | "word_list = []\n", 94 | "for elm in sentences:\n", 95 | " word_list += elm\n", 96 | "word_list = list(set(word_list))\n", 97 | "word_list.sort()\n", 98 | "word_list = [''] + word_list\n", 99 | "\n", 100 | "word_dic = {word : idx for idx, word in enumerate(word_list)}\n", 101 | "print(word_dic)" 102 | ] 103 | }, 104 | { 105 | "cell_type": "code", 106 | "execution_count": 4, 107 | "metadata": {}, 108 | "outputs": [ 109 | { 110 | "name": "stdout", 111 | "output_type": "stream", 112 | "text": [ 113 | "['', 'adjective', 'adverb', 'determiner', 'noun', 'preposition', 'pronoun', 'verb']\n" 114 | ] 115 | }, 116 | { 117 | "data": { 118 | "text/plain": [ 119 | "{'': 0,\n", 120 | " 'adjective': 1,\n", 121 | " 'adverb': 2,\n", 122 | " 'determiner': 3,\n", 123 | " 'noun': 4,\n", 124 | " 'preposition': 5,\n", 125 | " 'pronoun': 6,\n", 126 | " 'verb': 7}" 127 | ] 128 | }, 129 | "execution_count": 4, 130 | "metadata": {}, 131 | "output_type": "execute_result" 132 | } 133 | ], 134 | "source": [ 135 | "# pos dic\n", 136 | "pos_list = []\n", 137 | "for elm in pos:\n", 138 | " pos_list += elm\n", 139 | "pos_list = list(set(pos_list))\n", 140 | "pos_list.sort()\n", 141 | "pos_list = [''] + pos_list\n", 142 | "print(pos_list)\n", 143 | "\n", 144 | "pos_dic = {pos : idx for idx, pos in enumerate(pos_list)}\n", 145 | "pos_dic" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": 5, 151 | "metadata": {}, 152 | "outputs": [ 153 | { 154 | "data": { 155 | "text/plain": [ 156 | "{0: '',\n", 157 | " 1: 'adjective',\n", 158 | " 2: 'adverb',\n", 159 | " 3: 'determiner',\n", 160 | " 4: 'noun',\n", 161 | " 5: 'preposition',\n", 162 | " 6: 'pronoun',\n", 163 | " 7: 'verb'}" 164 | ] 165 | }, 166 | "execution_count": 5, 167 | "metadata": {}, 168 | "output_type": "execute_result" 169 | } 170 | ], 171 | "source": [ 172 | "pos_idx_to_dic = {elm[1] : elm[0] for elm in pos_dic.items()}\n", 173 | "pos_idx_to_dic" 174 | ] 175 | }, 176 | { 177 | "cell_type": "markdown", 178 | "metadata": {}, 179 | "source": [ 180 | "### Create pad_seq function" 181 | ] 182 | }, 183 | { 184 | "cell_type": "code", 185 | "execution_count": 6, 186 | "metadata": {}, 187 | "outputs": [], 188 | "source": [ 189 | "def pad_seq(sequences, max_len, dic):\n", 190 | " seq_len, seq_indices = [], []\n", 191 | " for seq in sequences:\n", 192 | " seq_len.append(len(seq))\n", 193 | " seq_idx = [dic.get(char) for char in seq]\n", 194 | " seq_idx += (max_len - len(seq_idx)) * [dic.get('')] # 0 is idx of meaningless token \"\"\n", 195 | " seq_indices.append(seq_idx)\n", 196 | " return seq_len, seq_indices" 197 | ] 198 | }, 199 | { 200 | "cell_type": "markdown", 201 | "metadata": {}, 202 | "source": [ 203 | "### Pre-process data" 204 | ] 205 | }, 206 | { 207 | "cell_type": "code", 208 | "execution_count": 7, 209 | "metadata": {}, 210 | "outputs": [ 211 | { 212 | "name": "stdout", 213 | "output_type": "stream", 214 | "text": [ 215 | "[3, 4, 7, 5] (4, 10)\n" 216 | ] 217 | } 218 | ], 219 | "source": [ 220 | "max_length = 10\n", 221 | "X_length, X_indices = pad_seq(sequences = sentences, max_len = max_length, dic = word_dic)\n", 222 | "print(X_length, np.shape(X_indices))" 223 | ] 224 | }, 225 | { 226 | "cell_type": "code", 227 | "execution_count": 8, 228 | "metadata": {}, 229 | "outputs": [ 230 | { 231 | "name": "stdout", 232 | "output_type": "stream", 233 | "text": [ 234 | "(4, 10)\n" 235 | ] 236 | } 237 | ], 238 | "source": [ 239 | "y = [elm + [''] * (max_length - len(elm)) for elm in pos]\n", 240 | "y = [list(map(lambda el : pos_dic.get(el), elm)) for elm in y]\n", 241 | "print(np.shape(y))" 242 | ] 243 | }, 244 | { 245 | "cell_type": "code", 246 | "execution_count": 9, 247 | "metadata": {}, 248 | "outputs": [ 249 | { 250 | "data": { 251 | "text/plain": [ 252 | "[[6, 7, 1, 0, 0, 0, 0, 0, 0, 0],\n", 253 | " [4, 7, 2, 1, 0, 0, 0, 0, 0, 0],\n", 254 | " [4, 7, 3, 4, 5, 1, 4, 0, 0, 0],\n", 255 | " [4, 7, 2, 1, 7, 0, 0, 0, 0, 0]]" 256 | ] 257 | }, 258 | "execution_count": 9, 259 | "metadata": {}, 260 | "output_type": "execute_result" 261 | } 262 | ], 263 | "source": [ 264 | "y" 265 | ] 266 | }, 267 | { 268 | "cell_type": "markdown", 269 | "metadata": {}, 270 | "source": [ 271 | "### Define SimPosLSTM" 272 | ] 273 | }, 274 | { 275 | "cell_type": "code", 276 | "execution_count": 10, 277 | "metadata": {}, 278 | "outputs": [], 279 | "source": [ 280 | "class SimPosLSTM:\n", 281 | " def __init__(self, X_length, X_indices, y, n_of_classes, hidden_dim, max_len, word_dic):\n", 282 | " \n", 283 | " # Data pipeline\n", 284 | " with tf.variable_scope('input_layer'):\n", 285 | " self._X_length = X_length\n", 286 | " self._X_indices = X_indices\n", 287 | " self._y = y\n", 288 | " \n", 289 | " one_hot = tf.eye(len(word_dic), dtype = tf.float32)\n", 290 | " self._one_hot = tf.get_variable(name='one_hot_embedding', initializer = one_hot,\n", 291 | " trainable = False) # embedding vector training 안할 것이기 때문\n", 292 | " self._X_batch = tf.nn.embedding_lookup(params = self._one_hot, ids = self._X_indices)\n", 293 | " \n", 294 | " # LSTM cell (many to many)\n", 295 | " with tf.variable_scope('lstm_cell'):\n", 296 | " lstm_cell = tf.contrib.rnn.BasicLSTMCell(num_units = hidden_dim,\n", 297 | " activation = tf.nn.tanh)\n", 298 | " score_cell = tf.contrib.rnn.OutputProjectionWrapper(cell = lstm_cell, output_size = n_of_classes)\n", 299 | " self._outputs, _ = tf.nn.dynamic_rnn(cell = score_cell, inputs = self._X_batch,\n", 300 | " sequence_length = self._X_length,\n", 301 | " dtype = tf.float32)\n", 302 | " \n", 303 | " with tf.variable_scope('seq2seq_loss'):\n", 304 | " masks = tf.sequence_mask(lengths = self._X_length, maxlen = max_len, dtype = tf.float32)\n", 305 | " self.seq2seq_loss = tf.contrib.seq2seq.sequence_loss(logits = self._outputs, targets = self._y,\n", 306 | " weights = masks)\n", 307 | " \n", 308 | " with tf.variable_scope('prediction'):\n", 309 | " self._prediction = tf.argmax(input = self._outputs,\n", 310 | " axis = 2, output_type = tf.int32)\n", 311 | " \n", 312 | " def predict(self, sess, X_length, X_indices):\n", 313 | " feed_prediction = {self._X_length : X_length, self._X_indices : X_indices}\n", 314 | " return sess.run(self._prediction, feed_dict = feed_prediction)" 315 | ] 316 | }, 317 | { 318 | "cell_type": "markdown", 319 | "metadata": {}, 320 | "source": [ 321 | "### Create a model of SimPosLSTM" 322 | ] 323 | }, 324 | { 325 | "cell_type": "code", 326 | "execution_count": 11, 327 | "metadata": {}, 328 | "outputs": [ 329 | { 330 | "name": "stdout", 331 | "output_type": "stream", 332 | "text": [ 333 | "2\n" 334 | ] 335 | } 336 | ], 337 | "source": [ 338 | "# hyper-parameter#\n", 339 | "lr = .003\n", 340 | "epochs = 100\n", 341 | "batch_size = 2\n", 342 | "total_step = int(np.shape(X_indices)[0] / batch_size)\n", 343 | "print(total_step)" 344 | ] 345 | }, 346 | { 347 | "cell_type": "code", 348 | "execution_count": 12, 349 | "metadata": {}, 350 | "outputs": [ 351 | { 352 | "name": "stdout", 353 | "output_type": "stream", 354 | "text": [ 355 | "\n" 356 | ] 357 | } 358 | ], 359 | "source": [ 360 | "## create data pipeline with tf.data\n", 361 | "tr_dataset = tf.data.Dataset.from_tensor_slices((X_length, X_indices, y))\n", 362 | "tr_dataset = tr_dataset.shuffle(buffer_size = 20)\n", 363 | "tr_dataset = tr_dataset.batch(batch_size = batch_size)\n", 364 | "tr_iterator = tr_dataset.make_initializable_iterator()\n", 365 | "print(tr_dataset)" 366 | ] 367 | }, 368 | { 369 | "cell_type": "code", 370 | "execution_count": 13, 371 | "metadata": {}, 372 | "outputs": [], 373 | "source": [ 374 | "X_length_mb, X_indices_mb, y_mb = tr_iterator.get_next()" 375 | ] 376 | }, 377 | { 378 | "cell_type": "code", 379 | "execution_count": 14, 380 | "metadata": {}, 381 | "outputs": [], 382 | "source": [ 383 | "sim_pos_lstm = SimPosLSTM(X_length = X_length_mb, X_indices = X_indices_mb, y = y_mb,\n", 384 | " n_of_classes = 8, hidden_dim = 16, max_len = max_length, word_dic = word_dic)" 385 | ] 386 | }, 387 | { 388 | "cell_type": "markdown", 389 | "metadata": {}, 390 | "source": [ 391 | "### Creat training op and train model" 392 | ] 393 | }, 394 | { 395 | "cell_type": "code", 396 | "execution_count": 15, 397 | "metadata": {}, 398 | "outputs": [], 399 | "source": [ 400 | "## create training op\n", 401 | "opt = tf.train.AdamOptimizer(learning_rate = lr)\n", 402 | "training_op = opt.minimize(loss = sim_pos_lstm.seq2seq_loss)" 403 | ] 404 | }, 405 | { 406 | "cell_type": "code", 407 | "execution_count": 16, 408 | "metadata": {}, 409 | "outputs": [ 410 | { 411 | "name": "stdout", 412 | "output_type": "stream", 413 | "text": [ 414 | "epoch : 10, tr_loss : 1.943\n", 415 | "epoch : 20, tr_loss : 1.743\n", 416 | "epoch : 30, tr_loss : 1.473\n", 417 | "epoch : 40, tr_loss : 1.163\n", 418 | "epoch : 50, tr_loss : 0.826\n", 419 | "epoch : 60, tr_loss : 0.569\n", 420 | "epoch : 70, tr_loss : 0.413\n", 421 | "epoch : 80, tr_loss : 0.300\n", 422 | "epoch : 90, tr_loss : 0.212\n", 423 | "epoch : 100, tr_loss : 0.160\n" 424 | ] 425 | } 426 | ], 427 | "source": [ 428 | "sess = tf.Session()\n", 429 | "sess.run(tf.global_variables_initializer())\n", 430 | "\n", 431 | "tr_loss_hist = []\n", 432 | "\n", 433 | "for epoch in range(epochs):\n", 434 | " avg_tr_loss = 0\n", 435 | " tr_step = 0\n", 436 | " \n", 437 | " sess.run(tr_iterator.initializer)\n", 438 | " try:\n", 439 | " while True:\n", 440 | " _, tr_loss = sess.run(fetches = [training_op, sim_pos_lstm.seq2seq_loss])\n", 441 | " avg_tr_loss += tr_loss\n", 442 | " tr_step += 1\n", 443 | " \n", 444 | " except tf.errors.OutOfRangeError:\n", 445 | " pass\n", 446 | " \n", 447 | " avg_tr_loss /= tr_step\n", 448 | " tr_loss_hist.append(avg_tr_loss)\n", 449 | " if (epoch + 1) % 10 == 0:\n", 450 | " print('epoch : {:3}, tr_loss : {:.3f}'.format(epoch + 1, avg_tr_loss))" 451 | ] 452 | }, 453 | { 454 | "cell_type": "code", 455 | "execution_count": 17, 456 | "metadata": {}, 457 | "outputs": [ 458 | { 459 | "data": { 460 | "text/plain": [ 461 | "array([[6, 7, 1, 0, 0, 0, 0, 0, 0, 0],\n", 462 | " [4, 7, 2, 1, 0, 0, 0, 0, 0, 0],\n", 463 | " [4, 7, 3, 4, 5, 1, 4, 0, 0, 0],\n", 464 | " [4, 7, 2, 1, 7, 0, 0, 0, 0, 0]], dtype=int32)" 465 | ] 466 | }, 467 | "execution_count": 17, 468 | "metadata": {}, 469 | "output_type": "execute_result" 470 | } 471 | ], 472 | "source": [ 473 | "yhat = sim_pos_lstm.predict(sess = sess, X_length = X_length, X_indices = X_indices)\n", 474 | "yhat" 475 | ] 476 | }, 477 | { 478 | "cell_type": "code", 479 | "execution_count": 18, 480 | "metadata": {}, 481 | "outputs": [ 482 | { 483 | "data": { 484 | "text/plain": [ 485 | "[[6, 7, 1, 0, 0, 0, 0, 0, 0, 0],\n", 486 | " [4, 7, 2, 1, 0, 0, 0, 0, 0, 0],\n", 487 | " [4, 7, 3, 4, 5, 1, 4, 0, 0, 0],\n", 488 | " [4, 7, 2, 1, 7, 0, 0, 0, 0, 0]]" 489 | ] 490 | }, 491 | "execution_count": 18, 492 | "metadata": {}, 493 | "output_type": "execute_result" 494 | } 495 | ], 496 | "source": [ 497 | "y" 498 | ] 499 | }, 500 | { 501 | "cell_type": "code", 502 | "execution_count": 19, 503 | "metadata": {}, 504 | "outputs": [ 505 | { 506 | "name": "stdout", 507 | "output_type": "stream", 508 | "text": [ 509 | "['pronoun', 'verb', 'adjective', '', '', '', '', '', '', '']\n", 510 | "['noun', 'verb', 'adverb', 'adjective', '', '', '', '', '', '']\n", 511 | "['noun', 'verb', 'determiner', 'noun', 'preposition', 'adjective', 'noun', '', '', '']\n", 512 | "['noun', 'verb', 'adverb', 'adjective', 'verb', '', '', '', '', '']\n" 513 | ] 514 | } 515 | ], 516 | "source": [ 517 | "yhat = [list(map(lambda elm : pos_idx_to_dic.get(elm), row)) for row in yhat]\n", 518 | "for elm in yhat:\n", 519 | " print(elm)" 520 | ] 521 | } 522 | ], 523 | "metadata": { 524 | "kernelspec": { 525 | "display_name": "Python 3", 526 | "language": "python", 527 | "name": "python3" 528 | }, 529 | "language_info": { 530 | "codemirror_mode": { 531 | "name": "ipython", 532 | "version": 3 533 | }, 534 | "file_extension": ".py", 535 | "mimetype": "text/x-python", 536 | "name": "python", 537 | "nbconvert_exporter": "python", 538 | "pygments_lexer": "ipython3", 539 | "version": "3.6.5" 540 | } 541 | }, 542 | "nbformat": 4, 543 | "nbformat_minor": 2 544 | } 545 | -------------------------------------------------------------------------------- /Lec11_Recurrent Neural Networks/Lec11_Many to Many Classification by RNN.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# CS 20 : TensorFlow for Deep Learning Research\n", 8 | "## Lecture 11 : Recurrent Neural Networks\n", 9 | "Simple example for Many to Many Classification (Simple pos tagger) by Recurrent Neural Networks. \n", 10 | "\n", 11 | "### Many to Many Classification by RNN\n", 12 | "- Creating the **data pipeline** with `tf.data`\n", 13 | "- Preprocessing word sequences (variable input sequence length) using `padding technique` by `user function (pad_seq)`\n", 14 | "- Using `tf.nn.embedding_lookup` for getting vector of tokens (eg. word, character)\n", 15 | "- Training **many to many classification** with `tf.contrib.seq2seq.sequence_loss`\n", 16 | "- Masking unvalid token with `tf.sequence_mask`\n", 17 | "- Creating the model as **Class**\n", 18 | "- Reference\n", 19 | " - https://github.com/aisolab/sample_code_of_Deep_learning_Basics/blob/master/DLEL/DLEL_12_2_RNN_(toy_example).ipynb" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": {}, 25 | "source": [ 26 | "### Setup" 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": 1, 32 | "metadata": {}, 33 | "outputs": [ 34 | { 35 | "name": "stdout", 36 | "output_type": "stream", 37 | "text": [ 38 | "1.8.0\n" 39 | ] 40 | } 41 | ], 42 | "source": [ 43 | "import os, sys\n", 44 | "import numpy as np\n", 45 | "import pandas as pd\n", 46 | "import matplotlib.pyplot as plt\n", 47 | "import tensorflow as tf\n", 48 | "import string\n", 49 | "%matplotlib inline\n", 50 | "\n", 51 | "slim = tf.contrib.slim\n", 52 | "print(tf.__version__)" 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "metadata": {}, 58 | "source": [ 59 | "### Prepare example data " 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 2, 65 | "metadata": {}, 66 | "outputs": [], 67 | "source": [ 68 | "sentences = [['I', 'feel', 'hungry'],\n", 69 | " ['tensorflow', 'is', 'very', 'difficult'],\n", 70 | " ['tensorflow', 'is', 'a', 'framework', 'for', 'deep', 'learning'],\n", 71 | " ['tensorflow', 'is', 'very', 'fast', 'changing']]\n", 72 | "pos = [['pronoun', 'verb', 'adjective'],\n", 73 | " ['noun', 'verb', 'adverb', 'adjective'],\n", 74 | " ['noun', 'verb', 'determiner', 'noun', 'preposition', 'adjective', 'noun'],\n", 75 | " ['noun', 'verb', 'adverb', 'adjective', 'verb']]" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": 3, 81 | "metadata": {}, 82 | "outputs": [ 83 | { 84 | "name": "stdout", 85 | "output_type": "stream", 86 | "text": [ 87 | "{'': 0, 'I': 1, 'a': 2, 'changing': 3, 'deep': 4, 'difficult': 5, 'fast': 6, 'feel': 7, 'for': 8, 'framework': 9, 'hungry': 10, 'is': 11, 'learning': 12, 'tensorflow': 13, 'very': 14}\n" 88 | ] 89 | } 90 | ], 91 | "source": [ 92 | "# word dic\n", 93 | "word_list = []\n", 94 | "for elm in sentences:\n", 95 | " word_list += elm\n", 96 | "word_list = list(set(word_list))\n", 97 | "word_list.sort()\n", 98 | "word_list = [''] + word_list\n", 99 | "\n", 100 | "word_dic = {word : idx for idx, word in enumerate(word_list)}\n", 101 | "print(word_dic)" 102 | ] 103 | }, 104 | { 105 | "cell_type": "code", 106 | "execution_count": 4, 107 | "metadata": {}, 108 | "outputs": [ 109 | { 110 | "name": "stdout", 111 | "output_type": "stream", 112 | "text": [ 113 | "['', 'adjective', 'adverb', 'determiner', 'noun', 'preposition', 'pronoun', 'verb']\n" 114 | ] 115 | }, 116 | { 117 | "data": { 118 | "text/plain": [ 119 | "{'': 0,\n", 120 | " 'adjective': 1,\n", 121 | " 'adverb': 2,\n", 122 | " 'determiner': 3,\n", 123 | " 'noun': 4,\n", 124 | " 'preposition': 5,\n", 125 | " 'pronoun': 6,\n", 126 | " 'verb': 7}" 127 | ] 128 | }, 129 | "execution_count": 4, 130 | "metadata": {}, 131 | "output_type": "execute_result" 132 | } 133 | ], 134 | "source": [ 135 | "# pos dic\n", 136 | "pos_list = []\n", 137 | "for elm in pos:\n", 138 | " pos_list += elm\n", 139 | "pos_list = list(set(pos_list))\n", 140 | "pos_list.sort()\n", 141 | "pos_list = [''] + pos_list\n", 142 | "print(pos_list)\n", 143 | "\n", 144 | "pos_dic = {pos : idx for idx, pos in enumerate(pos_list)}\n", 145 | "pos_dic" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": 5, 151 | "metadata": {}, 152 | "outputs": [ 153 | { 154 | "data": { 155 | "text/plain": [ 156 | "{0: '',\n", 157 | " 1: 'adjective',\n", 158 | " 2: 'adverb',\n", 159 | " 3: 'determiner',\n", 160 | " 4: 'noun',\n", 161 | " 5: 'preposition',\n", 162 | " 6: 'pronoun',\n", 163 | " 7: 'verb'}" 164 | ] 165 | }, 166 | "execution_count": 5, 167 | "metadata": {}, 168 | "output_type": "execute_result" 169 | } 170 | ], 171 | "source": [ 172 | "pos_idx_to_dic = {elm[1] : elm[0] for elm in pos_dic.items()}\n", 173 | "pos_idx_to_dic" 174 | ] 175 | }, 176 | { 177 | "cell_type": "markdown", 178 | "metadata": {}, 179 | "source": [ 180 | "### Create pad_seq function" 181 | ] 182 | }, 183 | { 184 | "cell_type": "code", 185 | "execution_count": 6, 186 | "metadata": {}, 187 | "outputs": [], 188 | "source": [ 189 | "def pad_seq(sequences, max_len, dic):\n", 190 | " seq_len, seq_indices = [], []\n", 191 | " for seq in sequences:\n", 192 | " seq_len.append(len(seq))\n", 193 | " seq_idx = [dic.get(char) for char in seq]\n", 194 | " seq_idx += (max_len - len(seq_idx)) * [dic.get('')] # 0 is idx of meaningless token \"\"\n", 195 | " seq_indices.append(seq_idx)\n", 196 | " return seq_len, seq_indices" 197 | ] 198 | }, 199 | { 200 | "cell_type": "markdown", 201 | "metadata": {}, 202 | "source": [ 203 | "### Pre-process data" 204 | ] 205 | }, 206 | { 207 | "cell_type": "code", 208 | "execution_count": 7, 209 | "metadata": {}, 210 | "outputs": [ 211 | { 212 | "name": "stdout", 213 | "output_type": "stream", 214 | "text": [ 215 | "[3, 4, 7, 5] (4, 10)\n" 216 | ] 217 | } 218 | ], 219 | "source": [ 220 | "max_length = 10\n", 221 | "X_length, X_indices = pad_seq(sequences = sentences, max_len = max_length, dic = word_dic)\n", 222 | "print(X_length, np.shape(X_indices))" 223 | ] 224 | }, 225 | { 226 | "cell_type": "code", 227 | "execution_count": 8, 228 | "metadata": {}, 229 | "outputs": [ 230 | { 231 | "name": "stdout", 232 | "output_type": "stream", 233 | "text": [ 234 | "(4, 10)\n" 235 | ] 236 | } 237 | ], 238 | "source": [ 239 | "y = [elm + [''] * (max_length - len(elm)) for elm in pos]\n", 240 | "y = [list(map(lambda el : pos_dic.get(el), elm)) for elm in y]\n", 241 | "print(np.shape(y))" 242 | ] 243 | }, 244 | { 245 | "cell_type": "code", 246 | "execution_count": 9, 247 | "metadata": {}, 248 | "outputs": [ 249 | { 250 | "data": { 251 | "text/plain": [ 252 | "[[6, 7, 1, 0, 0, 0, 0, 0, 0, 0],\n", 253 | " [4, 7, 2, 1, 0, 0, 0, 0, 0, 0],\n", 254 | " [4, 7, 3, 4, 5, 1, 4, 0, 0, 0],\n", 255 | " [4, 7, 2, 1, 7, 0, 0, 0, 0, 0]]" 256 | ] 257 | }, 258 | "execution_count": 9, 259 | "metadata": {}, 260 | "output_type": "execute_result" 261 | } 262 | ], 263 | "source": [ 264 | "y" 265 | ] 266 | }, 267 | { 268 | "cell_type": "markdown", 269 | "metadata": {}, 270 | "source": [ 271 | "### Define SimPosRNN" 272 | ] 273 | }, 274 | { 275 | "cell_type": "code", 276 | "execution_count": 10, 277 | "metadata": {}, 278 | "outputs": [], 279 | "source": [ 280 | "class SimPosRNN:\n", 281 | " def __init__(self, X_length, X_indices, y, n_of_classes, hidden_dim, max_len, word_dic):\n", 282 | " \n", 283 | " # Data pipeline\n", 284 | " with tf.variable_scope('input_layer'):\n", 285 | " self._X_length = X_length\n", 286 | " self._X_indices = X_indices\n", 287 | " self._y = y\n", 288 | " \n", 289 | " one_hot = tf.eye(len(word_dic), dtype = tf.float32)\n", 290 | " self._one_hot = tf.get_variable(name='one_hot_embedding', initializer = one_hot,\n", 291 | " trainable = False) # embedding vector training 안할 것이기 때문\n", 292 | " self._X_batch = tf.nn.embedding_lookup(params = self._one_hot, ids = self._X_indices)\n", 293 | " \n", 294 | " # RNN cell (many to many)\n", 295 | " with tf.variable_scope('rnn_cell'):\n", 296 | " rnn_cell = tf.contrib.rnn.BasicRNNCell(num_units = hidden_dim,\n", 297 | " activation = tf.nn.tanh)\n", 298 | " score_cell = tf.contrib.rnn.OutputProjectionWrapper(cell = rnn_cell, output_size = n_of_classes)\n", 299 | " self._outputs, _ = tf.nn.dynamic_rnn(cell = score_cell, inputs = self._X_batch,\n", 300 | " sequence_length = self._X_length,\n", 301 | " dtype = tf.float32)\n", 302 | " \n", 303 | " with tf.variable_scope('seq2seq_loss'):\n", 304 | " masks = tf.sequence_mask(lengths = self._X_length, maxlen = max_len, dtype = tf.float32)\n", 305 | " self.seq2seq_loss = tf.contrib.seq2seq.sequence_loss(logits = self._outputs, targets = self._y,\n", 306 | " weights = masks)\n", 307 | " \n", 308 | " with tf.variable_scope('prediction'):\n", 309 | " self._prediction = tf.argmax(input = self._outputs,\n", 310 | " axis = 2, output_type = tf.int32)\n", 311 | " \n", 312 | " def predict(self, sess, X_length, X_indices):\n", 313 | " feed_prediction = {self._X_length : X_length, self._X_indices : X_indices}\n", 314 | " return sess.run(self._prediction, feed_dict = feed_prediction)" 315 | ] 316 | }, 317 | { 318 | "cell_type": "markdown", 319 | "metadata": {}, 320 | "source": [ 321 | "### Create a model of SimPosRNN" 322 | ] 323 | }, 324 | { 325 | "cell_type": "code", 326 | "execution_count": 11, 327 | "metadata": {}, 328 | "outputs": [ 329 | { 330 | "name": "stdout", 331 | "output_type": "stream", 332 | "text": [ 333 | "2\n" 334 | ] 335 | } 336 | ], 337 | "source": [ 338 | "# hyper-parameter#\n", 339 | "lr = .003\n", 340 | "epochs = 100\n", 341 | "batch_size = 2\n", 342 | "total_step = int(np.shape(X_indices)[0] / batch_size)\n", 343 | "print(total_step)" 344 | ] 345 | }, 346 | { 347 | "cell_type": "code", 348 | "execution_count": 12, 349 | "metadata": {}, 350 | "outputs": [ 351 | { 352 | "name": "stdout", 353 | "output_type": "stream", 354 | "text": [ 355 | "\n" 356 | ] 357 | } 358 | ], 359 | "source": [ 360 | "## create data pipeline with tf.data\n", 361 | "tr_dataset = tf.data.Dataset.from_tensor_slices((X_length, X_indices, y))\n", 362 | "tr_dataset = tr_dataset.shuffle(buffer_size = 20)\n", 363 | "tr_dataset = tr_dataset.batch(batch_size = batch_size)\n", 364 | "tr_iterator = tr_dataset.make_initializable_iterator()\n", 365 | "print(tr_dataset)" 366 | ] 367 | }, 368 | { 369 | "cell_type": "code", 370 | "execution_count": 13, 371 | "metadata": {}, 372 | "outputs": [], 373 | "source": [ 374 | "X_length_mb, X_indices_mb, y_mb = tr_iterator.get_next()" 375 | ] 376 | }, 377 | { 378 | "cell_type": "code", 379 | "execution_count": 14, 380 | "metadata": {}, 381 | "outputs": [], 382 | "source": [ 383 | "sim_pos_rnn = SimPosRNN(X_length = X_length_mb, X_indices = X_indices_mb, y = y_mb,\n", 384 | " n_of_classes = 8, hidden_dim = 16, max_len = max_length, word_dic = word_dic)" 385 | ] 386 | }, 387 | { 388 | "cell_type": "markdown", 389 | "metadata": {}, 390 | "source": [ 391 | "### Creat training op and train model" 392 | ] 393 | }, 394 | { 395 | "cell_type": "code", 396 | "execution_count": 15, 397 | "metadata": {}, 398 | "outputs": [], 399 | "source": [ 400 | "## create training op\n", 401 | "opt = tf.train.AdamOptimizer(learning_rate = lr)\n", 402 | "training_op = opt.minimize(loss = sim_pos_rnn.seq2seq_loss)" 403 | ] 404 | }, 405 | { 406 | "cell_type": "code", 407 | "execution_count": 16, 408 | "metadata": {}, 409 | "outputs": [ 410 | { 411 | "name": "stdout", 412 | "output_type": "stream", 413 | "text": [ 414 | "epoch : 10, tr_loss : 1.716\n", 415 | "epoch : 20, tr_loss : 1.214\n", 416 | "epoch : 30, tr_loss : 0.785\n", 417 | "epoch : 40, tr_loss : 0.502\n", 418 | "epoch : 50, tr_loss : 0.347\n", 419 | "epoch : 60, tr_loss : 0.246\n", 420 | "epoch : 70, tr_loss : 0.183\n", 421 | "epoch : 80, tr_loss : 0.142\n", 422 | "epoch : 90, tr_loss : 0.109\n", 423 | "epoch : 100, tr_loss : 0.087\n" 424 | ] 425 | } 426 | ], 427 | "source": [ 428 | "sess = tf.Session()\n", 429 | "sess.run(tf.global_variables_initializer())\n", 430 | "\n", 431 | "tr_loss_hist = []\n", 432 | "\n", 433 | "for epoch in range(epochs):\n", 434 | " avg_tr_loss = 0\n", 435 | " tr_step = 0\n", 436 | " \n", 437 | " sess.run(tr_iterator.initializer)\n", 438 | " try:\n", 439 | " while True:\n", 440 | " _, tr_loss = sess.run(fetches = [training_op, sim_pos_rnn.seq2seq_loss])\n", 441 | " avg_tr_loss += tr_loss\n", 442 | " tr_step += 1\n", 443 | " \n", 444 | " except tf.errors.OutOfRangeError:\n", 445 | " pass\n", 446 | " \n", 447 | " avg_tr_loss /= tr_step\n", 448 | " tr_loss_hist.append(avg_tr_loss)\n", 449 | " if (epoch + 1) % 10 == 0:\n", 450 | " print('epoch : {:3}, tr_loss : {:.3f}'.format(epoch + 1, avg_tr_loss))" 451 | ] 452 | }, 453 | { 454 | "cell_type": "code", 455 | "execution_count": 17, 456 | "metadata": {}, 457 | "outputs": [ 458 | { 459 | "data": { 460 | "text/plain": [ 461 | "array([[6, 7, 1, 0, 0, 0, 0, 0, 0, 0],\n", 462 | " [4, 7, 2, 1, 0, 0, 0, 0, 0, 0],\n", 463 | " [4, 7, 3, 4, 5, 1, 4, 0, 0, 0],\n", 464 | " [4, 7, 2, 1, 7, 0, 0, 0, 0, 0]], dtype=int32)" 465 | ] 466 | }, 467 | "execution_count": 17, 468 | "metadata": {}, 469 | "output_type": "execute_result" 470 | } 471 | ], 472 | "source": [ 473 | "yhat = sim_pos_rnn.predict(sess = sess, X_length = X_length, X_indices = X_indices)\n", 474 | "yhat" 475 | ] 476 | }, 477 | { 478 | "cell_type": "code", 479 | "execution_count": 18, 480 | "metadata": {}, 481 | "outputs": [ 482 | { 483 | "data": { 484 | "text/plain": [ 485 | "[[6, 7, 1, 0, 0, 0, 0, 0, 0, 0],\n", 486 | " [4, 7, 2, 1, 0, 0, 0, 0, 0, 0],\n", 487 | " [4, 7, 3, 4, 5, 1, 4, 0, 0, 0],\n", 488 | " [4, 7, 2, 1, 7, 0, 0, 0, 0, 0]]" 489 | ] 490 | }, 491 | "execution_count": 18, 492 | "metadata": {}, 493 | "output_type": "execute_result" 494 | } 495 | ], 496 | "source": [ 497 | "y" 498 | ] 499 | }, 500 | { 501 | "cell_type": "code", 502 | "execution_count": 19, 503 | "metadata": {}, 504 | "outputs": [ 505 | { 506 | "name": "stdout", 507 | "output_type": "stream", 508 | "text": [ 509 | "['pronoun', 'verb', 'adjective', '', '', '', '', '', '', '']\n", 510 | "['noun', 'verb', 'adverb', 'adjective', '', '', '', '', '', '']\n", 511 | "['noun', 'verb', 'determiner', 'noun', 'preposition', 'adjective', 'noun', '', '', '']\n", 512 | "['noun', 'verb', 'adverb', 'adjective', 'verb', '', '', '', '', '']\n" 513 | ] 514 | } 515 | ], 516 | "source": [ 517 | "yhat = [list(map(lambda elm : pos_idx_to_dic.get(elm), row)) for row in yhat]\n", 518 | "for elm in yhat:\n", 519 | " print(elm)" 520 | ] 521 | } 522 | ], 523 | "metadata": { 524 | "kernelspec": { 525 | "display_name": "Python 3", 526 | "language": "python", 527 | "name": "python3" 528 | }, 529 | "language_info": { 530 | "codemirror_mode": { 531 | "name": "ipython", 532 | "version": 3 533 | }, 534 | "file_extension": ".py", 535 | "mimetype": "text/x-python", 536 | "name": "python", 537 | "nbconvert_exporter": "python", 538 | "pygments_lexer": "ipython3", 539 | "version": "3.6.5" 540 | } 541 | }, 542 | "nbformat": 4, 543 | "nbformat_minor": 2 544 | } 545 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # CS 20 : Tensorflow for Deep Learning Research 2 | Refactoring code examples of CS 20 : Tensorflow for Deep Learning Research following tensorflow 2.0 (current tf 1.12) 3 | 4 | * notice 5 | + `{filename}_kd.ipynb` is implemented by using `tf.keras` and `tf.data` 6 | + `{filename}_de.ipynb` is implemented by using `tf.data` and `eager execution` 7 | + `{filename}_kde.ipynb` is implemented by using `tf.keras`, `tf.data` and `eager execution` 8 | 9 | * syllabus : http://web.stanford.edu/class/cs20si/syllabus.html 10 | * github : https://github.com/chiphuyen/stanford-tensorflow-tutorials 11 | - - - 12 | 13 | ### 01. Overview of Tensorflow 14 | - Lec01 Overview of Tensorflow example code 15 | - [Lec01_Overview of Tensorflow.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec01_Overview%20of%20Tensorflow/Lec01_Overview%20of%20Tensorflow.ipynb) 16 | 17 | ### 02. Operations 18 | - Lec02 Operations example code 19 | - [Lec02_Operations.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec02_Operations/Lec02_Operations.ipynb) 20 | 21 | ### 03. Linear and Logistic Regression 22 | - Simple usage of tf.data 23 | - [How to simply use tf.data.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec03_Linear%20and%20Logistic%20Regression/How%20to%20simply%20use%20tf.data.ipynb) 24 | 25 | 26 | - Lec03 Linear and Logistic Regression example code 27 | - [Lec03_Linear Regression with mse loss.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec03_Linear%20and%20Logistic%20Regression/Lec03_Linear%20Regression%20with%20mse%20loss.ipynb) 28 | - [Lec03_Linear Regression with huber loss by low-level.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec03_Linear%20and%20Logistic%20Regression/Lec03_Linear%20Regression%20with%20huber%20loss%20by%20low-level.ipynb) 29 | - [Lec03_Linear Regression with huber loss by high-level.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec03_Linear%20and%20Logistic%20Regression/Lec03_Linear%20Regression%20with%20huber%20loss%20by%20high-level.ipynb) 30 | - [Lec03_Linear Regression with tf.data.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec03_Linear%20and%20Logistic%20Regression/Lec03_Linear%20Regression%20with%20tf.data.ipynb) 31 | - [Lec03_Linear Regression with tf.data_de.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec03_Linear%20and%20Logistic%20Regression/Lec03_Linear%20Regression%20with%20tf.data_de.ipynb) 32 | - [Lec03_Logistic Regression with ce loss.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec03_Linear%20and%20Logistic%20Regression/Lec03_Logistic%20Regression%20with%20ce%20loss.ipynb) 33 | - [Lec03_Logistic Regression with tf.data.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec03_Linear%20and%20Logistic%20Regression/Lec03_Logistic%20Regression%20with%20tf.data.ipynb) 34 | - [Lec03_Logistic Regression with tf.data_de.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec03_Linear%20and%20Logistic%20Regression/Lec03_Logistic%20Regression%20with%20tf.data_de.ipynb) 35 | 36 | ### 04. Eager Execution 37 | + Lec04 Eager execution example code 38 | * [Lec04_Eager execution.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec04_Eager%20execution/Lec04_Eager%20execution.ipynb) 39 | * [Lec04_Automatic differentiation and gradient tape.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec04_Eager%20execution/Lec04_Automatic%20differentiation%20and%20gradient%20tape.ipynb) 40 | * [Lec04_Custom training basics.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec04_Eager%20execution/Lec04_Custom%20training%20basics.ipynb) 41 | * [Lec04_Custom training walkthrough.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec04_Eager%20execution/Lec04_Custom%20training%20walkthrough.ipynb) 42 | * [Lec04_Custom training subclassing.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec04_Eager%20execution/Lec04_Custom%20training%20subclassing.ipynb) 43 | ### 05. Variable sharing and managing experiments 44 | - Simple usage of tf.keras 45 | - [How to use keras.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec05_Variable%20sharing%20and%20managing%20experiments/How%20to%20use%20keras.ipynb) 46 | 47 | - Lec05 Variable sharing and managing experiments example code 48 | - [Lec05_Variable sharing.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec05_Variable%20sharing%20and%20managing%20experiments/Lec05_Variable%20sharing.ipynb) 49 | - [Lec05_Randomization.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec05_Variable%20sharing%20and%20managing%20experiments/Lec05_Randomization.ipynb) 50 | - [Lec05_Applied example with tf.placeholder.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec05_Variable%20sharing%20and%20managing%20experiments/Lec05_Applied%20example%20with%20tf.placeholder.ipynb) 51 | - [Lec05_Applied example with tf.data.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec05_Variable%20sharing%20and%20managing%20experiments/Lec05_Applied%20example%20with%20tf.data.ipynb) 52 | - [Lec05_Applied example with tf.data_kde.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec05_Variable%20sharing%20and%20managing%20experiments/Lec05_Applied%20example%20with%20tf.data_kde.ipynb) 53 | - [Lec05_Word2vec_simple.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec05_Variable%20sharing%20and%20managing%20experiments/Lec05_Word2vec_simple.ipynb) 54 | 55 | ### 06. Introduction to ConvNet 56 | ### 07. ConvNet in TensorFlow 57 | - Lec07 ConvNet in TensorFlow example code 58 | - [Lec07_ConvNet mnist by low-level.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec07_ConvNet%20in%20Tensorflow/Lec07_ConvNet%20mnist%20by%20low-level.ipynb) 59 | - [Lec07_ConvNet mnist by high-level.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec07_ConvNet%20in%20Tensorflow/Lec07_ConvNet%20mnist%20by%20high-level.ipynb) 60 | - [Lec07_ConvNet mnist by high-level_kd.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec07_ConvNet%20in%20Tensorflow/Lec07_ConvNet%20mnist%20by%20high-level_kd.ipynb) 61 | - [Lec07_ConvNet mnist by high-level_kde.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec07_ConvNet%20in%20Tensorflow/Lec07_ConvNet%20mnist%20by%20high-level_kde.ipynb) 62 | - [Lec07_ConvNet mnist with Weight initialization and Drop out.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec07_ConvNet%20in%20Tensorflow/Lec07_ConvNet%20mnist%20with%20Weight%20initialization%20and%20Drop%20out.ipynb) 63 | - [Lec07_ConvNet mnist with Weight initialization and Drop out_kde.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec07_ConvNet%20in%20Tensorflow/Lec07_ConvNet%20mnist%20with%20Weight%20initialization%20and%20Drop%20out_kde.ipynb) 64 | - [Lec07_ConvNet mnist with Weight initialization and Batch norm.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec07_ConvNet%20in%20Tensorflow/Lec07_ConvNet%20mnist%20with%20Weight%20initialization%20and%20Batch%20norm.ipynb) 65 | - [Lec07_ConvNet mnist with Weight initialization and Batch norm_kde.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec07_ConvNet%20in%20Tensorflow/Lec07_ConvNet%20mnist%20with%20Weight%20initialization%20and%20Batch%20norm_kde.ipynb) 66 | 67 | ### 08. Style Transfer 68 | ### 09. Variational Auto-Encoders 69 | ### 10. Generative Adversarial Networks 70 | ### 11. Recurrent Neural Networks 71 | - Presentation 72 | - [To quickly implementing RNN.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/To%20quickly%20implementing%20RNN.ipynb) 73 | 74 | - Lec11 Recurrent Neural Networks example code 75 | - many to one, word sentiment classification example 76 | - [Lec11_Many to One Classification by RNN.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20One%20Classification%20by%20RNN.ipynb) 77 | - [Lec11_Many to One Classification by RNN_kde.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20One%20Classification%20by%20RNN_kde.ipynb) 78 | - [Lec11_Many to One Classification by LSTM.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20One%20Classification%20by%20LSTM.ipynb) 79 | - [Lec11_Many to One Classification by GRU.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20One%20Classification%20by%20GRU.ipynb) 80 | - [Lec11_Many to One Classification by Bi-directional RNN.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20One%20Classification%20by%20Bi-directional%20RNN.ipynb) 81 | - [Lec11_Many to One Classification by Bi-directional LSTM.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20One%20Classification%20by%20Bi-directional%20LSTM.ipynb) 82 | - [Lec11_Many to One Classification by Bi-directional GRU.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20One%20Classification%20by%20Bi-directional%20GRU.ipynb) 83 | - [Lec11_Many to One Classification by Stacked RNN with Drop out.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20One%20Classification%20by%20Stacked%20RNN%20with%20Drop%20out.ipynb) 84 | - [Lec11_Many to One Classification by Stacked LSTM with Drop out.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20One%20Classification%20by%20Stacked%20LSTM%20with%20Drop%20out.ipynb) 85 | - [Lec11_Many to One Classification by Stacked GRU with Drop out.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20One%20Classification%20by%20Stacked%20GRU%20with%20Drop%20out.ipynb) 86 | - [Lec11_Many to One Classification by Stacked Bi-directional RNN with Drop out.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20One%20Classification%20by%20Stacked%20Bi-directional%20RNN%20with%20Drop%20out.ipynb) 87 | - [Lec11_Many to One Classification by Stacked Bi-directional LSTM with Drop out.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20One%20Classification%20by%20Stacked%20Bi-directional%20LSTM%20with%20Drop%20out.ipynb) 88 | - [Lec11_Many to One Classification by Stacked Bi-directional GRU with Drop out.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20One%20Classification%20by%20Stacked%20Bi-directional%20GRU%20with%20Drop%20out.ipynb) 89 | 90 | - many to many, simple pos-tagger example 91 | - [Lec11_Many to Many Classification by RNN.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20Many%20Classification%20by%20RNN.ipynb) 92 | - [Lec11_Many to Many Classification by LSTM.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20Many%20Classification%20by%20LSTM.ipynb) 93 | - [Lec11_Many to Many Classification by GRU.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20Many%20Classification%20by%20GRU.ipynb) 94 | - [Lec11_Many to Many Classification by Bi-directional RNN.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20Many%20Classification%20by%20Bi-directional%20RNN.ipynb) 95 | - [Lec11_Many to Many Classification by Bi-directional LSTM.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20Many%20Classification%20by%20Bi-directional%20LSTM.ipynb) 96 | - [Lec11_Many to Many Classification by Bi-directional GRU.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20Many%20Classification%20by%20Bi-directional%20GRU.ipynb) 97 | - [Lec11_Many to Many Classification by Stacked RNN with Drop out.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20Many%20Classification%20by%20Stacked%20RNN%20with%20Drop%20out.ipynb) 98 | - [Lec11_Many to Many Classification by Stacked LSTM with Drop out.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20Many%20Classification%20by%20Stacked%20LSTM%20with%20Drop%20out.ipynb) 99 | - [Lec11_Many to Many Classification by Stacked GRU with Drop out.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20Many%20Classification%20by%20Stacked%20GRU%20with%20Drop%20out.ipynb) 100 | - [Lec11_Many to Many Classification by Stacked Bi-directional RNN with Drop out.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20Many%20Classification%20by%20Stacked%20Bi-directional%20RNN%20with%20Drop%20out.ipynb) 101 | - [Lec11_Many to Many Classification by Stacked Bi-directional LSTM with Drop out.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20Many%20Classification%20by%20Stacked%20Bi-directional%20LSTM%20with%20Drop%20out.ipynb) 102 | - [Lec11_Many to Many Classification by Stacked Bi-directional GRU with Drop out.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec11_Recurrent%20Neural%20Networks/Lec11_Many%20to%20Many%20Classification%20by%20Stacked%20Bi-directional%20GRU%20with%20Drop%20out.ipynb) 103 | 104 | ### 12. Seq2Seq with Attention 105 | - Lec12 Seq2Seq with Attention example code 106 | 107 | - encoder decoder (many to many), simple neural machine translation example 108 | - [Lec12_Seq2Seq by Encoder RNN and Decoder RNN.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec12_Seq2Seq%20with%20Attention/Lec12_Seq2Seq%20by%20Encoder%20RNN%20and%20Decoder%20RNN.ipynb) 109 | - [Lec12_Seq2Seq by Encoder Bi-directional RNN and Decoder RNN.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec12_Seq2Seq%20with%20Attention/Lec12_Seq2Seq%20by%20Encoder%20Bi-directional%20RNN%20and%20Decoder%20RNN.ipynb) 110 | - [Lec12_Seq2Seq with Attention by Encoder RNN and Decoder RNN.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec12_Seq2Seq%20with%20Attention/Lec12_Seq2Seq%20with%20Attention%20by%20Encoder%20RNN%20and%20Decoder%20RNN.ipynb) 111 | - [Lec12_Seq2Seq with Attention by Encoder Bi-directional RNN and Decoder RNN.ipynb](https://nbviewer.jupyter.org/github/aisolab/CS20/blob/master/Lec12_Seq2Seq%20with%20Attention/Lec12_Seq2Seq%20with%20Attention%20by%20Encoder%20Bi-directional%20RNN%20and%20Decoder%20RNN.ipynb) 112 | 113 | -------------------------------------------------------------------------------- /data/lecture03/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seopbo/CS20/467f0bb51b065886cbeafa1401381f5c6d3c5f30/data/lecture03/.DS_Store -------------------------------------------------------------------------------- /data/lecture03/example_with_data/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seopbo/CS20/467f0bb51b065886cbeafa1401381f5c6d3c5f30/data/lecture03/example_with_data/.DS_Store -------------------------------------------------------------------------------- /data/lecture03/example_with_data/train_dir/birth_life_2010_tr_1.txt: -------------------------------------------------------------------------------- 1 | Vanuatu 3.8689999999999998 70.819487805 2 | Tonga 3.911 72.150658537 3 | Timor-Leste 5.577999999999999 61.999853658999996 4 | Thailand 1.579 73.927658537 5 | Philippines 3.142 68.484317073 6 | Papua New Guinea 3.951 62.440609756 7 | New Zealand 2.16 80.702439024 8 | Myanmar 2.002 64.66209756100001 9 | Mongolia 2.504 68.19497561 10 | Micronesia 3.451 68.764829268 11 | -------------------------------------------------------------------------------- /data/lecture03/example_with_data/train_dir/birth_life_2010_tr_10.txt: -------------------------------------------------------------------------------- 1 | Jordan 3.8 73.289658537 2 | Israel 3.03 81.504878049 3 | Iran 1.67 72.75185365899999 4 | Djibouti 3.75 57.527390243999996 5 | Bahrain 2.54 75.023829268 6 | Algeria 2.2640000000000002 72.852536585 7 | United States 2.1 78.241463415 8 | Canada 1.6769999999999998 80.79780487800001 9 | Bermuda 1.764 79.288536585 10 | Sri Lanka 2.313 74.722609756 11 | -------------------------------------------------------------------------------- /data/lecture03/example_with_data/train_dir/birth_life_2010_tr_11.txt: -------------------------------------------------------------------------------- 1 | Pakistan 3.423 65.198853659 2 | Nepal 2.727 68.394829268 3 | Maldives 1.7519999999999998 76.551414634 4 | India 2.625 65.131341463 5 | Bhutan 2.399 66.908853659 6 | Bangladesh 2.245 68.634804878 7 | Zimbabwe 3.29 49.86087804899999 8 | Zambia 6.257999999999999 48.455487805 9 | Togo 4.072 56.588707317 10 | The Gambia 4.896 58.16002439 11 | -------------------------------------------------------------------------------- /data/lecture03/example_with_data/train_dir/birth_life_2010_tr_12.txt: -------------------------------------------------------------------------------- 1 | South Africa 2.458 52.081487805 2 | Somalia 6.3389999999999995 50.895536585 3 | Sierra Leone 4.982 47.402195121999995 4 | Seychelles 2.5 73.03414634100001 5 | Senegal 4.819 58.954073171000005 6 | Rwanda 5.371 55.057121951000006 7 | Niger 7.063 54.265634146000004 8 | Mauritius 1.47 72.967317073 9 | Mauritania 4.533 58.21695122 10 | Malawi 5.99 53.462634146000006 11 | -------------------------------------------------------------------------------- /data/lecture03/example_with_data/train_dir/birth_life_2010_tr_13.txt: -------------------------------------------------------------------------------- 1 | Madagascar 4.651 66.46707317100001 2 | Liberia 5.2379999999999995 56.147585366 3 | Lesotho 3.199 47.365073171000006 4 | Kenya 4.718 56.497073171000004 5 | Guinea-Bissau 5.063 47.700658536999995 6 | Guinea 5.246 53.638585366 7 | Ghana 4.17 63.837268293 8 | Gabon 3.25 62.286682927 9 | Equatorial Guinea 5.185 50.840804878 10 | Dem. Rep. Congo 5.775 48.069585366000005 11 | -------------------------------------------------------------------------------- /data/lecture03/example_with_data/train_dir/birth_life_2010_tr_14.txt: -------------------------------------------------------------------------------- 1 | Congo 4.544 56.960195121999995 2 | Comoros 4.919 60.626268293 3 | Chad 5.981 49.194829268 4 | Central African Republic 4.631 47.618463415 5 | Cape Verde 2.405 73.77404878 6 | Cameroon 4.487 51.062756098 7 | Burundi 4.338 49.877219511999996 8 | Botswana 2.75 53.109512195 9 | Benin 5.287000000000001 55.585585366000004 10 | Angola 5.443 50.653658537 11 | -------------------------------------------------------------------------------- /data/lecture03/example_with_data/train_dir/birth_life_2010_tr_2.txt: -------------------------------------------------------------------------------- 1 | Malaysia 2.635 74.024560976 2 | Lao PDR 2.747 67.064 3 | Japan 1.39 82.932682927 4 | Hong Kong SAR, China 1.1079999999999999 82.87804878 5 | Guam 2.47 75.994268293 6 | French Polynesia 2.09 75.076878049 7 | Fiji 2.668 69.225829268 8 | Dem. Rep. Korea 2.022 68.532146341 9 | China 1.598 73.273097561 10 | Cambodia 2.5810000000000004 62.536219511999995 11 | -------------------------------------------------------------------------------- /data/lecture03/example_with_data/train_dir/birth_life_2010_tr_3.txt: -------------------------------------------------------------------------------- 1 | Australia 1.92 81.695121951 2 | Uzbekistan 2.499 68.001 3 | United Kingdom 1.94 80.402439024 4 | Turkmenistan 2.399 64.863512195 5 | Tajikistan 3.2969999999999997 67.259902439 6 | Sweden 1.98 81.451219512 7 | Spain 1.39 81.62682926800001 8 | Slovak Republic 1.4 75.112195122 9 | Russia 1.54 68.804878049 10 | Romania 1.38 73.458536585 11 | -------------------------------------------------------------------------------- /data/lecture03/example_with_data/train_dir/birth_life_2010_tr_4.txt: -------------------------------------------------------------------------------- 1 | Poland 1.38 76.24634146300001 2 | Norway 1.95 80.997560976 3 | Netherlands 1.79 80.702439024 4 | Montenegro 1.656 74.311097561 5 | Moldova 1.475 68.903707317 6 | Macedonia 1.422 74.618853659 7 | Luxembourg 1.63 80.087804878 8 | Lithuania 1.55 73.268292683 9 | Latvia 1.17 73.482926829 10 | Kyrgyz Republic 2.898 69.368463415 11 | -------------------------------------------------------------------------------- /data/lecture03/example_with_data/train_dir/birth_life_2010_tr_5.txt: -------------------------------------------------------------------------------- 1 | Italy 1.4 81.736585366 2 | Ireland 2.07 80.29195122 3 | Iceland 2.2 81.451219512 4 | Greece 1.44 80.387804878 5 | France 2.0 81.368292683 6 | Finland 1.87 79.870731707 7 | Denmark 1.87 79.1 8 | Czech Republic 1.49 77.424390244 9 | Cyprus 1.476 79.380390244 10 | Croatia 1.46 76.475609756 11 | -------------------------------------------------------------------------------- /data/lecture03/example_with_data/train_dir/birth_life_2010_tr_6.txt: -------------------------------------------------------------------------------- 1 | Bulgaria 1.49 73.512195122 2 | Bosnia and Herzegovina 1.148 75.400439024 3 | Belarus 1.44 70.40487804899999 4 | Azerbaijan 2.3 70.506512195 5 | Austria 1.44 80.382926829 6 | Armenia 1.736 73.783560976 7 | Albania 1.536 76.90095122 8 | Virgin Islands 1.8 79.189439024 9 | Venezuela 2.49 74.127317073 10 | The Bahamas 1.896 75.222121951 11 | -------------------------------------------------------------------------------- /data/lecture03/example_with_data/train_dir/birth_life_2010_tr_7.txt: -------------------------------------------------------------------------------- 1 | Suriname 2.336 70.335317073 2 | St. Lucia 1.98 74.439902439 3 | Puerto Rico 1.797 78.91329268300001 4 | Peru 2.501 73.76497561 5 | Paraguay 2.9539999999999997 72.277 6 | Panama 2.484 75.974243902 7 | Mexico 2.32 76.683780488 8 | Jamaica 2.329 72.847121951 9 | Guyana 2.262 69.54914634100001 10 | Guatemala 3.983 70.825414634 11 | -------------------------------------------------------------------------------- /data/lecture03/example_with_data/train_dir/birth_life_2010_tr_8.txt: -------------------------------------------------------------------------------- 1 | El Salvador 2.25 71.732365854 2 | Ecuador 2.479 75.462292683 3 | Cuba 1.4669999999999999 78.964146341 4 | Costa Rica 1.848 79.19260975600001 5 | Colombia 2.1 73.429682927 6 | Chile 1.8619999999999999 78.885731707 7 | Brazil 1.83 73.099536585 8 | Barbados 1.551 76.572829268 9 | Aruba 1.699 74.975170732 10 | Argentina 2.211 75.63214634100001 11 | -------------------------------------------------------------------------------- /data/lecture03/example_with_data/train_dir/birth_life_2010_tr_9.txt: -------------------------------------------------------------------------------- 1 | Yemen 5.2 65.030463415 2 | West Bank and Gaza 4.453 72.643682927 3 | United Arab Emirates 1.749 76.57360975600001 4 | Tunisia 2.04 74.6 5 | Syrian Arab Republic 2.9339999999999997 75.702560976 6 | Saudi Arabia 2.8110000000000004 73.85041463399999 7 | Malta 1.38 80.948780488 8 | Libya 2.5639999999999996 74.753121951 9 | Lebanon 1.8 72.408756098 10 | Kuwait 2.295 74.604731707 11 | -------------------------------------------------------------------------------- /data/lecture03/example_with_data/val_dir/birth_life_2010_val.txt: -------------------------------------------------------------------------------- 1 | Portugal 1.32 79.026829268 2 | Korea 1.22 80.76195122 3 | Afghanistan 6.287999999999999 48.282195122 4 | Honduras 3.139 72.825926829 5 | Brunei 2.042 77.93202439 6 | Solomon Islands 4.229 67.465195122 7 | Bolivia 3.3480000000000003 66.268560976 8 | Nigeria 5.525 51.41002439 9 | Iraq 4.702 68.48604878 10 | Vietnam 1.8219999999999998 74.828243902 11 | Eritrea 4.453 60.994195122 12 | Belgium 1.84 79.936585366 13 | Kosovo 2.2893 69.884390244 14 | Singapore 1.15 81.641463415 15 | Mozambique 4.912 49.696926829 16 | Tanzania 5.544 57.387487805 17 | Estonia 1.63 75.429268293 18 | Indonesia 2.117 68.889658537 19 | Samoa 3.86 72.306390244 20 | Uganda 6.149 53.61463414600001 21 | Macao SAR, China 1.092 80.775317073 22 | Burkina Faso 5.85 54.924195122 23 | Dominican Republic 2.58 73.20002439 24 | Slovenia 1.57 79.42195122 25 | Nicaragua 2.622 73.729219512 26 | Kazakhstan 2.59 68.295365854 27 | Oman 2.309 73.124609756 28 | New Caledonia 2.141 76.301682927 29 | Grenada 2.235 75.660439024 30 | Germany 1.39 79.987804878 31 | Mali 6.294 50.954829268000005 32 | Channel Islands 1.46 79.832 33 | Namibia 3.217 62.070097561000004 34 | Trinidad and Tobago 1.639 69.755 35 | St. Vincent and the Grenadines 2.064 72.112536585 36 | Sudan 4.4 61.108243902 37 | Ethiopia 4.1930000000000005 58.715097561 38 | Turkey 2.088 73.696658537 39 | Morocco 2.279 71.864634146 40 | Georgia 1.555 73.327341463 41 | Qatar 2.271 78.097585366 42 | Hungary 1.25 74.207317073 43 | Uruguay 1.9856 76.23682926800001 44 | Egypt 2.733 72.975268293 45 | Ukraine 1.445 70.27560975600001 46 | Haiti 3.34 61.763000000000005 47 | Belize 2.79 75.83995122 48 | Switzerland 1.5 82.24634146300001 49 | Swaziland 3.364 48.342804878 50 | Serbia 1.4 73.936585366 51 | -------------------------------------------------------------------------------- /data/lecture03/example_with_placeholder/birth_life_2010.txt: -------------------------------------------------------------------------------- 1 | Country Birth rate Life expectancy 2 | Vietnam 1.822 74.828243902 3 | Vanuatu 3.869 70.819487805 4 | Tonga 3.911 72.150658537 5 | Timor-Leste 5.578 61.999853659 6 | Thailand 1.579 73.927658537 7 | Solomon Islands 4.229 67.465195122 8 | Singapore 1.15 81.641463415 9 | Samoa 3.86 72.306390244 10 | Philippines 3.142 68.484317073 11 | Papua New Guinea 3.951 62.440609756 12 | New Zealand 2.16 80.702439024 13 | New Caledonia 2.141 76.301682927 14 | Myanmar 2.002 64.662097561 15 | Mongolia 2.504 68.194975610 16 | Micronesia 3.451 68.764829268 17 | Malaysia 2.635 74.024560976 18 | Macao SAR, China 1.092 80.775317073 19 | Lao PDR 2.747 67.064 20 | Korea 1.22 80.761951220 21 | Japan 1.39 82.932682927 22 | Indonesia 2.117 68.889658537 23 | Hong Kong SAR, China 1.108 82.878048780 24 | Guam 2.47 75.994268293 25 | French Polynesia 2.09 75.076878049 26 | Fiji 2.668 69.225829268 27 | Dem. Rep. Korea 2.022 68.532146341 28 | China 1.598 73.273097561 29 | Cambodia 2.581 62.536219512 30 | Brunei 2.042 77.932024390 31 | Australia 1.92 81.695121951 32 | Uzbekistan 2.499 68.001 33 | United Kingdom 1.94 80.402439024 34 | Ukraine 1.445 70.275609756 35 | Turkmenistan 2.399 64.863512195 36 | Turkey 2.088 73.696658537 37 | Tajikistan 3.297 67.259902439 38 | Switzerland 1.5 82.246341463 39 | Sweden 1.98 81.451219512 40 | Spain 1.39 81.626829268 41 | Slovenia 1.57 79.421951220 42 | Slovak Republic 1.4 75.112195122 43 | Serbia 1.4 73.936585366 44 | Russia 1.54 68.804878049 45 | Romania 1.38 73.458536585 46 | Portugal 1.32 79.026829268 47 | Poland 1.38 76.246341463 48 | Norway 1.95 80.997560976 49 | Netherlands 1.79 80.702439024 50 | Montenegro 1.656 74.311097561 51 | Moldova 1.475 68.903707317 52 | Macedonia 1.422 74.618853659 53 | Luxembourg 1.63 80.087804878 54 | Lithuania 1.55 73.268292683 55 | Latvia 1.17 73.482926829 56 | Kyrgyz Republic 2.898 69.368463415 57 | Kosovo 2.2893 69.884390244 58 | Kazakhstan 2.59 68.295365854 59 | Italy 1.4 81.736585366 60 | Ireland 2.07 80.291951220 61 | Iceland 2.2 81.451219512 62 | Hungary 1.25 74.207317073 63 | Greece 1.44 80.387804878 64 | Germany 1.39 79.987804878 65 | Georgia 1.555 73.327341463 66 | France 2 81.368292683 67 | Finland 1.87 79.870731707 68 | Estonia 1.63 75.429268293 69 | Denmark 1.87 79.1 70 | Czech Republic 1.49 77.424390244 71 | Cyprus 1.476 79.380390244 72 | Croatia 1.46 76.475609756 73 | Channel Islands 1.46 79.832 74 | Bulgaria 1.49 73.512195122 75 | Bosnia and Herzegovina 1.148 75.400439024 76 | Belgium 1.84 79.936585366 77 | Belarus 1.44 70.404878049 78 | Azerbaijan 2.3 70.506512195 79 | Austria 1.44 80.382926829 80 | Armenia 1.736 73.783560976 81 | Albania 1.536 76.900951220 82 | Virgin Islands 1.8 79.189439024 83 | Venezuela 2.49 74.127317073 84 | Uruguay 1.9856 76.236829268 85 | Trinidad and Tobago 1.639 69.755 86 | The Bahamas 1.896 75.222121951 87 | Suriname 2.336 70.335317073 88 | St. Vincent and the Grenadines 2.064 72.112536585 89 | St. Lucia 1.98 74.439902439 90 | Puerto Rico 1.797 78.913292683 91 | Peru 2.501 73.764975610 92 | Paraguay 2.954 72.277 93 | Panama 2.484 75.974243902 94 | Nicaragua 2.622 73.729219512 95 | Mexico 2.32 76.683780488 96 | Jamaica 2.329 72.847121951 97 | Honduras 3.139 72.825926829 98 | Haiti 3.34 61.763 99 | Guyana 2.262 69.549146341 100 | Guatemala 3.983 70.825414634 101 | Grenada 2.235 75.660439024 102 | El Salvador 2.25 71.732365854 103 | Ecuador 2.479 75.462292683 104 | Dominican Republic 2.58 73.200024390 105 | Cuba 1.467 78.964146341 106 | Costa Rica 1.848 79.192609756 107 | Colombia 2.1 73.429682927 108 | Chile 1.862 78.885731707 109 | Brazil 1.83 73.099536585 110 | Bolivia 3.348 66.268560976 111 | Belize 2.79 75.839951220 112 | Barbados 1.551 76.572829268 113 | Aruba 1.699 74.975170732 114 | Argentina 2.211 75.632146341 115 | Yemen 5.2 65.030463415 116 | West Bank and Gaza 4.453 72.643682927 117 | United Arab Emirates 1.749 76.573609756 118 | Tunisia 2.04 74.6 119 | Syrian Arab Republic 2.934 75.702560976 120 | Saudi Arabia 2.811 73.850414634 121 | Qatar 2.271 78.097585366 122 | Oman 2.309 73.124609756 123 | Morocco 2.279 71.864634146 124 | Malta 1.38 80.948780488 125 | Libya 2.564 74.753121951 126 | Lebanon 1.8 72.408756098 127 | Kuwait 2.295 74.604731707 128 | Jordan 3.8 73.289658537 129 | Israel 3.03 81.504878049 130 | Iraq 4.702 68.486048780 131 | Iran 1.67 72.751853659 132 | Egypt 2.733 72.975268293 133 | Djibouti 3.75 57.527390244 134 | Bahrain 2.54 75.023829268 135 | Algeria 2.264 72.852536585 136 | United States 2.1 78.241463415 137 | Canada 1.677 80.797804878 138 | Bermuda 1.764 79.288536585 139 | Sri Lanka 2.313 74.722609756 140 | Pakistan 3.423 65.198853659 141 | Nepal 2.727 68.394829268 142 | Maldives 1.752 76.551414634 143 | India 2.625 65.131341463 144 | Bhutan 2.399 66.908853659 145 | Bangladesh 2.245 68.634804878 146 | Afghanistan 6.288 48.282195122 147 | Zimbabwe 3.29 49.860878049 148 | Zambia 6.258 48.455487805 149 | Uganda 6.149 53.614634146 150 | Togo 4.072 56.588707317 151 | The Gambia 4.896 58.160024390 152 | Tanzania 5.544 57.387487805 153 | Swaziland 3.364 48.342804878 154 | Sudan 4.4 61.108243902 155 | South Africa 2.458 52.081487805 156 | Somalia 6.339 50.895536585 157 | Sierra Leone 4.982 47.402195122 158 | Seychelles 2.5 73.034146341 159 | Senegal 4.819 58.954073171 160 | Rwanda 5.371 55.057121951 161 | Nigeria 5.525 51.410024390 162 | Niger 7.063 54.265634146 163 | Namibia 3.217 62.070097561 164 | Mozambique 4.912 49.696926829 165 | Mauritius 1.47 72.967317073 166 | Mauritania 4.533 58.216951220 167 | Mali 6.294 50.954829268 168 | Malawi 5.99 53.462634146 169 | Madagascar 4.651 66.467073171 170 | Liberia 5.238 56.147585366 171 | Lesotho 3.199 47.365073171 172 | Kenya 4.718 56.497073171 173 | Guinea-Bissau 5.063 47.700658537 174 | Guinea 5.246 53.638585366 175 | Ghana 4.17 63.837268293 176 | Gabon 3.25 62.286682927 177 | Ethiopia 4.193 58.715097561 178 | Eritrea 4.453 60.994195122 179 | Equatorial Guinea 5.185 50.840804878 180 | Dem. Rep. Congo 5.775 48.069585366 181 | Congo 4.544 56.960195122 182 | Comoros 4.919 60.626268293 183 | Chad 5.981 49.194829268 184 | Central African Republic 4.631 47.618463415 185 | Cape Verde 2.405 73.774048780 186 | Cameroon 4.487 51.062756098 187 | Burundi 4.338 49.877219512 188 | Burkina Faso 5.85 54.924195122 189 | Botswana 2.75 53.109512195 190 | Benin 5.287 55.585585366 191 | Angola 5.443 50.653658537 -------------------------------------------------------------------------------- /data/lecture04/iris_test.csv: -------------------------------------------------------------------------------- 1 | 5.9,3,4.2,1.5,1 2 | 6.9,3.1,5.4,2.1,2 3 | 5.1,3.3,1.7,0.5,0 4 | 6,3.4,4.5,1.6,1 5 | 5.5,2.5,4,1.3,1 6 | 6.2,2.9,4.3,1.3,1 7 | 5.5,4.2,1.4,0.2,0 8 | 6.3,2.8,5.1,1.5,2 9 | 5.6,3,4.1,1.3,1 10 | 6.7,2.5,5.8,1.8,2 11 | 7.1,3,5.9,2.1,2 12 | 4.3,3,1.1,0.1,0 13 | 5.6,2.8,4.9,2,2 14 | 5.5,2.3,4,1.3,1 15 | 6,2.2,4,1,1 16 | 5.1,3.5,1.4,0.2,0 17 | 5.7,2.6,3.5,1,1 18 | 4.8,3.4,1.9,0.2,0 19 | 5.1,3.4,1.5,0.2,0 20 | 5.7,2.5,5,2,2 21 | 5.4,3.4,1.7,0.2,0 22 | 5.6,3,4.5,1.5,1 23 | 6.3,2.9,5.6,1.8,2 24 | 6.3,2.5,4.9,1.5,1 25 | 5.8,2.7,3.9,1.2,1 26 | 6.1,3,4.6,1.4,1 27 | 5.2,4.1,1.5,0.1,0 28 | 6.7,3.1,4.7,1.5,1 29 | 6.7,3.3,5.7,2.5,2 30 | 6.4,2.9,4.3,1.3,1 31 | -------------------------------------------------------------------------------- /data/lecture04/iris_training.csv: -------------------------------------------------------------------------------- 1 | 6.4,2.8,5.6,2.2,2 2 | 5,2.3,3.3,1,1 3 | 4.9,2.5,4.5,1.7,2 4 | 4.9,3.1,1.5,0.1,0 5 | 5.7,3.8,1.7,0.3,0 6 | 4.4,3.2,1.3,0.2,0 7 | 5.4,3.4,1.5,0.4,0 8 | 6.9,3.1,5.1,2.3,2 9 | 6.7,3.1,4.4,1.4,1 10 | 5.1,3.7,1.5,0.4,0 11 | 5.2,2.7,3.9,1.4,1 12 | 6.9,3.1,4.9,1.5,1 13 | 5.8,4,1.2,0.2,0 14 | 5.4,3.9,1.7,0.4,0 15 | 7.7,3.8,6.7,2.2,2 16 | 6.3,3.3,4.7,1.6,1 17 | 6.8,3.2,5.9,2.3,2 18 | 7.6,3,6.6,2.1,2 19 | 6.4,3.2,5.3,2.3,2 20 | 5.7,4.4,1.5,0.4,0 21 | 6.7,3.3,5.7,2.1,2 22 | 6.4,2.8,5.6,2.1,2 23 | 5.4,3.9,1.3,0.4,0 24 | 6.1,2.6,5.6,1.4,2 25 | 7.2,3,5.8,1.6,2 26 | 5.2,3.5,1.5,0.2,0 27 | 5.8,2.6,4,1.2,1 28 | 5.9,3,5.1,1.8,2 29 | 5.4,3,4.5,1.5,1 30 | 6.7,3,5,1.7,1 31 | 6.3,2.3,4.4,1.3,1 32 | 5.1,2.5,3,1.1,1 33 | 6.4,3.2,4.5,1.5,1 34 | 6.8,3,5.5,2.1,2 35 | 6.2,2.8,4.8,1.8,2 36 | 6.9,3.2,5.7,2.3,2 37 | 6.5,3.2,5.1,2,2 38 | 5.8,2.8,5.1,2.4,2 39 | 5.1,3.8,1.5,0.3,0 40 | 4.8,3,1.4,0.3,0 41 | 7.9,3.8,6.4,2,2 42 | 5.8,2.7,5.1,1.9,2 43 | 6.7,3,5.2,2.3,2 44 | 5.1,3.8,1.9,0.4,0 45 | 4.7,3.2,1.6,0.2,0 46 | 6,2.2,5,1.5,2 47 | 4.8,3.4,1.6,0.2,0 48 | 7.7,2.6,6.9,2.3,2 49 | 4.6,3.6,1,0.2,0 50 | 7.2,3.2,6,1.8,2 51 | 5,3.3,1.4,0.2,0 52 | 6.6,3,4.4,1.4,1 53 | 6.1,2.8,4,1.3,1 54 | 5,3.2,1.2,0.2,0 55 | 7,3.2,4.7,1.4,1 56 | 6,3,4.8,1.8,2 57 | 7.4,2.8,6.1,1.9,2 58 | 5.8,2.7,5.1,1.9,2 59 | 6.2,3.4,5.4,2.3,2 60 | 5,2,3.5,1,1 61 | 5.6,2.5,3.9,1.1,1 62 | 6.7,3.1,5.6,2.4,2 63 | 6.3,2.5,5,1.9,2 64 | 6.4,3.1,5.5,1.8,2 65 | 6.2,2.2,4.5,1.5,1 66 | 7.3,2.9,6.3,1.8,2 67 | 4.4,3,1.3,0.2,0 68 | 7.2,3.6,6.1,2.5,2 69 | 6.5,3,5.5,1.8,2 70 | 5,3.4,1.5,0.2,0 71 | 4.7,3.2,1.3,0.2,0 72 | 6.6,2.9,4.6,1.3,1 73 | 5.5,3.5,1.3,0.2,0 74 | 7.7,3,6.1,2.3,2 75 | 6.1,3,4.9,1.8,2 76 | 4.9,3.1,1.5,0.1,0 77 | 5.5,2.4,3.8,1.1,1 78 | 5.7,2.9,4.2,1.3,1 79 | 6,2.9,4.5,1.5,1 80 | 6.4,2.7,5.3,1.9,2 81 | 5.4,3.7,1.5,0.2,0 82 | 6.1,2.9,4.7,1.4,1 83 | 6.5,2.8,4.6,1.5,1 84 | 5.6,2.7,4.2,1.3,1 85 | 6.3,3.4,5.6,2.4,2 86 | 4.9,3.1,1.5,0.1,0 87 | 6.8,2.8,4.8,1.4,1 88 | 5.7,2.8,4.5,1.3,1 89 | 6,2.7,5.1,1.6,1 90 | 5,3.5,1.3,0.3,0 91 | 6.5,3,5.2,2,2 92 | 6.1,2.8,4.7,1.2,1 93 | 5.1,3.5,1.4,0.3,0 94 | 4.6,3.1,1.5,0.2,0 95 | 6.5,3,5.8,2.2,2 96 | 4.6,3.4,1.4,0.3,0 97 | 4.6,3.2,1.4,0.2,0 98 | 7.7,2.8,6.7,2,2 99 | 5.9,3.2,4.8,1.8,1 100 | 5.1,3.8,1.6,0.2,0 101 | 4.9,3,1.4,0.2,0 102 | 4.9,2.4,3.3,1,1 103 | 4.5,2.3,1.3,0.3,0 104 | 5.8,2.7,4.1,1,1 105 | 5,3.4,1.6,0.4,0 106 | 5.2,3.4,1.4,0.2,0 107 | 5.3,3.7,1.5,0.2,0 108 | 5,3.6,1.4,0.2,0 109 | 5.6,2.9,3.6,1.3,1 110 | 4.8,3.1,1.6,0.2,0 111 | 6.3,2.7,4.9,1.8,2 112 | 5.7,2.8,4.1,1.3,1 113 | 5,3,1.6,0.2,0 114 | 6.3,3.3,6,2.5,2 115 | 5,3.5,1.6,0.6,0 116 | 5.5,2.6,4.4,1.2,1 117 | 5.7,3,4.2,1.2,1 118 | 4.4,2.9,1.4,0.2,0 119 | 4.8,3,1.4,0.1,0 120 | 5.5,2.4,3.7,1,1 121 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | absl-py==0.6.1 2 | appnope==0.1.0 3 | astor==0.7.1 4 | backcall==0.1.0 5 | bleach==3.0.2 6 | cycler==0.10.0 7 | decorator==4.3.0 8 | defusedxml==0.5.0 9 | entrypoints==0.2.3 10 | gast==0.2.0 11 | grpcio==1.17.1 12 | h5py==2.9.0 13 | ipykernel==5.1.0 14 | ipython==7.2.0 15 | ipython-genutils==0.2.0 16 | ipywidgets==7.4.2 17 | jedi==0.13.2 18 | Jinja2==2.10 19 | jsonschema==2.6.0 20 | jupyter==1.0.0 21 | jupyter-client==5.2.4 22 | jupyter-console==6.0.0 23 | jupyter-core==4.4.0 24 | Keras-Applications==1.0.6 25 | Keras-Preprocessing==1.0.5 26 | kiwisolver==1.0.1 27 | Markdown==3.0.1 28 | MarkupSafe==1.1.0 29 | matplotlib==3.0.2 30 | mistune==0.8.4 31 | nbconvert==5.4.0 32 | nbformat==4.4.0 33 | notebook==5.7.4 34 | numpy==1.15.4 35 | pandas==0.23.4 36 | pandocfilters==1.4.2 37 | parso==0.3.1 38 | pexpect==4.6.0 39 | pickleshare==0.7.5 40 | prometheus-client==0.5.0 41 | prompt-toolkit==2.0.7 42 | protobuf==3.6.1 43 | ptyprocess==0.6.0 44 | Pygments==2.3.1 45 | pyparsing==2.3.0 46 | python-dateutil==2.7.5 47 | pytz==2018.7 48 | pyzmq==17.1.2 49 | qtconsole==4.4.3 50 | Send2Trash==1.5.0 51 | six==1.12.0 52 | tensorboard==1.12.1 53 | tensorflow==1.12.0 54 | termcolor==1.1.0 55 | terminado==0.8.1 56 | testpath==0.4.2 57 | tornado==5.1.1 58 | traitlets==4.3.2 59 | wcwidth==0.1.7 60 | webencodings==0.5.1 61 | Werkzeug==0.14.1 62 | widgetsnbextension==3.4.2 63 | --------------------------------------------------------------------------------