├── .gitignore ├── README.md ├── code ├── Tutorial1_Regression.ipynb ├── Tutorial2_MLP_MNIST.ipynb ├── Tutorial3-1_CNN_MNIST.ipynb ├── Tutorial3-2_CNN_CIFAR10.ipynb └── Tutorial4_RNN_Name.ipynb ├── data ├── regression_data.txt └── woman_name_dataset.csv └── image └── tensorboard.png /.gitignore: -------------------------------------------------------------------------------- 1 | output/* 2 | .ipython_checkpoints/* 3 | .ipython_checkpoints 4 | answer -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Deep Learning Tutorial 2 | 3 | ### First Class 4 | - [Lecture - Introduction to deep learning 1](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/EZ_M4cpii7FPjywju0dcV9kBueyH0jjxCd_apKKT5xiLyQ?e=yKwg5J) 5 | - [Tutorial - Tensorflow and regression](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/EZaJDyjVX5lLpFSG9XL4MaoBGwBjRAgzgqmArvJDagMhJw?e=0Mydl8) 6 | 7 | ### Second Class 8 | - [Lecture - Introduction to deep learning 2](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/EX-Ob5JBQKhFkjI9JX2x9QgBZ87l5keRsUYe5-ekZllcug?e=fxceUO) 9 | - [Tutorial - Multi Layer Perceptrons](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/EUf_X57J1MNOuuHzre0QlbsBCnUQz0ppIEMsiCMxolD7gQ?e=MZyZA3) 10 | 11 | ### Third Class 12 | - [Lecture - Convolutional Neural Networks 1](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/ETz9uaskUHBKiKUyxFk2b48BEsejUfDciPLYGXBUipp6fA?e=0sS79W) 13 | - [Lecture - Convolutional Neural Networks 2](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/EQbk7z_8q8hIsiDImNHFJLYB_T5RkyIzH8Mj7-KBrHMZkA?e=r4WyOr) 14 | - [Tutorial - Convolutional Neural Networks(MNIST)](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/ESi06xRGkppJhookanl8pC8BWh-qhWMTrYhWwKU-KQHeyw?e=uRuZEu) 15 | - [Tutorial - Convolutional Neural Networks(CIFAR10)](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/EXjVnUGynA1EhOfkuDuS5LoBWHsDGj4xjwM259JWbuP2WA?e=ZZmtAF) 16 | 17 | ### Fourth Class 18 | - [Lecture - Recurrent Neural Networks](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/EVrxTDoIIGpNlWx0YqEkF5sBJ8Sns_uNIK1llLJifhEVyw?e=HjhIZl) 19 | - [Lecture - Advanced topics](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/EW6BL5tzgCFJnx_tNDJ1fW4B2QqE3fIXMpPiO2WPHt_98w?e=ajsode) 20 | - [Tutorial - Recurrent Neural Networks](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/EUl-ZNocKbVClPfLAQvPmp8BF1pC93MWfOdQDTiKlzR3og?e=lvObU6) 21 | -------------------------------------------------------------------------------- /code/Tutorial1_Regression.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Practice Day 1. Regression" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "### Load File" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": null, 20 | "metadata": {}, 21 | "outputs": [], 22 | "source": [ 23 | "file = open('../data/regression_data.txt','r') # open the file with read-only\n", 24 | "text = file.readlines() # read all line texts\n", 25 | "file.close() # close the file\n", 26 | "\n", 27 | "x_data = []\n", 28 | "y_data = []\n", 29 | "\n", 30 | "# convert to float\n", 31 | "for s in text:\n", 32 | " data = s.split()\n", 33 | " x_data.append(float(data[0]))\n", 34 | " y_data.append(float(data[1])) " 35 | ] 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "metadata": {}, 40 | "source": [ 41 | "### Plot data" 42 | ] 43 | }, 44 | { 45 | "cell_type": "code", 46 | "execution_count": null, 47 | "metadata": { 48 | "scrolled": false 49 | }, 50 | "outputs": [], 51 | "source": [ 52 | "%matplotlib notebook\n", 53 | "import matplotlib.pyplot as plt\n", 54 | "import numpy as np\n", 55 | "\n", 56 | "plt.figure(1)\n", 57 | "plt.plot(x_data, y_data, 'ro') # plot data\n", 58 | "\n", 59 | "plt.xlabel('x-axis') \n", 60 | "plt.ylabel('y-axis')\n", 61 | "plt.title('My data')\n", 62 | "\n", 63 | "plt.show()" 64 | ] 65 | }, 66 | { 67 | "cell_type": "markdown", 68 | "metadata": {}, 69 | "source": [ 70 | "### Closed Form Linear Regression" 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": null, 76 | "metadata": { 77 | "scrolled": true 78 | }, 79 | "outputs": [], 80 | "source": [ 81 | "# convert to numpy-array\n", 82 | "x_data = np.asarray(x_data)\n", 83 | "y_data = np.asarray(y_data)\n", 84 | "\n", 85 | "N = len(x_data) # the size of data\n", 86 | "\n", 87 | "sum_of_x =\n", 88 | "sum_of_x_square = \n", 89 | "sum_of_y = \n", 90 | "sum_of_xy = \n", 91 | "\n", 92 | "a = \n", 93 | "b = \n", 94 | "\n", 95 | "print ('a : {}'.format(a))\n", 96 | "print ('b : {}'.format(b))" 97 | ] 98 | }, 99 | { 100 | "cell_type": "code", 101 | "execution_count": null, 102 | "metadata": { 103 | "scrolled": false 104 | }, 105 | "outputs": [], 106 | "source": [ 107 | "plt.figure(2)\n", 108 | "\n", 109 | "y_regression = a * x_data + b\n", 110 | "\n", 111 | "plt.plot(x_data, y_data, 'ro')\n", 112 | "plt.plot(x_data, y_regression, 'b')\n", 113 | "\n", 114 | "plt.xlabel('x-axis') \n", 115 | "plt.ylabel('y-axis')\n", 116 | "plt.title('Closed Form Regression')\n", 117 | "\n", 118 | "plt.show()" 119 | ] 120 | }, 121 | { 122 | "cell_type": "markdown", 123 | "metadata": {}, 124 | "source": [ 125 | "### Tensorflow" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": null, 131 | "metadata": {}, 132 | "outputs": [], 133 | "source": [ 134 | "import tensorflow as tf\n", 135 | "a = tf.add(3,5)\n", 136 | "print(a)" 137 | ] 138 | }, 139 | { 140 | "cell_type": "code", 141 | "execution_count": null, 142 | "metadata": {}, 143 | "outputs": [], 144 | "source": [ 145 | "a = tf.add(3,5)\n", 146 | "sess = tf.Session()\n", 147 | "print(sess.run(a))\n", 148 | "sess.close()" 149 | ] 150 | }, 151 | { 152 | "cell_type": "code", 153 | "execution_count": null, 154 | "metadata": {}, 155 | "outputs": [], 156 | "source": [ 157 | "x = 2\n", 158 | "y = 3\n", 159 | "add_op = tf.add(x,y)\n", 160 | "mul_op = tf.multiply(x,y)\n", 161 | "useless = tf.multiply(x, add_op)\n", 162 | "pow_op = tf.pow(add_op, mul_op)\n", 163 | "with tf.Session() as sess:\n", 164 | " z, not_useless = sess.run([pow_op, useless])" 165 | ] 166 | }, 167 | { 168 | "cell_type": "code", 169 | "execution_count": null, 170 | "metadata": {}, 171 | "outputs": [], 172 | "source": [ 173 | "# create variables with tf.get variable\n", 174 | "s = tf.get_variable(\"scalar\", initializer=tf.constant(2))\n", 175 | "m = tf.get_variable(\"matrix\", initializer=tf.constant([[0,1],[2,3]]))\n", 176 | "W = tf.get_variable(\"big_matrix\", shape=(784,10), initializer=tf.zeros_initializer())" 177 | ] 178 | }, 179 | { 180 | "cell_type": "code", 181 | "execution_count": null, 182 | "metadata": {}, 183 | "outputs": [], 184 | "source": [ 185 | "with tf.Session() as sess:\n", 186 | " sess.run(tf.global_variables_initializer())" 187 | ] 188 | }, 189 | { 190 | "cell_type": "code", 191 | "execution_count": null, 192 | "metadata": {}, 193 | "outputs": [], 194 | "source": [ 195 | "a = tf.placeholder(tf.float32, shape=[3])\n", 196 | "\n", 197 | "b = tf.constant([5,5,5], tf.float32)\n", 198 | "\n", 199 | "c = a + b\n", 200 | "\n", 201 | "with tf.Session() as sess:\n", 202 | " print(sess.run(c,{a:[1,2,3]}))" 203 | ] 204 | }, 205 | { 206 | "cell_type": "markdown", 207 | "metadata": {}, 208 | "source": [ 209 | "### Linear Regression with Gradient Descent" 210 | ] 211 | }, 212 | { 213 | "cell_type": "code", 214 | "execution_count": null, 215 | "metadata": {}, 216 | "outputs": [], 217 | "source": [ 218 | "import tensorflow as tf\n", 219 | "\n", 220 | "# define constant\n", 221 | "c = tf.constant(0)\n", 222 | "\n", 223 | "# define variables (not initalized)\n", 224 | "w = tf.Variable(tf.random_uniform([1],-0.1,0.1))\n", 225 | "b = tf.Variable(tf.zeros([1]))\n", 226 | "\n", 227 | "# define placeholder\n", 228 | "x_data_placeholder = tf.placeholder('float32', shape=x_data.shape)\n", 229 | "y_data_placeholder = tf.placeholder('float32', shape=y_data.shape)\n", 230 | "\n", 231 | "# define loss function\n", 232 | "y_regression = w * x_data_placeholder + b\n", 233 | "\n", 234 | "loss = \n", 235 | "\n", 236 | "# define optimizer\n", 237 | "train = tf.train.GradientDescentOptimizer(learning_rate=0.7).minimize(loss)\n", 238 | "\n", 239 | "# initialize variables\n", 240 | "init = tf.global_variables_initializer()\n", 241 | "\n", 242 | "sess = tf.Session()\n", 243 | "sess.run(init) # run variable initializer : now you can use tensorflow variables" 244 | ] 245 | }, 246 | { 247 | "cell_type": "code", 248 | "execution_count": null, 249 | "metadata": {}, 250 | "outputs": [], 251 | "source": [ 252 | "# see initial variables\n", 253 | "my_feed_dict = {x_data_placeholder:x_data, y_data_placeholder:y_data}\n", 254 | "\n", 255 | "initial_w, initial_b = sess.run([w,b], feed_dict=my_feed_dict)\n", 256 | "\n", 257 | "print(\"Initial variables w : {}, b : {}\".format(initial_w, initial_b)) " 258 | ] 259 | }, 260 | { 261 | "cell_type": "code", 262 | "execution_count": null, 263 | "metadata": { 264 | "scrolled": false 265 | }, 266 | "outputs": [], 267 | "source": [ 268 | "plt.figure(3)\n", 269 | "plt.xlabel('x-axis') \n", 270 | "plt.ylabel('y-axis')\n", 271 | "plt.title('Linear Regression with Gradient Descent')\n", 272 | "\n", 273 | "plt.plot(x_data, y_data, 'ro')\n", 274 | "plt.plot(x_data, (initial_w * x_data + initial_b), label='initial')\n", 275 | "\n", 276 | "# training step\n", 277 | "\n", 278 | "for i in range():\n", 279 | " sess.run() # train operator\n", 280 | " current_w, current_b, current_loss = sess.run()# variable and loss \n", 281 | " print(\"iteration : {}, w : {}, b : {}, loss : {}\".format(i,current_w, current_b, current_loss))\n", 282 | " \n", 283 | " # display\n", 284 | " plt.plot(x_data, (current_w * x_data + current_b), label='iter : {}'.format(i))\n", 285 | " \n", 286 | "plt.legend()\n", 287 | "\n", 288 | "plt.show()" 289 | ] 290 | } 291 | ], 292 | "metadata": { 293 | "celltoolbar": "Raw Cell Format", 294 | "kernelspec": { 295 | "display_name": "Python 3", 296 | "language": "python", 297 | "name": "python3" 298 | }, 299 | "language_info": { 300 | "codemirror_mode": { 301 | "name": "ipython", 302 | "version": 3 303 | }, 304 | "file_extension": ".py", 305 | "mimetype": "text/x-python", 306 | "name": "python", 307 | "nbconvert_exporter": "python", 308 | "pygments_lexer": "ipython3", 309 | "version": "3.6.8" 310 | } 311 | }, 312 | "nbformat": 4, 313 | "nbformat_minor": 2 314 | } 315 | -------------------------------------------------------------------------------- /code/Tutorial2_MLP_MNIST.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## IMPORT PACKAGES" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 3, 13 | "metadata": {}, 14 | "outputs": [ 15 | { 16 | "name": "stdout", 17 | "output_type": "stream", 18 | "text": [ 19 | "CURRENT TF VERSION IS [1.2.1]\n", 20 | "PACKAGES LOADED\n" 21 | ] 22 | } 23 | ], 24 | "source": [ 25 | "import numpy as np\n", 26 | "import tensorflow as tf\n", 27 | "import matplotlib.pyplot as plt\n", 28 | "from tensorflow.examples.tutorials.mnist import input_data\n", 29 | "print (\"CURRENT TF VERSION IS [%s]\" % (tf.__version__))\n", 30 | "print (\"PACKAGES LOADED\")" 31 | ] 32 | }, 33 | { 34 | "cell_type": "markdown", 35 | "metadata": {}, 36 | "source": [ 37 | "## LOAD MNIST DATA" 38 | ] 39 | }, 40 | { 41 | "cell_type": "code", 42 | "execution_count": null, 43 | "metadata": { 44 | "collapsed": true 45 | }, 46 | "outputs": [], 47 | "source": [ 48 | "mnist = input_data.read_data_sets('data/', one_hot=True)" 49 | ] 50 | }, 51 | { 52 | "cell_type": "markdown", 53 | "metadata": {}, 54 | "source": [ 55 | "## INVESTIGATE MNIST DATA" 56 | ] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "execution_count": null, 61 | "metadata": { 62 | "collapsed": true 63 | }, 64 | "outputs": [], 65 | "source": [ 66 | "def print_np(x):\n", 67 | " print ('SHAPE OF is %s' % (x.shape,))\n", 68 | " print ('VALUES LOOK LIKE \\n %s' % (x))" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": null, 74 | "metadata": { 75 | "collapsed": true 76 | }, 77 | "outputs": [], 78 | "source": [ 79 | "print_np(mnist.train.images)\n", 80 | "print_np(mnist.train.labels)" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": null, 86 | "metadata": { 87 | "collapsed": true 88 | }, 89 | "outputs": [], 90 | "source": [ 91 | "print_np(mnist.test.images)\n", 92 | "print_np(mnist.test.labels)" 93 | ] 94 | }, 95 | { 96 | "cell_type": "code", 97 | "execution_count": null, 98 | "metadata": { 99 | "collapsed": true 100 | }, 101 | "outputs": [], 102 | "source": [ 103 | "print_np(mnist.validation.images)\n", 104 | "print_np(mnist.validation.labels)" 105 | ] 106 | }, 107 | { 108 | "cell_type": "markdown", 109 | "metadata": {}, 110 | "source": [ 111 | "## DRAW MNIST DATA " 112 | ] 113 | }, 114 | { 115 | "cell_type": "code", 116 | "execution_count": null, 117 | "metadata": { 118 | "collapsed": true 119 | }, 120 | "outputs": [], 121 | "source": [ 122 | "ntrain = mnist.train.images.shape[0]\n", 123 | "nsample = 3\n", 124 | "randidx = np.random.randint(ntrain, size=nsample)\n", 125 | "for i in randidx:\n", 126 | " imgvec = mnist.train.images[i, :]\n", 127 | " labelvec = mnist.train.labels[i, :]\n", 128 | " img = np.reshape(imgvec, (28, 28))\n", 129 | " label = np.argmax(labelvec)\n", 130 | " plt.matshow(img, cmap=plt.get_cmap('gray'))\n", 131 | " plt.title(\"[%d] DATA / LABEL IS [%d]\"%(i, label))" 132 | ] 133 | }, 134 | { 135 | "cell_type": "markdown", 136 | "metadata": {}, 137 | "source": [ 138 | "## GET RANDOM BATCH" 139 | ] 140 | }, 141 | { 142 | "cell_type": "code", 143 | "execution_count": null, 144 | "metadata": { 145 | "collapsed": true 146 | }, 147 | "outputs": [], 148 | "source": [ 149 | "ntrain = 10\n", 150 | "randindices = np.random.permutation(ntrain)\n", 151 | "print (randindices.shape)" 152 | ] 153 | }, 154 | { 155 | "cell_type": "code", 156 | "execution_count": null, 157 | "metadata": { 158 | "collapsed": true 159 | }, 160 | "outputs": [], 161 | "source": [ 162 | "ntrain = 10\n", 163 | "nbatch = 4\n", 164 | "niter = ntrain // nbatch + 1\n", 165 | "for i in range(niter):\n", 166 | " currindices = randindices[i*nbatch:(i+1)*nbatch]\n", 167 | " print (\"ITER: [%d] BATCH INDEX: %s\" % (i, currindices))\n", 168 | " # GET BATCH\n", 169 | " xbatch = mnist.train.images[currindices, :]\n", 170 | " ybatch = mnist.train.labels[currindices, :]\n", 171 | " print (\" - SHAPE OF 'XBATCH' IS %s\" % (xbatch.shape,))\n", 172 | " print (\" - SHAPE OF 'YBATCH' IS %s\" % (ybatch.shape,))" 173 | ] 174 | }, 175 | { 176 | "cell_type": "markdown", 177 | "metadata": {}, 178 | "source": [ 179 | "## DEFINE MODEL" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": null, 185 | "metadata": { 186 | "collapsed": true 187 | }, 188 | "outputs": [], 189 | "source": [ 190 | "# NETWORK TOPOLOGIES\n", 191 | "n_hidden_1 = 256 \n", 192 | "n_hidden_2 = 128 \n", 193 | "n_input = 784 \n", 194 | "n_classes = 10\n", 195 | "\n", 196 | "# INPUTS AND OUTPUTS\n", 197 | "x = # [EXERCISE 1]\n", 198 | "y = # [EXERCISE 1]\n", 199 | " \n", 200 | "# NETWORK PARAMETERS\n", 201 | "stddev = 0.1\n", 202 | "weights = {\n", 203 | " 'h1': # [EXERCISE 1],\n", 204 | " 'h2': # [EXERCISE 1],\n", 205 | " 'out': # [EXERCISE 1]\n", 206 | "}\n", 207 | "biases = {\n", 208 | " 'b1': # [EXERCISE 1],\n", 209 | " 'b2': # [EXERCISE 1],\n", 210 | " 'out': # [EXERCISE 1]\n", 211 | "}\n", 212 | "print (\"NETWORK READY\")" 213 | ] 214 | }, 215 | { 216 | "cell_type": "markdown", 217 | "metadata": {}, 218 | "source": [ 219 | "## BUILD GRAPH" 220 | ] 221 | }, 222 | { 223 | "cell_type": "code", 224 | "execution_count": null, 225 | "metadata": { 226 | "collapsed": true 227 | }, 228 | "outputs": [], 229 | "source": [ 230 | "# MODEL\n", 231 | "def multilayer_perceptron(_X, _weights, _biases):\n", 232 | " # [EXERCISE 2]\n", 233 | "\n", 234 | "# PREDICTION\n", 235 | "pred = multilayer_perceptron(x, weights, biases)\n", 236 | "\n", 237 | "# LOSS AND OPTIMIZER\n", 238 | "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=# [EXERCISE 3], logits=# [EXERCISE 3]))\n", 239 | "optm = tf.train.AdamOptimizer(learning_rate=0.001).minimize(cost) \n", 240 | "corr = # [EXERCISE 3]\n", 241 | "accr = tf.reduce_mean(tf.cast(corr, \"float\"))\n", 242 | "\n", 243 | "# INITIALIZER\n", 244 | "init = tf.global_variables_initializer()\n", 245 | "print (\"FUNCTIONS READY\")" 246 | ] 247 | }, 248 | { 249 | "cell_type": "markdown", 250 | "metadata": {}, 251 | "source": [ 252 | "## RUN" 253 | ] 254 | }, 255 | { 256 | "cell_type": "code", 257 | "execution_count": null, 258 | "metadata": { 259 | "collapsed": true 260 | }, 261 | "outputs": [], 262 | "source": [ 263 | "# PARAMETERS\n", 264 | "training_epochs = 20\n", 265 | "batch_size = 100\n", 266 | "display_step = 4\n", 267 | "# LAUNCH THE GRAPH\n", 268 | "sess = tf.Session()\n", 269 | "sess.run(init)\n", 270 | "# OPTIMIZE\n", 271 | "for epoch in range(training_epochs):\n", 272 | " avg_cost = 0.\n", 273 | " total_batch = int(mnist.train.num_examples/batch_size)\n", 274 | " # ITERATION\n", 275 | " for i in range(total_batch):\n", 276 | " batch_xs, batch_ys = mnist.train.next_batch(batch_size)\n", 277 | " feeds = {x: batch_xs, y: batch_ys}\n", 278 | " sess.run(optm, feed_dict=feeds)\n", 279 | " avg_cost += sess.run(cost, feed_dict=feeds)\n", 280 | " avg_cost = avg_cost / total_batch\n", 281 | " # DISPLAY\n", 282 | " if (epoch+1) % display_step == 0:\n", 283 | " print (\"Epoch: %03d/%03d cost: %.9f\" % (epoch, training_epochs, avg_cost))\n", 284 | " feeds = {x: batch_xs, y: batch_ys}\n", 285 | " train_acc = sess.run(accr, feed_dict=feeds)\n", 286 | " print (\"TRAIN ACCURACY: %.3f\" % (train_acc))\n", 287 | " feeds = {x: mnist.test.images, y: mnist.test.labels}\n", 288 | " test_acc = sess.run(accr, feed_dict=feeds)\n", 289 | " print (\"TEST ACCURACY: %.3f\" % (test_acc))\n", 290 | "print (\"OPTIMIZATION FINISHED\")" 291 | ] 292 | } 293 | ], 294 | "metadata": { 295 | "kernelspec": { 296 | "display_name": "Python [conda env:tensorflow]", 297 | "language": "python", 298 | "name": "conda-env-tensorflow-py" 299 | }, 300 | "language_info": { 301 | "codemirror_mode": { 302 | "name": "ipython", 303 | "version": 3 304 | }, 305 | "file_extension": ".py", 306 | "mimetype": "text/x-python", 307 | "name": "python", 308 | "nbconvert_exporter": "python", 309 | "pygments_lexer": "ipython3", 310 | "version": "3.6.1" 311 | } 312 | }, 313 | "nbformat": 4, 314 | "nbformat_minor": 2 315 | } 316 | -------------------------------------------------------------------------------- /code/Tutorial3-1_CNN_MNIST.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## CONVOLUTIONAL NEURAL NETWORK (MNIST)" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": null, 13 | "metadata": {}, 14 | "outputs": [], 15 | "source": [ 16 | "import os\n", 17 | "import numpy as np\n", 18 | "import matplotlib.pyplot as plt\n", 19 | "import tensorflow as tf\n", 20 | "from tensorflow.examples.tutorials.mnist import input_data\n", 21 | "%matplotlib inline \n", 22 | "print (\"CURRENT TF VERSION IS [%s]\" % (tf.__version__))\n", 23 | "print (\"PACKAGES LOADED\")" 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "## LOAD MNIST" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": null, 36 | "metadata": {}, 37 | "outputs": [], 38 | "source": [ 39 | "mnist = input_data.read_data_sets('data/', one_hot=True)\n", 40 | "trainimg = mnist.train.images\n", 41 | "trainlabel = mnist.train.labels\n", 42 | "testimg = mnist.test.images\n", 43 | "testlabel = mnist.test.labels\n", 44 | "print (\"MNIST ready\")" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": {}, 50 | "source": [ 51 | "## DEFINE MODEL" 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": null, 57 | "metadata": {}, 58 | "outputs": [], 59 | "source": [ 60 | "# NETWORK TOPOLOGIES\n", 61 | "n_input = 784\n", 62 | "n_channel = 64 \n", 63 | "n_classes = 10 \n", 64 | "\n", 65 | "# INPUTS AND OUTPUTS\n", 66 | "x = tf.placeholder(\"float\", [None, n_input])\n", 67 | "y = tf.placeholder(\"float\", [None, n_classes])\n", 68 | " \n", 69 | "# NETWORK PARAMETERS\n", 70 | "stddev = 0.1\n", 71 | "weights = {\n", 72 | " 'c1': tf.Variable(tf.random_normal([7, 7, 1, n_channel], stddev=stddev)),\n", 73 | " 'd1': tf.Variable(tf.random_normal([14*14*64, n_classes], stddev=stddev))\n", 74 | "}\n", 75 | "biases = {\n", 76 | " 'c1': tf.Variable(tf.random_normal([n_channel], stddev=stddev)),\n", 77 | " 'd1': tf.Variable(tf.random_normal([n_classes], stddev=stddev))\n", 78 | "}\n", 79 | "print (\"NETWORK READY\")" 80 | ] 81 | }, 82 | { 83 | "cell_type": "markdown", 84 | "metadata": {}, 85 | "source": [ 86 | "## DEFINE GRAPH" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": null, 92 | "metadata": { 93 | "collapsed": true 94 | }, 95 | "outputs": [], 96 | "source": [ 97 | "# MODEL\n", 98 | "def CNN(_x, _w, _b):\n", 99 | " # RESHAPE\n", 100 | " _x_r = tf.reshape(_x, shape=[-1, 28, 28, 1])\n", 101 | " # TODO1: CONVOLUTION\n", 102 | " # _conv1 = \n", 103 | " # ADD BIAS\n", 104 | " _conv2 = tf.nn.bias_add(_conv1, _b['c1'])\n", 105 | " # RELU\n", 106 | " _conv3 = tf.nn.relu(_conv2)\n", 107 | " # TODO2: MAX-POOL\n", 108 | " #_pool = \n", 109 | " # VECTORIZE\n", 110 | " _dense = tf.reshape(_pool, [-1, _w['d1'].get_shape().as_list()[0]])\n", 111 | " # DENSE\n", 112 | " _logit = tf.add(tf.matmul(_dense, _w['d1']), _b['d1'])\n", 113 | " _out = {\n", 114 | " 'x_r': _x_r, 'conv1': _conv1, 'conv2': _conv2, 'conv3': _conv3\n", 115 | " , 'pool': _pool, 'dense': _dense, 'logit': _logit\n", 116 | " }\n", 117 | " return _out\n", 118 | "\n", 119 | "# PREDICTION\n", 120 | "cnnout = CNN(x, weights, biases)\n", 121 | "\n", 122 | "# LOSS AND OPTIMIZER\n", 123 | "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n", 124 | " labels=y, logits=cnnout['logit']))\n", 125 | "optm = tf.train.AdamOptimizer(learning_rate=0.001).minimize(cost) \n", 126 | "corr = tf.equal(tf.argmax(cnnout['logit'], 1), tf.argmax(y, 1)) \n", 127 | "accr = tf.reduce_mean(tf.cast(corr, \"float\"))\n", 128 | "\n", 129 | "# INITIALIZER\n", 130 | "init = tf.global_variables_initializer()\n", 131 | "print (\"FUNCTIONS READY\")" 132 | ] 133 | }, 134 | { 135 | "cell_type": "markdown", 136 | "metadata": {}, 137 | "source": [ 138 | "## SAVER" 139 | ] 140 | }, 141 | { 142 | "cell_type": "code", 143 | "execution_count": null, 144 | "metadata": {}, 145 | "outputs": [], 146 | "source": [ 147 | "dirpath = \"data/nets\"\n", 148 | "if not os.path.exists(dirpath):\n", 149 | " os.makedirs(dirpath)\n", 150 | "savedir = \"data/nets/cnn_mnist_simple/\"\n", 151 | "saver = tf.train.Saver(max_to_keep=3)\n", 152 | "save_step = 5\n", 153 | "if not os.path.exists(savedir):\n", 154 | " os.makedirs(savedir)\n", 155 | "print (\"SAVER READY\")" 156 | ] 157 | }, 158 | { 159 | "cell_type": "markdown", 160 | "metadata": {}, 161 | "source": [ 162 | "## RUN" 163 | ] 164 | }, 165 | { 166 | "cell_type": "code", 167 | "execution_count": null, 168 | "metadata": {}, 169 | "outputs": [], 170 | "source": [ 171 | "# PARAMETERS\n", 172 | "training_epochs = 5\n", 173 | "batch_size = 32\n", 174 | "display_step = 1\n", 175 | "# LAUNCH THE GRAPH\n", 176 | "sess = tf.Session()\n", 177 | "sess.run(init)\n", 178 | "# OPTIMIZE\n", 179 | "for epoch in range(training_epochs):\n", 180 | " avg_cost = 0.\n", 181 | " total_batch = int(mnist.train.num_examples/batch_size)\n", 182 | " # ITERATION\n", 183 | " for i in range(total_batch):\n", 184 | " # TODO3: Fill in the for loop where we run the graph, compute the cost, and optimize.\n", 185 | " #\n", 186 | " #\n", 187 | " #\n", 188 | " #\n", 189 | " avg_cost = avg_cost / total_batch\n", 190 | " # DISPLAY\n", 191 | " if (epoch+1) % display_step == 0:\n", 192 | " print (\"Epoch: %03d/%03d cost: %.9f\" % (epoch+1, training_epochs, avg_cost))\n", 193 | " feeds = {x: batch_xs, y: batch_ys}\n", 194 | " train_acc = sess.run(accr, feed_dict=feeds)\n", 195 | " print (\"TRAIN ACCURACY: %.3f\" % (train_acc))\n", 196 | " feeds = {x: mnist.test.images, y: mnist.test.labels}\n", 197 | " test_acc = sess.run(accr, feed_dict=feeds)\n", 198 | " print (\"TEST ACCURACY: %.3f\" % (test_acc))\n", 199 | " # SAVE\n", 200 | " if (epoch+1) % save_step == 0:\n", 201 | " savename = savedir+\"net-\"+str(epoch+1)+\".ckpt\"\n", 202 | " saver.save(sess, savename)\n", 203 | " print (\"[%s] SAVED.\" % (savename))\n", 204 | "print (\"OPTIMIZATION FINISHED\")" 205 | ] 206 | }, 207 | { 208 | "cell_type": "markdown", 209 | "metadata": {}, 210 | "source": [ 211 | "## RESTORE" 212 | ] 213 | }, 214 | { 215 | "cell_type": "code", 216 | "execution_count": null, 217 | "metadata": {}, 218 | "outputs": [], 219 | "source": [ 220 | "do_restore = 0\n", 221 | "if do_restore == 1:\n", 222 | " sess = tf.Session()\n", 223 | " epoch = 5\n", 224 | " savename = savedir+\"net-\"+str(epoch)+\".ckpt\"\n", 225 | " saver.restore(sess, savename)\n", 226 | " print (\"NETWORK RESTORED\")\n", 227 | "else:\n", 228 | " print (\"DO NOTHING\")" 229 | ] 230 | }, 231 | { 232 | "cell_type": "markdown", 233 | "metadata": {}, 234 | "source": [ 235 | "## SEE HOW IT WORKS" 236 | ] 237 | }, 238 | { 239 | "cell_type": "code", 240 | "execution_count": null, 241 | "metadata": { 242 | "collapsed": true 243 | }, 244 | "outputs": [], 245 | "source": [ 246 | "# FEEDFORWARD THE FIRST TRAIN IMAGE\n", 247 | "input_r = sess.run(cnnout['x_r'], feed_dict={x: trainimg[0:1, :]})\n", 248 | "conv1 = sess.run(cnnout['conv1'], feed_dict={x: trainimg[0:1, :]})\n", 249 | "conv2 = sess.run(cnnout['conv2'], feed_dict={x: trainimg[0:1, :]})\n", 250 | "conv3 = sess.run(cnnout['conv3'], feed_dict={x: trainimg[0:1, :]})\n", 251 | "pool = sess.run(cnnout['pool'], feed_dict={x: trainimg[0:1, :]})\n", 252 | "dense = sess.run(cnnout['dense'], feed_dict={x: trainimg[0:1, :]})\n", 253 | "out = sess.run(cnnout['logit'], feed_dict={x: trainimg[0:1, :]})" 254 | ] 255 | }, 256 | { 257 | "cell_type": "markdown", 258 | "metadata": {}, 259 | "source": [ 260 | "## INPUT" 261 | ] 262 | }, 263 | { 264 | "cell_type": "code", 265 | "execution_count": null, 266 | "metadata": {}, 267 | "outputs": [], 268 | "source": [ 269 | "print (\"Size of 'input_r' is %s\" % (input_r.shape,))\n", 270 | "label = np.argmax(trainlabel[0, :])\n", 271 | "print (\"Label is %d\" % (label))\n", 272 | "\n", 273 | "# PLOT\n", 274 | "plt.matshow(input_r[0, :, :, 0], cmap=plt.get_cmap('gray'))\n", 275 | "plt.title(\"Label of this image is \" + str(label) + \"\")\n", 276 | "plt.colorbar()\n", 277 | "plt.show()" 278 | ] 279 | }, 280 | { 281 | "cell_type": "markdown", 282 | "metadata": {}, 283 | "source": [ 284 | "## CONV" 285 | ] 286 | }, 287 | { 288 | "cell_type": "code", 289 | "execution_count": null, 290 | "metadata": { 291 | "scrolled": true 292 | }, 293 | "outputs": [], 294 | "source": [ 295 | "print (\"SIZE OF 'CONV1' IS %s\" % (conv1.shape,))\n", 296 | "for i in range(3):\n", 297 | " plt.matshow(conv1[0, :, :, i], cmap=plt.get_cmap('gray'))\n", 298 | " plt.title(str(i) + \"th conv1\")\n", 299 | " plt.colorbar()\n", 300 | " plt.show()" 301 | ] 302 | }, 303 | { 304 | "cell_type": "markdown", 305 | "metadata": {}, 306 | "source": [ 307 | "## CONV + BIAS" 308 | ] 309 | }, 310 | { 311 | "cell_type": "code", 312 | "execution_count": null, 313 | "metadata": {}, 314 | "outputs": [], 315 | "source": [ 316 | "print (\"SIZE OF 'CONV2' IS %s\" % (conv2.shape,))\n", 317 | "for i in range(3):\n", 318 | " plt.matshow(conv2[0, :, :, i], cmap=plt.get_cmap('gray'))\n", 319 | " plt.title(str(i) + \"th conv2\")\n", 320 | " plt.colorbar()\n", 321 | " plt.show()" 322 | ] 323 | }, 324 | { 325 | "cell_type": "markdown", 326 | "metadata": {}, 327 | "source": [ 328 | "## CONV + BIAS + RELU" 329 | ] 330 | }, 331 | { 332 | "cell_type": "code", 333 | "execution_count": null, 334 | "metadata": {}, 335 | "outputs": [], 336 | "source": [ 337 | "print (\"SIZE OF 'CONV3' IS %s\" % (conv3.shape,))\n", 338 | "for i in range(3):\n", 339 | " plt.matshow(conv3[0, :, :, i], cmap=plt.get_cmap('gray'))\n", 340 | " plt.title(str(i) + \"th conv3\")\n", 341 | " plt.colorbar()\n", 342 | " plt.show()" 343 | ] 344 | }, 345 | { 346 | "cell_type": "markdown", 347 | "metadata": {}, 348 | "source": [ 349 | "## POOL" 350 | ] 351 | }, 352 | { 353 | "cell_type": "code", 354 | "execution_count": null, 355 | "metadata": {}, 356 | "outputs": [], 357 | "source": [ 358 | "print (\"SIZE OF 'POOL' IS %s\" % (pool.shape,))\n", 359 | "for i in range(3):\n", 360 | " plt.matshow(pool[0, :, :, i], cmap=plt.get_cmap('gray'))\n", 361 | " plt.title(str(i) + \"th pool\")\n", 362 | " plt.colorbar()\n", 363 | " plt.show()" 364 | ] 365 | }, 366 | { 367 | "cell_type": "markdown", 368 | "metadata": {}, 369 | "source": [ 370 | "## DENSE" 371 | ] 372 | }, 373 | { 374 | "cell_type": "code", 375 | "execution_count": null, 376 | "metadata": {}, 377 | "outputs": [], 378 | "source": [ 379 | "print (\"SIZE OF 'DENSE' IS %s\" % (dense.shape,))\n", 380 | "print (\"SIZE OF 'OUT' IS %s\" % (out.shape,))\n", 381 | "plt.matshow(out, cmap=plt.get_cmap('gray'))\n", 382 | "plt.title(\"OUT\")\n", 383 | "plt.colorbar()\n", 384 | "plt.show()\n", 385 | "\n", 386 | "print (\"PREDICTED LABEL IS [%d]\" % np.argmax(out))" 387 | ] 388 | }, 389 | { 390 | "cell_type": "markdown", 391 | "metadata": {}, 392 | "source": [ 393 | "## CONV FILTER" 394 | ] 395 | }, 396 | { 397 | "cell_type": "code", 398 | "execution_count": null, 399 | "metadata": {}, 400 | "outputs": [], 401 | "source": [ 402 | "wc1 = sess.run(weights['c1'])\n", 403 | "print (\"SIZE OF 'WC1' IS %s\" % (wc1.shape,))\n", 404 | "for i in range(3):\n", 405 | " plt.matshow(wc1[:, :, 0, i], cmap=plt.get_cmap('gray'))\n", 406 | " plt.title(str(i) + \"th conv filter\")\n", 407 | " plt.colorbar()\n", 408 | " plt.show()" 409 | ] 410 | } 411 | ], 412 | "metadata": { 413 | "kernelspec": { 414 | "display_name": "Python 3", 415 | "language": "python", 416 | "name": "python3" 417 | }, 418 | "language_info": { 419 | "codemirror_mode": { 420 | "name": "ipython", 421 | "version": 3 422 | }, 423 | "file_extension": ".py", 424 | "mimetype": "text/x-python", 425 | "name": "python", 426 | "nbconvert_exporter": "python", 427 | "pygments_lexer": "ipython3", 428 | "version": "3.5.4" 429 | } 430 | }, 431 | "nbformat": 4, 432 | "nbformat_minor": 1 433 | } 434 | -------------------------------------------------------------------------------- /code/Tutorial3-2_CNN_CIFAR10.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Deep Learning Tutorial
Day 4. CIFAR10 Classification\n", 8 | "\n", 9 | "Written by Nuri Kim.\n", 10 | "This code can be downloaded from [GitHub](https://github.com/bareblackfoot/deep_learning_tutorial).\n", 11 | "\n", 12 | "References of this notebook: \n", 13 | "*TensorFlow-Tutorials by [Magnus Erik Hvass Pedersen](http://www.hvass-labs.org/)* and \n", 14 | "*CIFAR-10 by [dhanushkamath](https://github.com/dhanushkamath/CIFAR-10/blob/master/Cifar10.ipynb)*" 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "## Introduction\n", 22 | "\n", 23 | "This tutorial demonstrates the workflow of an image classification using TensorFlow with convolutional networks. Unlike MNIST dataset, CIFAR10 dataset has color images of 10 categories. Here, we define and optimize a simple mathematical model in TensorFlow. The results are then plotted." 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "## Imports" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": null, 36 | "metadata": {}, 37 | "outputs": [], 38 | "source": [ 39 | "from glob import glob\n", 40 | "from time import time\n", 41 | "from sklearn.manifold import TSNE\n", 42 | "from sklearn.preprocessing import LabelEncoder\n", 43 | "from sklearn.metrics import confusion_matrix\n", 44 | "try:\n", 45 | " from urllib.request import urlretrieve\n", 46 | "except:\n", 47 | " from urllib import urlretrieve\n", 48 | "import pandas as pd\n", 49 | "import numpy as np\n", 50 | "import sys, os, math\n", 51 | "import tensorflow as tf\n", 52 | "import tensorflow.contrib.slim as slim\n", 53 | "import matplotlib.pyplot as plt\n", 54 | "import itertools\n", 55 | "import tarfile\n", 56 | "import pickle\n", 57 | "\n", 58 | "print (\"Packages loaded\")" 59 | ] 60 | }, 61 | { 62 | "cell_type": "markdown", 63 | "metadata": {}, 64 | "source": [ 65 | "### Set parameters" 66 | ] 67 | }, 68 | { 69 | "cell_type": "code", 70 | "execution_count": null, 71 | "metadata": {}, 72 | "outputs": [], 73 | "source": [ 74 | "# PARAMS\n", 75 | "_BATCH_SIZE = 32\n", 76 | "_EPOCH = 10\n", 77 | "_SNAPSHOT_PREFIX = \"cifar10\"\n", 78 | "_NUM_VIS_EMBEDDING = 500\n", 79 | "_IMAGE_HEIGHT = 32\n", 80 | "_IMAGE_WIDTH = 32\n", 81 | "_IMAGE_CHANNELS = 3\n", 82 | "class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']\n", 83 | "_NUM_CLASSES = len(class_names)\n", 84 | "class_to_num = dict(zip(class_names, range(_NUM_CLASSES)))\n", 85 | "num_to_class = dict(zip(range(_NUM_CLASSES), class_names))\n", 86 | "lr = 1e-3" 87 | ] 88 | }, 89 | { 90 | "cell_type": "markdown", 91 | "metadata": {}, 92 | "source": [ 93 | "This was developed using Python 3.5 and TensorFlow version 1.10.1. Please check yours:" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": null, 99 | "metadata": {}, 100 | "outputs": [], 101 | "source": [ 102 | "tf.__version__" 103 | ] 104 | }, 105 | { 106 | "cell_type": "markdown", 107 | "metadata": {}, 108 | "source": [ 109 | "### Helper-functions for loading images" 110 | ] 111 | }, 112 | { 113 | "cell_type": "markdown", 114 | "metadata": {}, 115 | "source": [ 116 | "Functions used to download dataset and pre-process images." 117 | ] 118 | }, 119 | { 120 | "cell_type": "code", 121 | "execution_count": null, 122 | "metadata": {}, 123 | "outputs": [], 124 | "source": [ 125 | "def get_data_set(name=\"train\"):\n", 126 | " x = y = None\n", 127 | " maybe_download_and_extract()\n", 128 | " folder_name = \"cifar10\"\n", 129 | " f = open('./data/'+folder_name+'/batches.meta', 'rb')\n", 130 | " f.close()\n", 131 | "\n", 132 | " if name is \"train\":\n", 133 | " for i in range(5):\n", 134 | " f = open('./data/'+folder_name+'/data_batch_' + str(i + 1), 'rb')\n", 135 | " try:\n", 136 | " datadict = pickle.load(f, encoding='latin1') \n", 137 | " except:\n", 138 | " datadict = pickle.load(f)\n", 139 | " f.close()\n", 140 | "\n", 141 | " _X = datadict[\"data\"]\n", 142 | " _Y = datadict['labels']\n", 143 | "\n", 144 | " _X = np.array(_X, dtype=float) / 255.0\n", 145 | " _X = _X.reshape([-1, _IMAGE_CHANNELS, _IMAGE_HEIGHT, _IMAGE_WIDTH])\n", 146 | " _X = _X.transpose([0, 2, 3, 1])\n", 147 | " _X = _X.reshape(-1, _IMAGE_HEIGHT*_IMAGE_WIDTH*_IMAGE_CHANNELS)\n", 148 | "\n", 149 | " if x is None:\n", 150 | " x = _X\n", 151 | " y = _Y\n", 152 | " else:\n", 153 | " x = np.concatenate((x, _X), axis=0)\n", 154 | " y = np.concatenate((y, _Y), axis=0)\n", 155 | "\n", 156 | " elif name is \"test\":\n", 157 | " f = open('./data/'+folder_name+'/test_batch', 'rb')\n", 158 | " try:\n", 159 | " datadict = pickle.load(f, encoding='latin1') \n", 160 | " except:\n", 161 | " datadict = pickle.load(f)\n", 162 | " f.close()\n", 163 | " x = datadict[\"data\"]\n", 164 | " y = np.array(datadict['labels'])\n", 165 | " x = np.array(x, dtype=float) / 255.0\n", 166 | " x = x.reshape([-1, _IMAGE_CHANNELS, _IMAGE_HEIGHT, _IMAGE_WIDTH])\n", 167 | " x = x.transpose([0, 2, 3, 1])\n", 168 | " x = x.reshape(-1, _IMAGE_HEIGHT*_IMAGE_WIDTH*_IMAGE_CHANNELS)\n", 169 | "\n", 170 | " return x, dense_to_one_hot(y)\n", 171 | "\n", 172 | "def dense_to_one_hot(labels_dense, num_classes=10):\n", 173 | " num_labels = labels_dense.shape[0]\n", 174 | " index_offset = np.arange(num_labels) * num_classes\n", 175 | " labels_one_hot = np.zeros((num_labels, num_classes))\n", 176 | " labels_one_hot.flat[index_offset + labels_dense.ravel()] = 1\n", 177 | " return labels_one_hot\n", 178 | "\n", 179 | "def print_download_progress(count, block_size, total_size):\n", 180 | " pct_complete = float(count * block_size) / total_size\n", 181 | " msg = \"\\r- Download progress: {0:.1%}\".format(pct_complete)\n", 182 | " sys.stdout.write(msg)\n", 183 | " sys.stdout.flush()\n", 184 | " \n", 185 | "def maybe_download_and_extract():\n", 186 | " main_directory = \"./data/\"\n", 187 | " if not os.path.exists(main_directory):\n", 188 | " os.makedirs(main_directory)\n", 189 | " cifar_10_directory = main_directory+\"cifar10/\"\n", 190 | " if not os.path.exists(cifar_10_directory):\n", 191 | " url = \"http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\"\n", 192 | " filename = url.split('/')[-1]\n", 193 | " file_path = os.path.join(main_directory, filename)\n", 194 | " zip_cifar_10 = file_path\n", 195 | " file_path, _ = urlretrieve(url=url, filename=file_path, \n", 196 | " reporthook=print_download_progress)\n", 197 | "\n", 198 | " print()\n", 199 | " print(\"Download finished. Extracting files.\")\n", 200 | " if file_path.endswith(\".zip\"):\n", 201 | " zipfile.ZipFile(file=file_path, mode=\"r\").extractall(main_directory)\n", 202 | " elif file_path.endswith((\".tar.gz\", \".tgz\")):\n", 203 | " tarfile.open(name=file_path, mode=\"r:gz\").extractall(main_directory)\n", 204 | " print(\"Done.\")\n", 205 | "\n", 206 | " os.rename(main_directory+\"cifar-10-batches-py\", cifar_10_directory)\n", 207 | " os.remove(zip_cifar_10)" 208 | ] 209 | }, 210 | { 211 | "cell_type": "markdown", 212 | "metadata": {}, 213 | "source": [ 214 | "## Load Data\n", 215 | "The size of CIFAR10 dataset is about 180MB and it will be downloaded automatically if it is not located in the given path." 216 | ] 217 | }, 218 | { 219 | "cell_type": "code", 220 | "execution_count": null, 221 | "metadata": {}, 222 | "outputs": [], 223 | "source": [ 224 | "train_x, train_y = get_data_set(\"train\")\n", 225 | "test_x, test_y = get_data_set(\"test\")\n", 226 | "\n", 227 | "print (\"Dataset loaded\")" 228 | ] 229 | }, 230 | { 231 | "cell_type": "markdown", 232 | "metadata": {}, 233 | "source": [ 234 | "The CIFAR10 dataset has now been loaded and consists of 60000 32x32 color images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. " 235 | ] 236 | }, 237 | { 238 | "cell_type": "markdown", 239 | "metadata": {}, 240 | "source": [ 241 | "### Plot a few images to see if data is correct" 242 | ] 243 | }, 244 | { 245 | "cell_type": "code", 246 | "execution_count": null, 247 | "metadata": {}, 248 | "outputs": [], 249 | "source": [ 250 | "def show_images():\n", 251 | " fig, axes = plt.subplots(3, 3)\n", 252 | " fig.subplots_adjust(hspace=0.6, wspace=0.3)\n", 253 | " \n", 254 | " for i, ax in enumerate(axes.flat):\n", 255 | " # Plot image.\n", 256 | " ax.imshow(train_x[i].reshape(_IMAGE_HEIGHT, _IMAGE_WIDTH, _IMAGE_CHANNELS))\n", 257 | " \n", 258 | " # Name of the true class.\n", 259 | " cls_true_name = num_to_class[train_y[i].argmax()]\n", 260 | " xlabel = \"class: {0}\".format(cls_true_name)\n", 261 | " \n", 262 | " # Show the classes as the label on the x-axis.\n", 263 | " ax.set_xlabel(xlabel)\n", 264 | " \n", 265 | " # Remove ticks from the plot.\n", 266 | " ax.set_xticks([])\n", 267 | " ax.set_yticks([])\n", 268 | " return\n", 269 | "\n", 270 | "print(train_x.shape)\n", 271 | "print(train_y.shape)\n", 272 | "show_images()" 273 | ] 274 | }, 275 | { 276 | "cell_type": "markdown", 277 | "metadata": {}, 278 | "source": [ 279 | "## TensorFlow Graph\n", 280 | "\n", 281 | "The entire purpose of TensorFlow is to have a so-called computational graph that can be executed much more efficiently than if the same calculations were to be performed directly in Python. TensorFlow can be more efficient than NumPy because TensorFlow knows the entire computation graph that must be executed, while NumPy only knows the computation of a single mathematical operation at a time.\n", 282 | "\n", 283 | "TensorFlow can also automatically calculate the gradients that are needed to optimize the variables of the graph so as to make the model perform better. This is because the graph is a combination of simple mathematical expressions so the gradient of the entire graph can be calculated using the chain-rule for derivatives.\n", 284 | "\n", 285 | "TensorFlow can also take advantage of multi-core CPUs as well as GPUs.\n", 286 | "\n", 287 | "A TensorFlow graph consists of the following parts which will be detailed below:\n", 288 | "\n", 289 | "* Placeholder variables used to feed input into the graph.\n", 290 | "* Model variables that are going to be optimized so as to make the model perform better.\n", 291 | "* The model which is essentially just a mathematical function that calculates some output given the input in the placeholder variables and the model variables.\n", 292 | "* A loss measure that can be used to guide the optimization of the variables.\n", 293 | "* An optimization method which updates the variables of the model.\n", 294 | "\n", 295 | "In addition, the TensorFlow graph may also contain various debugging statements e.g. for logging data to be displayed using TensorBoard, which is not covered in this tutorial." 296 | ] 297 | }, 298 | { 299 | "cell_type": "markdown", 300 | "metadata": {}, 301 | "source": [ 302 | "### Placeholder variables" 303 | ] 304 | }, 305 | { 306 | "cell_type": "markdown", 307 | "metadata": {}, 308 | "source": [ 309 | "Placeholder variables serve as the input to the graph that we may change each time we execute the graph. We call this feeding the placeholder variables and it is demonstrated further below.\n", 310 | "\n", 311 | "First we define the placeholder variable for the input images. This allows us to change the images that are input to the TensorFlow graph. This is a so-called tensor, which just means that it is a multi-dimensional vector or matrix. The data-type is set to `float32` and the shape is set to `[None, _IMAGE_HEIGHT * _IMAGE_WIDTH * _IMAGE_CHANNEL]`, where `None` means that the tensor may hold an arbitrary number of images with each image being a vector of length `_IMAGE_HEIGHT * _IMAGE_WIDTH * _IMAGE_CHANNEL`." 312 | ] 313 | }, 314 | { 315 | "cell_type": "code", 316 | "execution_count": null, 317 | "metadata": {}, 318 | "outputs": [], 319 | "source": [ 320 | "x = tf.placeholder(tf.float32, shape=[None, _IMAGE_HEIGHT * _IMAGE_WIDTH * _IMAGE_CHANNELS], name='Input')" 321 | ] 322 | }, 323 | { 324 | "cell_type": "markdown", 325 | "metadata": {}, 326 | "source": [ 327 | "Next we have the placeholder variable for the true labels associated with the images that were input in the placeholder variable `x`. The shape of this placeholder variable is `[None, _NUM_CLASSES]` which means it may hold an arbitrary number of labels and each label is a vector of length `_NUM_CLASSES` which is 10 in this case." 328 | ] 329 | }, 330 | { 331 | "cell_type": "code", 332 | "execution_count": null, 333 | "metadata": {}, 334 | "outputs": [], 335 | "source": [ 336 | "y_true = tf.placeholder(tf.float32, shape=[None, _NUM_CLASSES], name='Output')" 337 | ] 338 | }, 339 | { 340 | "cell_type": "markdown", 341 | "metadata": {}, 342 | "source": [ 343 | "Since we use a dropout regularization method for a fully connected layer, we have the placeholder variable for the probability to keep variables. If `keep_probability` is 1.0, all variables are the same without the dropout regularization. When `keep_probability` is 0.5, half of the variables in a fully connected layer become 0.0. " 344 | ] 345 | }, 346 | { 347 | "cell_type": "code", 348 | "execution_count": null, 349 | "metadata": {}, 350 | "outputs": [], 351 | "source": [ 352 | "keep_probability = tf.placeholder(tf.float32, shape=[], name='keep_prob')" 353 | ] 354 | }, 355 | { 356 | "cell_type": "markdown", 357 | "metadata": {}, 358 | "source": [ 359 | "`global_step` saves how many optimization iterations are used." 360 | ] 361 | }, 362 | { 363 | "cell_type": "code", 364 | "execution_count": null, 365 | "metadata": {}, 366 | "outputs": [], 367 | "source": [ 368 | "global_step = tf.Variable(initial_value=0, trainable=False, name='global_step')" 369 | ] 370 | }, 371 | { 372 | "cell_type": "markdown", 373 | "metadata": {}, 374 | "source": [ 375 | "Tensorflow takes an image input in order of `[Batch, _IMAGE_HEIGHT, _IMAGE_WIDTH, _IMAGE_CHANNEL]`. We reshape the input image to make the tensorflow graph can understand the input. " 376 | ] 377 | }, 378 | { 379 | "cell_type": "code", 380 | "execution_count": null, 381 | "metadata": {}, 382 | "outputs": [], 383 | "source": [ 384 | "x_image = tf.reshape(x, [-1, _IMAGE_HEIGHT, _IMAGE_WIDTH, _IMAGE_CHANNELS], name='images')" 385 | ] 386 | }, 387 | { 388 | "cell_type": "markdown", 389 | "metadata": {}, 390 | "source": [ 391 | "### Network model to be optimized" 392 | ] 393 | }, 394 | { 395 | "cell_type": "markdown", 396 | "metadata": {}, 397 | "source": [ 398 | "The network consists of `4` convolutional networks and `2` fully connected layers. `conv1_1` layer consists of filters with weight size `[3, 3, 3, 32]` and bias for each output channel, size `[32]`. `conv1_2` layer consists of filters with weight size `[3, 3, 32, 32]` and bias for each output channel, size `[32]`. `conv2_1` layer contains filters with size `[3, 3, 32, 64]` and bias for each output channel, size `[64]`. `conv3_1` layer consists of filters with size `[3, 3, 64, 128]` and bias for each output channel, size `[128]`. `fully_connected` layer has a filter with weight size `[_IMAGE_HEIGHT/8 * _IMAGE_WIDTH/8 * 128, 1000]`, and bias for each output channel, size 1000. Lastly, the `feat` layer has a filter with weight size `[1000, 10]` and bias for each output channel, size `[10]`." 399 | ] 400 | }, 401 | { 402 | "cell_type": "code", 403 | "execution_count": null, 404 | "metadata": {}, 405 | "outputs": [], 406 | "source": [ 407 | "### [Exercise 1] Change the CNN architecture to VGG16\n", 408 | "feat = slim.repeat(x_image, 2, slim.conv2d, 32, [3, 3], scope='conv1')\n", 409 | "feat = slim.max_pool2d(feat, [2, 2], padding='SAME', scope='pool1')\n", 410 | "feat = slim.repeat(feat, 1, slim.conv2d, 64, [3, 3], scope='conv2')\n", 411 | "feat = slim.max_pool2d(feat, [2, 2], padding='SAME', scope='pool2')\n", 412 | "feat = slim.repeat(feat, 1, slim.conv2d, 128, [3, 3], scope='conv3')\n", 413 | "feat = slim.max_pool2d(feat, [2, 2], padding='SAME', scope='pool3')\n", 414 | "feat = slim.flatten(feat, scope='flatten')\n", 415 | "feat = slim.fully_connected(feat, 1000, scope='fully_connected')" 416 | ] 417 | }, 418 | { 419 | "cell_type": "code", 420 | "execution_count": null, 421 | "metadata": {}, 422 | "outputs": [], 423 | "source": [ 424 | "### [Exercise 2] Add a regularization method\n", 425 | "feat = " 426 | ] 427 | }, 428 | { 429 | "cell_type": "code", 430 | "execution_count": null, 431 | "metadata": {}, 432 | "outputs": [], 433 | "source": [ 434 | "y_pred = slim.fully_connected(feat, _NUM_CLASSES, scope='feat', activation_fn=None)\n", 435 | "softmax = tf.nn.softmax(y_pred)\n", 436 | "y_pred_cls = tf.argmax(softmax, axis=1)" 437 | ] 438 | }, 439 | { 440 | "cell_type": "markdown", 441 | "metadata": {}, 442 | "source": [ 443 | "Let's see the variables in the network model which we defined above." 444 | ] 445 | }, 446 | { 447 | "cell_type": "code", 448 | "execution_count": null, 449 | "metadata": {}, 450 | "outputs": [], 451 | "source": [ 452 | "print(tf.global_variables())" 453 | ] 454 | }, 455 | { 456 | "cell_type": "markdown", 457 | "metadata": {}, 458 | "source": [ 459 | "### Loss-function to be optimized" 460 | ] 461 | }, 462 | { 463 | "cell_type": "markdown", 464 | "metadata": {}, 465 | "source": [ 466 | "To make the model better at classifying the input images, we must somehow change the variables for `weights` and `biases`. To do this we first need to know how well the model currently performs by comparing the predicted output of the model `y_pred` to the desired output `y_true`.\n", 467 | "\n", 468 | "The cross-entropy is a performance measure used in classification. The cross-entropy is a continuous function that is always positive and if the predicted output of the model exactly matches the desired output then the cross-entropy equals zero. The goal of optimization is therefore to minimize the cross-entropy so it gets as close to zero as possible by changing the `weights` and `biases` of the model.\n", 469 | "\n", 470 | "TensorFlow has a built-in function for calculating the cross-entropy. Note that it uses the values of the `logits` because it also calculates the softmax internally." 471 | ] 472 | }, 473 | { 474 | "cell_type": "code", 475 | "execution_count": null, 476 | "metadata": {}, 477 | "outputs": [], 478 | "source": [ 479 | "try:\n", 480 | " cross_entropy = tf.nn.softmax_cross_entropy_with_logits_v2(logits=y_pred, labels=y_true)\n", 481 | "except:\n", 482 | " cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits=y_pred, labels=y_true)" 483 | ] 484 | }, 485 | { 486 | "cell_type": "markdown", 487 | "metadata": {}, 488 | "source": [ 489 | "We have now calculated the cross-entropy for each of the image classifications so we have a measure of how well the model performs on each image individually. But in order to use the cross-entropy to guide the optimization of the model's variables we need a single scalar value, so we simply take the average of the cross-entropy for all the image classifications." 490 | ] 491 | }, 492 | { 493 | "cell_type": "code", 494 | "execution_count": null, 495 | "metadata": {}, 496 | "outputs": [], 497 | "source": [ 498 | "loss = tf.reduce_mean(cross_entropy)" 499 | ] 500 | }, 501 | { 502 | "cell_type": "markdown", 503 | "metadata": {}, 504 | "source": [ 505 | "### Optimization method" 506 | ] 507 | }, 508 | { 509 | "cell_type": "markdown", 510 | "metadata": {}, 511 | "source": [ 512 | "Now that we have a loss measure that must be minimized, we can then create an optimizer. In this case it is the basic form of Gradient Descent where the learning rate (step-size) is set to 0.001.\n", 513 | "\n", 514 | "Note that optimization is not performed at this point. In fact, nothing is calculated at all, we just add the optimizer-object to the TensorFlow graph for later execution." 515 | ] 516 | }, 517 | { 518 | "cell_type": "code", 519 | "execution_count": null, 520 | "metadata": {}, 521 | "outputs": [], 522 | "source": [ 523 | "global_accuracy = 0\n", 524 | "epoch_start = 0\n", 525 | "\n", 526 | "### [Exercise 3] Write your own optimizer\n", 527 | "optimizer = " 528 | ] 529 | }, 530 | { 531 | "cell_type": "markdown", 532 | "metadata": {}, 533 | "source": [ 534 | "### Performance measures" 535 | ] 536 | }, 537 | { 538 | "cell_type": "markdown", 539 | "metadata": {}, 540 | "source": [ 541 | "We need a few more performance measures to display the progress to the user.\n", 542 | "\n", 543 | "This is a vector of booleans whether the predicted class equals the true class of each image." 544 | ] 545 | }, 546 | { 547 | "cell_type": "code", 548 | "execution_count": null, 549 | "metadata": {}, 550 | "outputs": [], 551 | "source": [ 552 | "correct_prediction = tf.equal(y_pred_cls, tf.argmax(y_true, axis=1))" 553 | ] 554 | }, 555 | { 556 | "cell_type": "markdown", 557 | "metadata": {}, 558 | "source": [ 559 | "This calculates the classification accuracy by first type-casting the vector of booleans to floats, so that False becomes 0 and True becomes 1, and then calculating the average of these numbers." 560 | ] 561 | }, 562 | { 563 | "cell_type": "code", 564 | "execution_count": null, 565 | "metadata": {}, 566 | "outputs": [], 567 | "source": [ 568 | "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))*100.0" 569 | ] 570 | }, 571 | { 572 | "cell_type": "markdown", 573 | "metadata": {}, 574 | "source": [ 575 | "## TensorFlow Run" 576 | ] 577 | }, 578 | { 579 | "cell_type": "markdown", 580 | "metadata": {}, 581 | "source": [ 582 | "### Create TensorFlow session\n", 583 | "\n", 584 | "Once the TensorFlow graph has been created, we have to create a TensorFlow session which is used to execute the graph." 585 | ] 586 | }, 587 | { 588 | "cell_type": "code", 589 | "execution_count": null, 590 | "metadata": {}, 591 | "outputs": [], 592 | "source": [ 593 | "sess = tf.Session()" 594 | ] 595 | }, 596 | { 597 | "cell_type": "markdown", 598 | "metadata": {}, 599 | "source": [ 600 | "### Initialize network parameters" 601 | ] 602 | }, 603 | { 604 | "cell_type": "markdown", 605 | "metadata": {}, 606 | "source": [ 607 | "The variables for `weights` and `biases` in convolutional networks and fully connected layers must be initialized before we start optimizing them." 608 | ] 609 | }, 610 | { 611 | "cell_type": "code", 612 | "execution_count": null, 613 | "metadata": {}, 614 | "outputs": [], 615 | "source": [ 616 | "sess.run(tf.global_variables_initializer())\n", 617 | "print(\"Initialized variables.\")" 618 | ] 619 | }, 620 | { 621 | "cell_type": "markdown", 622 | "metadata": {}, 623 | "source": [ 624 | "### Helper-functions to plot results" 625 | ] 626 | }, 627 | { 628 | "cell_type": "markdown", 629 | "metadata": {}, 630 | "source": [ 631 | "Function for plotting confusion matrix of test images." 632 | ] 633 | }, 634 | { 635 | "cell_type": "code", 636 | "execution_count": null, 637 | "metadata": {}, 638 | "outputs": [], 639 | "source": [ 640 | "def plot_confusion_matrix(cm, classes,\n", 641 | " normalize=False,\n", 642 | " title='Confusion matrix',\n", 643 | " cmap=plt.cm.Blues):\n", 644 | " \"\"\"\n", 645 | " This function prints and plots the confusion matrix.\n", 646 | " Normalization can be applied by setting `normalize=True`.\n", 647 | " \"\"\"\n", 648 | " print('Drawing confusion matrix...')\n", 649 | " if normalize:\n", 650 | " cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n", 651 | " print(\"Normalized confusion matrix\")\n", 652 | " else:\n", 653 | " print('Confusion matrix, without normalization')\n", 654 | "\n", 655 | " plt.imshow(cm, interpolation='nearest', cmap=cmap)\n", 656 | " plt.title(title)\n", 657 | " plt.colorbar()\n", 658 | " tick_marks = np.arange(len(classes))\n", 659 | " plt.xticks(tick_marks, classes, rotation=45)\n", 660 | " plt.yticks(tick_marks, classes)\n", 661 | "\n", 662 | " fmt = '.2f' if normalize else 'd'\n", 663 | " thresh = cm.max() / 2.\n", 664 | " for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n", 665 | " plt.text(j, i, format(cm[i, j], fmt),\n", 666 | " horizontalalignment=\"center\",\n", 667 | " color=\"white\" if cm[i, j] > thresh else \"black\")\n", 668 | "\n", 669 | " plt.tight_layout()\n", 670 | " plt.ylabel('True label')\n", 671 | " plt.xlabel('Predicted label')\n", 672 | " plt.show()" 673 | ] 674 | }, 675 | { 676 | "cell_type": "markdown", 677 | "metadata": {}, 678 | "source": [ 679 | "Function for plotting tsne examples of test images." 680 | ] 681 | }, 682 | { 683 | "cell_type": "code", 684 | "execution_count": null, 685 | "metadata": {}, 686 | "outputs": [], 687 | "source": [ 688 | "def plot_tsne(y_pred_val, y_true_val):\n", 689 | " numbered = LabelEncoder().fit_transform(list(y_true_val))\n", 690 | " c = plt.cm.get_cmap('hsv', len(set(numbered)))\n", 691 | " colors=[]\n", 692 | " for num in numbered:\n", 693 | " temp=c(num)\n", 694 | " colors.append(temp)\n", 695 | " \n", 696 | " print('Fitting TSNE...')\n", 697 | " X_embedded = TSNE(n_components=2).fit_transform(y_pred_val[:10000])\n", 698 | " X_embedded = np.array(X_embedded)\n", 699 | " vis_x = X_embedded[:, 0]\n", 700 | " vis_y = X_embedded[:, 1]\n", 701 | " numbered = LabelEncoder().fit_transform(list(y_true_val))\n", 702 | " c = plt.cm.get_cmap('hsv', len(set(numbered)))\n", 703 | " plt.figure(num=None, figsize=(5, 5), dpi=80, facecolor='w', edgecolor='k')\n", 704 | " plt.scatter(vis_x, vis_y, c=colors,alpha=1,linewidths=1,s=10, edgecolors='none')\n", 705 | " plt.show()\n", 706 | " return X_embedded" 707 | ] 708 | }, 709 | { 710 | "cell_type": "markdown", 711 | "metadata": {}, 712 | "source": [ 713 | "### Helper-function to test the model at the middle of training" 714 | ] 715 | }, 716 | { 717 | "cell_type": "code", 718 | "execution_count": null, 719 | "metadata": {}, 720 | "outputs": [], 721 | "source": [ 722 | "def test(_global_step, epoch, y_pred_val, y_true_val):\n", 723 | " global global_accuracy\n", 724 | " global epoch_start\n", 725 | "\n", 726 | " i = 0\n", 727 | " predicted_class = np.zeros(shape=len(test_x), dtype=np.int)\n", 728 | " while i < len(test_x):\n", 729 | " j = min(i + _BATCH_SIZE, len(test_x))\n", 730 | " batch_xs = test_x[i:j, :]\n", 731 | " batch_ys = test_y[i:j, :]\n", 732 | " predicted_class[i:j] = sess.run(\n", 733 | " y_pred_cls,\n", 734 | " feed_dict={x: batch_xs, y_true: batch_ys, keep_probability: 1.0}\n", 735 | " ) \n", 736 | " i = j\n", 737 | " text_y_argmax = np.argmax(test_y, axis=1) \n", 738 | " correct = (text_y_argmax == predicted_class)\n", 739 | " acc = correct.mean()*100\n", 740 | " correct_numbers = correct.sum()\n", 741 | "\n", 742 | " hours, rem = divmod(time() - epoch_start, 3600)\n", 743 | " minutes, seconds = divmod(rem, 60)\n", 744 | " mes = \"\\nEpoch {} - accuracy: {:.1f}% ({}/{}) - time: {:0>2}:{:0>2}:{:05.2f}\"\n", 745 | " print(mes.format((epoch+1), acc, correct_numbers, len(test_x), int(hours), int(minutes), seconds))\n", 746 | "\n", 747 | " if global_accuracy != 0 and global_accuracy < acc:\n", 748 | " mes = \"This epoch receive better accuracy: {:.1f}% > {:.1f}%.\"\n", 749 | " print(mes.format(acc, global_accuracy))\n", 750 | " global_accuracy = acc\n", 751 | "\n", 752 | " elif global_accuracy == 0:\n", 753 | " global_accuracy = acc\n", 754 | " \n", 755 | " # Plot non-normalized confusion matrix\n", 756 | " cnf_matrix = confusion_matrix(text_y_argmax, predicted_class)\n", 757 | " np.set_printoptions(precision=2)\n", 758 | " plot_confusion_matrix(cnf_matrix, classes=class_names,\n", 759 | " title='Confusion matrix, without normalization')\n", 760 | " plot_tsne(y_pred_val, y_true_val)\n", 761 | " \n", 762 | " print(\"###########################################################################################################\")" 763 | ] 764 | }, 765 | { 766 | "cell_type": "markdown", 767 | "metadata": {}, 768 | "source": [ 769 | "### Helper-function to perform optimization iterations" 770 | ] 771 | }, 772 | { 773 | "cell_type": "markdown", 774 | "metadata": {}, 775 | "source": [ 776 | "Function for performing a number of optimization iterations so as to gradually improve the `weights` and `biases` of the model. In each iteration, a new batch of data is selected from the training-set and then TensorFlow executes the optimizer using those training samples." 777 | ] 778 | }, 779 | { 780 | "cell_type": "code", 781 | "execution_count": null, 782 | "metadata": {}, 783 | "outputs": [], 784 | "source": [ 785 | "def train(epoch):\n", 786 | " global epoch_start\n", 787 | " epoch_start = time()\n", 788 | " batch_size = int(math.ceil(len(train_x) / _BATCH_SIZE))\n", 789 | " i_global = 0\n", 790 | "\n", 791 | " for s in range(batch_size):\n", 792 | " batch_xs = train_x[s*_BATCH_SIZE: (s+1)*_BATCH_SIZE]\n", 793 | " batch_ys = train_y[s*_BATCH_SIZE: (s+1)*_BATCH_SIZE]\n", 794 | "\n", 795 | " start_time = time()\n", 796 | " i_global, _, batch_loss, batch_acc = sess.run(\n", 797 | " [global_step, optimizer, loss, accuracy],\n", 798 | " feed_dict={x: batch_xs, y_true: batch_ys, keep_probability: 0.5})\n", 799 | " duration = time() - start_time\n", 800 | "\n", 801 | " if s % 100 == 0:\n", 802 | " percentage = int(float(s)/float(batch_size)*100.0)\n", 803 | " bar_len = 29\n", 804 | " filled_len = int((bar_len*int(percentage))/100)\n", 805 | " bar = '=' * filled_len + '>' + '-' * (bar_len - filled_len)\n", 806 | "\n", 807 | " msg = \"Global step: {:>5} - [{}] {:>3}% - accuracy: {:.1f}% - loss: {:.4f} - {:.1f} sample/sec\"\n", 808 | " print(msg.format(i_global, bar, percentage, batch_acc, batch_loss, _BATCH_SIZE / duration))\n", 809 | " y_pred_val = sess.run(y_pred, feed_dict={x: train_x[:_NUM_VIS_EMBEDDING], keep_probability: 1.0})\n", 810 | " y_true_val = train_y[:_NUM_VIS_EMBEDDING]\n", 811 | " y_true_val = y_true_val.argmax(1)\n", 812 | "\n", 813 | " test(i_global, epoch, y_pred_val, y_true_val)" 814 | ] 815 | }, 816 | { 817 | "cell_type": "markdown", 818 | "metadata": {}, 819 | "source": [ 820 | "### Start training networks" 821 | ] 822 | }, 823 | { 824 | "cell_type": "markdown", 825 | "metadata": {}, 826 | "source": [ 827 | "Weights are optimized and the accuracy on the test set increases as global step increases." 828 | ] 829 | }, 830 | { 831 | "cell_type": "code", 832 | "execution_count": null, 833 | "metadata": {}, 834 | "outputs": [], 835 | "source": [ 836 | "print(\"Start training\")\n", 837 | "train_start = time()\n", 838 | "\n", 839 | "for i in range(_EPOCH):\n", 840 | " print(\"\\nEpoch: {}/{}\\n\".format((i+1), _EPOCH))\n", 841 | " train(i)\n", 842 | "\n", 843 | "hours, rem = divmod(time() - train_start, 3600)\n", 844 | "minutes, seconds = divmod(rem, 60)\n", 845 | "mes = \"Best accuracy pre session: {:.2f}%, time: {:0>2}:{:0>2}:{:05.2f}\"\n", 846 | "print(mes.format(global_accuracy, int(hours), int(minutes), seconds))\n", 847 | "print(\"Done training\")" 848 | ] 849 | } 850 | ], 851 | "metadata": { 852 | "kernelspec": { 853 | "display_name": "Python 3", 854 | "language": "python", 855 | "name": "python3" 856 | }, 857 | "language_info": { 858 | "codemirror_mode": { 859 | "name": "ipython", 860 | "version": 3 861 | }, 862 | "file_extension": ".py", 863 | "mimetype": "text/x-python", 864 | "name": "python", 865 | "nbconvert_exporter": "python", 866 | "pygments_lexer": "ipython3", 867 | "version": "3.5.2" 868 | } 869 | }, 870 | "nbformat": 4, 871 | "nbformat_minor": 2 872 | } 873 | -------------------------------------------------------------------------------- /code/Tutorial4_RNN_Name.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## load prerequisites" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": {}, 14 | "outputs": [ 15 | { 16 | "name": "stderr", 17 | "output_type": "stream", 18 | "text": [ 19 | "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", 20 | " _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n", 21 | "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", 22 | " _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n", 23 | "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", 24 | " _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n", 25 | "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", 26 | " _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n", 27 | "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:521: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", 28 | " _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n", 29 | "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n", 30 | " np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n" 31 | ] 32 | } 33 | ], 34 | "source": [ 35 | "import csv\n", 36 | "import random\n", 37 | "import numpy as np\n", 38 | "import tensorflow as tf" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": {}, 44 | "source": [ 45 | "## Preprocess data : names" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 2, 51 | "metadata": {}, 52 | "outputs": [ 53 | { 54 | "name": "stdout", 55 | "output_type": "stream", 56 | "text": [ 57 | "name total : 1219\n", 58 | "maximum name length : 11\n" 59 | ] 60 | } 61 | ], 62 | "source": [ 63 | "data_name = set()\n", 64 | "max_len = 0\n", 65 | "with open('../data/woman_name_dataset.csv') as csv_file:\n", 66 | " csv_reader = csv.reader(csv_file, delimiter=',')\n", 67 | " line_count = 0\n", 68 | " for row in csv_reader:\n", 69 | " if line_count == 0:\n", 70 | " line_count += 1\n", 71 | " else:\n", 72 | " tmp_name = row[1].split()[0]\n", 73 | " data_name.add(row[1].split()[0])\n", 74 | " if len(tmp_name) > max_len:\n", 75 | " max_len = len(tmp_name)\n", 76 | "data_name = list(data_name)\n", 77 | "print('name total : {}'.format(len(data_name)))\n", 78 | "print('maximum name length : {}'.format(max_len))" 79 | ] 80 | }, 81 | { 82 | "cell_type": "markdown", 83 | "metadata": {}, 84 | "source": [ 85 | "## Preprocess data : Characters" 86 | ] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": 3, 91 | "metadata": {}, 92 | "outputs": [ 93 | { 94 | "name": "stdout", 95 | "output_type": "stream", 96 | "text": [ 97 | "26 alphabets : ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n" 98 | ] 99 | } 100 | ], 101 | "source": [ 102 | "chars = set()\n", 103 | "for name in data_name:\n", 104 | " for char in name:\n", 105 | " chars.add(char)\n", 106 | "chars = list(np.sort(list(chars)))\n", 107 | "print('{} alphabets : '.format(len(chars)), chars)" 108 | ] 109 | }, 110 | { 111 | "cell_type": "markdown", 112 | "metadata": {}, 113 | "source": [ 114 | "## Define function to convert name to onehot" 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": 4, 120 | "metadata": {}, 121 | "outputs": [ 122 | { 123 | "data": { 124 | "text/plain": [ 125 | "array([[[0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.,\n", 126 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n", 127 | " [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n", 128 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n", 129 | " [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.,\n", 130 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n", 131 | " [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n", 132 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n", 133 | " [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n", 134 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n", 135 | " [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n", 136 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n", 137 | " [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n", 138 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n", 139 | " [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n", 140 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n", 141 | " [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n", 142 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n", 143 | " [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n", 144 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n", 145 | " [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n", 146 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.]]])" 147 | ] 148 | }, 149 | "execution_count": 4, 150 | "metadata": {}, 151 | "output_type": "execute_result" 152 | } 153 | ], 154 | "source": [ 155 | "def name_to_onehot(names, chars, max_len):\n", 156 | " onehot = np.zeros((len(names), max_len, len(chars)+1))\n", 157 | " for idx_1, name in enumerate(names):\n", 158 | " for idx_2 in range(max_len):\n", 159 | " if idx_2 < len(name):\n", 160 | " idx_3 = chars.index(name[idx_2])\n", 161 | " onehot[idx_1, idx_2, idx_3] = 1\n", 162 | " else:\n", 163 | " onehot[idx_1, idx_2, -1] = 1\n", 164 | " return onehot\n", 165 | "\n", 166 | "onehot_ex = name_to_onehot(['jane'], chars, max_len)\n", 167 | "onehot_ex" 168 | ] 169 | }, 170 | { 171 | "cell_type": "markdown", 172 | "metadata": {}, 173 | "source": [ 174 | "## Define dimension and Placeholders" 175 | ] 176 | }, 177 | { 178 | "cell_type": "code", 179 | "execution_count": 5, 180 | "metadata": {}, 181 | "outputs": [], 182 | "source": [ 183 | "num_data = len(data_name)\n", 184 | "seq_len = max_len - 1\n", 185 | "dim_data = len(chars) + 1\n", 186 | "\n", 187 | "ph_input_name = tf.placeholder(dtype=tf.float32, shape=[None, seq_len, dim_data])\n", 188 | "ph_output_name = tf.placeholder(dtype=tf.float32, shape=[None, seq_len, dim_data])" 189 | ] 190 | }, 191 | { 192 | "cell_type": "markdown", 193 | "metadata": {}, 194 | "source": [ 195 | "## Define weight variables" 196 | ] 197 | }, 198 | { 199 | "cell_type": "code", 200 | "execution_count": 6, 201 | "metadata": {}, 202 | "outputs": [], 203 | "source": [ 204 | "dim_rnn_cell = 128\n", 205 | "stddev = 0.02\n", 206 | "\n", 207 | "with tf.variable_scope('weights'):\n", 208 | " W_i = tf.get_variable('W_i', dtype=tf.float32,\n", 209 | " initializer=tf.random_normal([dim_data, dim_rnn_cell],\n", 210 | " stddev = stddev))\n", 211 | " b_i = tf.get_variable('b_i', dtype=tf.float32,\n", 212 | " initializer=tf.random_normal([dim_rnn_cell],\n", 213 | " stddev = stddev))\n", 214 | " W_o = tf.get_variable('W_o', dtype=tf.float32,\n", 215 | " initializer=tf.random_normal([dim_rnn_cell, dim_data],\n", 216 | " stddev = stddev))\n", 217 | " b_o = tf.get_variable('b_o', dtype=tf.float32,\n", 218 | " initializer=tf.random_normal([dim_data],\n", 219 | " stddev = stddev))" 220 | ] 221 | }, 222 | { 223 | "cell_type": "markdown", 224 | "metadata": {}, 225 | "source": [ 226 | "## Define RNN for training" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": 21, 232 | "metadata": {}, 233 | "outputs": [], 234 | "source": [ 235 | "def name_rnn_train(_x, _seq_len, _dim_data, _dim_rnn_cell):\n", 236 | " _x_split = tf.transpose(_x, [1, 0, 2]) # seq_len, batch, dim_data\n", 237 | " _x_split = tf.reshape(_x_split, [-1, _dim_data])\n", 238 | " \n", 239 | " # Linear Operation for Input\n", 240 | " # Fill here\n", 241 | " \n", 242 | " # Define LSTM Cell && RNN\n", 243 | " with tf.variable_scope('rnn', reuse=tf.AUTO_REUSE):\n", 244 | " _rnn_cell = tf.nn.rnn_cell.BasicLSTMCell(_dim_rnn_cell)\n", 245 | " _output, _state = tf.nn.static_rnn(_rnn_cell, _h_split, dtype=tf.float32)\n", 246 | " \n", 247 | " # Linear Operation for Output\n", 248 | " _total_out = []\n", 249 | " # Fill here\n", 250 | " \n", 251 | " return tf.transpose(tf.stack(_total_out), [1, 0, 2])" 252 | ] 253 | }, 254 | { 255 | "cell_type": "markdown", 256 | "metadata": {}, 257 | "source": [ 258 | "## Define result graph" 259 | ] 260 | }, 261 | { 262 | "cell_type": "code", 263 | "execution_count": 17, 264 | "metadata": {}, 265 | "outputs": [ 266 | { 267 | "name": "stdout", 268 | "output_type": "stream", 269 | "text": [ 270 | "result_shape : (?, 10, 27)\n" 271 | ] 272 | } 273 | ], 274 | "source": [ 275 | "result_name = name_rnn_train(ph_input_name, seq_len, dim_data, dim_rnn_cell)\n", 276 | "print('result_shape :', result_name.shape)" 277 | ] 278 | }, 279 | { 280 | "cell_type": "markdown", 281 | "metadata": {}, 282 | "source": [ 283 | "## Define Loss function" 284 | ] 285 | }, 286 | { 287 | "cell_type": "code", 288 | "execution_count": 18, 289 | "metadata": {}, 290 | "outputs": [ 291 | { 292 | "name": "stdout", 293 | "output_type": "stream", 294 | "text": [ 295 | "WARNING:tensorflow:From :7: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.\n", 296 | "Instructions for updating:\n", 297 | "\n", 298 | "Future major versions of TensorFlow will allow gradients to flow\n", 299 | "into the labels input on backprop by default.\n", 300 | "\n", 301 | "See tf.nn.softmax_cross_entropy_with_logits_v2.\n", 302 | "\n" 303 | ] 304 | } 305 | ], 306 | "source": [ 307 | "def name_loss(_gt_name, _result_name, _seq_len):\n", 308 | " total_loss = 0\n", 309 | " for i in range(_seq_len):\n", 310 | " # Fill here\n", 311 | " return total_loss\n", 312 | "rnn_loss = name_loss(ph_output_name, result_name, seq_len)" 313 | ] 314 | }, 315 | { 316 | "cell_type": "markdown", 317 | "metadata": {}, 318 | "source": [ 319 | "## Define Optimizer and Get Ready" 320 | ] 321 | }, 322 | { 323 | "cell_type": "code", 324 | "execution_count": 19, 325 | "metadata": {}, 326 | "outputs": [ 327 | { 328 | "name": "stdout", 329 | "output_type": "stream", 330 | "text": [ 331 | "Now ready to start the session\n" 332 | ] 333 | } 334 | ], 335 | "source": [ 336 | "learning_rate = 1e-3\n", 337 | "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(rnn_loss)\n", 338 | "\n", 339 | "init = tf.global_variables_initializer()\n", 340 | "saver = tf.train.Saver(var_list=tf.trainable_variables())\n", 341 | "\n", 342 | "print('Now ready to start the session')" 343 | ] 344 | }, 345 | { 346 | "cell_type": "markdown", 347 | "metadata": { 348 | "scrolled": true 349 | }, 350 | "source": [ 351 | "## Session Run" 352 | ] 353 | }, 354 | { 355 | "cell_type": "code", 356 | "execution_count": 20, 357 | "metadata": { 358 | "scrolled": true 359 | }, 360 | "outputs": [ 361 | { 362 | "name": "stdout", 363 | "output_type": "stream", 364 | "text": [ 365 | "epoch : 1, train_loss : 28.54256660059879\n", 366 | "Model saved in file: ./RNN_model/model.ckpt-1\n", 367 | "epoch : 2, train_loss : 18.67396987111945\n", 368 | "Model saved in file: ./RNN_model/model.ckpt-2\n", 369 | "epoch : 3, train_loss : 17.6947021484375\n", 370 | "Model saved in file: ./RNN_model/model.ckpt-3\n", 371 | "epoch : 4, train_loss : 17.15771614877801\n", 372 | "Model saved in file: ./RNN_model/model.ckpt-4\n", 373 | "epoch : 5, train_loss : 16.822392915424548\n", 374 | "Model saved in file: ./RNN_model/model.ckpt-5\n", 375 | "epoch : 6, train_loss : 16.50236872622841\n", 376 | "Model saved in file: ./RNN_model/model.ckpt-6\n", 377 | "epoch : 7, train_loss : 16.15971851348877\n", 378 | "Model saved in file: ./RNN_model/model.ckpt-7\n", 379 | "epoch : 8, train_loss : 15.863567603261846\n", 380 | "Model saved in file: ./RNN_model/model.ckpt-8\n", 381 | "epoch : 9, train_loss : 15.546000480651854\n", 382 | "Model saved in file: ./RNN_model/model.ckpt-9\n", 383 | "epoch : 10, train_loss : 15.251013906378496\n", 384 | "Model saved in file: ./RNN_model/model.ckpt-10\n", 385 | "epoch : 11, train_loss : 14.958294015181693\n", 386 | "Model saved in file: ./RNN_model/model.ckpt-11\n", 387 | "epoch : 12, train_loss : 14.696805903786107\n", 388 | "Model saved in file: ./RNN_model/model.ckpt-12\n", 389 | "epoch : 13, train_loss : 14.449218097485993\n", 390 | "Model saved in file: ./RNN_model/model.ckpt-13\n", 391 | "epoch : 14, train_loss : 14.164149234169411\n", 392 | "Model saved in file: ./RNN_model/model.ckpt-14\n", 393 | "epoch : 15, train_loss : 13.922977648283306\n", 394 | "Model saved in file: ./RNN_model/model.ckpt-15\n", 395 | "epoch : 16, train_loss : 13.688792931406121\n", 396 | "Model saved in file: ./RNN_model/model.ckpt-16\n", 397 | "epoch : 17, train_loss : 13.500556142706621\n", 398 | "Model saved in file: ./RNN_model/model.ckpt-17\n", 399 | "epoch : 18, train_loss : 13.379649513646175\n", 400 | "Model saved in file: ./RNN_model/model.ckpt-18\n", 401 | "epoch : 19, train_loss : 13.261106491088869\n", 402 | "Model saved in file: ./RNN_model/model.ckpt-19\n", 403 | "epoch : 20, train_loss : 13.178394016466644\n", 404 | "Model saved in file: ./RNN_model/model.ckpt-20\n", 405 | "epoch : 21, train_loss : 13.10595422042044\n", 406 | "Model saved in file: ./RNN_model/model.ckpt-21\n", 407 | "epoch : 22, train_loss : 13.019753255342184\n", 408 | "Model saved in file: ./RNN_model/model.ckpt-22\n", 409 | "epoch : 23, train_loss : 12.941644317225407\n", 410 | "Model saved in file: ./RNN_model/model.ckpt-23\n", 411 | "epoch : 24, train_loss : 12.854022628382634\n", 412 | "Model saved in file: ./RNN_model/model.ckpt-24\n", 413 | "epoch : 25, train_loss : 12.777433897319591\n", 414 | "Model saved in file: ./RNN_model/model.ckpt-25\n", 415 | "epoch : 26, train_loss : 12.726191018757067\n", 416 | "Model saved in file: ./RNN_model/model.ckpt-26\n", 417 | "epoch : 27, train_loss : 12.633992948030171\n", 418 | "Model saved in file: ./RNN_model/model.ckpt-27\n", 419 | "epoch : 28, train_loss : 12.554255284761128\n", 420 | "Model saved in file: ./RNN_model/model.ckpt-28\n", 421 | "epoch : 29, train_loss : 12.472216806913677\n", 422 | "Model saved in file: ./RNN_model/model.ckpt-29\n", 423 | "epoch : 30, train_loss : 12.378323755766216\n", 424 | "Model saved in file: ./RNN_model/model.ckpt-30\n", 425 | "epoch : 31, train_loss : 12.29627257899234\n", 426 | "Model saved in file: ./RNN_model/model.ckpt-31\n", 427 | "epoch : 32, train_loss : 12.201784937005293\n", 428 | "Model saved in file: ./RNN_model/model.ckpt-32\n", 429 | "epoch : 33, train_loss : 12.131009202254447\n", 430 | "Model saved in file: ./RNN_model/model.ckpt-33\n", 431 | "epoch : 34, train_loss : 12.030012632671156\n", 432 | "Model saved in file: ./RNN_model/model.ckpt-34\n", 433 | "epoch : 35, train_loss : 11.948686900891753\n", 434 | "Model saved in file: ./RNN_model/model.ckpt-35\n", 435 | "epoch : 36, train_loss : 11.872862564890008\n", 436 | "Model saved in file: ./RNN_model/model.ckpt-36\n", 437 | "epoch : 37, train_loss : 11.797504926982675\n", 438 | "Model saved in file: ./RNN_model/model.ckpt-37\n", 439 | "epoch : 38, train_loss : 11.700841652719598\n", 440 | "Model saved in file: ./RNN_model/model.ckpt-38\n", 441 | "epoch : 39, train_loss : 11.630409039949114\n", 442 | "Model saved in file: ./RNN_model/model.ckpt-39\n", 443 | "epoch : 40, train_loss : 11.542844320598403\n", 444 | "Model saved in file: ./RNN_model/model.ckpt-40\n", 445 | "epoch : 41, train_loss : 11.463216731422827\n", 446 | "Model saved in file: ./RNN_model/model.ckpt-41\n", 447 | "epoch : 42, train_loss : 11.410016461422568\n", 448 | "Model saved in file: ./RNN_model/model.ckpt-42\n", 449 | "epoch : 43, train_loss : 11.323547564054788\n", 450 | "Model saved in file: ./RNN_model/model.ckpt-43\n", 451 | "epoch : 44, train_loss : 11.257381991336217\n", 452 | "Model saved in file: ./RNN_model/model.ckpt-44\n", 453 | "epoch : 45, train_loss : 11.184963878832363\n", 454 | "Model saved in file: ./RNN_model/model.ckpt-45\n", 455 | "epoch : 46, train_loss : 11.095906508596322\n", 456 | "Model saved in file: ./RNN_model/model.ckpt-46\n", 457 | "epoch : 47, train_loss : 11.023151146738153\n", 458 | "Model saved in file: ./RNN_model/model.ckpt-47\n", 459 | "epoch : 48, train_loss : 10.94416974720202\n", 460 | "Model saved in file: ./RNN_model/model.ckpt-48\n", 461 | "epoch : 49, train_loss : 10.855457556875129\n", 462 | "Model saved in file: ./RNN_model/model.ckpt-49\n", 463 | "epoch : 50, train_loss : 10.76490462453742\n", 464 | "Model saved in file: ./RNN_model/model.ckpt-50\n", 465 | "epoch : 51, train_loss : 10.699354021172775\n", 466 | "Model saved in file: ./RNN_model/model.ckpt-51\n", 467 | "epoch : 52, train_loss : 10.617735461184852\n", 468 | "Model saved in file: ./RNN_model/model.ckpt-52\n", 469 | "epoch : 53, train_loss : 10.505781976800218\n", 470 | "Model saved in file: ./RNN_model/model.ckpt-53\n", 471 | "epoch : 54, train_loss : 10.439787061590899\n", 472 | "Model saved in file: ./RNN_model/model.ckpt-54\n", 473 | "epoch : 55, train_loss : 10.355367208781997\n", 474 | "Model saved in file: ./RNN_model/model.ckpt-55\n", 475 | "epoch : 56, train_loss : 10.283977910092002\n", 476 | "Model saved in file: ./RNN_model/model.ckpt-56\n", 477 | "epoch : 57, train_loss : 10.18765484659295\n", 478 | "Model saved in file: ./RNN_model/model.ckpt-57\n", 479 | "epoch : 58, train_loss : 10.093573670638236\n", 480 | "Model saved in file: ./RNN_model/model.ckpt-58\n", 481 | "epoch : 59, train_loss : 10.022882511741239\n", 482 | "Model saved in file: ./RNN_model/model.ckpt-59\n", 483 | "epoch : 60, train_loss : 9.933478907534951\n", 484 | "Model saved in file: ./RNN_model/model.ckpt-60\n", 485 | "epoch : 61, train_loss : 9.846686915347448\n", 486 | "Model saved in file: ./RNN_model/model.ckpt-61\n", 487 | "epoch : 62, train_loss : 9.761049822757117\n", 488 | "Model saved in file: ./RNN_model/model.ckpt-62\n", 489 | "epoch : 63, train_loss : 9.676333327042428\n", 490 | "Model saved in file: ./RNN_model/model.ckpt-63\n", 491 | "epoch : 64, train_loss : 9.594584665800395\n", 492 | "Model saved in file: ./RNN_model/model.ckpt-64\n", 493 | "epoch : 65, train_loss : 9.528890910901522\n", 494 | "Model saved in file: ./RNN_model/model.ckpt-65\n", 495 | "epoch : 66, train_loss : 9.422712426436576\n", 496 | "Model saved in file: ./RNN_model/model.ckpt-66\n", 497 | "epoch : 67, train_loss : 9.35149017133211\n", 498 | "Model saved in file: ./RNN_model/model.ckpt-67\n", 499 | "epoch : 68, train_loss : 9.263297633120889\n", 500 | "Model saved in file: ./RNN_model/model.ckpt-68\n", 501 | "epoch : 69, train_loss : 9.18922223542866\n", 502 | "Model saved in file: ./RNN_model/model.ckpt-69\n", 503 | "epoch : 70, train_loss : 9.105483255888284\n", 504 | "Model saved in file: ./RNN_model/model.ckpt-70\n", 505 | "epoch : 71, train_loss : 9.025170326232912\n", 506 | "Model saved in file: ./RNN_model/model.ckpt-71\n", 507 | "epoch : 72, train_loss : 8.950168208072062\n", 508 | "Model saved in file: ./RNN_model/model.ckpt-72\n", 509 | "epoch : 73, train_loss : 8.869566013938503\n", 510 | "Model saved in file: ./RNN_model/model.ckpt-73\n", 511 | "epoch : 74, train_loss : 8.783397649463854\n", 512 | "Model saved in file: ./RNN_model/model.ckpt-74\n", 513 | "epoch : 75, train_loss : 8.73381529356304\n", 514 | "Model saved in file: ./RNN_model/model.ckpt-75\n", 515 | "epoch : 76, train_loss : 8.646269346538341\n", 516 | "Model saved in file: ./RNN_model/model.ckpt-76\n", 517 | "epoch : 77, train_loss : 8.568772868106239\n", 518 | "Model saved in file: ./RNN_model/model.ckpt-77\n", 519 | "epoch : 78, train_loss : 8.4884763014944\n", 520 | "Model saved in file: ./RNN_model/model.ckpt-78\n", 521 | "epoch : 79, train_loss : 8.4184637571636\n", 522 | "Model saved in file: ./RNN_model/model.ckpt-79\n", 523 | "epoch : 80, train_loss : 8.361174156791286\n", 524 | "Model saved in file: ./RNN_model/model.ckpt-80\n", 525 | "epoch : 81, train_loss : 8.268802893789191\n", 526 | "Model saved in file: ./RNN_model/model.ckpt-81\n", 527 | "epoch : 82, train_loss : 8.21933236875032\n", 528 | "Model saved in file: ./RNN_model/model.ckpt-82\n", 529 | "epoch : 83, train_loss : 8.134884106485467\n", 530 | "Model saved in file: ./RNN_model/model.ckpt-83\n", 531 | "epoch : 84, train_loss : 8.07383999071623\n", 532 | "Model saved in file: ./RNN_model/model.ckpt-84\n", 533 | "epoch : 85, train_loss : 8.025716580842671\n", 534 | "Model saved in file: ./RNN_model/model.ckpt-85\n", 535 | "epoch : 86, train_loss : 7.934834053641873\n", 536 | "Model saved in file: ./RNN_model/model.ckpt-86\n", 537 | "epoch : 87, train_loss : 7.877349903709008\n", 538 | "Model saved in file: ./RNN_model/model.ckpt-87\n", 539 | "epoch : 88, train_loss : 7.807283652456182\n", 540 | "Model saved in file: ./RNN_model/model.ckpt-88\n", 541 | "epoch : 89, train_loss : 7.7494699327569245\n", 542 | "Model saved in file: ./RNN_model/model.ckpt-89\n", 543 | "epoch : 90, train_loss : 7.675424701289128\n", 544 | "Model saved in file: ./RNN_model/model.ckpt-90\n", 545 | "epoch : 91, train_loss : 7.608195681320993\n", 546 | "Model saved in file: ./RNN_model/model.ckpt-91\n" 547 | ] 548 | }, 549 | { 550 | "name": "stdout", 551 | "output_type": "stream", 552 | "text": [ 553 | "epoch : 92, train_loss : 7.5594291937978655\n", 554 | "Model saved in file: ./RNN_model/model.ckpt-92\n", 555 | "epoch : 93, train_loss : 7.497586049531637\n", 556 | "Model saved in file: ./RNN_model/model.ckpt-93\n", 557 | "epoch : 94, train_loss : 7.4298533640409765\n", 558 | "Model saved in file: ./RNN_model/model.ckpt-94\n", 559 | "epoch : 95, train_loss : 7.387549375232897\n", 560 | "Model saved in file: ./RNN_model/model.ckpt-95\n", 561 | "epoch : 96, train_loss : 7.342127197667171\n", 562 | "Model saved in file: ./RNN_model/model.ckpt-96\n", 563 | "epoch : 97, train_loss : 7.279497246993215\n", 564 | "Model saved in file: ./RNN_model/model.ckpt-97\n", 565 | "epoch : 98, train_loss : 7.227611642134817\n", 566 | "Model saved in file: ./RNN_model/model.ckpt-98\n", 567 | "epoch : 99, train_loss : 7.185787100540964\n", 568 | "Model saved in file: ./RNN_model/model.ckpt-99\n", 569 | "epoch : 100, train_loss : 7.120706583324233\n", 570 | "Model saved in file: ./RNN_model/model.ckpt-100\n", 571 | "epoch : 101, train_loss : 7.06613658603869\n", 572 | "Model saved in file: ./RNN_model/model.ckpt-101\n", 573 | "epoch : 102, train_loss : 7.016613608912419\n", 574 | "Model saved in file: ./RNN_model/model.ckpt-102\n", 575 | "epoch : 103, train_loss : 6.979021875481855\n", 576 | "Model saved in file: ./RNN_model/model.ckpt-103\n", 577 | "epoch : 104, train_loss : 6.9431470067877505\n", 578 | "Model saved in file: ./RNN_model/model.ckpt-104\n", 579 | "epoch : 105, train_loss : 6.876120241064775\n", 580 | "Model saved in file: ./RNN_model/model.ckpt-105\n", 581 | "epoch : 106, train_loss : 6.84489819878026\n", 582 | "Model saved in file: ./RNN_model/model.ckpt-106\n", 583 | "epoch : 107, train_loss : 6.782469799644068\n", 584 | "Model saved in file: ./RNN_model/model.ckpt-107\n", 585 | "epoch : 108, train_loss : 6.738975022968492\n", 586 | "Model saved in file: ./RNN_model/model.ckpt-108\n", 587 | "epoch : 109, train_loss : 6.705347186640689\n", 588 | "Model saved in file: ./RNN_model/model.ckpt-109\n", 589 | "epoch : 110, train_loss : 6.660457811857524\n", 590 | "Model saved in file: ./RNN_model/model.ckpt-110\n", 591 | "epoch : 111, train_loss : 6.612845169870477\n", 592 | "Model saved in file: ./RNN_model/model.ckpt-111\n", 593 | "epoch : 112, train_loss : 6.573223766527677\n", 594 | "Model saved in file: ./RNN_model/model.ckpt-112\n", 595 | "epoch : 113, train_loss : 6.524199711649041\n", 596 | "Model saved in file: ./RNN_model/model.ckpt-113\n", 597 | "epoch : 114, train_loss : 6.475720832222386\n", 598 | "Model saved in file: ./RNN_model/model.ckpt-114\n", 599 | "epoch : 115, train_loss : 6.454433416065416\n", 600 | "Model saved in file: ./RNN_model/model.ckpt-115\n", 601 | "epoch : 116, train_loss : 6.414958502116957\n", 602 | "Model saved in file: ./RNN_model/model.ckpt-116\n", 603 | "epoch : 117, train_loss : 6.377824808421888\n", 604 | "Model saved in file: ./RNN_model/model.ckpt-117\n", 605 | "epoch : 118, train_loss : 6.347458437869425\n", 606 | "Model saved in file: ./RNN_model/model.ckpt-118\n", 607 | "epoch : 119, train_loss : 6.309866679342169\n", 608 | "Model saved in file: ./RNN_model/model.ckpt-119\n", 609 | "epoch : 120, train_loss : 6.269866366135448\n", 610 | "Model saved in file: ./RNN_model/model.ckpt-120\n", 611 | "epoch : 121, train_loss : 6.230119178169653\n", 612 | "Model saved in file: ./RNN_model/model.ckpt-121\n", 613 | "epoch : 122, train_loss : 6.200611365468877\n", 614 | "Model saved in file: ./RNN_model/model.ckpt-122\n", 615 | "epoch : 123, train_loss : 6.167604220540901\n", 616 | "Model saved in file: ./RNN_model/model.ckpt-123\n", 617 | "epoch : 124, train_loss : 6.1351448862176206\n", 618 | "Model saved in file: ./RNN_model/model.ckpt-124\n", 619 | "epoch : 125, train_loss : 6.096212211408113\n", 620 | "Model saved in file: ./RNN_model/model.ckpt-125\n", 621 | "epoch : 126, train_loss : 6.081643982937461\n", 622 | "Model saved in file: ./RNN_model/model.ckpt-126\n", 623 | "epoch : 127, train_loss : 6.0440531278911385\n", 624 | "Model saved in file: ./RNN_model/model.ckpt-127\n", 625 | "epoch : 128, train_loss : 6.014468142860814\n", 626 | "Model saved in file: ./RNN_model/model.ckpt-128\n", 627 | "epoch : 129, train_loss : 5.974940751728258\n", 628 | "Model saved in file: ./RNN_model/model.ckpt-129\n", 629 | "epoch : 130, train_loss : 5.942427936353183\n", 630 | "Model saved in file: ./RNN_model/model.ckpt-130\n", 631 | "epoch : 131, train_loss : 5.911266075937372\n", 632 | "Model saved in file: ./RNN_model/model.ckpt-131\n", 633 | "epoch : 132, train_loss : 5.901675676044666\n", 634 | "Model saved in file: ./RNN_model/model.ckpt-132\n", 635 | "epoch : 133, train_loss : 5.863052619130988\n", 636 | "Model saved in file: ./RNN_model/model.ckpt-133\n", 637 | "epoch : 134, train_loss : 5.842718249873111\n", 638 | "Model saved in file: ./RNN_model/model.ckpt-134\n", 639 | "epoch : 135, train_loss : 5.815403059909219\n", 640 | "Model saved in file: ./RNN_model/model.ckpt-135\n", 641 | "epoch : 136, train_loss : 5.790516301205284\n", 642 | "Model saved in file: ./RNN_model/model.ckpt-136\n", 643 | "epoch : 137, train_loss : 5.765897474790873\n", 644 | "Model saved in file: ./RNN_model/model.ckpt-137\n", 645 | "epoch : 138, train_loss : 5.734498801984285\n", 646 | "Model saved in file: ./RNN_model/model.ckpt-138\n", 647 | "epoch : 139, train_loss : 5.709458024878252\n", 648 | "Model saved in file: ./RNN_model/model.ckpt-139\n", 649 | "epoch : 140, train_loss : 5.711953213340358\n", 650 | "Model saved in file: ./RNN_model/model.ckpt-140\n", 651 | "epoch : 141, train_loss : 5.671023946059378\n", 652 | "Model saved in file: ./RNN_model/model.ckpt-141\n", 653 | "epoch : 142, train_loss : 5.637541896418521\n", 654 | "Model saved in file: ./RNN_model/model.ckpt-142\n", 655 | "epoch : 143, train_loss : 5.616442956422506\n", 656 | "Model saved in file: ./RNN_model/model.ckpt-143\n", 657 | "epoch : 144, train_loss : 5.598658561706543\n", 658 | "Model saved in file: ./RNN_model/model.ckpt-144\n", 659 | "epoch : 145, train_loss : 5.581714780707109\n", 660 | "Model saved in file: ./RNN_model/model.ckpt-145\n", 661 | "epoch : 146, train_loss : 5.5535081060309155\n", 662 | "Model saved in file: ./RNN_model/model.ckpt-146\n", 663 | "epoch : 147, train_loss : 5.540834201009649\n", 664 | "Model saved in file: ./RNN_model/model.ckpt-147\n", 665 | "epoch : 148, train_loss : 5.522211752439801\n", 666 | "Model saved in file: ./RNN_model/model.ckpt-148\n", 667 | "epoch : 149, train_loss : 5.494936541507118\n", 668 | "Model saved in file: ./RNN_model/model.ckpt-149\n", 669 | "epoch : 150, train_loss : 5.487423168985467\n", 670 | "Model saved in file: ./RNN_model/model.ckpt-150\n", 671 | "epoch : 151, train_loss : 5.47228539617438\n", 672 | "Model saved in file: ./RNN_model/model.ckpt-151\n", 673 | "epoch : 152, train_loss : 5.44294625834415\n", 674 | "Model saved in file: ./RNN_model/model.ckpt-152\n", 675 | "epoch : 153, train_loss : 5.421394398337917\n", 676 | "Model saved in file: ./RNN_model/model.ckpt-153\n", 677 | "epoch : 154, train_loss : 5.401166388863012\n", 678 | "Model saved in file: ./RNN_model/model.ckpt-154\n", 679 | "epoch : 155, train_loss : 5.383386360971551\n", 680 | "Model saved in file: ./RNN_model/model.ckpt-155\n", 681 | "epoch : 156, train_loss : 5.367568794049713\n", 682 | "Model saved in file: ./RNN_model/model.ckpt-156\n", 683 | "epoch : 157, train_loss : 5.348022737001118\n", 684 | "Model saved in file: ./RNN_model/model.ckpt-157\n", 685 | "epoch : 158, train_loss : 5.337232564624987\n", 686 | "Model saved in file: ./RNN_model/model.ckpt-158\n", 687 | "epoch : 159, train_loss : 5.311242630607205\n", 688 | "Model saved in file: ./RNN_model/model.ckpt-159\n", 689 | "epoch : 160, train_loss : 5.301685232865183\n", 690 | "Model saved in file: ./RNN_model/model.ckpt-160\n", 691 | "epoch : 161, train_loss : 5.2913457970870175\n", 692 | "Model saved in file: ./RNN_model/model.ckpt-161\n", 693 | "epoch : 162, train_loss : 5.268940825211374\n", 694 | "Model saved in file: ./RNN_model/model.ckpt-162\n", 695 | "epoch : 163, train_loss : 5.257901492871735\n", 696 | "Model saved in file: ./RNN_model/model.ckpt-163\n", 697 | "epoch : 164, train_loss : 5.241432340521562\n", 698 | "Model saved in file: ./RNN_model/model.ckpt-164\n", 699 | "epoch : 165, train_loss : 5.224895125941227\n", 700 | "Model saved in file: ./RNN_model/model.ckpt-165\n", 701 | "epoch : 166, train_loss : 5.217037978925203\n", 702 | "Model saved in file: ./RNN_model/model.ckpt-166\n", 703 | "epoch : 167, train_loss : 5.202302204935175\n", 704 | "Model saved in file: ./RNN_model/model.ckpt-167\n", 705 | "epoch : 168, train_loss : 5.191168483934905\n", 706 | "Model saved in file: ./RNN_model/model.ckpt-168\n", 707 | "epoch : 169, train_loss : 5.171945145255641\n", 708 | "Model saved in file: ./RNN_model/model.ckpt-169\n", 709 | "epoch : 170, train_loss : 5.158411728708368\n", 710 | "Model saved in file: ./RNN_model/model.ckpt-170\n", 711 | "epoch : 171, train_loss : 5.146283149719238\n", 712 | "Model saved in file: ./RNN_model/model.ckpt-171\n", 713 | "epoch : 172, train_loss : 5.144351833745054\n", 714 | "Model saved in file: ./RNN_model/model.ckpt-172\n", 715 | "epoch : 173, train_loss : 5.117720403169331\n", 716 | "Model saved in file: ./RNN_model/model.ckpt-173\n", 717 | "epoch : 174, train_loss : 5.115271141654568\n", 718 | "Model saved in file: ./RNN_model/model.ckpt-174\n", 719 | "epoch : 175, train_loss : 5.099593388406854\n", 720 | "Model saved in file: ./RNN_model/model.ckpt-175\n", 721 | "epoch : 176, train_loss : 5.0907128986559425\n", 722 | "Model saved in file: ./RNN_model/model.ckpt-176\n", 723 | "epoch : 177, train_loss : 5.075802777942858\n", 724 | "Model saved in file: ./RNN_model/model.ckpt-177\n", 725 | "epoch : 178, train_loss : 5.062778146643387\n", 726 | "Model saved in file: ./RNN_model/model.ckpt-178\n", 727 | "epoch : 179, train_loss : 5.056196388445402\n", 728 | "Model saved in file: ./RNN_model/model.ckpt-179\n", 729 | "epoch : 180, train_loss : 5.053058724654347\n", 730 | "Model saved in file: ./RNN_model/model.ckpt-180\n", 731 | "epoch : 181, train_loss : 5.042384298224198\n", 732 | "Model saved in file: ./RNN_model/model.ckpt-181\n" 733 | ] 734 | }, 735 | { 736 | "name": "stdout", 737 | "output_type": "stream", 738 | "text": [ 739 | "epoch : 182, train_loss : 5.035394317225406\n", 740 | "Model saved in file: ./RNN_model/model.ckpt-182\n", 741 | "epoch : 183, train_loss : 5.022335805391011\n", 742 | "Model saved in file: ./RNN_model/model.ckpt-183\n", 743 | "epoch : 184, train_loss : 5.015330239346152\n", 744 | "Model saved in file: ./RNN_model/model.ckpt-184\n", 745 | "epoch : 185, train_loss : 5.001781639299894\n", 746 | "Model saved in file: ./RNN_model/model.ckpt-185\n", 747 | "epoch : 186, train_loss : 4.991091376856755\n", 748 | "Model saved in file: ./RNN_model/model.ckpt-186\n", 749 | "epoch : 187, train_loss : 4.975522994995117\n", 750 | "Model saved in file: ./RNN_model/model.ckpt-187\n", 751 | "epoch : 188, train_loss : 4.9709730650249275\n", 752 | "Model saved in file: ./RNN_model/model.ckpt-188\n", 753 | "epoch : 189, train_loss : 4.958301217932451\n", 754 | "Model saved in file: ./RNN_model/model.ckpt-189\n", 755 | "epoch : 190, train_loss : 4.954532548000939\n", 756 | "Model saved in file: ./RNN_model/model.ckpt-190\n", 757 | "epoch : 191, train_loss : 4.949857485921759\n", 758 | "Model saved in file: ./RNN_model/model.ckpt-191\n", 759 | "epoch : 192, train_loss : 4.932167404576353\n", 760 | "Model saved in file: ./RNN_model/model.ckpt-192\n", 761 | "epoch : 193, train_loss : 4.932374226419549\n", 762 | "Model saved in file: ./RNN_model/model.ckpt-193\n", 763 | "epoch : 194, train_loss : 4.912426848160593\n", 764 | "Model saved in file: ./RNN_model/model.ckpt-194\n", 765 | "epoch : 195, train_loss : 4.904873044867264\n", 766 | "Model saved in file: ./RNN_model/model.ckpt-195\n", 767 | "epoch : 196, train_loss : 4.897598818728799\n", 768 | "Model saved in file: ./RNN_model/model.ckpt-196\n", 769 | "epoch : 197, train_loss : 4.904120244477925\n", 770 | "Model saved in file: ./RNN_model/model.ckpt-197\n", 771 | "epoch : 198, train_loss : 4.894826562781083\n", 772 | "Model saved in file: ./RNN_model/model.ckpt-198\n", 773 | "epoch : 199, train_loss : 4.894350127169961\n", 774 | "Model saved in file: ./RNN_model/model.ckpt-199\n", 775 | "epoch : 200, train_loss : 4.883309740769237\n", 776 | "Model saved in file: ./RNN_model/model.ckpt-200\n", 777 | "epoch : 201, train_loss : 4.865750463385331\n", 778 | "Model saved in file: ./RNN_model/model.ckpt-201\n", 779 | "epoch : 202, train_loss : 4.856913390912507\n", 780 | "Model saved in file: ./RNN_model/model.ckpt-202\n", 781 | "epoch : 203, train_loss : 4.850797402231317\n", 782 | "Model saved in file: ./RNN_model/model.ckpt-203\n", 783 | "epoch : 204, train_loss : 4.84923287441856\n", 784 | "Model saved in file: ./RNN_model/model.ckpt-204\n", 785 | "epoch : 205, train_loss : 4.83483520307039\n", 786 | "Model saved in file: ./RNN_model/model.ckpt-205\n", 787 | "epoch : 206, train_loss : 4.830684185028076\n", 788 | "Model saved in file: ./RNN_model/model.ckpt-206\n", 789 | "epoch : 207, train_loss : 4.830576143766704\n", 790 | "Model saved in file: ./RNN_model/model.ckpt-207\n", 791 | "epoch : 208, train_loss : 4.825661609047338\n", 792 | "Model saved in file: ./RNN_model/model.ckpt-208\n", 793 | "epoch : 209, train_loss : 4.815081194827431\n", 794 | "Model saved in file: ./RNN_model/model.ckpt-209\n", 795 | "epoch : 210, train_loss : 4.8050768249913265\n", 796 | "Model saved in file: ./RNN_model/model.ckpt-210\n", 797 | "epoch : 211, train_loss : 4.80832408603869\n", 798 | "Model saved in file: ./RNN_model/model.ckpt-211\n", 799 | "epoch : 212, train_loss : 4.7996390242325635\n", 800 | "Model saved in file: ./RNN_model/model.ckpt-212\n", 801 | "epoch : 213, train_loss : 4.790082755841707\n", 802 | "Model saved in file: ./RNN_model/model.ckpt-213\n", 803 | "epoch : 214, train_loss : 4.791138096859581\n", 804 | "Model saved in file: ./RNN_model/model.ckpt-214\n", 805 | "epoch : 215, train_loss : 4.7881072696886555\n", 806 | "Model saved in file: ./RNN_model/model.ckpt-215\n", 807 | "epoch : 216, train_loss : 4.785542387711375\n", 808 | "Model saved in file: ./RNN_model/model.ckpt-216\n", 809 | "epoch : 217, train_loss : 4.775825701261822\n", 810 | "Model saved in file: ./RNN_model/model.ckpt-217\n", 811 | "epoch : 218, train_loss : 4.765661666267794\n", 812 | "Model saved in file: ./RNN_model/model.ckpt-218\n", 813 | "epoch : 219, train_loss : 4.758792124296489\n", 814 | "Model saved in file: ./RNN_model/model.ckpt-219\n", 815 | "epoch : 220, train_loss : 4.761143985547518\n", 816 | "Model saved in file: ./RNN_model/model.ckpt-220\n", 817 | "epoch : 221, train_loss : 4.7590883154618115\n", 818 | "Model saved in file: ./RNN_model/model.ckpt-221\n", 819 | "epoch : 222, train_loss : 4.749942177220394\n", 820 | "Model saved in file: ./RNN_model/model.ckpt-222\n", 821 | "epoch : 223, train_loss : 4.735043249632183\n", 822 | "Model saved in file: ./RNN_model/model.ckpt-223\n", 823 | "epoch : 224, train_loss : 4.740250361593146\n", 824 | "Model saved in file: ./RNN_model/model.ckpt-224\n", 825 | "epoch : 225, train_loss : 4.740147138896741\n", 826 | "Model saved in file: ./RNN_model/model.ckpt-225\n", 827 | "epoch : 226, train_loss : 4.740278620468942\n", 828 | "Model saved in file: ./RNN_model/model.ckpt-226\n", 829 | "epoch : 227, train_loss : 4.734238473992598\n", 830 | "Model saved in file: ./RNN_model/model.ckpt-227\n", 831 | "epoch : 228, train_loss : 4.731205864956505\n", 832 | "Model saved in file: ./RNN_model/model.ckpt-228\n", 833 | "epoch : 229, train_loss : 4.715315517626311\n", 834 | "Model saved in file: ./RNN_model/model.ckpt-229\n", 835 | "epoch : 230, train_loss : 4.709307018079255\n", 836 | "Model saved in file: ./RNN_model/model.ckpt-230\n", 837 | "epoch : 231, train_loss : 4.710273165451854\n", 838 | "Model saved in file: ./RNN_model/model.ckpt-231\n", 839 | "epoch : 232, train_loss : 4.707971447392514\n", 840 | "Model saved in file: ./RNN_model/model.ckpt-232\n", 841 | "epoch : 233, train_loss : 4.700421785053454\n", 842 | "Model saved in file: ./RNN_model/model.ckpt-233\n", 843 | "epoch : 234, train_loss : 4.707987433985661\n", 844 | "Model saved in file: ./RNN_model/model.ckpt-234\n", 845 | "epoch : 235, train_loss : 4.692016476079038\n", 846 | "Model saved in file: ./RNN_model/model.ckpt-235\n", 847 | "epoch : 236, train_loss : 4.690692048323781\n", 848 | "Model saved in file: ./RNN_model/model.ckpt-236\n", 849 | "epoch : 237, train_loss : 4.691278758801911\n", 850 | "Model saved in file: ./RNN_model/model.ckpt-237\n", 851 | "epoch : 238, train_loss : 4.67991966950266\n", 852 | "Model saved in file: ./RNN_model/model.ckpt-238\n", 853 | "epoch : 239, train_loss : 4.68348483035439\n", 854 | "Model saved in file: ./RNN_model/model.ckpt-239\n", 855 | "epoch : 240, train_loss : 4.677710759012323\n", 856 | "Model saved in file: ./RNN_model/model.ckpt-240\n", 857 | "epoch : 241, train_loss : 4.6693857343573315\n", 858 | "Model saved in file: ./RNN_model/model.ckpt-241\n", 859 | "epoch : 242, train_loss : 4.671603529076827\n", 860 | "Model saved in file: ./RNN_model/model.ckpt-242\n", 861 | "epoch : 243, train_loss : 4.659419737364116\n", 862 | "Model saved in file: ./RNN_model/model.ckpt-243\n", 863 | "epoch : 244, train_loss : 4.659056312159488\n", 864 | "Model saved in file: ./RNN_model/model.ckpt-244\n", 865 | "epoch : 245, train_loss : 4.664423290051911\n", 866 | "Model saved in file: ./RNN_model/model.ckpt-245\n", 867 | "epoch : 246, train_loss : 4.6546941556428605\n", 868 | "Model saved in file: ./RNN_model/model.ckpt-246\n", 869 | "epoch : 247, train_loss : 4.6566291859275415\n", 870 | "Model saved in file: ./RNN_model/model.ckpt-247\n", 871 | "epoch : 248, train_loss : 4.64633153614245\n", 872 | "Model saved in file: ./RNN_model/model.ckpt-248\n", 873 | "epoch : 249, train_loss : 4.656420506929097\n", 874 | "Model saved in file: ./RNN_model/model.ckpt-249\n", 875 | "epoch : 250, train_loss : 4.652243714583547\n", 876 | "Model saved in file: ./RNN_model/model.ckpt-250\n", 877 | "epoch : 251, train_loss : 4.650367460752788\n", 878 | "Model saved in file: ./RNN_model/model.ckpt-251\n", 879 | "epoch : 252, train_loss : 4.651430004521421\n", 880 | "Model saved in file: ./RNN_model/model.ckpt-252\n", 881 | "epoch : 253, train_loss : 4.642912789394981\n", 882 | "Model saved in file: ./RNN_model/model.ckpt-253\n", 883 | "epoch : 254, train_loss : 4.6306385492023665\n", 884 | "Model saved in file: ./RNN_model/model.ckpt-254\n", 885 | "epoch : 255, train_loss : 4.635086310537238\n", 886 | "Model saved in file: ./RNN_model/model.ckpt-255\n", 887 | "epoch : 256, train_loss : 4.630607002659848\n", 888 | "Model saved in file: ./RNN_model/model.ckpt-256\n", 889 | "epoch : 257, train_loss : 4.617120642411081\n", 890 | "Model saved in file: ./RNN_model/model.ckpt-257\n", 891 | "epoch : 258, train_loss : 4.6210913156208235\n", 892 | "Model saved in file: ./RNN_model/model.ckpt-258\n", 893 | "epoch : 259, train_loss : 4.627978600953755\n", 894 | "Model saved in file: ./RNN_model/model.ckpt-259\n", 895 | "epoch : 260, train_loss : 4.613597568712736\n", 896 | "Model saved in file: ./RNN_model/model.ckpt-260\n", 897 | "epoch : 261, train_loss : 4.615750513578716\n", 898 | "Model saved in file: ./RNN_model/model.ckpt-261\n", 899 | "epoch : 262, train_loss : 4.61929740403828\n", 900 | "Model saved in file: ./RNN_model/model.ckpt-262\n", 901 | "epoch : 263, train_loss : 4.607439241911235\n", 902 | "Model saved in file: ./RNN_model/model.ckpt-263\n", 903 | "epoch : 264, train_loss : 4.608241558074951\n", 904 | "Model saved in file: ./RNN_model/model.ckpt-264\n", 905 | "epoch : 265, train_loss : 4.605861463044819\n", 906 | "Model saved in file: ./RNN_model/model.ckpt-265\n", 907 | "epoch : 266, train_loss : 4.607786178588866\n", 908 | "Model saved in file: ./RNN_model/model.ckpt-266\n", 909 | "epoch : 267, train_loss : 4.604507496482447\n", 910 | "Model saved in file: ./RNN_model/model.ckpt-267\n", 911 | "epoch : 268, train_loss : 4.598436129720588\n", 912 | "Model saved in file: ./RNN_model/model.ckpt-268\n", 913 | "epoch : 269, train_loss : 4.595844068025288\n", 914 | "Model saved in file: ./RNN_model/model.ckpt-269\n", 915 | "epoch : 270, train_loss : 4.5908743456790315\n", 916 | "Model saved in file: ./RNN_model/model.ckpt-270\n", 917 | "epoch : 271, train_loss : 4.594808302427594\n", 918 | "Model saved in file: ./RNN_model/model.ckpt-271\n" 919 | ] 920 | }, 921 | { 922 | "name": "stdout", 923 | "output_type": "stream", 924 | "text": [ 925 | "epoch : 272, train_loss : 4.581478319670024\n", 926 | "Model saved in file: ./RNN_model/model.ckpt-272\n", 927 | "epoch : 273, train_loss : 4.599064099161248\n", 928 | "Model saved in file: ./RNN_model/model.ckpt-273\n", 929 | "epoch : 274, train_loss : 4.6004664019534465\n", 930 | "Model saved in file: ./RNN_model/model.ckpt-274\n", 931 | "epoch : 275, train_loss : 4.581155300140381\n", 932 | "Model saved in file: ./RNN_model/model.ckpt-275\n", 933 | "epoch : 276, train_loss : 4.590859287663509\n", 934 | "Model saved in file: ./RNN_model/model.ckpt-276\n", 935 | "epoch : 277, train_loss : 4.590206020756772\n", 936 | "Model saved in file: ./RNN_model/model.ckpt-277\n", 937 | "epoch : 278, train_loss : 4.578938358708433\n", 938 | "Model saved in file: ./RNN_model/model.ckpt-278\n", 939 | "epoch : 279, train_loss : 4.5749553881193465\n", 940 | "Model saved in file: ./RNN_model/model.ckpt-279\n", 941 | "epoch : 280, train_loss : 4.570446190081145\n", 942 | "Model saved in file: ./RNN_model/model.ckpt-280\n", 943 | "epoch : 281, train_loss : 4.578030310179058\n", 944 | "Model saved in file: ./RNN_model/model.ckpt-281\n", 945 | "epoch : 282, train_loss : 4.576507693842838\n", 946 | "Model saved in file: ./RNN_model/model.ckpt-282\n", 947 | "epoch : 283, train_loss : 4.571079229053698\n", 948 | "Model saved in file: ./RNN_model/model.ckpt-283\n", 949 | "epoch : 284, train_loss : 4.580598028082596\n", 950 | "Model saved in file: ./RNN_model/model.ckpt-284\n", 951 | "epoch : 285, train_loss : 4.56740319101434\n", 952 | "Model saved in file: ./RNN_model/model.ckpt-285\n", 953 | "epoch : 286, train_loss : 4.571033302106356\n", 954 | "Model saved in file: ./RNN_model/model.ckpt-286\n", 955 | "epoch : 287, train_loss : 4.570549764131245\n", 956 | "Model saved in file: ./RNN_model/model.ckpt-287\n", 957 | "epoch : 288, train_loss : 4.562460472709254\n", 958 | "Model saved in file: ./RNN_model/model.ckpt-288\n", 959 | "epoch : 289, train_loss : 4.557164367876554\n", 960 | "Model saved in file: ./RNN_model/model.ckpt-289\n", 961 | "epoch : 290, train_loss : 4.558068702095434\n", 962 | "Model saved in file: ./RNN_model/model.ckpt-290\n", 963 | "epoch : 291, train_loss : 4.550961042705335\n", 964 | "Model saved in file: ./RNN_model/model.ckpt-291\n", 965 | "epoch : 292, train_loss : 4.554549417997661\n", 966 | "Model saved in file: ./RNN_model/model.ckpt-292\n", 967 | "epoch : 293, train_loss : 4.551793926640561\n", 968 | "Model saved in file: ./RNN_model/model.ckpt-293\n", 969 | "epoch : 294, train_loss : 4.552393009788112\n", 970 | "Model saved in file: ./RNN_model/model.ckpt-294\n", 971 | "epoch : 295, train_loss : 4.5522077711004965\n", 972 | "Model saved in file: ./RNN_model/model.ckpt-295\n", 973 | "epoch : 296, train_loss : 4.561055509667646\n", 974 | "Model saved in file: ./RNN_model/model.ckpt-296\n", 975 | "epoch : 297, train_loss : 4.551473642650404\n", 976 | "Model saved in file: ./RNN_model/model.ckpt-297\n", 977 | "epoch : 298, train_loss : 4.548797808195415\n", 978 | "Model saved in file: ./RNN_model/model.ckpt-298\n", 979 | "epoch : 299, train_loss : 4.550925857142397\n", 980 | "Model saved in file: ./RNN_model/model.ckpt-299\n", 981 | "epoch : 300, train_loss : 4.551664352416992\n", 982 | "Model saved in file: ./RNN_model/model.ckpt-300\n" 983 | ] 984 | } 985 | ], 986 | "source": [ 987 | "max_epoch = 300\n", 988 | "batch_size = 64\n", 989 | "num_batch = int(num_data/batch_size)\n", 990 | "with tf.Session() as sess:\n", 991 | " sess.run(init)\n", 992 | " \n", 993 | " for _epoch in range(max_epoch):\n", 994 | " random.seed(_epoch)\n", 995 | " batch_shuffle = list(range(num_data))\n", 996 | " random.shuffle(batch_shuffle)\n", 997 | " \n", 998 | " total_train_loss = 0\n", 999 | " \n", 1000 | " for i in range(num_batch):\n", 1001 | " batch_idx = [batch_shuffle[idx] for idx in range(i*batch_size,\n", 1002 | " (i+1)*batch_size)]\n", 1003 | " batch_names = [name for name in data_name if data_name.index(name) in batch_idx]\n", 1004 | " batch_onehots = name_to_onehot(batch_names, chars, max_len)\n", 1005 | " \n", 1006 | " input_onehot = batch_onehots[:, 0:(max_len-1), :]\n", 1007 | " output_onehot = batch_onehots[:, 1:max_len, :]\n", 1008 | "\n", 1009 | " train_feed_dict = {ph_input_name: input_onehot, \n", 1010 | " ph_output_name: output_onehot}\n", 1011 | " \n", 1012 | " sess.run(optimizer, feed_dict = train_feed_dict)\n", 1013 | " curr_loss = sess.run(rnn_loss, feed_dict=train_feed_dict)\n", 1014 | " \n", 1015 | " total_train_loss += curr_loss/num_batch\n", 1016 | "\n", 1017 | " print('epoch : {}, train_loss : {}'.format(_epoch+1, total_train_loss))\n", 1018 | " \n", 1019 | " model_save_path = saver.save(sess, './RNN_model/model.ckpt', global_step=_epoch+1)\n", 1020 | " print('Model saved in file: {}'.format(model_save_path))" 1021 | ] 1022 | }, 1023 | { 1024 | "cell_type": "markdown", 1025 | "metadata": {}, 1026 | "source": [ 1027 | "## Define RNN to test" 1028 | ] 1029 | }, 1030 | { 1031 | "cell_type": "code", 1032 | "execution_count": 23, 1033 | "metadata": {}, 1034 | "outputs": [], 1035 | "source": [ 1036 | "ph_test_input_name = tf.placeholder(dtype=tf.float32, shape=[1, 1, dim_data])\n", 1037 | "ph_h = tf.placeholder(dtype=tf.float32, shape=[1, dim_rnn_cell])\n", 1038 | "ph_c = tf.placeholder(dtype=tf.float32, shape=[1, dim_rnn_cell])\n", 1039 | "\n", 1040 | "def name_rnn_test(_x, _dim_data, _dim_rnn_cell, _prev_h, _prev_c):\n", 1041 | " _x_split = tf.transpose(_x, [1, 0, 2]) # seq_len, batch, dim_data\n", 1042 | " _x_split = tf.reshape(_x_split, [-1, _dim_data])\n", 1043 | " \n", 1044 | " # Linear Operation for Input\n", 1045 | " # Fill here\n", 1046 | "\n", 1047 | " # Define LSTM Cell && RNN including (h, c)\n", 1048 | " # Fill here\n", 1049 | " \n", 1050 | " # Linear Operation for Ouput\n", 1051 | " _total_out = [] \n", 1052 | " # Fill here" 1053 | ] 1054 | }, 1055 | { 1056 | "cell_type": "markdown", 1057 | "metadata": {}, 1058 | "source": [ 1059 | "## Run Test" 1060 | ] 1061 | }, 1062 | { 1063 | "cell_type": "code", 1064 | "execution_count": 24, 1065 | "metadata": {}, 1066 | "outputs": [ 1067 | { 1068 | "name": "stdout", 1069 | "output_type": "stream", 1070 | "text": [ 1071 | "INFO:tensorflow:Restoring parameters from ./RNN_model/model.ckpt-300\n", 1072 | "Result Name : anique\n" 1073 | ] 1074 | } 1075 | ], 1076 | "source": [ 1077 | "test_result_name, test_state = name_rnn_test(ph_test_input_name, dim_data, dim_rnn_cell,\n", 1078 | " ph_h, ph_c)\n", 1079 | "\n", 1080 | "with tf.Session() as sess:\n", 1081 | " sess.run(init)\n", 1082 | " \n", 1083 | " saver.restore(sess, './RNN_model/model.ckpt-300')\n", 1084 | " \n", 1085 | " total_name = ''\n", 1086 | " prev_char = 'a'\n", 1087 | " total_name += prev_char\n", 1088 | " prev_state = (np.zeros((1, dim_rnn_cell)), np.zeros((1, dim_rnn_cell)))\n", 1089 | " for i in range(seq_len):\n", 1090 | " input_onehot = np.zeros((1, 1, dim_data))\n", 1091 | " prev_char_idx = chars.index(prev_char)\n", 1092 | " input_onehot[:, :, prev_char_idx] = 1\n", 1093 | " \n", 1094 | " test_feed_dict = {ph_test_input_name: input_onehot,\n", 1095 | " ph_h: prev_state[0], ph_c: prev_state[1]}\n", 1096 | " curr_result, curr_state = sess.run([test_result_name, test_state], test_feed_dict)\n", 1097 | " if np.argmax(curr_result) == dim_data-1:\n", 1098 | " break\n", 1099 | " else:\n", 1100 | " softmax_result = sess.run(tf.nn.softmax(test_result_name), test_feed_dict)\n", 1101 | " softmax_result = np.squeeze(softmax_result)\n", 1102 | " softmax_result = softmax_result[:dim_data-1]/sum(softmax_result[:dim_data-1])\n", 1103 | " prev_char = np.random.choice(chars, 1, p=softmax_result)\n", 1104 | " total_name += prev_char[0]\n", 1105 | " prev_state = curr_state \n", 1106 | " print('Result Name :', total_name)" 1107 | ] 1108 | } 1109 | ], 1110 | "metadata": { 1111 | "kernelspec": { 1112 | "display_name": "Python 3", 1113 | "language": "python", 1114 | "name": "python3" 1115 | }, 1116 | "language_info": { 1117 | "codemirror_mode": { 1118 | "name": "ipython", 1119 | "version": 3 1120 | }, 1121 | "file_extension": ".py", 1122 | "mimetype": "text/x-python", 1123 | "name": "python", 1124 | "nbconvert_exporter": "python", 1125 | "pygments_lexer": "ipython3", 1126 | "version": "3.6.8" 1127 | }, 1128 | "varInspector": { 1129 | "cols": { 1130 | "lenName": 16, 1131 | "lenType": 16, 1132 | "lenVar": 40 1133 | }, 1134 | "kernels_config": { 1135 | "python": { 1136 | "delete_cmd_postfix": "", 1137 | "delete_cmd_prefix": "del ", 1138 | "library": "var_list.py", 1139 | "varRefreshCmd": "print(var_dic_list())" 1140 | }, 1141 | "r": { 1142 | "delete_cmd_postfix": ") ", 1143 | "delete_cmd_prefix": "rm(", 1144 | "library": "var_list.r", 1145 | "varRefreshCmd": "cat(var_dic_list()) " 1146 | } 1147 | }, 1148 | "types_to_exclude": [ 1149 | "module", 1150 | "function", 1151 | "builtin_function_or_method", 1152 | "instance", 1153 | "_Feature" 1154 | ], 1155 | "window_display": false 1156 | } 1157 | }, 1158 | "nbformat": 4, 1159 | "nbformat_minor": 2 1160 | } 1161 | -------------------------------------------------------------------------------- /data/regression_data.txt: -------------------------------------------------------------------------------- 1 | 0.07344104893026171 0.30577654340464877 2 | -0.6671361041630213 0.21484387904157468 3 | 0.6110255704162972 0.35501391366259544 4 | -0.9845774527321076 0.20112862280820337 5 | -0.8837433910334389 0.2082755715198671 6 | -0.12854759010314276 0.3027415514693224 7 | 0.22099135724553243 0.3316209902246986 8 | 0.8534127709556332 0.38163625638979287 9 | 0.6242369517172701 0.3556166815262491 10 | -0.9884481919394896 0.18689766580127848 11 | -0.19720628238168114 0.26657175813563644 12 | 0.40892109591833226 0.3441350190637889 13 | 0.30833442465401917 0.33391826602343727 14 | -0.7683662711499288 0.2484192484125181 15 | 0.5810093587501926 0.36320411308762074 16 | -0.6130423180488918 0.22633849283964744 17 | 0.37080779728646673 0.3126308632491387 18 | 0.2689146356815888 0.33740516201145293 19 | 0.06920308546332121 0.3047139374541547 20 | 0.2582535575581715 0.3553355585400768 21 | 0.646076592630676 0.3695065507898268 22 | 0.06043786393526185 0.31084587768470995 23 | -0.15156313196207938 0.2760793081065102 24 | -0.930058893085125 0.21387698080707523 25 | 0.036524641335194064 0.30345621937957273 26 | 0.6304748305749586 0.3676698640539604 27 | -0.07516740680855216 0.2941553031464959 28 | -0.31744264827504054 0.27497961517334646 29 | -0.530769649065824 0.25596054269450985 30 | 0.832539226011729 0.38363705013933963 31 | -0.5377918900681617 0.24087606990797308 32 | 0.41861797886723906 0.34889992586169494 33 | 0.1623015779456236 0.3147683967398835 34 | 0.22209006549848076 0.32006745957964217 35 | -0.5552588651204258 0.2587647658231737 36 | 0.4721245881055858 0.3463014535533262 37 | 0.7529675286636983 0.3840323818043389 38 | 0.7415627461497138 0.3732452555671697 39 | 0.8829149304286472 0.3804828578513638 40 | 0.8060363026316897 0.3821564893581173 41 | 0.07683324784802203 0.2835805437966362 42 | -0.04241254210041734 0.29576905089163086 43 | 0.529685938257727 0.3395512139749959 44 | 0.0003760236074303869 0.3058898167419914 45 | 0.26242265067749715 0.31159207310504095 46 | -0.3804550924918553 0.2655427548710176 47 | 0.9113962745189543 0.3928625661091734 48 | 0.1830214848198568 0.32933621351115755 49 | 0.6652377843088599 0.37993359383835845 50 | 0.2462102541127391 0.31216380570837565 51 | -0.7655938018983295 0.21120322070459682 52 | 0.12891646349998864 0.32072340722783177 53 | 0.5975051754856993 0.358046175143507 54 | 0.6132702466298365 0.36853162758867714 55 | -0.32857986166521114 0.29571146543597493 56 | 0.35673858245777246 0.35152327189372484 57 | 0.018313416895856305 0.2875541287948366 58 | 0.6374087769466188 0.3573532551994217 59 | -0.5083941951560929 0.2447003301323157 60 | -0.3798196676918688 0.25691939102461053 61 | -0.7894805225544625 0.23140230758941666 62 | -0.1207824273538356 0.29929625578730806 63 | -0.9530285118985227 0.2075403750648572 64 | -0.311370647581825 0.27641447459832724 65 | -0.4912825554267988 0.259867778499509 66 | 0.8089974781758611 0.3744296044175936 67 | -0.029339651848966808 0.30973104778627997 68 | -0.3629689399567875 0.2713338385563311 69 | -0.28085773893119925 0.2829996343601426 70 | -0.3433383050664902 0.261235805827517 71 | 0.26819021337722804 0.31512381358502417 72 | 0.8379749185037964 0.3887617954552294 73 | -0.9721217602104253 0.21795242356774774 74 | 0.380942166529068 0.339779869625644 75 | -0.7028279008436107 0.2323194173868055 76 | -0.2484023903325714 0.2823861947490793 77 | -0.43535488718672366 0.24916216430359805 78 | 0.1930005450718788 0.3353753519312524 79 | -0.12975251016629397 0.29723894398369094 80 | 0.9537897512414495 0.3968293471866555 81 | -0.4674718628893888 0.2656561736924804 82 | -0.7521595268287959 0.2278348103272739 83 | -0.19006698759297302 0.27157786416830515 84 | -0.19192104501312923 0.2914769099490614 85 | 0.19410225598679975 0.329892830646774 86 | 0.19166765679801756 0.3098383895290911 87 | 0.03917784229508059 0.30601206544169496 88 | 0.18153546167642953 0.33479577937825894 89 | 0.5739667972748239 0.34436754509312273 90 | -0.16065727380382788 0.29868282753613895 91 | 0.8320151074446431 0.4081725110522518 92 | 0.006562278756251816 0.2927020475399281 93 | -0.05950895933915401 0.3082183282688187 94 | 0.9833690684791196 0.40469345506706506 95 | -0.23000484706137847 0.2710686912913266 96 | 0.8929405125595964 0.3897941565900382 97 | 0.37392170368698285 0.3176849445645886 98 | 0.6638252849338457 0.37162713059265773 99 | 0.0807277272988256 0.3253643285629878 100 | 0.18306645332943972 0.32224581911573247 101 | -------------------------------------------------------------------------------- /image/tensorboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rllab-snu/deep_learning_tutorial/a7beb2f012660af51341a7adaf4926c1abb3eac2/image/tensorboard.png --------------------------------------------------------------------------------