├── .gitignore
├── README.md
├── code
├── Tutorial1_Regression.ipynb
├── Tutorial2_MLP_MNIST.ipynb
├── Tutorial3-1_CNN_MNIST.ipynb
├── Tutorial3-2_CNN_CIFAR10.ipynb
└── Tutorial4_RNN_Name.ipynb
├── data
├── regression_data.txt
└── woman_name_dataset.csv
└── image
└── tensorboard.png
/.gitignore:
--------------------------------------------------------------------------------
1 | output/*
2 | .ipython_checkpoints/*
3 | .ipython_checkpoints
4 | answer
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Deep Learning Tutorial
2 |
3 | ### First Class
4 | - [Lecture - Introduction to deep learning 1](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/EZ_M4cpii7FPjywju0dcV9kBueyH0jjxCd_apKKT5xiLyQ?e=yKwg5J)
5 | - [Tutorial - Tensorflow and regression](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/EZaJDyjVX5lLpFSG9XL4MaoBGwBjRAgzgqmArvJDagMhJw?e=0Mydl8)
6 |
7 | ### Second Class
8 | - [Lecture - Introduction to deep learning 2](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/EX-Ob5JBQKhFkjI9JX2x9QgBZ87l5keRsUYe5-ekZllcug?e=fxceUO)
9 | - [Tutorial - Multi Layer Perceptrons](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/EUf_X57J1MNOuuHzre0QlbsBCnUQz0ppIEMsiCMxolD7gQ?e=MZyZA3)
10 |
11 | ### Third Class
12 | - [Lecture - Convolutional Neural Networks 1](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/ETz9uaskUHBKiKUyxFk2b48BEsejUfDciPLYGXBUipp6fA?e=0sS79W)
13 | - [Lecture - Convolutional Neural Networks 2](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/EQbk7z_8q8hIsiDImNHFJLYB_T5RkyIzH8Mj7-KBrHMZkA?e=r4WyOr)
14 | - [Tutorial - Convolutional Neural Networks(MNIST)](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/ESi06xRGkppJhookanl8pC8BWh-qhWMTrYhWwKU-KQHeyw?e=uRuZEu)
15 | - [Tutorial - Convolutional Neural Networks(CIFAR10)](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/EXjVnUGynA1EhOfkuDuS5LoBWHsDGj4xjwM259JWbuP2WA?e=ZZmtAF)
16 |
17 | ### Fourth Class
18 | - [Lecture - Recurrent Neural Networks](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/EVrxTDoIIGpNlWx0YqEkF5sBJ8Sns_uNIK1llLJifhEVyw?e=HjhIZl)
19 | - [Lecture - Advanced topics](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/EW6BL5tzgCFJnx_tNDJ1fW4B2QqE3fIXMpPiO2WPHt_98w?e=ajsode)
20 | - [Tutorial - Recurrent Neural Networks](https://mysnu-my.sharepoint.com/:b:/g/personal/blackfoot_seoul_ac_kr/EUl-ZNocKbVClPfLAQvPmp8BF1pC93MWfOdQDTiKlzR3og?e=lvObU6)
21 |
--------------------------------------------------------------------------------
/code/Tutorial1_Regression.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Practice Day 1. Regression"
8 | ]
9 | },
10 | {
11 | "cell_type": "markdown",
12 | "metadata": {},
13 | "source": [
14 | "### Load File"
15 | ]
16 | },
17 | {
18 | "cell_type": "code",
19 | "execution_count": null,
20 | "metadata": {},
21 | "outputs": [],
22 | "source": [
23 | "file = open('../data/regression_data.txt','r') # open the file with read-only\n",
24 | "text = file.readlines() # read all line texts\n",
25 | "file.close() # close the file\n",
26 | "\n",
27 | "x_data = []\n",
28 | "y_data = []\n",
29 | "\n",
30 | "# convert to float\n",
31 | "for s in text:\n",
32 | " data = s.split()\n",
33 | " x_data.append(float(data[0]))\n",
34 | " y_data.append(float(data[1])) "
35 | ]
36 | },
37 | {
38 | "cell_type": "markdown",
39 | "metadata": {},
40 | "source": [
41 | "### Plot data"
42 | ]
43 | },
44 | {
45 | "cell_type": "code",
46 | "execution_count": null,
47 | "metadata": {
48 | "scrolled": false
49 | },
50 | "outputs": [],
51 | "source": [
52 | "%matplotlib notebook\n",
53 | "import matplotlib.pyplot as plt\n",
54 | "import numpy as np\n",
55 | "\n",
56 | "plt.figure(1)\n",
57 | "plt.plot(x_data, y_data, 'ro') # plot data\n",
58 | "\n",
59 | "plt.xlabel('x-axis') \n",
60 | "plt.ylabel('y-axis')\n",
61 | "plt.title('My data')\n",
62 | "\n",
63 | "plt.show()"
64 | ]
65 | },
66 | {
67 | "cell_type": "markdown",
68 | "metadata": {},
69 | "source": [
70 | "### Closed Form Linear Regression"
71 | ]
72 | },
73 | {
74 | "cell_type": "code",
75 | "execution_count": null,
76 | "metadata": {
77 | "scrolled": true
78 | },
79 | "outputs": [],
80 | "source": [
81 | "# convert to numpy-array\n",
82 | "x_data = np.asarray(x_data)\n",
83 | "y_data = np.asarray(y_data)\n",
84 | "\n",
85 | "N = len(x_data) # the size of data\n",
86 | "\n",
87 | "sum_of_x =\n",
88 | "sum_of_x_square = \n",
89 | "sum_of_y = \n",
90 | "sum_of_xy = \n",
91 | "\n",
92 | "a = \n",
93 | "b = \n",
94 | "\n",
95 | "print ('a : {}'.format(a))\n",
96 | "print ('b : {}'.format(b))"
97 | ]
98 | },
99 | {
100 | "cell_type": "code",
101 | "execution_count": null,
102 | "metadata": {
103 | "scrolled": false
104 | },
105 | "outputs": [],
106 | "source": [
107 | "plt.figure(2)\n",
108 | "\n",
109 | "y_regression = a * x_data + b\n",
110 | "\n",
111 | "plt.plot(x_data, y_data, 'ro')\n",
112 | "plt.plot(x_data, y_regression, 'b')\n",
113 | "\n",
114 | "plt.xlabel('x-axis') \n",
115 | "plt.ylabel('y-axis')\n",
116 | "plt.title('Closed Form Regression')\n",
117 | "\n",
118 | "plt.show()"
119 | ]
120 | },
121 | {
122 | "cell_type": "markdown",
123 | "metadata": {},
124 | "source": [
125 | "### Tensorflow"
126 | ]
127 | },
128 | {
129 | "cell_type": "code",
130 | "execution_count": null,
131 | "metadata": {},
132 | "outputs": [],
133 | "source": [
134 | "import tensorflow as tf\n",
135 | "a = tf.add(3,5)\n",
136 | "print(a)"
137 | ]
138 | },
139 | {
140 | "cell_type": "code",
141 | "execution_count": null,
142 | "metadata": {},
143 | "outputs": [],
144 | "source": [
145 | "a = tf.add(3,5)\n",
146 | "sess = tf.Session()\n",
147 | "print(sess.run(a))\n",
148 | "sess.close()"
149 | ]
150 | },
151 | {
152 | "cell_type": "code",
153 | "execution_count": null,
154 | "metadata": {},
155 | "outputs": [],
156 | "source": [
157 | "x = 2\n",
158 | "y = 3\n",
159 | "add_op = tf.add(x,y)\n",
160 | "mul_op = tf.multiply(x,y)\n",
161 | "useless = tf.multiply(x, add_op)\n",
162 | "pow_op = tf.pow(add_op, mul_op)\n",
163 | "with tf.Session() as sess:\n",
164 | " z, not_useless = sess.run([pow_op, useless])"
165 | ]
166 | },
167 | {
168 | "cell_type": "code",
169 | "execution_count": null,
170 | "metadata": {},
171 | "outputs": [],
172 | "source": [
173 | "# create variables with tf.get variable\n",
174 | "s = tf.get_variable(\"scalar\", initializer=tf.constant(2))\n",
175 | "m = tf.get_variable(\"matrix\", initializer=tf.constant([[0,1],[2,3]]))\n",
176 | "W = tf.get_variable(\"big_matrix\", shape=(784,10), initializer=tf.zeros_initializer())"
177 | ]
178 | },
179 | {
180 | "cell_type": "code",
181 | "execution_count": null,
182 | "metadata": {},
183 | "outputs": [],
184 | "source": [
185 | "with tf.Session() as sess:\n",
186 | " sess.run(tf.global_variables_initializer())"
187 | ]
188 | },
189 | {
190 | "cell_type": "code",
191 | "execution_count": null,
192 | "metadata": {},
193 | "outputs": [],
194 | "source": [
195 | "a = tf.placeholder(tf.float32, shape=[3])\n",
196 | "\n",
197 | "b = tf.constant([5,5,5], tf.float32)\n",
198 | "\n",
199 | "c = a + b\n",
200 | "\n",
201 | "with tf.Session() as sess:\n",
202 | " print(sess.run(c,{a:[1,2,3]}))"
203 | ]
204 | },
205 | {
206 | "cell_type": "markdown",
207 | "metadata": {},
208 | "source": [
209 | "### Linear Regression with Gradient Descent"
210 | ]
211 | },
212 | {
213 | "cell_type": "code",
214 | "execution_count": null,
215 | "metadata": {},
216 | "outputs": [],
217 | "source": [
218 | "import tensorflow as tf\n",
219 | "\n",
220 | "# define constant\n",
221 | "c = tf.constant(0)\n",
222 | "\n",
223 | "# define variables (not initalized)\n",
224 | "w = tf.Variable(tf.random_uniform([1],-0.1,0.1))\n",
225 | "b = tf.Variable(tf.zeros([1]))\n",
226 | "\n",
227 | "# define placeholder\n",
228 | "x_data_placeholder = tf.placeholder('float32', shape=x_data.shape)\n",
229 | "y_data_placeholder = tf.placeholder('float32', shape=y_data.shape)\n",
230 | "\n",
231 | "# define loss function\n",
232 | "y_regression = w * x_data_placeholder + b\n",
233 | "\n",
234 | "loss = \n",
235 | "\n",
236 | "# define optimizer\n",
237 | "train = tf.train.GradientDescentOptimizer(learning_rate=0.7).minimize(loss)\n",
238 | "\n",
239 | "# initialize variables\n",
240 | "init = tf.global_variables_initializer()\n",
241 | "\n",
242 | "sess = tf.Session()\n",
243 | "sess.run(init) # run variable initializer : now you can use tensorflow variables"
244 | ]
245 | },
246 | {
247 | "cell_type": "code",
248 | "execution_count": null,
249 | "metadata": {},
250 | "outputs": [],
251 | "source": [
252 | "# see initial variables\n",
253 | "my_feed_dict = {x_data_placeholder:x_data, y_data_placeholder:y_data}\n",
254 | "\n",
255 | "initial_w, initial_b = sess.run([w,b], feed_dict=my_feed_dict)\n",
256 | "\n",
257 | "print(\"Initial variables w : {}, b : {}\".format(initial_w, initial_b)) "
258 | ]
259 | },
260 | {
261 | "cell_type": "code",
262 | "execution_count": null,
263 | "metadata": {
264 | "scrolled": false
265 | },
266 | "outputs": [],
267 | "source": [
268 | "plt.figure(3)\n",
269 | "plt.xlabel('x-axis') \n",
270 | "plt.ylabel('y-axis')\n",
271 | "plt.title('Linear Regression with Gradient Descent')\n",
272 | "\n",
273 | "plt.plot(x_data, y_data, 'ro')\n",
274 | "plt.plot(x_data, (initial_w * x_data + initial_b), label='initial')\n",
275 | "\n",
276 | "# training step\n",
277 | "\n",
278 | "for i in range():\n",
279 | " sess.run() # train operator\n",
280 | " current_w, current_b, current_loss = sess.run()# variable and loss \n",
281 | " print(\"iteration : {}, w : {}, b : {}, loss : {}\".format(i,current_w, current_b, current_loss))\n",
282 | " \n",
283 | " # display\n",
284 | " plt.plot(x_data, (current_w * x_data + current_b), label='iter : {}'.format(i))\n",
285 | " \n",
286 | "plt.legend()\n",
287 | "\n",
288 | "plt.show()"
289 | ]
290 | }
291 | ],
292 | "metadata": {
293 | "celltoolbar": "Raw Cell Format",
294 | "kernelspec": {
295 | "display_name": "Python 3",
296 | "language": "python",
297 | "name": "python3"
298 | },
299 | "language_info": {
300 | "codemirror_mode": {
301 | "name": "ipython",
302 | "version": 3
303 | },
304 | "file_extension": ".py",
305 | "mimetype": "text/x-python",
306 | "name": "python",
307 | "nbconvert_exporter": "python",
308 | "pygments_lexer": "ipython3",
309 | "version": "3.6.8"
310 | }
311 | },
312 | "nbformat": 4,
313 | "nbformat_minor": 2
314 | }
315 |
--------------------------------------------------------------------------------
/code/Tutorial2_MLP_MNIST.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "## IMPORT PACKAGES"
8 | ]
9 | },
10 | {
11 | "cell_type": "code",
12 | "execution_count": 3,
13 | "metadata": {},
14 | "outputs": [
15 | {
16 | "name": "stdout",
17 | "output_type": "stream",
18 | "text": [
19 | "CURRENT TF VERSION IS [1.2.1]\n",
20 | "PACKAGES LOADED\n"
21 | ]
22 | }
23 | ],
24 | "source": [
25 | "import numpy as np\n",
26 | "import tensorflow as tf\n",
27 | "import matplotlib.pyplot as plt\n",
28 | "from tensorflow.examples.tutorials.mnist import input_data\n",
29 | "print (\"CURRENT TF VERSION IS [%s]\" % (tf.__version__))\n",
30 | "print (\"PACKAGES LOADED\")"
31 | ]
32 | },
33 | {
34 | "cell_type": "markdown",
35 | "metadata": {},
36 | "source": [
37 | "## LOAD MNIST DATA"
38 | ]
39 | },
40 | {
41 | "cell_type": "code",
42 | "execution_count": null,
43 | "metadata": {
44 | "collapsed": true
45 | },
46 | "outputs": [],
47 | "source": [
48 | "mnist = input_data.read_data_sets('data/', one_hot=True)"
49 | ]
50 | },
51 | {
52 | "cell_type": "markdown",
53 | "metadata": {},
54 | "source": [
55 | "## INVESTIGATE MNIST DATA"
56 | ]
57 | },
58 | {
59 | "cell_type": "code",
60 | "execution_count": null,
61 | "metadata": {
62 | "collapsed": true
63 | },
64 | "outputs": [],
65 | "source": [
66 | "def print_np(x):\n",
67 | " print ('SHAPE OF is %s' % (x.shape,))\n",
68 | " print ('VALUES LOOK LIKE \\n %s' % (x))"
69 | ]
70 | },
71 | {
72 | "cell_type": "code",
73 | "execution_count": null,
74 | "metadata": {
75 | "collapsed": true
76 | },
77 | "outputs": [],
78 | "source": [
79 | "print_np(mnist.train.images)\n",
80 | "print_np(mnist.train.labels)"
81 | ]
82 | },
83 | {
84 | "cell_type": "code",
85 | "execution_count": null,
86 | "metadata": {
87 | "collapsed": true
88 | },
89 | "outputs": [],
90 | "source": [
91 | "print_np(mnist.test.images)\n",
92 | "print_np(mnist.test.labels)"
93 | ]
94 | },
95 | {
96 | "cell_type": "code",
97 | "execution_count": null,
98 | "metadata": {
99 | "collapsed": true
100 | },
101 | "outputs": [],
102 | "source": [
103 | "print_np(mnist.validation.images)\n",
104 | "print_np(mnist.validation.labels)"
105 | ]
106 | },
107 | {
108 | "cell_type": "markdown",
109 | "metadata": {},
110 | "source": [
111 | "## DRAW MNIST DATA "
112 | ]
113 | },
114 | {
115 | "cell_type": "code",
116 | "execution_count": null,
117 | "metadata": {
118 | "collapsed": true
119 | },
120 | "outputs": [],
121 | "source": [
122 | "ntrain = mnist.train.images.shape[0]\n",
123 | "nsample = 3\n",
124 | "randidx = np.random.randint(ntrain, size=nsample)\n",
125 | "for i in randidx:\n",
126 | " imgvec = mnist.train.images[i, :]\n",
127 | " labelvec = mnist.train.labels[i, :]\n",
128 | " img = np.reshape(imgvec, (28, 28))\n",
129 | " label = np.argmax(labelvec)\n",
130 | " plt.matshow(img, cmap=plt.get_cmap('gray'))\n",
131 | " plt.title(\"[%d] DATA / LABEL IS [%d]\"%(i, label))"
132 | ]
133 | },
134 | {
135 | "cell_type": "markdown",
136 | "metadata": {},
137 | "source": [
138 | "## GET RANDOM BATCH"
139 | ]
140 | },
141 | {
142 | "cell_type": "code",
143 | "execution_count": null,
144 | "metadata": {
145 | "collapsed": true
146 | },
147 | "outputs": [],
148 | "source": [
149 | "ntrain = 10\n",
150 | "randindices = np.random.permutation(ntrain)\n",
151 | "print (randindices.shape)"
152 | ]
153 | },
154 | {
155 | "cell_type": "code",
156 | "execution_count": null,
157 | "metadata": {
158 | "collapsed": true
159 | },
160 | "outputs": [],
161 | "source": [
162 | "ntrain = 10\n",
163 | "nbatch = 4\n",
164 | "niter = ntrain // nbatch + 1\n",
165 | "for i in range(niter):\n",
166 | " currindices = randindices[i*nbatch:(i+1)*nbatch]\n",
167 | " print (\"ITER: [%d] BATCH INDEX: %s\" % (i, currindices))\n",
168 | " # GET BATCH\n",
169 | " xbatch = mnist.train.images[currindices, :]\n",
170 | " ybatch = mnist.train.labels[currindices, :]\n",
171 | " print (\" - SHAPE OF 'XBATCH' IS %s\" % (xbatch.shape,))\n",
172 | " print (\" - SHAPE OF 'YBATCH' IS %s\" % (ybatch.shape,))"
173 | ]
174 | },
175 | {
176 | "cell_type": "markdown",
177 | "metadata": {},
178 | "source": [
179 | "## DEFINE MODEL"
180 | ]
181 | },
182 | {
183 | "cell_type": "code",
184 | "execution_count": null,
185 | "metadata": {
186 | "collapsed": true
187 | },
188 | "outputs": [],
189 | "source": [
190 | "# NETWORK TOPOLOGIES\n",
191 | "n_hidden_1 = 256 \n",
192 | "n_hidden_2 = 128 \n",
193 | "n_input = 784 \n",
194 | "n_classes = 10\n",
195 | "\n",
196 | "# INPUTS AND OUTPUTS\n",
197 | "x = # [EXERCISE 1]\n",
198 | "y = # [EXERCISE 1]\n",
199 | " \n",
200 | "# NETWORK PARAMETERS\n",
201 | "stddev = 0.1\n",
202 | "weights = {\n",
203 | " 'h1': # [EXERCISE 1],\n",
204 | " 'h2': # [EXERCISE 1],\n",
205 | " 'out': # [EXERCISE 1]\n",
206 | "}\n",
207 | "biases = {\n",
208 | " 'b1': # [EXERCISE 1],\n",
209 | " 'b2': # [EXERCISE 1],\n",
210 | " 'out': # [EXERCISE 1]\n",
211 | "}\n",
212 | "print (\"NETWORK READY\")"
213 | ]
214 | },
215 | {
216 | "cell_type": "markdown",
217 | "metadata": {},
218 | "source": [
219 | "## BUILD GRAPH"
220 | ]
221 | },
222 | {
223 | "cell_type": "code",
224 | "execution_count": null,
225 | "metadata": {
226 | "collapsed": true
227 | },
228 | "outputs": [],
229 | "source": [
230 | "# MODEL\n",
231 | "def multilayer_perceptron(_X, _weights, _biases):\n",
232 | " # [EXERCISE 2]\n",
233 | "\n",
234 | "# PREDICTION\n",
235 | "pred = multilayer_perceptron(x, weights, biases)\n",
236 | "\n",
237 | "# LOSS AND OPTIMIZER\n",
238 | "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=# [EXERCISE 3], logits=# [EXERCISE 3]))\n",
239 | "optm = tf.train.AdamOptimizer(learning_rate=0.001).minimize(cost) \n",
240 | "corr = # [EXERCISE 3]\n",
241 | "accr = tf.reduce_mean(tf.cast(corr, \"float\"))\n",
242 | "\n",
243 | "# INITIALIZER\n",
244 | "init = tf.global_variables_initializer()\n",
245 | "print (\"FUNCTIONS READY\")"
246 | ]
247 | },
248 | {
249 | "cell_type": "markdown",
250 | "metadata": {},
251 | "source": [
252 | "## RUN"
253 | ]
254 | },
255 | {
256 | "cell_type": "code",
257 | "execution_count": null,
258 | "metadata": {
259 | "collapsed": true
260 | },
261 | "outputs": [],
262 | "source": [
263 | "# PARAMETERS\n",
264 | "training_epochs = 20\n",
265 | "batch_size = 100\n",
266 | "display_step = 4\n",
267 | "# LAUNCH THE GRAPH\n",
268 | "sess = tf.Session()\n",
269 | "sess.run(init)\n",
270 | "# OPTIMIZE\n",
271 | "for epoch in range(training_epochs):\n",
272 | " avg_cost = 0.\n",
273 | " total_batch = int(mnist.train.num_examples/batch_size)\n",
274 | " # ITERATION\n",
275 | " for i in range(total_batch):\n",
276 | " batch_xs, batch_ys = mnist.train.next_batch(batch_size)\n",
277 | " feeds = {x: batch_xs, y: batch_ys}\n",
278 | " sess.run(optm, feed_dict=feeds)\n",
279 | " avg_cost += sess.run(cost, feed_dict=feeds)\n",
280 | " avg_cost = avg_cost / total_batch\n",
281 | " # DISPLAY\n",
282 | " if (epoch+1) % display_step == 0:\n",
283 | " print (\"Epoch: %03d/%03d cost: %.9f\" % (epoch, training_epochs, avg_cost))\n",
284 | " feeds = {x: batch_xs, y: batch_ys}\n",
285 | " train_acc = sess.run(accr, feed_dict=feeds)\n",
286 | " print (\"TRAIN ACCURACY: %.3f\" % (train_acc))\n",
287 | " feeds = {x: mnist.test.images, y: mnist.test.labels}\n",
288 | " test_acc = sess.run(accr, feed_dict=feeds)\n",
289 | " print (\"TEST ACCURACY: %.3f\" % (test_acc))\n",
290 | "print (\"OPTIMIZATION FINISHED\")"
291 | ]
292 | }
293 | ],
294 | "metadata": {
295 | "kernelspec": {
296 | "display_name": "Python [conda env:tensorflow]",
297 | "language": "python",
298 | "name": "conda-env-tensorflow-py"
299 | },
300 | "language_info": {
301 | "codemirror_mode": {
302 | "name": "ipython",
303 | "version": 3
304 | },
305 | "file_extension": ".py",
306 | "mimetype": "text/x-python",
307 | "name": "python",
308 | "nbconvert_exporter": "python",
309 | "pygments_lexer": "ipython3",
310 | "version": "3.6.1"
311 | }
312 | },
313 | "nbformat": 4,
314 | "nbformat_minor": 2
315 | }
316 |
--------------------------------------------------------------------------------
/code/Tutorial3-1_CNN_MNIST.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "## CONVOLUTIONAL NEURAL NETWORK (MNIST)"
8 | ]
9 | },
10 | {
11 | "cell_type": "code",
12 | "execution_count": null,
13 | "metadata": {},
14 | "outputs": [],
15 | "source": [
16 | "import os\n",
17 | "import numpy as np\n",
18 | "import matplotlib.pyplot as plt\n",
19 | "import tensorflow as tf\n",
20 | "from tensorflow.examples.tutorials.mnist import input_data\n",
21 | "%matplotlib inline \n",
22 | "print (\"CURRENT TF VERSION IS [%s]\" % (tf.__version__))\n",
23 | "print (\"PACKAGES LOADED\")"
24 | ]
25 | },
26 | {
27 | "cell_type": "markdown",
28 | "metadata": {},
29 | "source": [
30 | "## LOAD MNIST"
31 | ]
32 | },
33 | {
34 | "cell_type": "code",
35 | "execution_count": null,
36 | "metadata": {},
37 | "outputs": [],
38 | "source": [
39 | "mnist = input_data.read_data_sets('data/', one_hot=True)\n",
40 | "trainimg = mnist.train.images\n",
41 | "trainlabel = mnist.train.labels\n",
42 | "testimg = mnist.test.images\n",
43 | "testlabel = mnist.test.labels\n",
44 | "print (\"MNIST ready\")"
45 | ]
46 | },
47 | {
48 | "cell_type": "markdown",
49 | "metadata": {},
50 | "source": [
51 | "## DEFINE MODEL"
52 | ]
53 | },
54 | {
55 | "cell_type": "code",
56 | "execution_count": null,
57 | "metadata": {},
58 | "outputs": [],
59 | "source": [
60 | "# NETWORK TOPOLOGIES\n",
61 | "n_input = 784\n",
62 | "n_channel = 64 \n",
63 | "n_classes = 10 \n",
64 | "\n",
65 | "# INPUTS AND OUTPUTS\n",
66 | "x = tf.placeholder(\"float\", [None, n_input])\n",
67 | "y = tf.placeholder(\"float\", [None, n_classes])\n",
68 | " \n",
69 | "# NETWORK PARAMETERS\n",
70 | "stddev = 0.1\n",
71 | "weights = {\n",
72 | " 'c1': tf.Variable(tf.random_normal([7, 7, 1, n_channel], stddev=stddev)),\n",
73 | " 'd1': tf.Variable(tf.random_normal([14*14*64, n_classes], stddev=stddev))\n",
74 | "}\n",
75 | "biases = {\n",
76 | " 'c1': tf.Variable(tf.random_normal([n_channel], stddev=stddev)),\n",
77 | " 'd1': tf.Variable(tf.random_normal([n_classes], stddev=stddev))\n",
78 | "}\n",
79 | "print (\"NETWORK READY\")"
80 | ]
81 | },
82 | {
83 | "cell_type": "markdown",
84 | "metadata": {},
85 | "source": [
86 | "## DEFINE GRAPH"
87 | ]
88 | },
89 | {
90 | "cell_type": "code",
91 | "execution_count": null,
92 | "metadata": {
93 | "collapsed": true
94 | },
95 | "outputs": [],
96 | "source": [
97 | "# MODEL\n",
98 | "def CNN(_x, _w, _b):\n",
99 | " # RESHAPE\n",
100 | " _x_r = tf.reshape(_x, shape=[-1, 28, 28, 1])\n",
101 | " # TODO1: CONVOLUTION\n",
102 | " # _conv1 = \n",
103 | " # ADD BIAS\n",
104 | " _conv2 = tf.nn.bias_add(_conv1, _b['c1'])\n",
105 | " # RELU\n",
106 | " _conv3 = tf.nn.relu(_conv2)\n",
107 | " # TODO2: MAX-POOL\n",
108 | " #_pool = \n",
109 | " # VECTORIZE\n",
110 | " _dense = tf.reshape(_pool, [-1, _w['d1'].get_shape().as_list()[0]])\n",
111 | " # DENSE\n",
112 | " _logit = tf.add(tf.matmul(_dense, _w['d1']), _b['d1'])\n",
113 | " _out = {\n",
114 | " 'x_r': _x_r, 'conv1': _conv1, 'conv2': _conv2, 'conv3': _conv3\n",
115 | " , 'pool': _pool, 'dense': _dense, 'logit': _logit\n",
116 | " }\n",
117 | " return _out\n",
118 | "\n",
119 | "# PREDICTION\n",
120 | "cnnout = CNN(x, weights, biases)\n",
121 | "\n",
122 | "# LOSS AND OPTIMIZER\n",
123 | "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n",
124 | " labels=y, logits=cnnout['logit']))\n",
125 | "optm = tf.train.AdamOptimizer(learning_rate=0.001).minimize(cost) \n",
126 | "corr = tf.equal(tf.argmax(cnnout['logit'], 1), tf.argmax(y, 1)) \n",
127 | "accr = tf.reduce_mean(tf.cast(corr, \"float\"))\n",
128 | "\n",
129 | "# INITIALIZER\n",
130 | "init = tf.global_variables_initializer()\n",
131 | "print (\"FUNCTIONS READY\")"
132 | ]
133 | },
134 | {
135 | "cell_type": "markdown",
136 | "metadata": {},
137 | "source": [
138 | "## SAVER"
139 | ]
140 | },
141 | {
142 | "cell_type": "code",
143 | "execution_count": null,
144 | "metadata": {},
145 | "outputs": [],
146 | "source": [
147 | "dirpath = \"data/nets\"\n",
148 | "if not os.path.exists(dirpath):\n",
149 | " os.makedirs(dirpath)\n",
150 | "savedir = \"data/nets/cnn_mnist_simple/\"\n",
151 | "saver = tf.train.Saver(max_to_keep=3)\n",
152 | "save_step = 5\n",
153 | "if not os.path.exists(savedir):\n",
154 | " os.makedirs(savedir)\n",
155 | "print (\"SAVER READY\")"
156 | ]
157 | },
158 | {
159 | "cell_type": "markdown",
160 | "metadata": {},
161 | "source": [
162 | "## RUN"
163 | ]
164 | },
165 | {
166 | "cell_type": "code",
167 | "execution_count": null,
168 | "metadata": {},
169 | "outputs": [],
170 | "source": [
171 | "# PARAMETERS\n",
172 | "training_epochs = 5\n",
173 | "batch_size = 32\n",
174 | "display_step = 1\n",
175 | "# LAUNCH THE GRAPH\n",
176 | "sess = tf.Session()\n",
177 | "sess.run(init)\n",
178 | "# OPTIMIZE\n",
179 | "for epoch in range(training_epochs):\n",
180 | " avg_cost = 0.\n",
181 | " total_batch = int(mnist.train.num_examples/batch_size)\n",
182 | " # ITERATION\n",
183 | " for i in range(total_batch):\n",
184 | " # TODO3: Fill in the for loop where we run the graph, compute the cost, and optimize.\n",
185 | " #\n",
186 | " #\n",
187 | " #\n",
188 | " #\n",
189 | " avg_cost = avg_cost / total_batch\n",
190 | " # DISPLAY\n",
191 | " if (epoch+1) % display_step == 0:\n",
192 | " print (\"Epoch: %03d/%03d cost: %.9f\" % (epoch+1, training_epochs, avg_cost))\n",
193 | " feeds = {x: batch_xs, y: batch_ys}\n",
194 | " train_acc = sess.run(accr, feed_dict=feeds)\n",
195 | " print (\"TRAIN ACCURACY: %.3f\" % (train_acc))\n",
196 | " feeds = {x: mnist.test.images, y: mnist.test.labels}\n",
197 | " test_acc = sess.run(accr, feed_dict=feeds)\n",
198 | " print (\"TEST ACCURACY: %.3f\" % (test_acc))\n",
199 | " # SAVE\n",
200 | " if (epoch+1) % save_step == 0:\n",
201 | " savename = savedir+\"net-\"+str(epoch+1)+\".ckpt\"\n",
202 | " saver.save(sess, savename)\n",
203 | " print (\"[%s] SAVED.\" % (savename))\n",
204 | "print (\"OPTIMIZATION FINISHED\")"
205 | ]
206 | },
207 | {
208 | "cell_type": "markdown",
209 | "metadata": {},
210 | "source": [
211 | "## RESTORE"
212 | ]
213 | },
214 | {
215 | "cell_type": "code",
216 | "execution_count": null,
217 | "metadata": {},
218 | "outputs": [],
219 | "source": [
220 | "do_restore = 0\n",
221 | "if do_restore == 1:\n",
222 | " sess = tf.Session()\n",
223 | " epoch = 5\n",
224 | " savename = savedir+\"net-\"+str(epoch)+\".ckpt\"\n",
225 | " saver.restore(sess, savename)\n",
226 | " print (\"NETWORK RESTORED\")\n",
227 | "else:\n",
228 | " print (\"DO NOTHING\")"
229 | ]
230 | },
231 | {
232 | "cell_type": "markdown",
233 | "metadata": {},
234 | "source": [
235 | "## SEE HOW IT WORKS"
236 | ]
237 | },
238 | {
239 | "cell_type": "code",
240 | "execution_count": null,
241 | "metadata": {
242 | "collapsed": true
243 | },
244 | "outputs": [],
245 | "source": [
246 | "# FEEDFORWARD THE FIRST TRAIN IMAGE\n",
247 | "input_r = sess.run(cnnout['x_r'], feed_dict={x: trainimg[0:1, :]})\n",
248 | "conv1 = sess.run(cnnout['conv1'], feed_dict={x: trainimg[0:1, :]})\n",
249 | "conv2 = sess.run(cnnout['conv2'], feed_dict={x: trainimg[0:1, :]})\n",
250 | "conv3 = sess.run(cnnout['conv3'], feed_dict={x: trainimg[0:1, :]})\n",
251 | "pool = sess.run(cnnout['pool'], feed_dict={x: trainimg[0:1, :]})\n",
252 | "dense = sess.run(cnnout['dense'], feed_dict={x: trainimg[0:1, :]})\n",
253 | "out = sess.run(cnnout['logit'], feed_dict={x: trainimg[0:1, :]})"
254 | ]
255 | },
256 | {
257 | "cell_type": "markdown",
258 | "metadata": {},
259 | "source": [
260 | "## INPUT"
261 | ]
262 | },
263 | {
264 | "cell_type": "code",
265 | "execution_count": null,
266 | "metadata": {},
267 | "outputs": [],
268 | "source": [
269 | "print (\"Size of 'input_r' is %s\" % (input_r.shape,))\n",
270 | "label = np.argmax(trainlabel[0, :])\n",
271 | "print (\"Label is %d\" % (label))\n",
272 | "\n",
273 | "# PLOT\n",
274 | "plt.matshow(input_r[0, :, :, 0], cmap=plt.get_cmap('gray'))\n",
275 | "plt.title(\"Label of this image is \" + str(label) + \"\")\n",
276 | "plt.colorbar()\n",
277 | "plt.show()"
278 | ]
279 | },
280 | {
281 | "cell_type": "markdown",
282 | "metadata": {},
283 | "source": [
284 | "## CONV"
285 | ]
286 | },
287 | {
288 | "cell_type": "code",
289 | "execution_count": null,
290 | "metadata": {
291 | "scrolled": true
292 | },
293 | "outputs": [],
294 | "source": [
295 | "print (\"SIZE OF 'CONV1' IS %s\" % (conv1.shape,))\n",
296 | "for i in range(3):\n",
297 | " plt.matshow(conv1[0, :, :, i], cmap=plt.get_cmap('gray'))\n",
298 | " plt.title(str(i) + \"th conv1\")\n",
299 | " plt.colorbar()\n",
300 | " plt.show()"
301 | ]
302 | },
303 | {
304 | "cell_type": "markdown",
305 | "metadata": {},
306 | "source": [
307 | "## CONV + BIAS"
308 | ]
309 | },
310 | {
311 | "cell_type": "code",
312 | "execution_count": null,
313 | "metadata": {},
314 | "outputs": [],
315 | "source": [
316 | "print (\"SIZE OF 'CONV2' IS %s\" % (conv2.shape,))\n",
317 | "for i in range(3):\n",
318 | " plt.matshow(conv2[0, :, :, i], cmap=plt.get_cmap('gray'))\n",
319 | " plt.title(str(i) + \"th conv2\")\n",
320 | " plt.colorbar()\n",
321 | " plt.show()"
322 | ]
323 | },
324 | {
325 | "cell_type": "markdown",
326 | "metadata": {},
327 | "source": [
328 | "## CONV + BIAS + RELU"
329 | ]
330 | },
331 | {
332 | "cell_type": "code",
333 | "execution_count": null,
334 | "metadata": {},
335 | "outputs": [],
336 | "source": [
337 | "print (\"SIZE OF 'CONV3' IS %s\" % (conv3.shape,))\n",
338 | "for i in range(3):\n",
339 | " plt.matshow(conv3[0, :, :, i], cmap=plt.get_cmap('gray'))\n",
340 | " plt.title(str(i) + \"th conv3\")\n",
341 | " plt.colorbar()\n",
342 | " plt.show()"
343 | ]
344 | },
345 | {
346 | "cell_type": "markdown",
347 | "metadata": {},
348 | "source": [
349 | "## POOL"
350 | ]
351 | },
352 | {
353 | "cell_type": "code",
354 | "execution_count": null,
355 | "metadata": {},
356 | "outputs": [],
357 | "source": [
358 | "print (\"SIZE OF 'POOL' IS %s\" % (pool.shape,))\n",
359 | "for i in range(3):\n",
360 | " plt.matshow(pool[0, :, :, i], cmap=plt.get_cmap('gray'))\n",
361 | " plt.title(str(i) + \"th pool\")\n",
362 | " plt.colorbar()\n",
363 | " plt.show()"
364 | ]
365 | },
366 | {
367 | "cell_type": "markdown",
368 | "metadata": {},
369 | "source": [
370 | "## DENSE"
371 | ]
372 | },
373 | {
374 | "cell_type": "code",
375 | "execution_count": null,
376 | "metadata": {},
377 | "outputs": [],
378 | "source": [
379 | "print (\"SIZE OF 'DENSE' IS %s\" % (dense.shape,))\n",
380 | "print (\"SIZE OF 'OUT' IS %s\" % (out.shape,))\n",
381 | "plt.matshow(out, cmap=plt.get_cmap('gray'))\n",
382 | "plt.title(\"OUT\")\n",
383 | "plt.colorbar()\n",
384 | "plt.show()\n",
385 | "\n",
386 | "print (\"PREDICTED LABEL IS [%d]\" % np.argmax(out))"
387 | ]
388 | },
389 | {
390 | "cell_type": "markdown",
391 | "metadata": {},
392 | "source": [
393 | "## CONV FILTER"
394 | ]
395 | },
396 | {
397 | "cell_type": "code",
398 | "execution_count": null,
399 | "metadata": {},
400 | "outputs": [],
401 | "source": [
402 | "wc1 = sess.run(weights['c1'])\n",
403 | "print (\"SIZE OF 'WC1' IS %s\" % (wc1.shape,))\n",
404 | "for i in range(3):\n",
405 | " plt.matshow(wc1[:, :, 0, i], cmap=plt.get_cmap('gray'))\n",
406 | " plt.title(str(i) + \"th conv filter\")\n",
407 | " plt.colorbar()\n",
408 | " plt.show()"
409 | ]
410 | }
411 | ],
412 | "metadata": {
413 | "kernelspec": {
414 | "display_name": "Python 3",
415 | "language": "python",
416 | "name": "python3"
417 | },
418 | "language_info": {
419 | "codemirror_mode": {
420 | "name": "ipython",
421 | "version": 3
422 | },
423 | "file_extension": ".py",
424 | "mimetype": "text/x-python",
425 | "name": "python",
426 | "nbconvert_exporter": "python",
427 | "pygments_lexer": "ipython3",
428 | "version": "3.5.4"
429 | }
430 | },
431 | "nbformat": 4,
432 | "nbformat_minor": 1
433 | }
434 |
--------------------------------------------------------------------------------
/code/Tutorial3-2_CNN_CIFAR10.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Deep Learning Tutorial
Day 4. CIFAR10 Classification\n",
8 | "\n",
9 | "Written by Nuri Kim.\n",
10 | "This code can be downloaded from [GitHub](https://github.com/bareblackfoot/deep_learning_tutorial).\n",
11 | "\n",
12 | "References of this notebook: \n",
13 | "*TensorFlow-Tutorials by [Magnus Erik Hvass Pedersen](http://www.hvass-labs.org/)* and \n",
14 | "*CIFAR-10 by [dhanushkamath](https://github.com/dhanushkamath/CIFAR-10/blob/master/Cifar10.ipynb)*"
15 | ]
16 | },
17 | {
18 | "cell_type": "markdown",
19 | "metadata": {},
20 | "source": [
21 | "## Introduction\n",
22 | "\n",
23 | "This tutorial demonstrates the workflow of an image classification using TensorFlow with convolutional networks. Unlike MNIST dataset, CIFAR10 dataset has color images of 10 categories. Here, we define and optimize a simple mathematical model in TensorFlow. The results are then plotted."
24 | ]
25 | },
26 | {
27 | "cell_type": "markdown",
28 | "metadata": {},
29 | "source": [
30 | "## Imports"
31 | ]
32 | },
33 | {
34 | "cell_type": "code",
35 | "execution_count": null,
36 | "metadata": {},
37 | "outputs": [],
38 | "source": [
39 | "from glob import glob\n",
40 | "from time import time\n",
41 | "from sklearn.manifold import TSNE\n",
42 | "from sklearn.preprocessing import LabelEncoder\n",
43 | "from sklearn.metrics import confusion_matrix\n",
44 | "try:\n",
45 | " from urllib.request import urlretrieve\n",
46 | "except:\n",
47 | " from urllib import urlretrieve\n",
48 | "import pandas as pd\n",
49 | "import numpy as np\n",
50 | "import sys, os, math\n",
51 | "import tensorflow as tf\n",
52 | "import tensorflow.contrib.slim as slim\n",
53 | "import matplotlib.pyplot as plt\n",
54 | "import itertools\n",
55 | "import tarfile\n",
56 | "import pickle\n",
57 | "\n",
58 | "print (\"Packages loaded\")"
59 | ]
60 | },
61 | {
62 | "cell_type": "markdown",
63 | "metadata": {},
64 | "source": [
65 | "### Set parameters"
66 | ]
67 | },
68 | {
69 | "cell_type": "code",
70 | "execution_count": null,
71 | "metadata": {},
72 | "outputs": [],
73 | "source": [
74 | "# PARAMS\n",
75 | "_BATCH_SIZE = 32\n",
76 | "_EPOCH = 10\n",
77 | "_SNAPSHOT_PREFIX = \"cifar10\"\n",
78 | "_NUM_VIS_EMBEDDING = 500\n",
79 | "_IMAGE_HEIGHT = 32\n",
80 | "_IMAGE_WIDTH = 32\n",
81 | "_IMAGE_CHANNELS = 3\n",
82 | "class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']\n",
83 | "_NUM_CLASSES = len(class_names)\n",
84 | "class_to_num = dict(zip(class_names, range(_NUM_CLASSES)))\n",
85 | "num_to_class = dict(zip(range(_NUM_CLASSES), class_names))\n",
86 | "lr = 1e-3"
87 | ]
88 | },
89 | {
90 | "cell_type": "markdown",
91 | "metadata": {},
92 | "source": [
93 | "This was developed using Python 3.5 and TensorFlow version 1.10.1. Please check yours:"
94 | ]
95 | },
96 | {
97 | "cell_type": "code",
98 | "execution_count": null,
99 | "metadata": {},
100 | "outputs": [],
101 | "source": [
102 | "tf.__version__"
103 | ]
104 | },
105 | {
106 | "cell_type": "markdown",
107 | "metadata": {},
108 | "source": [
109 | "### Helper-functions for loading images"
110 | ]
111 | },
112 | {
113 | "cell_type": "markdown",
114 | "metadata": {},
115 | "source": [
116 | "Functions used to download dataset and pre-process images."
117 | ]
118 | },
119 | {
120 | "cell_type": "code",
121 | "execution_count": null,
122 | "metadata": {},
123 | "outputs": [],
124 | "source": [
125 | "def get_data_set(name=\"train\"):\n",
126 | " x = y = None\n",
127 | " maybe_download_and_extract()\n",
128 | " folder_name = \"cifar10\"\n",
129 | " f = open('./data/'+folder_name+'/batches.meta', 'rb')\n",
130 | " f.close()\n",
131 | "\n",
132 | " if name is \"train\":\n",
133 | " for i in range(5):\n",
134 | " f = open('./data/'+folder_name+'/data_batch_' + str(i + 1), 'rb')\n",
135 | " try:\n",
136 | " datadict = pickle.load(f, encoding='latin1') \n",
137 | " except:\n",
138 | " datadict = pickle.load(f)\n",
139 | " f.close()\n",
140 | "\n",
141 | " _X = datadict[\"data\"]\n",
142 | " _Y = datadict['labels']\n",
143 | "\n",
144 | " _X = np.array(_X, dtype=float) / 255.0\n",
145 | " _X = _X.reshape([-1, _IMAGE_CHANNELS, _IMAGE_HEIGHT, _IMAGE_WIDTH])\n",
146 | " _X = _X.transpose([0, 2, 3, 1])\n",
147 | " _X = _X.reshape(-1, _IMAGE_HEIGHT*_IMAGE_WIDTH*_IMAGE_CHANNELS)\n",
148 | "\n",
149 | " if x is None:\n",
150 | " x = _X\n",
151 | " y = _Y\n",
152 | " else:\n",
153 | " x = np.concatenate((x, _X), axis=0)\n",
154 | " y = np.concatenate((y, _Y), axis=0)\n",
155 | "\n",
156 | " elif name is \"test\":\n",
157 | " f = open('./data/'+folder_name+'/test_batch', 'rb')\n",
158 | " try:\n",
159 | " datadict = pickle.load(f, encoding='latin1') \n",
160 | " except:\n",
161 | " datadict = pickle.load(f)\n",
162 | " f.close()\n",
163 | " x = datadict[\"data\"]\n",
164 | " y = np.array(datadict['labels'])\n",
165 | " x = np.array(x, dtype=float) / 255.0\n",
166 | " x = x.reshape([-1, _IMAGE_CHANNELS, _IMAGE_HEIGHT, _IMAGE_WIDTH])\n",
167 | " x = x.transpose([0, 2, 3, 1])\n",
168 | " x = x.reshape(-1, _IMAGE_HEIGHT*_IMAGE_WIDTH*_IMAGE_CHANNELS)\n",
169 | "\n",
170 | " return x, dense_to_one_hot(y)\n",
171 | "\n",
172 | "def dense_to_one_hot(labels_dense, num_classes=10):\n",
173 | " num_labels = labels_dense.shape[0]\n",
174 | " index_offset = np.arange(num_labels) * num_classes\n",
175 | " labels_one_hot = np.zeros((num_labels, num_classes))\n",
176 | " labels_one_hot.flat[index_offset + labels_dense.ravel()] = 1\n",
177 | " return labels_one_hot\n",
178 | "\n",
179 | "def print_download_progress(count, block_size, total_size):\n",
180 | " pct_complete = float(count * block_size) / total_size\n",
181 | " msg = \"\\r- Download progress: {0:.1%}\".format(pct_complete)\n",
182 | " sys.stdout.write(msg)\n",
183 | " sys.stdout.flush()\n",
184 | " \n",
185 | "def maybe_download_and_extract():\n",
186 | " main_directory = \"./data/\"\n",
187 | " if not os.path.exists(main_directory):\n",
188 | " os.makedirs(main_directory)\n",
189 | " cifar_10_directory = main_directory+\"cifar10/\"\n",
190 | " if not os.path.exists(cifar_10_directory):\n",
191 | " url = \"http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\"\n",
192 | " filename = url.split('/')[-1]\n",
193 | " file_path = os.path.join(main_directory, filename)\n",
194 | " zip_cifar_10 = file_path\n",
195 | " file_path, _ = urlretrieve(url=url, filename=file_path, \n",
196 | " reporthook=print_download_progress)\n",
197 | "\n",
198 | " print()\n",
199 | " print(\"Download finished. Extracting files.\")\n",
200 | " if file_path.endswith(\".zip\"):\n",
201 | " zipfile.ZipFile(file=file_path, mode=\"r\").extractall(main_directory)\n",
202 | " elif file_path.endswith((\".tar.gz\", \".tgz\")):\n",
203 | " tarfile.open(name=file_path, mode=\"r:gz\").extractall(main_directory)\n",
204 | " print(\"Done.\")\n",
205 | "\n",
206 | " os.rename(main_directory+\"cifar-10-batches-py\", cifar_10_directory)\n",
207 | " os.remove(zip_cifar_10)"
208 | ]
209 | },
210 | {
211 | "cell_type": "markdown",
212 | "metadata": {},
213 | "source": [
214 | "## Load Data\n",
215 | "The size of CIFAR10 dataset is about 180MB and it will be downloaded automatically if it is not located in the given path."
216 | ]
217 | },
218 | {
219 | "cell_type": "code",
220 | "execution_count": null,
221 | "metadata": {},
222 | "outputs": [],
223 | "source": [
224 | "train_x, train_y = get_data_set(\"train\")\n",
225 | "test_x, test_y = get_data_set(\"test\")\n",
226 | "\n",
227 | "print (\"Dataset loaded\")"
228 | ]
229 | },
230 | {
231 | "cell_type": "markdown",
232 | "metadata": {},
233 | "source": [
234 | "The CIFAR10 dataset has now been loaded and consists of 60000 32x32 color images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. "
235 | ]
236 | },
237 | {
238 | "cell_type": "markdown",
239 | "metadata": {},
240 | "source": [
241 | "### Plot a few images to see if data is correct"
242 | ]
243 | },
244 | {
245 | "cell_type": "code",
246 | "execution_count": null,
247 | "metadata": {},
248 | "outputs": [],
249 | "source": [
250 | "def show_images():\n",
251 | " fig, axes = plt.subplots(3, 3)\n",
252 | " fig.subplots_adjust(hspace=0.6, wspace=0.3)\n",
253 | " \n",
254 | " for i, ax in enumerate(axes.flat):\n",
255 | " # Plot image.\n",
256 | " ax.imshow(train_x[i].reshape(_IMAGE_HEIGHT, _IMAGE_WIDTH, _IMAGE_CHANNELS))\n",
257 | " \n",
258 | " # Name of the true class.\n",
259 | " cls_true_name = num_to_class[train_y[i].argmax()]\n",
260 | " xlabel = \"class: {0}\".format(cls_true_name)\n",
261 | " \n",
262 | " # Show the classes as the label on the x-axis.\n",
263 | " ax.set_xlabel(xlabel)\n",
264 | " \n",
265 | " # Remove ticks from the plot.\n",
266 | " ax.set_xticks([])\n",
267 | " ax.set_yticks([])\n",
268 | " return\n",
269 | "\n",
270 | "print(train_x.shape)\n",
271 | "print(train_y.shape)\n",
272 | "show_images()"
273 | ]
274 | },
275 | {
276 | "cell_type": "markdown",
277 | "metadata": {},
278 | "source": [
279 | "## TensorFlow Graph\n",
280 | "\n",
281 | "The entire purpose of TensorFlow is to have a so-called computational graph that can be executed much more efficiently than if the same calculations were to be performed directly in Python. TensorFlow can be more efficient than NumPy because TensorFlow knows the entire computation graph that must be executed, while NumPy only knows the computation of a single mathematical operation at a time.\n",
282 | "\n",
283 | "TensorFlow can also automatically calculate the gradients that are needed to optimize the variables of the graph so as to make the model perform better. This is because the graph is a combination of simple mathematical expressions so the gradient of the entire graph can be calculated using the chain-rule for derivatives.\n",
284 | "\n",
285 | "TensorFlow can also take advantage of multi-core CPUs as well as GPUs.\n",
286 | "\n",
287 | "A TensorFlow graph consists of the following parts which will be detailed below:\n",
288 | "\n",
289 | "* Placeholder variables used to feed input into the graph.\n",
290 | "* Model variables that are going to be optimized so as to make the model perform better.\n",
291 | "* The model which is essentially just a mathematical function that calculates some output given the input in the placeholder variables and the model variables.\n",
292 | "* A loss measure that can be used to guide the optimization of the variables.\n",
293 | "* An optimization method which updates the variables of the model.\n",
294 | "\n",
295 | "In addition, the TensorFlow graph may also contain various debugging statements e.g. for logging data to be displayed using TensorBoard, which is not covered in this tutorial."
296 | ]
297 | },
298 | {
299 | "cell_type": "markdown",
300 | "metadata": {},
301 | "source": [
302 | "### Placeholder variables"
303 | ]
304 | },
305 | {
306 | "cell_type": "markdown",
307 | "metadata": {},
308 | "source": [
309 | "Placeholder variables serve as the input to the graph that we may change each time we execute the graph. We call this feeding the placeholder variables and it is demonstrated further below.\n",
310 | "\n",
311 | "First we define the placeholder variable for the input images. This allows us to change the images that are input to the TensorFlow graph. This is a so-called tensor, which just means that it is a multi-dimensional vector or matrix. The data-type is set to `float32` and the shape is set to `[None, _IMAGE_HEIGHT * _IMAGE_WIDTH * _IMAGE_CHANNEL]`, where `None` means that the tensor may hold an arbitrary number of images with each image being a vector of length `_IMAGE_HEIGHT * _IMAGE_WIDTH * _IMAGE_CHANNEL`."
312 | ]
313 | },
314 | {
315 | "cell_type": "code",
316 | "execution_count": null,
317 | "metadata": {},
318 | "outputs": [],
319 | "source": [
320 | "x = tf.placeholder(tf.float32, shape=[None, _IMAGE_HEIGHT * _IMAGE_WIDTH * _IMAGE_CHANNELS], name='Input')"
321 | ]
322 | },
323 | {
324 | "cell_type": "markdown",
325 | "metadata": {},
326 | "source": [
327 | "Next we have the placeholder variable for the true labels associated with the images that were input in the placeholder variable `x`. The shape of this placeholder variable is `[None, _NUM_CLASSES]` which means it may hold an arbitrary number of labels and each label is a vector of length `_NUM_CLASSES` which is 10 in this case."
328 | ]
329 | },
330 | {
331 | "cell_type": "code",
332 | "execution_count": null,
333 | "metadata": {},
334 | "outputs": [],
335 | "source": [
336 | "y_true = tf.placeholder(tf.float32, shape=[None, _NUM_CLASSES], name='Output')"
337 | ]
338 | },
339 | {
340 | "cell_type": "markdown",
341 | "metadata": {},
342 | "source": [
343 | "Since we use a dropout regularization method for a fully connected layer, we have the placeholder variable for the probability to keep variables. If `keep_probability` is 1.0, all variables are the same without the dropout regularization. When `keep_probability` is 0.5, half of the variables in a fully connected layer become 0.0. "
344 | ]
345 | },
346 | {
347 | "cell_type": "code",
348 | "execution_count": null,
349 | "metadata": {},
350 | "outputs": [],
351 | "source": [
352 | "keep_probability = tf.placeholder(tf.float32, shape=[], name='keep_prob')"
353 | ]
354 | },
355 | {
356 | "cell_type": "markdown",
357 | "metadata": {},
358 | "source": [
359 | "`global_step` saves how many optimization iterations are used."
360 | ]
361 | },
362 | {
363 | "cell_type": "code",
364 | "execution_count": null,
365 | "metadata": {},
366 | "outputs": [],
367 | "source": [
368 | "global_step = tf.Variable(initial_value=0, trainable=False, name='global_step')"
369 | ]
370 | },
371 | {
372 | "cell_type": "markdown",
373 | "metadata": {},
374 | "source": [
375 | "Tensorflow takes an image input in order of `[Batch, _IMAGE_HEIGHT, _IMAGE_WIDTH, _IMAGE_CHANNEL]`. We reshape the input image to make the tensorflow graph can understand the input. "
376 | ]
377 | },
378 | {
379 | "cell_type": "code",
380 | "execution_count": null,
381 | "metadata": {},
382 | "outputs": [],
383 | "source": [
384 | "x_image = tf.reshape(x, [-1, _IMAGE_HEIGHT, _IMAGE_WIDTH, _IMAGE_CHANNELS], name='images')"
385 | ]
386 | },
387 | {
388 | "cell_type": "markdown",
389 | "metadata": {},
390 | "source": [
391 | "### Network model to be optimized"
392 | ]
393 | },
394 | {
395 | "cell_type": "markdown",
396 | "metadata": {},
397 | "source": [
398 | "The network consists of `4` convolutional networks and `2` fully connected layers. `conv1_1` layer consists of filters with weight size `[3, 3, 3, 32]` and bias for each output channel, size `[32]`. `conv1_2` layer consists of filters with weight size `[3, 3, 32, 32]` and bias for each output channel, size `[32]`. `conv2_1` layer contains filters with size `[3, 3, 32, 64]` and bias for each output channel, size `[64]`. `conv3_1` layer consists of filters with size `[3, 3, 64, 128]` and bias for each output channel, size `[128]`. `fully_connected` layer has a filter with weight size `[_IMAGE_HEIGHT/8 * _IMAGE_WIDTH/8 * 128, 1000]`, and bias for each output channel, size 1000. Lastly, the `feat` layer has a filter with weight size `[1000, 10]` and bias for each output channel, size `[10]`."
399 | ]
400 | },
401 | {
402 | "cell_type": "code",
403 | "execution_count": null,
404 | "metadata": {},
405 | "outputs": [],
406 | "source": [
407 | "### [Exercise 1] Change the CNN architecture to VGG16\n",
408 | "feat = slim.repeat(x_image, 2, slim.conv2d, 32, [3, 3], scope='conv1')\n",
409 | "feat = slim.max_pool2d(feat, [2, 2], padding='SAME', scope='pool1')\n",
410 | "feat = slim.repeat(feat, 1, slim.conv2d, 64, [3, 3], scope='conv2')\n",
411 | "feat = slim.max_pool2d(feat, [2, 2], padding='SAME', scope='pool2')\n",
412 | "feat = slim.repeat(feat, 1, slim.conv2d, 128, [3, 3], scope='conv3')\n",
413 | "feat = slim.max_pool2d(feat, [2, 2], padding='SAME', scope='pool3')\n",
414 | "feat = slim.flatten(feat, scope='flatten')\n",
415 | "feat = slim.fully_connected(feat, 1000, scope='fully_connected')"
416 | ]
417 | },
418 | {
419 | "cell_type": "code",
420 | "execution_count": null,
421 | "metadata": {},
422 | "outputs": [],
423 | "source": [
424 | "### [Exercise 2] Add a regularization method\n",
425 | "feat = "
426 | ]
427 | },
428 | {
429 | "cell_type": "code",
430 | "execution_count": null,
431 | "metadata": {},
432 | "outputs": [],
433 | "source": [
434 | "y_pred = slim.fully_connected(feat, _NUM_CLASSES, scope='feat', activation_fn=None)\n",
435 | "softmax = tf.nn.softmax(y_pred)\n",
436 | "y_pred_cls = tf.argmax(softmax, axis=1)"
437 | ]
438 | },
439 | {
440 | "cell_type": "markdown",
441 | "metadata": {},
442 | "source": [
443 | "Let's see the variables in the network model which we defined above."
444 | ]
445 | },
446 | {
447 | "cell_type": "code",
448 | "execution_count": null,
449 | "metadata": {},
450 | "outputs": [],
451 | "source": [
452 | "print(tf.global_variables())"
453 | ]
454 | },
455 | {
456 | "cell_type": "markdown",
457 | "metadata": {},
458 | "source": [
459 | "### Loss-function to be optimized"
460 | ]
461 | },
462 | {
463 | "cell_type": "markdown",
464 | "metadata": {},
465 | "source": [
466 | "To make the model better at classifying the input images, we must somehow change the variables for `weights` and `biases`. To do this we first need to know how well the model currently performs by comparing the predicted output of the model `y_pred` to the desired output `y_true`.\n",
467 | "\n",
468 | "The cross-entropy is a performance measure used in classification. The cross-entropy is a continuous function that is always positive and if the predicted output of the model exactly matches the desired output then the cross-entropy equals zero. The goal of optimization is therefore to minimize the cross-entropy so it gets as close to zero as possible by changing the `weights` and `biases` of the model.\n",
469 | "\n",
470 | "TensorFlow has a built-in function for calculating the cross-entropy. Note that it uses the values of the `logits` because it also calculates the softmax internally."
471 | ]
472 | },
473 | {
474 | "cell_type": "code",
475 | "execution_count": null,
476 | "metadata": {},
477 | "outputs": [],
478 | "source": [
479 | "try:\n",
480 | " cross_entropy = tf.nn.softmax_cross_entropy_with_logits_v2(logits=y_pred, labels=y_true)\n",
481 | "except:\n",
482 | " cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits=y_pred, labels=y_true)"
483 | ]
484 | },
485 | {
486 | "cell_type": "markdown",
487 | "metadata": {},
488 | "source": [
489 | "We have now calculated the cross-entropy for each of the image classifications so we have a measure of how well the model performs on each image individually. But in order to use the cross-entropy to guide the optimization of the model's variables we need a single scalar value, so we simply take the average of the cross-entropy for all the image classifications."
490 | ]
491 | },
492 | {
493 | "cell_type": "code",
494 | "execution_count": null,
495 | "metadata": {},
496 | "outputs": [],
497 | "source": [
498 | "loss = tf.reduce_mean(cross_entropy)"
499 | ]
500 | },
501 | {
502 | "cell_type": "markdown",
503 | "metadata": {},
504 | "source": [
505 | "### Optimization method"
506 | ]
507 | },
508 | {
509 | "cell_type": "markdown",
510 | "metadata": {},
511 | "source": [
512 | "Now that we have a loss measure that must be minimized, we can then create an optimizer. In this case it is the basic form of Gradient Descent where the learning rate (step-size) is set to 0.001.\n",
513 | "\n",
514 | "Note that optimization is not performed at this point. In fact, nothing is calculated at all, we just add the optimizer-object to the TensorFlow graph for later execution."
515 | ]
516 | },
517 | {
518 | "cell_type": "code",
519 | "execution_count": null,
520 | "metadata": {},
521 | "outputs": [],
522 | "source": [
523 | "global_accuracy = 0\n",
524 | "epoch_start = 0\n",
525 | "\n",
526 | "### [Exercise 3] Write your own optimizer\n",
527 | "optimizer = "
528 | ]
529 | },
530 | {
531 | "cell_type": "markdown",
532 | "metadata": {},
533 | "source": [
534 | "### Performance measures"
535 | ]
536 | },
537 | {
538 | "cell_type": "markdown",
539 | "metadata": {},
540 | "source": [
541 | "We need a few more performance measures to display the progress to the user.\n",
542 | "\n",
543 | "This is a vector of booleans whether the predicted class equals the true class of each image."
544 | ]
545 | },
546 | {
547 | "cell_type": "code",
548 | "execution_count": null,
549 | "metadata": {},
550 | "outputs": [],
551 | "source": [
552 | "correct_prediction = tf.equal(y_pred_cls, tf.argmax(y_true, axis=1))"
553 | ]
554 | },
555 | {
556 | "cell_type": "markdown",
557 | "metadata": {},
558 | "source": [
559 | "This calculates the classification accuracy by first type-casting the vector of booleans to floats, so that False becomes 0 and True becomes 1, and then calculating the average of these numbers."
560 | ]
561 | },
562 | {
563 | "cell_type": "code",
564 | "execution_count": null,
565 | "metadata": {},
566 | "outputs": [],
567 | "source": [
568 | "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))*100.0"
569 | ]
570 | },
571 | {
572 | "cell_type": "markdown",
573 | "metadata": {},
574 | "source": [
575 | "## TensorFlow Run"
576 | ]
577 | },
578 | {
579 | "cell_type": "markdown",
580 | "metadata": {},
581 | "source": [
582 | "### Create TensorFlow session\n",
583 | "\n",
584 | "Once the TensorFlow graph has been created, we have to create a TensorFlow session which is used to execute the graph."
585 | ]
586 | },
587 | {
588 | "cell_type": "code",
589 | "execution_count": null,
590 | "metadata": {},
591 | "outputs": [],
592 | "source": [
593 | "sess = tf.Session()"
594 | ]
595 | },
596 | {
597 | "cell_type": "markdown",
598 | "metadata": {},
599 | "source": [
600 | "### Initialize network parameters"
601 | ]
602 | },
603 | {
604 | "cell_type": "markdown",
605 | "metadata": {},
606 | "source": [
607 | "The variables for `weights` and `biases` in convolutional networks and fully connected layers must be initialized before we start optimizing them."
608 | ]
609 | },
610 | {
611 | "cell_type": "code",
612 | "execution_count": null,
613 | "metadata": {},
614 | "outputs": [],
615 | "source": [
616 | "sess.run(tf.global_variables_initializer())\n",
617 | "print(\"Initialized variables.\")"
618 | ]
619 | },
620 | {
621 | "cell_type": "markdown",
622 | "metadata": {},
623 | "source": [
624 | "### Helper-functions to plot results"
625 | ]
626 | },
627 | {
628 | "cell_type": "markdown",
629 | "metadata": {},
630 | "source": [
631 | "Function for plotting confusion matrix of test images."
632 | ]
633 | },
634 | {
635 | "cell_type": "code",
636 | "execution_count": null,
637 | "metadata": {},
638 | "outputs": [],
639 | "source": [
640 | "def plot_confusion_matrix(cm, classes,\n",
641 | " normalize=False,\n",
642 | " title='Confusion matrix',\n",
643 | " cmap=plt.cm.Blues):\n",
644 | " \"\"\"\n",
645 | " This function prints and plots the confusion matrix.\n",
646 | " Normalization can be applied by setting `normalize=True`.\n",
647 | " \"\"\"\n",
648 | " print('Drawing confusion matrix...')\n",
649 | " if normalize:\n",
650 | " cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
651 | " print(\"Normalized confusion matrix\")\n",
652 | " else:\n",
653 | " print('Confusion matrix, without normalization')\n",
654 | "\n",
655 | " plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
656 | " plt.title(title)\n",
657 | " plt.colorbar()\n",
658 | " tick_marks = np.arange(len(classes))\n",
659 | " plt.xticks(tick_marks, classes, rotation=45)\n",
660 | " plt.yticks(tick_marks, classes)\n",
661 | "\n",
662 | " fmt = '.2f' if normalize else 'd'\n",
663 | " thresh = cm.max() / 2.\n",
664 | " for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
665 | " plt.text(j, i, format(cm[i, j], fmt),\n",
666 | " horizontalalignment=\"center\",\n",
667 | " color=\"white\" if cm[i, j] > thresh else \"black\")\n",
668 | "\n",
669 | " plt.tight_layout()\n",
670 | " plt.ylabel('True label')\n",
671 | " plt.xlabel('Predicted label')\n",
672 | " plt.show()"
673 | ]
674 | },
675 | {
676 | "cell_type": "markdown",
677 | "metadata": {},
678 | "source": [
679 | "Function for plotting tsne examples of test images."
680 | ]
681 | },
682 | {
683 | "cell_type": "code",
684 | "execution_count": null,
685 | "metadata": {},
686 | "outputs": [],
687 | "source": [
688 | "def plot_tsne(y_pred_val, y_true_val):\n",
689 | " numbered = LabelEncoder().fit_transform(list(y_true_val))\n",
690 | " c = plt.cm.get_cmap('hsv', len(set(numbered)))\n",
691 | " colors=[]\n",
692 | " for num in numbered:\n",
693 | " temp=c(num)\n",
694 | " colors.append(temp)\n",
695 | " \n",
696 | " print('Fitting TSNE...')\n",
697 | " X_embedded = TSNE(n_components=2).fit_transform(y_pred_val[:10000])\n",
698 | " X_embedded = np.array(X_embedded)\n",
699 | " vis_x = X_embedded[:, 0]\n",
700 | " vis_y = X_embedded[:, 1]\n",
701 | " numbered = LabelEncoder().fit_transform(list(y_true_val))\n",
702 | " c = plt.cm.get_cmap('hsv', len(set(numbered)))\n",
703 | " plt.figure(num=None, figsize=(5, 5), dpi=80, facecolor='w', edgecolor='k')\n",
704 | " plt.scatter(vis_x, vis_y, c=colors,alpha=1,linewidths=1,s=10, edgecolors='none')\n",
705 | " plt.show()\n",
706 | " return X_embedded"
707 | ]
708 | },
709 | {
710 | "cell_type": "markdown",
711 | "metadata": {},
712 | "source": [
713 | "### Helper-function to test the model at the middle of training"
714 | ]
715 | },
716 | {
717 | "cell_type": "code",
718 | "execution_count": null,
719 | "metadata": {},
720 | "outputs": [],
721 | "source": [
722 | "def test(_global_step, epoch, y_pred_val, y_true_val):\n",
723 | " global global_accuracy\n",
724 | " global epoch_start\n",
725 | "\n",
726 | " i = 0\n",
727 | " predicted_class = np.zeros(shape=len(test_x), dtype=np.int)\n",
728 | " while i < len(test_x):\n",
729 | " j = min(i + _BATCH_SIZE, len(test_x))\n",
730 | " batch_xs = test_x[i:j, :]\n",
731 | " batch_ys = test_y[i:j, :]\n",
732 | " predicted_class[i:j] = sess.run(\n",
733 | " y_pred_cls,\n",
734 | " feed_dict={x: batch_xs, y_true: batch_ys, keep_probability: 1.0}\n",
735 | " ) \n",
736 | " i = j\n",
737 | " text_y_argmax = np.argmax(test_y, axis=1) \n",
738 | " correct = (text_y_argmax == predicted_class)\n",
739 | " acc = correct.mean()*100\n",
740 | " correct_numbers = correct.sum()\n",
741 | "\n",
742 | " hours, rem = divmod(time() - epoch_start, 3600)\n",
743 | " minutes, seconds = divmod(rem, 60)\n",
744 | " mes = \"\\nEpoch {} - accuracy: {:.1f}% ({}/{}) - time: {:0>2}:{:0>2}:{:05.2f}\"\n",
745 | " print(mes.format((epoch+1), acc, correct_numbers, len(test_x), int(hours), int(minutes), seconds))\n",
746 | "\n",
747 | " if global_accuracy != 0 and global_accuracy < acc:\n",
748 | " mes = \"This epoch receive better accuracy: {:.1f}% > {:.1f}%.\"\n",
749 | " print(mes.format(acc, global_accuracy))\n",
750 | " global_accuracy = acc\n",
751 | "\n",
752 | " elif global_accuracy == 0:\n",
753 | " global_accuracy = acc\n",
754 | " \n",
755 | " # Plot non-normalized confusion matrix\n",
756 | " cnf_matrix = confusion_matrix(text_y_argmax, predicted_class)\n",
757 | " np.set_printoptions(precision=2)\n",
758 | " plot_confusion_matrix(cnf_matrix, classes=class_names,\n",
759 | " title='Confusion matrix, without normalization')\n",
760 | " plot_tsne(y_pred_val, y_true_val)\n",
761 | " \n",
762 | " print(\"###########################################################################################################\")"
763 | ]
764 | },
765 | {
766 | "cell_type": "markdown",
767 | "metadata": {},
768 | "source": [
769 | "### Helper-function to perform optimization iterations"
770 | ]
771 | },
772 | {
773 | "cell_type": "markdown",
774 | "metadata": {},
775 | "source": [
776 | "Function for performing a number of optimization iterations so as to gradually improve the `weights` and `biases` of the model. In each iteration, a new batch of data is selected from the training-set and then TensorFlow executes the optimizer using those training samples."
777 | ]
778 | },
779 | {
780 | "cell_type": "code",
781 | "execution_count": null,
782 | "metadata": {},
783 | "outputs": [],
784 | "source": [
785 | "def train(epoch):\n",
786 | " global epoch_start\n",
787 | " epoch_start = time()\n",
788 | " batch_size = int(math.ceil(len(train_x) / _BATCH_SIZE))\n",
789 | " i_global = 0\n",
790 | "\n",
791 | " for s in range(batch_size):\n",
792 | " batch_xs = train_x[s*_BATCH_SIZE: (s+1)*_BATCH_SIZE]\n",
793 | " batch_ys = train_y[s*_BATCH_SIZE: (s+1)*_BATCH_SIZE]\n",
794 | "\n",
795 | " start_time = time()\n",
796 | " i_global, _, batch_loss, batch_acc = sess.run(\n",
797 | " [global_step, optimizer, loss, accuracy],\n",
798 | " feed_dict={x: batch_xs, y_true: batch_ys, keep_probability: 0.5})\n",
799 | " duration = time() - start_time\n",
800 | "\n",
801 | " if s % 100 == 0:\n",
802 | " percentage = int(float(s)/float(batch_size)*100.0)\n",
803 | " bar_len = 29\n",
804 | " filled_len = int((bar_len*int(percentage))/100)\n",
805 | " bar = '=' * filled_len + '>' + '-' * (bar_len - filled_len)\n",
806 | "\n",
807 | " msg = \"Global step: {:>5} - [{}] {:>3}% - accuracy: {:.1f}% - loss: {:.4f} - {:.1f} sample/sec\"\n",
808 | " print(msg.format(i_global, bar, percentage, batch_acc, batch_loss, _BATCH_SIZE / duration))\n",
809 | " y_pred_val = sess.run(y_pred, feed_dict={x: train_x[:_NUM_VIS_EMBEDDING], keep_probability: 1.0})\n",
810 | " y_true_val = train_y[:_NUM_VIS_EMBEDDING]\n",
811 | " y_true_val = y_true_val.argmax(1)\n",
812 | "\n",
813 | " test(i_global, epoch, y_pred_val, y_true_val)"
814 | ]
815 | },
816 | {
817 | "cell_type": "markdown",
818 | "metadata": {},
819 | "source": [
820 | "### Start training networks"
821 | ]
822 | },
823 | {
824 | "cell_type": "markdown",
825 | "metadata": {},
826 | "source": [
827 | "Weights are optimized and the accuracy on the test set increases as global step increases."
828 | ]
829 | },
830 | {
831 | "cell_type": "code",
832 | "execution_count": null,
833 | "metadata": {},
834 | "outputs": [],
835 | "source": [
836 | "print(\"Start training\")\n",
837 | "train_start = time()\n",
838 | "\n",
839 | "for i in range(_EPOCH):\n",
840 | " print(\"\\nEpoch: {}/{}\\n\".format((i+1), _EPOCH))\n",
841 | " train(i)\n",
842 | "\n",
843 | "hours, rem = divmod(time() - train_start, 3600)\n",
844 | "minutes, seconds = divmod(rem, 60)\n",
845 | "mes = \"Best accuracy pre session: {:.2f}%, time: {:0>2}:{:0>2}:{:05.2f}\"\n",
846 | "print(mes.format(global_accuracy, int(hours), int(minutes), seconds))\n",
847 | "print(\"Done training\")"
848 | ]
849 | }
850 | ],
851 | "metadata": {
852 | "kernelspec": {
853 | "display_name": "Python 3",
854 | "language": "python",
855 | "name": "python3"
856 | },
857 | "language_info": {
858 | "codemirror_mode": {
859 | "name": "ipython",
860 | "version": 3
861 | },
862 | "file_extension": ".py",
863 | "mimetype": "text/x-python",
864 | "name": "python",
865 | "nbconvert_exporter": "python",
866 | "pygments_lexer": "ipython3",
867 | "version": "3.5.2"
868 | }
869 | },
870 | "nbformat": 4,
871 | "nbformat_minor": 2
872 | }
873 |
--------------------------------------------------------------------------------
/code/Tutorial4_RNN_Name.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "## load prerequisites"
8 | ]
9 | },
10 | {
11 | "cell_type": "code",
12 | "execution_count": 1,
13 | "metadata": {},
14 | "outputs": [
15 | {
16 | "name": "stderr",
17 | "output_type": "stream",
18 | "text": [
19 | "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
20 | " _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
21 | "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
22 | " _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
23 | "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
24 | " _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
25 | "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
26 | " _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
27 | "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:521: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
28 | " _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
29 | "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
30 | " np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n"
31 | ]
32 | }
33 | ],
34 | "source": [
35 | "import csv\n",
36 | "import random\n",
37 | "import numpy as np\n",
38 | "import tensorflow as tf"
39 | ]
40 | },
41 | {
42 | "cell_type": "markdown",
43 | "metadata": {},
44 | "source": [
45 | "## Preprocess data : names"
46 | ]
47 | },
48 | {
49 | "cell_type": "code",
50 | "execution_count": 2,
51 | "metadata": {},
52 | "outputs": [
53 | {
54 | "name": "stdout",
55 | "output_type": "stream",
56 | "text": [
57 | "name total : 1219\n",
58 | "maximum name length : 11\n"
59 | ]
60 | }
61 | ],
62 | "source": [
63 | "data_name = set()\n",
64 | "max_len = 0\n",
65 | "with open('../data/woman_name_dataset.csv') as csv_file:\n",
66 | " csv_reader = csv.reader(csv_file, delimiter=',')\n",
67 | " line_count = 0\n",
68 | " for row in csv_reader:\n",
69 | " if line_count == 0:\n",
70 | " line_count += 1\n",
71 | " else:\n",
72 | " tmp_name = row[1].split()[0]\n",
73 | " data_name.add(row[1].split()[0])\n",
74 | " if len(tmp_name) > max_len:\n",
75 | " max_len = len(tmp_name)\n",
76 | "data_name = list(data_name)\n",
77 | "print('name total : {}'.format(len(data_name)))\n",
78 | "print('maximum name length : {}'.format(max_len))"
79 | ]
80 | },
81 | {
82 | "cell_type": "markdown",
83 | "metadata": {},
84 | "source": [
85 | "## Preprocess data : Characters"
86 | ]
87 | },
88 | {
89 | "cell_type": "code",
90 | "execution_count": 3,
91 | "metadata": {},
92 | "outputs": [
93 | {
94 | "name": "stdout",
95 | "output_type": "stream",
96 | "text": [
97 | "26 alphabets : ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n"
98 | ]
99 | }
100 | ],
101 | "source": [
102 | "chars = set()\n",
103 | "for name in data_name:\n",
104 | " for char in name:\n",
105 | " chars.add(char)\n",
106 | "chars = list(np.sort(list(chars)))\n",
107 | "print('{} alphabets : '.format(len(chars)), chars)"
108 | ]
109 | },
110 | {
111 | "cell_type": "markdown",
112 | "metadata": {},
113 | "source": [
114 | "## Define function to convert name to onehot"
115 | ]
116 | },
117 | {
118 | "cell_type": "code",
119 | "execution_count": 4,
120 | "metadata": {},
121 | "outputs": [
122 | {
123 | "data": {
124 | "text/plain": [
125 | "array([[[0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.,\n",
126 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
127 | " [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
128 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
129 | " [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.,\n",
130 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
131 | " [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
132 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
133 | " [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
134 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
135 | " [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
136 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
137 | " [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
138 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
139 | " [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
140 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
141 | " [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
142 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
143 | " [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
144 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
145 | " [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
146 | " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.]]])"
147 | ]
148 | },
149 | "execution_count": 4,
150 | "metadata": {},
151 | "output_type": "execute_result"
152 | }
153 | ],
154 | "source": [
155 | "def name_to_onehot(names, chars, max_len):\n",
156 | " onehot = np.zeros((len(names), max_len, len(chars)+1))\n",
157 | " for idx_1, name in enumerate(names):\n",
158 | " for idx_2 in range(max_len):\n",
159 | " if idx_2 < len(name):\n",
160 | " idx_3 = chars.index(name[idx_2])\n",
161 | " onehot[idx_1, idx_2, idx_3] = 1\n",
162 | " else:\n",
163 | " onehot[idx_1, idx_2, -1] = 1\n",
164 | " return onehot\n",
165 | "\n",
166 | "onehot_ex = name_to_onehot(['jane'], chars, max_len)\n",
167 | "onehot_ex"
168 | ]
169 | },
170 | {
171 | "cell_type": "markdown",
172 | "metadata": {},
173 | "source": [
174 | "## Define dimension and Placeholders"
175 | ]
176 | },
177 | {
178 | "cell_type": "code",
179 | "execution_count": 5,
180 | "metadata": {},
181 | "outputs": [],
182 | "source": [
183 | "num_data = len(data_name)\n",
184 | "seq_len = max_len - 1\n",
185 | "dim_data = len(chars) + 1\n",
186 | "\n",
187 | "ph_input_name = tf.placeholder(dtype=tf.float32, shape=[None, seq_len, dim_data])\n",
188 | "ph_output_name = tf.placeholder(dtype=tf.float32, shape=[None, seq_len, dim_data])"
189 | ]
190 | },
191 | {
192 | "cell_type": "markdown",
193 | "metadata": {},
194 | "source": [
195 | "## Define weight variables"
196 | ]
197 | },
198 | {
199 | "cell_type": "code",
200 | "execution_count": 6,
201 | "metadata": {},
202 | "outputs": [],
203 | "source": [
204 | "dim_rnn_cell = 128\n",
205 | "stddev = 0.02\n",
206 | "\n",
207 | "with tf.variable_scope('weights'):\n",
208 | " W_i = tf.get_variable('W_i', dtype=tf.float32,\n",
209 | " initializer=tf.random_normal([dim_data, dim_rnn_cell],\n",
210 | " stddev = stddev))\n",
211 | " b_i = tf.get_variable('b_i', dtype=tf.float32,\n",
212 | " initializer=tf.random_normal([dim_rnn_cell],\n",
213 | " stddev = stddev))\n",
214 | " W_o = tf.get_variable('W_o', dtype=tf.float32,\n",
215 | " initializer=tf.random_normal([dim_rnn_cell, dim_data],\n",
216 | " stddev = stddev))\n",
217 | " b_o = tf.get_variable('b_o', dtype=tf.float32,\n",
218 | " initializer=tf.random_normal([dim_data],\n",
219 | " stddev = stddev))"
220 | ]
221 | },
222 | {
223 | "cell_type": "markdown",
224 | "metadata": {},
225 | "source": [
226 | "## Define RNN for training"
227 | ]
228 | },
229 | {
230 | "cell_type": "code",
231 | "execution_count": 21,
232 | "metadata": {},
233 | "outputs": [],
234 | "source": [
235 | "def name_rnn_train(_x, _seq_len, _dim_data, _dim_rnn_cell):\n",
236 | " _x_split = tf.transpose(_x, [1, 0, 2]) # seq_len, batch, dim_data\n",
237 | " _x_split = tf.reshape(_x_split, [-1, _dim_data])\n",
238 | " \n",
239 | " # Linear Operation for Input\n",
240 | " # Fill here\n",
241 | " \n",
242 | " # Define LSTM Cell && RNN\n",
243 | " with tf.variable_scope('rnn', reuse=tf.AUTO_REUSE):\n",
244 | " _rnn_cell = tf.nn.rnn_cell.BasicLSTMCell(_dim_rnn_cell)\n",
245 | " _output, _state = tf.nn.static_rnn(_rnn_cell, _h_split, dtype=tf.float32)\n",
246 | " \n",
247 | " # Linear Operation for Output\n",
248 | " _total_out = []\n",
249 | " # Fill here\n",
250 | " \n",
251 | " return tf.transpose(tf.stack(_total_out), [1, 0, 2])"
252 | ]
253 | },
254 | {
255 | "cell_type": "markdown",
256 | "metadata": {},
257 | "source": [
258 | "## Define result graph"
259 | ]
260 | },
261 | {
262 | "cell_type": "code",
263 | "execution_count": 17,
264 | "metadata": {},
265 | "outputs": [
266 | {
267 | "name": "stdout",
268 | "output_type": "stream",
269 | "text": [
270 | "result_shape : (?, 10, 27)\n"
271 | ]
272 | }
273 | ],
274 | "source": [
275 | "result_name = name_rnn_train(ph_input_name, seq_len, dim_data, dim_rnn_cell)\n",
276 | "print('result_shape :', result_name.shape)"
277 | ]
278 | },
279 | {
280 | "cell_type": "markdown",
281 | "metadata": {},
282 | "source": [
283 | "## Define Loss function"
284 | ]
285 | },
286 | {
287 | "cell_type": "code",
288 | "execution_count": 18,
289 | "metadata": {},
290 | "outputs": [
291 | {
292 | "name": "stdout",
293 | "output_type": "stream",
294 | "text": [
295 | "WARNING:tensorflow:From :7: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.\n",
296 | "Instructions for updating:\n",
297 | "\n",
298 | "Future major versions of TensorFlow will allow gradients to flow\n",
299 | "into the labels input on backprop by default.\n",
300 | "\n",
301 | "See tf.nn.softmax_cross_entropy_with_logits_v2.\n",
302 | "\n"
303 | ]
304 | }
305 | ],
306 | "source": [
307 | "def name_loss(_gt_name, _result_name, _seq_len):\n",
308 | " total_loss = 0\n",
309 | " for i in range(_seq_len):\n",
310 | " # Fill here\n",
311 | " return total_loss\n",
312 | "rnn_loss = name_loss(ph_output_name, result_name, seq_len)"
313 | ]
314 | },
315 | {
316 | "cell_type": "markdown",
317 | "metadata": {},
318 | "source": [
319 | "## Define Optimizer and Get Ready"
320 | ]
321 | },
322 | {
323 | "cell_type": "code",
324 | "execution_count": 19,
325 | "metadata": {},
326 | "outputs": [
327 | {
328 | "name": "stdout",
329 | "output_type": "stream",
330 | "text": [
331 | "Now ready to start the session\n"
332 | ]
333 | }
334 | ],
335 | "source": [
336 | "learning_rate = 1e-3\n",
337 | "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(rnn_loss)\n",
338 | "\n",
339 | "init = tf.global_variables_initializer()\n",
340 | "saver = tf.train.Saver(var_list=tf.trainable_variables())\n",
341 | "\n",
342 | "print('Now ready to start the session')"
343 | ]
344 | },
345 | {
346 | "cell_type": "markdown",
347 | "metadata": {
348 | "scrolled": true
349 | },
350 | "source": [
351 | "## Session Run"
352 | ]
353 | },
354 | {
355 | "cell_type": "code",
356 | "execution_count": 20,
357 | "metadata": {
358 | "scrolled": true
359 | },
360 | "outputs": [
361 | {
362 | "name": "stdout",
363 | "output_type": "stream",
364 | "text": [
365 | "epoch : 1, train_loss : 28.54256660059879\n",
366 | "Model saved in file: ./RNN_model/model.ckpt-1\n",
367 | "epoch : 2, train_loss : 18.67396987111945\n",
368 | "Model saved in file: ./RNN_model/model.ckpt-2\n",
369 | "epoch : 3, train_loss : 17.6947021484375\n",
370 | "Model saved in file: ./RNN_model/model.ckpt-3\n",
371 | "epoch : 4, train_loss : 17.15771614877801\n",
372 | "Model saved in file: ./RNN_model/model.ckpt-4\n",
373 | "epoch : 5, train_loss : 16.822392915424548\n",
374 | "Model saved in file: ./RNN_model/model.ckpt-5\n",
375 | "epoch : 6, train_loss : 16.50236872622841\n",
376 | "Model saved in file: ./RNN_model/model.ckpt-6\n",
377 | "epoch : 7, train_loss : 16.15971851348877\n",
378 | "Model saved in file: ./RNN_model/model.ckpt-7\n",
379 | "epoch : 8, train_loss : 15.863567603261846\n",
380 | "Model saved in file: ./RNN_model/model.ckpt-8\n",
381 | "epoch : 9, train_loss : 15.546000480651854\n",
382 | "Model saved in file: ./RNN_model/model.ckpt-9\n",
383 | "epoch : 10, train_loss : 15.251013906378496\n",
384 | "Model saved in file: ./RNN_model/model.ckpt-10\n",
385 | "epoch : 11, train_loss : 14.958294015181693\n",
386 | "Model saved in file: ./RNN_model/model.ckpt-11\n",
387 | "epoch : 12, train_loss : 14.696805903786107\n",
388 | "Model saved in file: ./RNN_model/model.ckpt-12\n",
389 | "epoch : 13, train_loss : 14.449218097485993\n",
390 | "Model saved in file: ./RNN_model/model.ckpt-13\n",
391 | "epoch : 14, train_loss : 14.164149234169411\n",
392 | "Model saved in file: ./RNN_model/model.ckpt-14\n",
393 | "epoch : 15, train_loss : 13.922977648283306\n",
394 | "Model saved in file: ./RNN_model/model.ckpt-15\n",
395 | "epoch : 16, train_loss : 13.688792931406121\n",
396 | "Model saved in file: ./RNN_model/model.ckpt-16\n",
397 | "epoch : 17, train_loss : 13.500556142706621\n",
398 | "Model saved in file: ./RNN_model/model.ckpt-17\n",
399 | "epoch : 18, train_loss : 13.379649513646175\n",
400 | "Model saved in file: ./RNN_model/model.ckpt-18\n",
401 | "epoch : 19, train_loss : 13.261106491088869\n",
402 | "Model saved in file: ./RNN_model/model.ckpt-19\n",
403 | "epoch : 20, train_loss : 13.178394016466644\n",
404 | "Model saved in file: ./RNN_model/model.ckpt-20\n",
405 | "epoch : 21, train_loss : 13.10595422042044\n",
406 | "Model saved in file: ./RNN_model/model.ckpt-21\n",
407 | "epoch : 22, train_loss : 13.019753255342184\n",
408 | "Model saved in file: ./RNN_model/model.ckpt-22\n",
409 | "epoch : 23, train_loss : 12.941644317225407\n",
410 | "Model saved in file: ./RNN_model/model.ckpt-23\n",
411 | "epoch : 24, train_loss : 12.854022628382634\n",
412 | "Model saved in file: ./RNN_model/model.ckpt-24\n",
413 | "epoch : 25, train_loss : 12.777433897319591\n",
414 | "Model saved in file: ./RNN_model/model.ckpt-25\n",
415 | "epoch : 26, train_loss : 12.726191018757067\n",
416 | "Model saved in file: ./RNN_model/model.ckpt-26\n",
417 | "epoch : 27, train_loss : 12.633992948030171\n",
418 | "Model saved in file: ./RNN_model/model.ckpt-27\n",
419 | "epoch : 28, train_loss : 12.554255284761128\n",
420 | "Model saved in file: ./RNN_model/model.ckpt-28\n",
421 | "epoch : 29, train_loss : 12.472216806913677\n",
422 | "Model saved in file: ./RNN_model/model.ckpt-29\n",
423 | "epoch : 30, train_loss : 12.378323755766216\n",
424 | "Model saved in file: ./RNN_model/model.ckpt-30\n",
425 | "epoch : 31, train_loss : 12.29627257899234\n",
426 | "Model saved in file: ./RNN_model/model.ckpt-31\n",
427 | "epoch : 32, train_loss : 12.201784937005293\n",
428 | "Model saved in file: ./RNN_model/model.ckpt-32\n",
429 | "epoch : 33, train_loss : 12.131009202254447\n",
430 | "Model saved in file: ./RNN_model/model.ckpt-33\n",
431 | "epoch : 34, train_loss : 12.030012632671156\n",
432 | "Model saved in file: ./RNN_model/model.ckpt-34\n",
433 | "epoch : 35, train_loss : 11.948686900891753\n",
434 | "Model saved in file: ./RNN_model/model.ckpt-35\n",
435 | "epoch : 36, train_loss : 11.872862564890008\n",
436 | "Model saved in file: ./RNN_model/model.ckpt-36\n",
437 | "epoch : 37, train_loss : 11.797504926982675\n",
438 | "Model saved in file: ./RNN_model/model.ckpt-37\n",
439 | "epoch : 38, train_loss : 11.700841652719598\n",
440 | "Model saved in file: ./RNN_model/model.ckpt-38\n",
441 | "epoch : 39, train_loss : 11.630409039949114\n",
442 | "Model saved in file: ./RNN_model/model.ckpt-39\n",
443 | "epoch : 40, train_loss : 11.542844320598403\n",
444 | "Model saved in file: ./RNN_model/model.ckpt-40\n",
445 | "epoch : 41, train_loss : 11.463216731422827\n",
446 | "Model saved in file: ./RNN_model/model.ckpt-41\n",
447 | "epoch : 42, train_loss : 11.410016461422568\n",
448 | "Model saved in file: ./RNN_model/model.ckpt-42\n",
449 | "epoch : 43, train_loss : 11.323547564054788\n",
450 | "Model saved in file: ./RNN_model/model.ckpt-43\n",
451 | "epoch : 44, train_loss : 11.257381991336217\n",
452 | "Model saved in file: ./RNN_model/model.ckpt-44\n",
453 | "epoch : 45, train_loss : 11.184963878832363\n",
454 | "Model saved in file: ./RNN_model/model.ckpt-45\n",
455 | "epoch : 46, train_loss : 11.095906508596322\n",
456 | "Model saved in file: ./RNN_model/model.ckpt-46\n",
457 | "epoch : 47, train_loss : 11.023151146738153\n",
458 | "Model saved in file: ./RNN_model/model.ckpt-47\n",
459 | "epoch : 48, train_loss : 10.94416974720202\n",
460 | "Model saved in file: ./RNN_model/model.ckpt-48\n",
461 | "epoch : 49, train_loss : 10.855457556875129\n",
462 | "Model saved in file: ./RNN_model/model.ckpt-49\n",
463 | "epoch : 50, train_loss : 10.76490462453742\n",
464 | "Model saved in file: ./RNN_model/model.ckpt-50\n",
465 | "epoch : 51, train_loss : 10.699354021172775\n",
466 | "Model saved in file: ./RNN_model/model.ckpt-51\n",
467 | "epoch : 52, train_loss : 10.617735461184852\n",
468 | "Model saved in file: ./RNN_model/model.ckpt-52\n",
469 | "epoch : 53, train_loss : 10.505781976800218\n",
470 | "Model saved in file: ./RNN_model/model.ckpt-53\n",
471 | "epoch : 54, train_loss : 10.439787061590899\n",
472 | "Model saved in file: ./RNN_model/model.ckpt-54\n",
473 | "epoch : 55, train_loss : 10.355367208781997\n",
474 | "Model saved in file: ./RNN_model/model.ckpt-55\n",
475 | "epoch : 56, train_loss : 10.283977910092002\n",
476 | "Model saved in file: ./RNN_model/model.ckpt-56\n",
477 | "epoch : 57, train_loss : 10.18765484659295\n",
478 | "Model saved in file: ./RNN_model/model.ckpt-57\n",
479 | "epoch : 58, train_loss : 10.093573670638236\n",
480 | "Model saved in file: ./RNN_model/model.ckpt-58\n",
481 | "epoch : 59, train_loss : 10.022882511741239\n",
482 | "Model saved in file: ./RNN_model/model.ckpt-59\n",
483 | "epoch : 60, train_loss : 9.933478907534951\n",
484 | "Model saved in file: ./RNN_model/model.ckpt-60\n",
485 | "epoch : 61, train_loss : 9.846686915347448\n",
486 | "Model saved in file: ./RNN_model/model.ckpt-61\n",
487 | "epoch : 62, train_loss : 9.761049822757117\n",
488 | "Model saved in file: ./RNN_model/model.ckpt-62\n",
489 | "epoch : 63, train_loss : 9.676333327042428\n",
490 | "Model saved in file: ./RNN_model/model.ckpt-63\n",
491 | "epoch : 64, train_loss : 9.594584665800395\n",
492 | "Model saved in file: ./RNN_model/model.ckpt-64\n",
493 | "epoch : 65, train_loss : 9.528890910901522\n",
494 | "Model saved in file: ./RNN_model/model.ckpt-65\n",
495 | "epoch : 66, train_loss : 9.422712426436576\n",
496 | "Model saved in file: ./RNN_model/model.ckpt-66\n",
497 | "epoch : 67, train_loss : 9.35149017133211\n",
498 | "Model saved in file: ./RNN_model/model.ckpt-67\n",
499 | "epoch : 68, train_loss : 9.263297633120889\n",
500 | "Model saved in file: ./RNN_model/model.ckpt-68\n",
501 | "epoch : 69, train_loss : 9.18922223542866\n",
502 | "Model saved in file: ./RNN_model/model.ckpt-69\n",
503 | "epoch : 70, train_loss : 9.105483255888284\n",
504 | "Model saved in file: ./RNN_model/model.ckpt-70\n",
505 | "epoch : 71, train_loss : 9.025170326232912\n",
506 | "Model saved in file: ./RNN_model/model.ckpt-71\n",
507 | "epoch : 72, train_loss : 8.950168208072062\n",
508 | "Model saved in file: ./RNN_model/model.ckpt-72\n",
509 | "epoch : 73, train_loss : 8.869566013938503\n",
510 | "Model saved in file: ./RNN_model/model.ckpt-73\n",
511 | "epoch : 74, train_loss : 8.783397649463854\n",
512 | "Model saved in file: ./RNN_model/model.ckpt-74\n",
513 | "epoch : 75, train_loss : 8.73381529356304\n",
514 | "Model saved in file: ./RNN_model/model.ckpt-75\n",
515 | "epoch : 76, train_loss : 8.646269346538341\n",
516 | "Model saved in file: ./RNN_model/model.ckpt-76\n",
517 | "epoch : 77, train_loss : 8.568772868106239\n",
518 | "Model saved in file: ./RNN_model/model.ckpt-77\n",
519 | "epoch : 78, train_loss : 8.4884763014944\n",
520 | "Model saved in file: ./RNN_model/model.ckpt-78\n",
521 | "epoch : 79, train_loss : 8.4184637571636\n",
522 | "Model saved in file: ./RNN_model/model.ckpt-79\n",
523 | "epoch : 80, train_loss : 8.361174156791286\n",
524 | "Model saved in file: ./RNN_model/model.ckpt-80\n",
525 | "epoch : 81, train_loss : 8.268802893789191\n",
526 | "Model saved in file: ./RNN_model/model.ckpt-81\n",
527 | "epoch : 82, train_loss : 8.21933236875032\n",
528 | "Model saved in file: ./RNN_model/model.ckpt-82\n",
529 | "epoch : 83, train_loss : 8.134884106485467\n",
530 | "Model saved in file: ./RNN_model/model.ckpt-83\n",
531 | "epoch : 84, train_loss : 8.07383999071623\n",
532 | "Model saved in file: ./RNN_model/model.ckpt-84\n",
533 | "epoch : 85, train_loss : 8.025716580842671\n",
534 | "Model saved in file: ./RNN_model/model.ckpt-85\n",
535 | "epoch : 86, train_loss : 7.934834053641873\n",
536 | "Model saved in file: ./RNN_model/model.ckpt-86\n",
537 | "epoch : 87, train_loss : 7.877349903709008\n",
538 | "Model saved in file: ./RNN_model/model.ckpt-87\n",
539 | "epoch : 88, train_loss : 7.807283652456182\n",
540 | "Model saved in file: ./RNN_model/model.ckpt-88\n",
541 | "epoch : 89, train_loss : 7.7494699327569245\n",
542 | "Model saved in file: ./RNN_model/model.ckpt-89\n",
543 | "epoch : 90, train_loss : 7.675424701289128\n",
544 | "Model saved in file: ./RNN_model/model.ckpt-90\n",
545 | "epoch : 91, train_loss : 7.608195681320993\n",
546 | "Model saved in file: ./RNN_model/model.ckpt-91\n"
547 | ]
548 | },
549 | {
550 | "name": "stdout",
551 | "output_type": "stream",
552 | "text": [
553 | "epoch : 92, train_loss : 7.5594291937978655\n",
554 | "Model saved in file: ./RNN_model/model.ckpt-92\n",
555 | "epoch : 93, train_loss : 7.497586049531637\n",
556 | "Model saved in file: ./RNN_model/model.ckpt-93\n",
557 | "epoch : 94, train_loss : 7.4298533640409765\n",
558 | "Model saved in file: ./RNN_model/model.ckpt-94\n",
559 | "epoch : 95, train_loss : 7.387549375232897\n",
560 | "Model saved in file: ./RNN_model/model.ckpt-95\n",
561 | "epoch : 96, train_loss : 7.342127197667171\n",
562 | "Model saved in file: ./RNN_model/model.ckpt-96\n",
563 | "epoch : 97, train_loss : 7.279497246993215\n",
564 | "Model saved in file: ./RNN_model/model.ckpt-97\n",
565 | "epoch : 98, train_loss : 7.227611642134817\n",
566 | "Model saved in file: ./RNN_model/model.ckpt-98\n",
567 | "epoch : 99, train_loss : 7.185787100540964\n",
568 | "Model saved in file: ./RNN_model/model.ckpt-99\n",
569 | "epoch : 100, train_loss : 7.120706583324233\n",
570 | "Model saved in file: ./RNN_model/model.ckpt-100\n",
571 | "epoch : 101, train_loss : 7.06613658603869\n",
572 | "Model saved in file: ./RNN_model/model.ckpt-101\n",
573 | "epoch : 102, train_loss : 7.016613608912419\n",
574 | "Model saved in file: ./RNN_model/model.ckpt-102\n",
575 | "epoch : 103, train_loss : 6.979021875481855\n",
576 | "Model saved in file: ./RNN_model/model.ckpt-103\n",
577 | "epoch : 104, train_loss : 6.9431470067877505\n",
578 | "Model saved in file: ./RNN_model/model.ckpt-104\n",
579 | "epoch : 105, train_loss : 6.876120241064775\n",
580 | "Model saved in file: ./RNN_model/model.ckpt-105\n",
581 | "epoch : 106, train_loss : 6.84489819878026\n",
582 | "Model saved in file: ./RNN_model/model.ckpt-106\n",
583 | "epoch : 107, train_loss : 6.782469799644068\n",
584 | "Model saved in file: ./RNN_model/model.ckpt-107\n",
585 | "epoch : 108, train_loss : 6.738975022968492\n",
586 | "Model saved in file: ./RNN_model/model.ckpt-108\n",
587 | "epoch : 109, train_loss : 6.705347186640689\n",
588 | "Model saved in file: ./RNN_model/model.ckpt-109\n",
589 | "epoch : 110, train_loss : 6.660457811857524\n",
590 | "Model saved in file: ./RNN_model/model.ckpt-110\n",
591 | "epoch : 111, train_loss : 6.612845169870477\n",
592 | "Model saved in file: ./RNN_model/model.ckpt-111\n",
593 | "epoch : 112, train_loss : 6.573223766527677\n",
594 | "Model saved in file: ./RNN_model/model.ckpt-112\n",
595 | "epoch : 113, train_loss : 6.524199711649041\n",
596 | "Model saved in file: ./RNN_model/model.ckpt-113\n",
597 | "epoch : 114, train_loss : 6.475720832222386\n",
598 | "Model saved in file: ./RNN_model/model.ckpt-114\n",
599 | "epoch : 115, train_loss : 6.454433416065416\n",
600 | "Model saved in file: ./RNN_model/model.ckpt-115\n",
601 | "epoch : 116, train_loss : 6.414958502116957\n",
602 | "Model saved in file: ./RNN_model/model.ckpt-116\n",
603 | "epoch : 117, train_loss : 6.377824808421888\n",
604 | "Model saved in file: ./RNN_model/model.ckpt-117\n",
605 | "epoch : 118, train_loss : 6.347458437869425\n",
606 | "Model saved in file: ./RNN_model/model.ckpt-118\n",
607 | "epoch : 119, train_loss : 6.309866679342169\n",
608 | "Model saved in file: ./RNN_model/model.ckpt-119\n",
609 | "epoch : 120, train_loss : 6.269866366135448\n",
610 | "Model saved in file: ./RNN_model/model.ckpt-120\n",
611 | "epoch : 121, train_loss : 6.230119178169653\n",
612 | "Model saved in file: ./RNN_model/model.ckpt-121\n",
613 | "epoch : 122, train_loss : 6.200611365468877\n",
614 | "Model saved in file: ./RNN_model/model.ckpt-122\n",
615 | "epoch : 123, train_loss : 6.167604220540901\n",
616 | "Model saved in file: ./RNN_model/model.ckpt-123\n",
617 | "epoch : 124, train_loss : 6.1351448862176206\n",
618 | "Model saved in file: ./RNN_model/model.ckpt-124\n",
619 | "epoch : 125, train_loss : 6.096212211408113\n",
620 | "Model saved in file: ./RNN_model/model.ckpt-125\n",
621 | "epoch : 126, train_loss : 6.081643982937461\n",
622 | "Model saved in file: ./RNN_model/model.ckpt-126\n",
623 | "epoch : 127, train_loss : 6.0440531278911385\n",
624 | "Model saved in file: ./RNN_model/model.ckpt-127\n",
625 | "epoch : 128, train_loss : 6.014468142860814\n",
626 | "Model saved in file: ./RNN_model/model.ckpt-128\n",
627 | "epoch : 129, train_loss : 5.974940751728258\n",
628 | "Model saved in file: ./RNN_model/model.ckpt-129\n",
629 | "epoch : 130, train_loss : 5.942427936353183\n",
630 | "Model saved in file: ./RNN_model/model.ckpt-130\n",
631 | "epoch : 131, train_loss : 5.911266075937372\n",
632 | "Model saved in file: ./RNN_model/model.ckpt-131\n",
633 | "epoch : 132, train_loss : 5.901675676044666\n",
634 | "Model saved in file: ./RNN_model/model.ckpt-132\n",
635 | "epoch : 133, train_loss : 5.863052619130988\n",
636 | "Model saved in file: ./RNN_model/model.ckpt-133\n",
637 | "epoch : 134, train_loss : 5.842718249873111\n",
638 | "Model saved in file: ./RNN_model/model.ckpt-134\n",
639 | "epoch : 135, train_loss : 5.815403059909219\n",
640 | "Model saved in file: ./RNN_model/model.ckpt-135\n",
641 | "epoch : 136, train_loss : 5.790516301205284\n",
642 | "Model saved in file: ./RNN_model/model.ckpt-136\n",
643 | "epoch : 137, train_loss : 5.765897474790873\n",
644 | "Model saved in file: ./RNN_model/model.ckpt-137\n",
645 | "epoch : 138, train_loss : 5.734498801984285\n",
646 | "Model saved in file: ./RNN_model/model.ckpt-138\n",
647 | "epoch : 139, train_loss : 5.709458024878252\n",
648 | "Model saved in file: ./RNN_model/model.ckpt-139\n",
649 | "epoch : 140, train_loss : 5.711953213340358\n",
650 | "Model saved in file: ./RNN_model/model.ckpt-140\n",
651 | "epoch : 141, train_loss : 5.671023946059378\n",
652 | "Model saved in file: ./RNN_model/model.ckpt-141\n",
653 | "epoch : 142, train_loss : 5.637541896418521\n",
654 | "Model saved in file: ./RNN_model/model.ckpt-142\n",
655 | "epoch : 143, train_loss : 5.616442956422506\n",
656 | "Model saved in file: ./RNN_model/model.ckpt-143\n",
657 | "epoch : 144, train_loss : 5.598658561706543\n",
658 | "Model saved in file: ./RNN_model/model.ckpt-144\n",
659 | "epoch : 145, train_loss : 5.581714780707109\n",
660 | "Model saved in file: ./RNN_model/model.ckpt-145\n",
661 | "epoch : 146, train_loss : 5.5535081060309155\n",
662 | "Model saved in file: ./RNN_model/model.ckpt-146\n",
663 | "epoch : 147, train_loss : 5.540834201009649\n",
664 | "Model saved in file: ./RNN_model/model.ckpt-147\n",
665 | "epoch : 148, train_loss : 5.522211752439801\n",
666 | "Model saved in file: ./RNN_model/model.ckpt-148\n",
667 | "epoch : 149, train_loss : 5.494936541507118\n",
668 | "Model saved in file: ./RNN_model/model.ckpt-149\n",
669 | "epoch : 150, train_loss : 5.487423168985467\n",
670 | "Model saved in file: ./RNN_model/model.ckpt-150\n",
671 | "epoch : 151, train_loss : 5.47228539617438\n",
672 | "Model saved in file: ./RNN_model/model.ckpt-151\n",
673 | "epoch : 152, train_loss : 5.44294625834415\n",
674 | "Model saved in file: ./RNN_model/model.ckpt-152\n",
675 | "epoch : 153, train_loss : 5.421394398337917\n",
676 | "Model saved in file: ./RNN_model/model.ckpt-153\n",
677 | "epoch : 154, train_loss : 5.401166388863012\n",
678 | "Model saved in file: ./RNN_model/model.ckpt-154\n",
679 | "epoch : 155, train_loss : 5.383386360971551\n",
680 | "Model saved in file: ./RNN_model/model.ckpt-155\n",
681 | "epoch : 156, train_loss : 5.367568794049713\n",
682 | "Model saved in file: ./RNN_model/model.ckpt-156\n",
683 | "epoch : 157, train_loss : 5.348022737001118\n",
684 | "Model saved in file: ./RNN_model/model.ckpt-157\n",
685 | "epoch : 158, train_loss : 5.337232564624987\n",
686 | "Model saved in file: ./RNN_model/model.ckpt-158\n",
687 | "epoch : 159, train_loss : 5.311242630607205\n",
688 | "Model saved in file: ./RNN_model/model.ckpt-159\n",
689 | "epoch : 160, train_loss : 5.301685232865183\n",
690 | "Model saved in file: ./RNN_model/model.ckpt-160\n",
691 | "epoch : 161, train_loss : 5.2913457970870175\n",
692 | "Model saved in file: ./RNN_model/model.ckpt-161\n",
693 | "epoch : 162, train_loss : 5.268940825211374\n",
694 | "Model saved in file: ./RNN_model/model.ckpt-162\n",
695 | "epoch : 163, train_loss : 5.257901492871735\n",
696 | "Model saved in file: ./RNN_model/model.ckpt-163\n",
697 | "epoch : 164, train_loss : 5.241432340521562\n",
698 | "Model saved in file: ./RNN_model/model.ckpt-164\n",
699 | "epoch : 165, train_loss : 5.224895125941227\n",
700 | "Model saved in file: ./RNN_model/model.ckpt-165\n",
701 | "epoch : 166, train_loss : 5.217037978925203\n",
702 | "Model saved in file: ./RNN_model/model.ckpt-166\n",
703 | "epoch : 167, train_loss : 5.202302204935175\n",
704 | "Model saved in file: ./RNN_model/model.ckpt-167\n",
705 | "epoch : 168, train_loss : 5.191168483934905\n",
706 | "Model saved in file: ./RNN_model/model.ckpt-168\n",
707 | "epoch : 169, train_loss : 5.171945145255641\n",
708 | "Model saved in file: ./RNN_model/model.ckpt-169\n",
709 | "epoch : 170, train_loss : 5.158411728708368\n",
710 | "Model saved in file: ./RNN_model/model.ckpt-170\n",
711 | "epoch : 171, train_loss : 5.146283149719238\n",
712 | "Model saved in file: ./RNN_model/model.ckpt-171\n",
713 | "epoch : 172, train_loss : 5.144351833745054\n",
714 | "Model saved in file: ./RNN_model/model.ckpt-172\n",
715 | "epoch : 173, train_loss : 5.117720403169331\n",
716 | "Model saved in file: ./RNN_model/model.ckpt-173\n",
717 | "epoch : 174, train_loss : 5.115271141654568\n",
718 | "Model saved in file: ./RNN_model/model.ckpt-174\n",
719 | "epoch : 175, train_loss : 5.099593388406854\n",
720 | "Model saved in file: ./RNN_model/model.ckpt-175\n",
721 | "epoch : 176, train_loss : 5.0907128986559425\n",
722 | "Model saved in file: ./RNN_model/model.ckpt-176\n",
723 | "epoch : 177, train_loss : 5.075802777942858\n",
724 | "Model saved in file: ./RNN_model/model.ckpt-177\n",
725 | "epoch : 178, train_loss : 5.062778146643387\n",
726 | "Model saved in file: ./RNN_model/model.ckpt-178\n",
727 | "epoch : 179, train_loss : 5.056196388445402\n",
728 | "Model saved in file: ./RNN_model/model.ckpt-179\n",
729 | "epoch : 180, train_loss : 5.053058724654347\n",
730 | "Model saved in file: ./RNN_model/model.ckpt-180\n",
731 | "epoch : 181, train_loss : 5.042384298224198\n",
732 | "Model saved in file: ./RNN_model/model.ckpt-181\n"
733 | ]
734 | },
735 | {
736 | "name": "stdout",
737 | "output_type": "stream",
738 | "text": [
739 | "epoch : 182, train_loss : 5.035394317225406\n",
740 | "Model saved in file: ./RNN_model/model.ckpt-182\n",
741 | "epoch : 183, train_loss : 5.022335805391011\n",
742 | "Model saved in file: ./RNN_model/model.ckpt-183\n",
743 | "epoch : 184, train_loss : 5.015330239346152\n",
744 | "Model saved in file: ./RNN_model/model.ckpt-184\n",
745 | "epoch : 185, train_loss : 5.001781639299894\n",
746 | "Model saved in file: ./RNN_model/model.ckpt-185\n",
747 | "epoch : 186, train_loss : 4.991091376856755\n",
748 | "Model saved in file: ./RNN_model/model.ckpt-186\n",
749 | "epoch : 187, train_loss : 4.975522994995117\n",
750 | "Model saved in file: ./RNN_model/model.ckpt-187\n",
751 | "epoch : 188, train_loss : 4.9709730650249275\n",
752 | "Model saved in file: ./RNN_model/model.ckpt-188\n",
753 | "epoch : 189, train_loss : 4.958301217932451\n",
754 | "Model saved in file: ./RNN_model/model.ckpt-189\n",
755 | "epoch : 190, train_loss : 4.954532548000939\n",
756 | "Model saved in file: ./RNN_model/model.ckpt-190\n",
757 | "epoch : 191, train_loss : 4.949857485921759\n",
758 | "Model saved in file: ./RNN_model/model.ckpt-191\n",
759 | "epoch : 192, train_loss : 4.932167404576353\n",
760 | "Model saved in file: ./RNN_model/model.ckpt-192\n",
761 | "epoch : 193, train_loss : 4.932374226419549\n",
762 | "Model saved in file: ./RNN_model/model.ckpt-193\n",
763 | "epoch : 194, train_loss : 4.912426848160593\n",
764 | "Model saved in file: ./RNN_model/model.ckpt-194\n",
765 | "epoch : 195, train_loss : 4.904873044867264\n",
766 | "Model saved in file: ./RNN_model/model.ckpt-195\n",
767 | "epoch : 196, train_loss : 4.897598818728799\n",
768 | "Model saved in file: ./RNN_model/model.ckpt-196\n",
769 | "epoch : 197, train_loss : 4.904120244477925\n",
770 | "Model saved in file: ./RNN_model/model.ckpt-197\n",
771 | "epoch : 198, train_loss : 4.894826562781083\n",
772 | "Model saved in file: ./RNN_model/model.ckpt-198\n",
773 | "epoch : 199, train_loss : 4.894350127169961\n",
774 | "Model saved in file: ./RNN_model/model.ckpt-199\n",
775 | "epoch : 200, train_loss : 4.883309740769237\n",
776 | "Model saved in file: ./RNN_model/model.ckpt-200\n",
777 | "epoch : 201, train_loss : 4.865750463385331\n",
778 | "Model saved in file: ./RNN_model/model.ckpt-201\n",
779 | "epoch : 202, train_loss : 4.856913390912507\n",
780 | "Model saved in file: ./RNN_model/model.ckpt-202\n",
781 | "epoch : 203, train_loss : 4.850797402231317\n",
782 | "Model saved in file: ./RNN_model/model.ckpt-203\n",
783 | "epoch : 204, train_loss : 4.84923287441856\n",
784 | "Model saved in file: ./RNN_model/model.ckpt-204\n",
785 | "epoch : 205, train_loss : 4.83483520307039\n",
786 | "Model saved in file: ./RNN_model/model.ckpt-205\n",
787 | "epoch : 206, train_loss : 4.830684185028076\n",
788 | "Model saved in file: ./RNN_model/model.ckpt-206\n",
789 | "epoch : 207, train_loss : 4.830576143766704\n",
790 | "Model saved in file: ./RNN_model/model.ckpt-207\n",
791 | "epoch : 208, train_loss : 4.825661609047338\n",
792 | "Model saved in file: ./RNN_model/model.ckpt-208\n",
793 | "epoch : 209, train_loss : 4.815081194827431\n",
794 | "Model saved in file: ./RNN_model/model.ckpt-209\n",
795 | "epoch : 210, train_loss : 4.8050768249913265\n",
796 | "Model saved in file: ./RNN_model/model.ckpt-210\n",
797 | "epoch : 211, train_loss : 4.80832408603869\n",
798 | "Model saved in file: ./RNN_model/model.ckpt-211\n",
799 | "epoch : 212, train_loss : 4.7996390242325635\n",
800 | "Model saved in file: ./RNN_model/model.ckpt-212\n",
801 | "epoch : 213, train_loss : 4.790082755841707\n",
802 | "Model saved in file: ./RNN_model/model.ckpt-213\n",
803 | "epoch : 214, train_loss : 4.791138096859581\n",
804 | "Model saved in file: ./RNN_model/model.ckpt-214\n",
805 | "epoch : 215, train_loss : 4.7881072696886555\n",
806 | "Model saved in file: ./RNN_model/model.ckpt-215\n",
807 | "epoch : 216, train_loss : 4.785542387711375\n",
808 | "Model saved in file: ./RNN_model/model.ckpt-216\n",
809 | "epoch : 217, train_loss : 4.775825701261822\n",
810 | "Model saved in file: ./RNN_model/model.ckpt-217\n",
811 | "epoch : 218, train_loss : 4.765661666267794\n",
812 | "Model saved in file: ./RNN_model/model.ckpt-218\n",
813 | "epoch : 219, train_loss : 4.758792124296489\n",
814 | "Model saved in file: ./RNN_model/model.ckpt-219\n",
815 | "epoch : 220, train_loss : 4.761143985547518\n",
816 | "Model saved in file: ./RNN_model/model.ckpt-220\n",
817 | "epoch : 221, train_loss : 4.7590883154618115\n",
818 | "Model saved in file: ./RNN_model/model.ckpt-221\n",
819 | "epoch : 222, train_loss : 4.749942177220394\n",
820 | "Model saved in file: ./RNN_model/model.ckpt-222\n",
821 | "epoch : 223, train_loss : 4.735043249632183\n",
822 | "Model saved in file: ./RNN_model/model.ckpt-223\n",
823 | "epoch : 224, train_loss : 4.740250361593146\n",
824 | "Model saved in file: ./RNN_model/model.ckpt-224\n",
825 | "epoch : 225, train_loss : 4.740147138896741\n",
826 | "Model saved in file: ./RNN_model/model.ckpt-225\n",
827 | "epoch : 226, train_loss : 4.740278620468942\n",
828 | "Model saved in file: ./RNN_model/model.ckpt-226\n",
829 | "epoch : 227, train_loss : 4.734238473992598\n",
830 | "Model saved in file: ./RNN_model/model.ckpt-227\n",
831 | "epoch : 228, train_loss : 4.731205864956505\n",
832 | "Model saved in file: ./RNN_model/model.ckpt-228\n",
833 | "epoch : 229, train_loss : 4.715315517626311\n",
834 | "Model saved in file: ./RNN_model/model.ckpt-229\n",
835 | "epoch : 230, train_loss : 4.709307018079255\n",
836 | "Model saved in file: ./RNN_model/model.ckpt-230\n",
837 | "epoch : 231, train_loss : 4.710273165451854\n",
838 | "Model saved in file: ./RNN_model/model.ckpt-231\n",
839 | "epoch : 232, train_loss : 4.707971447392514\n",
840 | "Model saved in file: ./RNN_model/model.ckpt-232\n",
841 | "epoch : 233, train_loss : 4.700421785053454\n",
842 | "Model saved in file: ./RNN_model/model.ckpt-233\n",
843 | "epoch : 234, train_loss : 4.707987433985661\n",
844 | "Model saved in file: ./RNN_model/model.ckpt-234\n",
845 | "epoch : 235, train_loss : 4.692016476079038\n",
846 | "Model saved in file: ./RNN_model/model.ckpt-235\n",
847 | "epoch : 236, train_loss : 4.690692048323781\n",
848 | "Model saved in file: ./RNN_model/model.ckpt-236\n",
849 | "epoch : 237, train_loss : 4.691278758801911\n",
850 | "Model saved in file: ./RNN_model/model.ckpt-237\n",
851 | "epoch : 238, train_loss : 4.67991966950266\n",
852 | "Model saved in file: ./RNN_model/model.ckpt-238\n",
853 | "epoch : 239, train_loss : 4.68348483035439\n",
854 | "Model saved in file: ./RNN_model/model.ckpt-239\n",
855 | "epoch : 240, train_loss : 4.677710759012323\n",
856 | "Model saved in file: ./RNN_model/model.ckpt-240\n",
857 | "epoch : 241, train_loss : 4.6693857343573315\n",
858 | "Model saved in file: ./RNN_model/model.ckpt-241\n",
859 | "epoch : 242, train_loss : 4.671603529076827\n",
860 | "Model saved in file: ./RNN_model/model.ckpt-242\n",
861 | "epoch : 243, train_loss : 4.659419737364116\n",
862 | "Model saved in file: ./RNN_model/model.ckpt-243\n",
863 | "epoch : 244, train_loss : 4.659056312159488\n",
864 | "Model saved in file: ./RNN_model/model.ckpt-244\n",
865 | "epoch : 245, train_loss : 4.664423290051911\n",
866 | "Model saved in file: ./RNN_model/model.ckpt-245\n",
867 | "epoch : 246, train_loss : 4.6546941556428605\n",
868 | "Model saved in file: ./RNN_model/model.ckpt-246\n",
869 | "epoch : 247, train_loss : 4.6566291859275415\n",
870 | "Model saved in file: ./RNN_model/model.ckpt-247\n",
871 | "epoch : 248, train_loss : 4.64633153614245\n",
872 | "Model saved in file: ./RNN_model/model.ckpt-248\n",
873 | "epoch : 249, train_loss : 4.656420506929097\n",
874 | "Model saved in file: ./RNN_model/model.ckpt-249\n",
875 | "epoch : 250, train_loss : 4.652243714583547\n",
876 | "Model saved in file: ./RNN_model/model.ckpt-250\n",
877 | "epoch : 251, train_loss : 4.650367460752788\n",
878 | "Model saved in file: ./RNN_model/model.ckpt-251\n",
879 | "epoch : 252, train_loss : 4.651430004521421\n",
880 | "Model saved in file: ./RNN_model/model.ckpt-252\n",
881 | "epoch : 253, train_loss : 4.642912789394981\n",
882 | "Model saved in file: ./RNN_model/model.ckpt-253\n",
883 | "epoch : 254, train_loss : 4.6306385492023665\n",
884 | "Model saved in file: ./RNN_model/model.ckpt-254\n",
885 | "epoch : 255, train_loss : 4.635086310537238\n",
886 | "Model saved in file: ./RNN_model/model.ckpt-255\n",
887 | "epoch : 256, train_loss : 4.630607002659848\n",
888 | "Model saved in file: ./RNN_model/model.ckpt-256\n",
889 | "epoch : 257, train_loss : 4.617120642411081\n",
890 | "Model saved in file: ./RNN_model/model.ckpt-257\n",
891 | "epoch : 258, train_loss : 4.6210913156208235\n",
892 | "Model saved in file: ./RNN_model/model.ckpt-258\n",
893 | "epoch : 259, train_loss : 4.627978600953755\n",
894 | "Model saved in file: ./RNN_model/model.ckpt-259\n",
895 | "epoch : 260, train_loss : 4.613597568712736\n",
896 | "Model saved in file: ./RNN_model/model.ckpt-260\n",
897 | "epoch : 261, train_loss : 4.615750513578716\n",
898 | "Model saved in file: ./RNN_model/model.ckpt-261\n",
899 | "epoch : 262, train_loss : 4.61929740403828\n",
900 | "Model saved in file: ./RNN_model/model.ckpt-262\n",
901 | "epoch : 263, train_loss : 4.607439241911235\n",
902 | "Model saved in file: ./RNN_model/model.ckpt-263\n",
903 | "epoch : 264, train_loss : 4.608241558074951\n",
904 | "Model saved in file: ./RNN_model/model.ckpt-264\n",
905 | "epoch : 265, train_loss : 4.605861463044819\n",
906 | "Model saved in file: ./RNN_model/model.ckpt-265\n",
907 | "epoch : 266, train_loss : 4.607786178588866\n",
908 | "Model saved in file: ./RNN_model/model.ckpt-266\n",
909 | "epoch : 267, train_loss : 4.604507496482447\n",
910 | "Model saved in file: ./RNN_model/model.ckpt-267\n",
911 | "epoch : 268, train_loss : 4.598436129720588\n",
912 | "Model saved in file: ./RNN_model/model.ckpt-268\n",
913 | "epoch : 269, train_loss : 4.595844068025288\n",
914 | "Model saved in file: ./RNN_model/model.ckpt-269\n",
915 | "epoch : 270, train_loss : 4.5908743456790315\n",
916 | "Model saved in file: ./RNN_model/model.ckpt-270\n",
917 | "epoch : 271, train_loss : 4.594808302427594\n",
918 | "Model saved in file: ./RNN_model/model.ckpt-271\n"
919 | ]
920 | },
921 | {
922 | "name": "stdout",
923 | "output_type": "stream",
924 | "text": [
925 | "epoch : 272, train_loss : 4.581478319670024\n",
926 | "Model saved in file: ./RNN_model/model.ckpt-272\n",
927 | "epoch : 273, train_loss : 4.599064099161248\n",
928 | "Model saved in file: ./RNN_model/model.ckpt-273\n",
929 | "epoch : 274, train_loss : 4.6004664019534465\n",
930 | "Model saved in file: ./RNN_model/model.ckpt-274\n",
931 | "epoch : 275, train_loss : 4.581155300140381\n",
932 | "Model saved in file: ./RNN_model/model.ckpt-275\n",
933 | "epoch : 276, train_loss : 4.590859287663509\n",
934 | "Model saved in file: ./RNN_model/model.ckpt-276\n",
935 | "epoch : 277, train_loss : 4.590206020756772\n",
936 | "Model saved in file: ./RNN_model/model.ckpt-277\n",
937 | "epoch : 278, train_loss : 4.578938358708433\n",
938 | "Model saved in file: ./RNN_model/model.ckpt-278\n",
939 | "epoch : 279, train_loss : 4.5749553881193465\n",
940 | "Model saved in file: ./RNN_model/model.ckpt-279\n",
941 | "epoch : 280, train_loss : 4.570446190081145\n",
942 | "Model saved in file: ./RNN_model/model.ckpt-280\n",
943 | "epoch : 281, train_loss : 4.578030310179058\n",
944 | "Model saved in file: ./RNN_model/model.ckpt-281\n",
945 | "epoch : 282, train_loss : 4.576507693842838\n",
946 | "Model saved in file: ./RNN_model/model.ckpt-282\n",
947 | "epoch : 283, train_loss : 4.571079229053698\n",
948 | "Model saved in file: ./RNN_model/model.ckpt-283\n",
949 | "epoch : 284, train_loss : 4.580598028082596\n",
950 | "Model saved in file: ./RNN_model/model.ckpt-284\n",
951 | "epoch : 285, train_loss : 4.56740319101434\n",
952 | "Model saved in file: ./RNN_model/model.ckpt-285\n",
953 | "epoch : 286, train_loss : 4.571033302106356\n",
954 | "Model saved in file: ./RNN_model/model.ckpt-286\n",
955 | "epoch : 287, train_loss : 4.570549764131245\n",
956 | "Model saved in file: ./RNN_model/model.ckpt-287\n",
957 | "epoch : 288, train_loss : 4.562460472709254\n",
958 | "Model saved in file: ./RNN_model/model.ckpt-288\n",
959 | "epoch : 289, train_loss : 4.557164367876554\n",
960 | "Model saved in file: ./RNN_model/model.ckpt-289\n",
961 | "epoch : 290, train_loss : 4.558068702095434\n",
962 | "Model saved in file: ./RNN_model/model.ckpt-290\n",
963 | "epoch : 291, train_loss : 4.550961042705335\n",
964 | "Model saved in file: ./RNN_model/model.ckpt-291\n",
965 | "epoch : 292, train_loss : 4.554549417997661\n",
966 | "Model saved in file: ./RNN_model/model.ckpt-292\n",
967 | "epoch : 293, train_loss : 4.551793926640561\n",
968 | "Model saved in file: ./RNN_model/model.ckpt-293\n",
969 | "epoch : 294, train_loss : 4.552393009788112\n",
970 | "Model saved in file: ./RNN_model/model.ckpt-294\n",
971 | "epoch : 295, train_loss : 4.5522077711004965\n",
972 | "Model saved in file: ./RNN_model/model.ckpt-295\n",
973 | "epoch : 296, train_loss : 4.561055509667646\n",
974 | "Model saved in file: ./RNN_model/model.ckpt-296\n",
975 | "epoch : 297, train_loss : 4.551473642650404\n",
976 | "Model saved in file: ./RNN_model/model.ckpt-297\n",
977 | "epoch : 298, train_loss : 4.548797808195415\n",
978 | "Model saved in file: ./RNN_model/model.ckpt-298\n",
979 | "epoch : 299, train_loss : 4.550925857142397\n",
980 | "Model saved in file: ./RNN_model/model.ckpt-299\n",
981 | "epoch : 300, train_loss : 4.551664352416992\n",
982 | "Model saved in file: ./RNN_model/model.ckpt-300\n"
983 | ]
984 | }
985 | ],
986 | "source": [
987 | "max_epoch = 300\n",
988 | "batch_size = 64\n",
989 | "num_batch = int(num_data/batch_size)\n",
990 | "with tf.Session() as sess:\n",
991 | " sess.run(init)\n",
992 | " \n",
993 | " for _epoch in range(max_epoch):\n",
994 | " random.seed(_epoch)\n",
995 | " batch_shuffle = list(range(num_data))\n",
996 | " random.shuffle(batch_shuffle)\n",
997 | " \n",
998 | " total_train_loss = 0\n",
999 | " \n",
1000 | " for i in range(num_batch):\n",
1001 | " batch_idx = [batch_shuffle[idx] for idx in range(i*batch_size,\n",
1002 | " (i+1)*batch_size)]\n",
1003 | " batch_names = [name for name in data_name if data_name.index(name) in batch_idx]\n",
1004 | " batch_onehots = name_to_onehot(batch_names, chars, max_len)\n",
1005 | " \n",
1006 | " input_onehot = batch_onehots[:, 0:(max_len-1), :]\n",
1007 | " output_onehot = batch_onehots[:, 1:max_len, :]\n",
1008 | "\n",
1009 | " train_feed_dict = {ph_input_name: input_onehot, \n",
1010 | " ph_output_name: output_onehot}\n",
1011 | " \n",
1012 | " sess.run(optimizer, feed_dict = train_feed_dict)\n",
1013 | " curr_loss = sess.run(rnn_loss, feed_dict=train_feed_dict)\n",
1014 | " \n",
1015 | " total_train_loss += curr_loss/num_batch\n",
1016 | "\n",
1017 | " print('epoch : {}, train_loss : {}'.format(_epoch+1, total_train_loss))\n",
1018 | " \n",
1019 | " model_save_path = saver.save(sess, './RNN_model/model.ckpt', global_step=_epoch+1)\n",
1020 | " print('Model saved in file: {}'.format(model_save_path))"
1021 | ]
1022 | },
1023 | {
1024 | "cell_type": "markdown",
1025 | "metadata": {},
1026 | "source": [
1027 | "## Define RNN to test"
1028 | ]
1029 | },
1030 | {
1031 | "cell_type": "code",
1032 | "execution_count": 23,
1033 | "metadata": {},
1034 | "outputs": [],
1035 | "source": [
1036 | "ph_test_input_name = tf.placeholder(dtype=tf.float32, shape=[1, 1, dim_data])\n",
1037 | "ph_h = tf.placeholder(dtype=tf.float32, shape=[1, dim_rnn_cell])\n",
1038 | "ph_c = tf.placeholder(dtype=tf.float32, shape=[1, dim_rnn_cell])\n",
1039 | "\n",
1040 | "def name_rnn_test(_x, _dim_data, _dim_rnn_cell, _prev_h, _prev_c):\n",
1041 | " _x_split = tf.transpose(_x, [1, 0, 2]) # seq_len, batch, dim_data\n",
1042 | " _x_split = tf.reshape(_x_split, [-1, _dim_data])\n",
1043 | " \n",
1044 | " # Linear Operation for Input\n",
1045 | " # Fill here\n",
1046 | "\n",
1047 | " # Define LSTM Cell && RNN including (h, c)\n",
1048 | " # Fill here\n",
1049 | " \n",
1050 | " # Linear Operation for Ouput\n",
1051 | " _total_out = [] \n",
1052 | " # Fill here"
1053 | ]
1054 | },
1055 | {
1056 | "cell_type": "markdown",
1057 | "metadata": {},
1058 | "source": [
1059 | "## Run Test"
1060 | ]
1061 | },
1062 | {
1063 | "cell_type": "code",
1064 | "execution_count": 24,
1065 | "metadata": {},
1066 | "outputs": [
1067 | {
1068 | "name": "stdout",
1069 | "output_type": "stream",
1070 | "text": [
1071 | "INFO:tensorflow:Restoring parameters from ./RNN_model/model.ckpt-300\n",
1072 | "Result Name : anique\n"
1073 | ]
1074 | }
1075 | ],
1076 | "source": [
1077 | "test_result_name, test_state = name_rnn_test(ph_test_input_name, dim_data, dim_rnn_cell,\n",
1078 | " ph_h, ph_c)\n",
1079 | "\n",
1080 | "with tf.Session() as sess:\n",
1081 | " sess.run(init)\n",
1082 | " \n",
1083 | " saver.restore(sess, './RNN_model/model.ckpt-300')\n",
1084 | " \n",
1085 | " total_name = ''\n",
1086 | " prev_char = 'a'\n",
1087 | " total_name += prev_char\n",
1088 | " prev_state = (np.zeros((1, dim_rnn_cell)), np.zeros((1, dim_rnn_cell)))\n",
1089 | " for i in range(seq_len):\n",
1090 | " input_onehot = np.zeros((1, 1, dim_data))\n",
1091 | " prev_char_idx = chars.index(prev_char)\n",
1092 | " input_onehot[:, :, prev_char_idx] = 1\n",
1093 | " \n",
1094 | " test_feed_dict = {ph_test_input_name: input_onehot,\n",
1095 | " ph_h: prev_state[0], ph_c: prev_state[1]}\n",
1096 | " curr_result, curr_state = sess.run([test_result_name, test_state], test_feed_dict)\n",
1097 | " if np.argmax(curr_result) == dim_data-1:\n",
1098 | " break\n",
1099 | " else:\n",
1100 | " softmax_result = sess.run(tf.nn.softmax(test_result_name), test_feed_dict)\n",
1101 | " softmax_result = np.squeeze(softmax_result)\n",
1102 | " softmax_result = softmax_result[:dim_data-1]/sum(softmax_result[:dim_data-1])\n",
1103 | " prev_char = np.random.choice(chars, 1, p=softmax_result)\n",
1104 | " total_name += prev_char[0]\n",
1105 | " prev_state = curr_state \n",
1106 | " print('Result Name :', total_name)"
1107 | ]
1108 | }
1109 | ],
1110 | "metadata": {
1111 | "kernelspec": {
1112 | "display_name": "Python 3",
1113 | "language": "python",
1114 | "name": "python3"
1115 | },
1116 | "language_info": {
1117 | "codemirror_mode": {
1118 | "name": "ipython",
1119 | "version": 3
1120 | },
1121 | "file_extension": ".py",
1122 | "mimetype": "text/x-python",
1123 | "name": "python",
1124 | "nbconvert_exporter": "python",
1125 | "pygments_lexer": "ipython3",
1126 | "version": "3.6.8"
1127 | },
1128 | "varInspector": {
1129 | "cols": {
1130 | "lenName": 16,
1131 | "lenType": 16,
1132 | "lenVar": 40
1133 | },
1134 | "kernels_config": {
1135 | "python": {
1136 | "delete_cmd_postfix": "",
1137 | "delete_cmd_prefix": "del ",
1138 | "library": "var_list.py",
1139 | "varRefreshCmd": "print(var_dic_list())"
1140 | },
1141 | "r": {
1142 | "delete_cmd_postfix": ") ",
1143 | "delete_cmd_prefix": "rm(",
1144 | "library": "var_list.r",
1145 | "varRefreshCmd": "cat(var_dic_list()) "
1146 | }
1147 | },
1148 | "types_to_exclude": [
1149 | "module",
1150 | "function",
1151 | "builtin_function_or_method",
1152 | "instance",
1153 | "_Feature"
1154 | ],
1155 | "window_display": false
1156 | }
1157 | },
1158 | "nbformat": 4,
1159 | "nbformat_minor": 2
1160 | }
1161 |
--------------------------------------------------------------------------------
/data/regression_data.txt:
--------------------------------------------------------------------------------
1 | 0.07344104893026171 0.30577654340464877
2 | -0.6671361041630213 0.21484387904157468
3 | 0.6110255704162972 0.35501391366259544
4 | -0.9845774527321076 0.20112862280820337
5 | -0.8837433910334389 0.2082755715198671
6 | -0.12854759010314276 0.3027415514693224
7 | 0.22099135724553243 0.3316209902246986
8 | 0.8534127709556332 0.38163625638979287
9 | 0.6242369517172701 0.3556166815262491
10 | -0.9884481919394896 0.18689766580127848
11 | -0.19720628238168114 0.26657175813563644
12 | 0.40892109591833226 0.3441350190637889
13 | 0.30833442465401917 0.33391826602343727
14 | -0.7683662711499288 0.2484192484125181
15 | 0.5810093587501926 0.36320411308762074
16 | -0.6130423180488918 0.22633849283964744
17 | 0.37080779728646673 0.3126308632491387
18 | 0.2689146356815888 0.33740516201145293
19 | 0.06920308546332121 0.3047139374541547
20 | 0.2582535575581715 0.3553355585400768
21 | 0.646076592630676 0.3695065507898268
22 | 0.06043786393526185 0.31084587768470995
23 | -0.15156313196207938 0.2760793081065102
24 | -0.930058893085125 0.21387698080707523
25 | 0.036524641335194064 0.30345621937957273
26 | 0.6304748305749586 0.3676698640539604
27 | -0.07516740680855216 0.2941553031464959
28 | -0.31744264827504054 0.27497961517334646
29 | -0.530769649065824 0.25596054269450985
30 | 0.832539226011729 0.38363705013933963
31 | -0.5377918900681617 0.24087606990797308
32 | 0.41861797886723906 0.34889992586169494
33 | 0.1623015779456236 0.3147683967398835
34 | 0.22209006549848076 0.32006745957964217
35 | -0.5552588651204258 0.2587647658231737
36 | 0.4721245881055858 0.3463014535533262
37 | 0.7529675286636983 0.3840323818043389
38 | 0.7415627461497138 0.3732452555671697
39 | 0.8829149304286472 0.3804828578513638
40 | 0.8060363026316897 0.3821564893581173
41 | 0.07683324784802203 0.2835805437966362
42 | -0.04241254210041734 0.29576905089163086
43 | 0.529685938257727 0.3395512139749959
44 | 0.0003760236074303869 0.3058898167419914
45 | 0.26242265067749715 0.31159207310504095
46 | -0.3804550924918553 0.2655427548710176
47 | 0.9113962745189543 0.3928625661091734
48 | 0.1830214848198568 0.32933621351115755
49 | 0.6652377843088599 0.37993359383835845
50 | 0.2462102541127391 0.31216380570837565
51 | -0.7655938018983295 0.21120322070459682
52 | 0.12891646349998864 0.32072340722783177
53 | 0.5975051754856993 0.358046175143507
54 | 0.6132702466298365 0.36853162758867714
55 | -0.32857986166521114 0.29571146543597493
56 | 0.35673858245777246 0.35152327189372484
57 | 0.018313416895856305 0.2875541287948366
58 | 0.6374087769466188 0.3573532551994217
59 | -0.5083941951560929 0.2447003301323157
60 | -0.3798196676918688 0.25691939102461053
61 | -0.7894805225544625 0.23140230758941666
62 | -0.1207824273538356 0.29929625578730806
63 | -0.9530285118985227 0.2075403750648572
64 | -0.311370647581825 0.27641447459832724
65 | -0.4912825554267988 0.259867778499509
66 | 0.8089974781758611 0.3744296044175936
67 | -0.029339651848966808 0.30973104778627997
68 | -0.3629689399567875 0.2713338385563311
69 | -0.28085773893119925 0.2829996343601426
70 | -0.3433383050664902 0.261235805827517
71 | 0.26819021337722804 0.31512381358502417
72 | 0.8379749185037964 0.3887617954552294
73 | -0.9721217602104253 0.21795242356774774
74 | 0.380942166529068 0.339779869625644
75 | -0.7028279008436107 0.2323194173868055
76 | -0.2484023903325714 0.2823861947490793
77 | -0.43535488718672366 0.24916216430359805
78 | 0.1930005450718788 0.3353753519312524
79 | -0.12975251016629397 0.29723894398369094
80 | 0.9537897512414495 0.3968293471866555
81 | -0.4674718628893888 0.2656561736924804
82 | -0.7521595268287959 0.2278348103272739
83 | -0.19006698759297302 0.27157786416830515
84 | -0.19192104501312923 0.2914769099490614
85 | 0.19410225598679975 0.329892830646774
86 | 0.19166765679801756 0.3098383895290911
87 | 0.03917784229508059 0.30601206544169496
88 | 0.18153546167642953 0.33479577937825894
89 | 0.5739667972748239 0.34436754509312273
90 | -0.16065727380382788 0.29868282753613895
91 | 0.8320151074446431 0.4081725110522518
92 | 0.006562278756251816 0.2927020475399281
93 | -0.05950895933915401 0.3082183282688187
94 | 0.9833690684791196 0.40469345506706506
95 | -0.23000484706137847 0.2710686912913266
96 | 0.8929405125595964 0.3897941565900382
97 | 0.37392170368698285 0.3176849445645886
98 | 0.6638252849338457 0.37162713059265773
99 | 0.0807277272988256 0.3253643285629878
100 | 0.18306645332943972 0.32224581911573247
101 |
--------------------------------------------------------------------------------
/image/tensorboard.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/rllab-snu/deep_learning_tutorial/a7beb2f012660af51341a7adaf4926c1abb3eac2/image/tensorboard.png
--------------------------------------------------------------------------------