├── Chapter01 ├── Chapter 1 - Building efficient models with MXNet.ipynb ├── Chapter 1 - Building state-of-the-art, production-ready models with TensorFlow.ipynb ├── Chapter 1 - Defining networks using simple and efficient code with Gluon.ipynb ├── Chapter 1 - Implementing high-performance models with CNTK.ipynb ├── Chapter 1 - Intuitively building networks with Keras .ipynb └── Chapter 1 - Using PyTorchGÇÖs dynamic computation graphs for RNNs.ipynb ├── Chapter02 ├── Chapter 2 - Adding dropout to prevent overfitting.ipynb ├── Chapter 2 - Building a multi-layer neural network.ipynb ├── Chapter 2 - Experiment with hidden layers and hidden units.ipynb ├── Chapter 2 - Experimenting with different optimizers.ipynb ├── Chapter 2 - Getting started with activation functions.ipynb ├── Chapter 2 - Implementing a single-layer neural network.ipynb ├── Chapter 2 - Implementing an autoencoder.ipynb ├── Chapter 2 - Improving generalization with regularization.ipynb ├── Chapter 2 - Tuning the loss function.ipynb └── Chapter 2 - Understanding the Perceptron.ipynb ├── Chapter03 ├── Chapter 3 - Applying a 1D CNN to text.ipynb ├── Chapter 3 - Applying pooling layers.ipynb ├── Chapter 3 - Experimenting with different types of initialization.ipynb ├── Chapter 3 - Getting started with filters and parameter sharing.ipynb ├── Chapter 3 - Implementing a convolutional autencoder.ipynb ├── Chapter 3 - Optimizing with batch normalization.ipynb └── Chapter 3 - Understanding padding and strides.ipynb ├── Chapter04 ├── Chapter 4 - Adding Long Short-Term Memory (LSTM).ipynb ├── Chapter 4 - Adding attention.ipynb ├── Chapter 4 - Character-level text generation.ipynb ├── Chapter 4 - Implementing a simple RNN.ipynb ├── Chapter 4 - Implementing bidirectional RNNs.ipynb └── Chapter 4 - Using gated recurrent units (GRUs).ipynb ├── Chapter05 ├── Chapter 5 - Implementing a deep Q-learning algorithm.ipynb └── Chapter 5 - Implementing policy gradients.ipynb ├── Chapter06 ├── Chapter 6 - Implementing Deep Convolutional GANs (DCGANs) .ipynb ├── Chapter 6 - Understanding GANs.ipynb └── Chapter 6 - Upscaling the resolution of images with Super-Resolution GANs (SRGANs).ipynb ├── Chapter07 ├── Chapter 7 - Augmenting images with computer vision techniques.ipynb ├── Chapter 7 - Classifying objects in images.ipynb ├── Chapter 7 - Finding facial key points.ipynb ├── Chapter 7 - Localizing an object in images.ipynb ├── Chapter 7 - Recognizing faces.ipynb ├── Chapter 7 - Scene understanding (semantic segmentation).ipynb ├── Chapter 7 - Segmenting classes in images with U-net.ipynb └── Chapter 7 - Transferring styles to images.ipynb ├── Chapter08 ├── Chapter 8 - Analyzing sentiment.ipynb ├── Chapter 8 - Summarizing text.ipynb └── Chapter 8 - Translating sentences.ipynb ├── Chapter09 ├── Chapter 9 - Identifying speakers with voice recognition.ipynb ├── Chapter 9 - Implementing a speech recognition pipeline from scratch.ipynb └── Chapter 9 - Understanding videos with deep learning.ipynb ├── Chapter10 ├── Chapter 10 - Predicting bike sharing demand.ipynb ├── Chapter 10 - Predicting stock prices with neural networks.ipynb └── Chapter 10 - Using a shallow neural network for binary classification.ipynb ├── Chapter11 ├── Chapter 11 - Genetic Algorithm (GA) to optimize hyperparameters.ipynb ├── Chapter 11 - Learning to drive a car with end-to-end learning.ipynb └── Chapter 11 - Learning to play games with deep reinforcement learning.ipynb ├── Chapter12 ├── Chapter 12 - Adding dropouts to prevent overfitting.ipynb ├── Chapter 12 - Comparing optimizers.ipynb ├── Chapter 12 - Learning rates and learning rate schedulers.ipynb ├── Chapter 12 - Making a model more robust with data augmentation.ipynb ├── Chapter 12 - Using grid search for parameter tuning.ipynb ├── Chapter 12 - Visualizing training with TensorBoard and Keras.ipynb └── Chapter 12 - Working with batches and mini-batches.ipynb ├── Chapter13 ├── Chapter 13 - Analyzing network weights and more.ipynb ├── Chapter 13 - Freezing layers.ipynb ├── Chapter 13 - Storing the network topology and trained weights.ipynb └── Chapter 13 - Visualizing training with TensorBoard.ipynb ├── Chapter14 ├── Chapter 14 - Extracting bottleneck features with ResNet.ipynb ├── Chapter 14 - Fine-tuning with Xception.ipynb ├── Chapter 14 - Large-scale visual recognition with GoogLeNet_Inception.ipynb └── Chapter 14 - Leveraging pretrained VGG models for new classes.ipynb ├── LICENSE └── README.md /Chapter01/Chapter 1 - Building efficient models with MXNet.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import mxnet as mx" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": null, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "import numpy as np\n", 19 | "x_input = mx.nd.empty((1, 5), mx.gpu())\n", 20 | "x_input[:] = np.array([[1,2,3,4,5]], np.float32)\n", 21 | "\n", 22 | "y_input = mx.nd.empty((1, 5), mx.cpu())\n", 23 | "y_input[:] = np.array([[10, 15, 20, 22.5, 25]], np.float32)" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": null, 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "x_input\n", 33 | "w_input = x_input\n", 34 | "z_input = x_input.copyto(mx.cpu())\n", 35 | "x_input += 1\n", 36 | "w_input /= 2\n", 37 | "z_input *= 2" 38 | ] 39 | }, 40 | { 41 | "cell_type": "code", 42 | "execution_count": null, 43 | "metadata": {}, 44 | "outputs": [], 45 | "source": [ 46 | "print(x_input.asnumpy())\n", 47 | "print(w_input.asnumpy())\n", 48 | "print(z_input.asnumpy())" 49 | ] 50 | }, 51 | { 52 | "cell_type": "code", 53 | "execution_count": null, 54 | "metadata": {}, 55 | "outputs": [], 56 | "source": [ 57 | "batch_size = 1\n", 58 | "train_iter = mx.io.NDArrayIter(x_input, y_input, batch_size, shuffle=True, data_name='input', label_name='target')" 59 | ] 60 | }, 61 | { 62 | "cell_type": "code", 63 | "execution_count": null, 64 | "metadata": {}, 65 | "outputs": [], 66 | "source": [ 67 | "X = mx.sym.Variable('input')\n", 68 | "Y = mx.symbol.Variable('target')\n", 69 | "fc1 = mx.sym.FullyConnected(data=X, name='fc1', num_hidden = 5)\n", 70 | "lin_reg = mx.sym.LinearRegressionOutput(data=fc1, label=Y, name=\"lin_reg\")" 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": null, 76 | "metadata": {}, 77 | "outputs": [], 78 | "source": [ 79 | "model = mx.mod.Module(\n", 80 | " symbol = lin_reg,\n", 81 | " data_names=['input'], \n", 82 | " label_names = ['target']\n", 83 | ")" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": null, 89 | "metadata": {}, 90 | "outputs": [], 91 | "source": [ 92 | "model.fit(train_iter,\n", 93 | " optimizer_params={'learning_rate':0.01, 'momentum': 0.9},\n", 94 | " num_epoch=100,\n", 95 | " batch_end_callback = mx.callback.Speedometer(batch_size, 2))" 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": null, 101 | "metadata": {}, 102 | "outputs": [], 103 | "source": [ 104 | "model.predict(train_iter).asnumpy()" 105 | ] 106 | } 107 | ], 108 | "metadata": { 109 | "kernelspec": { 110 | "display_name": "Python 3", 111 | "language": "python", 112 | "name": "python3" 113 | }, 114 | "language_info": { 115 | "codemirror_mode": { 116 | "name": "ipython", 117 | "version": 3 118 | }, 119 | "file_extension": ".py", 120 | "mimetype": "text/x-python", 121 | "name": "python", 122 | "nbconvert_exporter": "python", 123 | "pygments_lexer": "ipython3", 124 | "version": "3.5.2" 125 | }, 126 | "widgets": { 127 | "state": {}, 128 | "version": "1.1.2" 129 | } 130 | }, 131 | "nbformat": 4, 132 | "nbformat_minor": 2 133 | } 134 | -------------------------------------------------------------------------------- /Chapter01/Chapter 1 - Building state-of-the-art, production-ready models with TensorFlow.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import tensorflow as tf" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 2, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "import numpy as np\n", 19 | "x_input = np.array([[1,2,3,4,5]])\n", 20 | "y_input = np.array([[10]])" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": 3, 26 | "metadata": {}, 27 | "outputs": [], 28 | "source": [ 29 | "x = tf.placeholder(tf.float32, [None, 5])\n", 30 | "y = tf.placeholder(tf.float32, [None, 1])" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": 4, 36 | "metadata": {}, 37 | "outputs": [], 38 | "source": [ 39 | "W = tf.Variable(tf.zeros([5, 1]))\n", 40 | "b = tf.Variable(tf.zeros([1]))\n", 41 | "y_pred = tf.matmul(x, W)+b" 42 | ] 43 | }, 44 | { 45 | "cell_type": "code", 46 | "execution_count": 5, 47 | "metadata": {}, 48 | "outputs": [], 49 | "source": [ 50 | "loss = tf.reduce_sum(tf.pow((y-y_pred), 2))" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": 6, 56 | "metadata": {}, 57 | "outputs": [], 58 | "source": [ 59 | "train = tf.train.GradientDescentOptimizer(0.0001).minimize(loss)" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 7, 65 | "metadata": {}, 66 | "outputs": [], 67 | "source": [ 68 | "init = tf.global_variables_initializer()" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": 8, 74 | "metadata": {}, 75 | "outputs": [], 76 | "source": [ 77 | "sess = tf.Session()\n", 78 | "sess.run(init)\n", 79 | "for i in range(10):\n", 80 | " feed_dict = {x: x_input, y: y_input}\n", 81 | " sess.run(train, feed_dict=feed_dict)" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": 9, 87 | "metadata": {}, 88 | "outputs": [ 89 | { 90 | "name": "stdout", 91 | "output_type": "stream", 92 | "text": [ 93 | "100.0\n", 94 | "97.7726\n", 95 | "95.5947\n", 96 | "93.4654\n", 97 | "91.3835\n", 98 | "89.3479\n", 99 | "87.3578\n", 100 | "85.4119\n", 101 | "83.5094\n", 102 | "81.6492\n" 103 | ] 104 | } 105 | ], 106 | "source": [ 107 | "sess = tf.Session()\n", 108 | "sess.run(init)\n", 109 | "for i in range(10):\n", 110 | " feed_dict = {x: x_input, y: y_input}\n", 111 | " _, loss_value = sess.run([train, loss], feed_dict=feed_dict)\n", 112 | " print(loss_value)" 113 | ] 114 | } 115 | ], 116 | "metadata": { 117 | "kernelspec": { 118 | "display_name": "Python 3", 119 | "language": "python", 120 | "name": "python3" 121 | }, 122 | "language_info": { 123 | "codemirror_mode": { 124 | "name": "ipython", 125 | "version": 3 126 | }, 127 | "file_extension": ".py", 128 | "mimetype": "text/x-python", 129 | "name": "python", 130 | "nbconvert_exporter": "python", 131 | "pygments_lexer": "ipython3", 132 | "version": "3.5.2" 133 | }, 134 | "widgets": { 135 | "state": {}, 136 | "version": "1.1.2" 137 | } 138 | }, 139 | "nbformat": 4, 140 | "nbformat_minor": 2 141 | } 142 | -------------------------------------------------------------------------------- /Chapter01/Chapter 1 - Defining networks using simple and efficient code with Gluon.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "from mxnet import gluon" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": null, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "import mxnet as mx\n", 19 | "import numpy as np\n", 20 | "x_input = mx.nd.empty((1, 5), mx.gpu())\n", 21 | "x_input[:] = np.array([[1,2,3,4,5]], np.float32)\n", 22 | "\n", 23 | "y_input = mx.nd.empty((1, 5), mx.gpu())\n", 24 | "y_input[:] = np.array([[10, 15, 20, 22.5, 25]], np.float32)" 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": null, 30 | "metadata": {}, 31 | "outputs": [], 32 | "source": [ 33 | "net = gluon.nn.Sequential()\n", 34 | "with net.name_scope():\n", 35 | " net.add(gluon.nn.Dense(16, activation=\"relu\"))\n", 36 | " net.add(gluon.nn.Dense(len(y_input)))" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": null, 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "net.collect_params().initialize(mx.init.Normal(), ctx=mx.gpu())" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": null, 51 | "metadata": {}, 52 | "outputs": [], 53 | "source": [ 54 | "softmax_cross_entropy = gluon.loss.SoftmaxCrossEntropyLoss()\n", 55 | "trainer = gluon.Trainer(net.collect_params(), 'adam', {'learning_rate': .1})" 56 | ] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "execution_count": null, 61 | "metadata": {}, 62 | "outputs": [], 63 | "source": [ 64 | "n_epochs = 10\n", 65 | "\n", 66 | "for e in range(n_epochs):\n", 67 | " for i in range(len(x_input)):\n", 68 | " input = x_input[i]\n", 69 | " target = y_input[i]\n", 70 | " with mx.autograd.record():\n", 71 | " output = net(input)\n", 72 | " loss = softmax_cross_entropy(output, target)\n", 73 | " loss.backward()\n", 74 | " trainer.step(input.shape[0])" 75 | ] 76 | } 77 | ], 78 | "metadata": { 79 | "kernelspec": { 80 | "display_name": "Python 3", 81 | "language": "python", 82 | "name": "python3" 83 | }, 84 | "language_info": { 85 | "codemirror_mode": { 86 | "name": "ipython", 87 | "version": 3 88 | }, 89 | "file_extension": ".py", 90 | "mimetype": "text/x-python", 91 | "name": "python", 92 | "nbconvert_exporter": "python", 93 | "pygments_lexer": "ipython3", 94 | "version": "3.5.2" 95 | }, 96 | "widgets": { 97 | "state": {}, 98 | "version": "1.1.2" 99 | } 100 | }, 101 | "nbformat": 4, 102 | "nbformat_minor": 2 103 | } 104 | -------------------------------------------------------------------------------- /Chapter01/Chapter 1 - Implementing high-performance models with CNTK.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import cntk" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 2, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "import numpy as np\n", 19 | "x_input = np.array([[1,2,3,4,5]], np.float32)\n", 20 | "y_input = np.array([[10]], np.float32)" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": 3, 26 | "metadata": {}, 27 | "outputs": [], 28 | "source": [ 29 | "X = cntk.input_variable(5, np.float32)\n", 30 | "y = cntk.input_variable(1, np.float32)" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": 4, 36 | "metadata": {}, 37 | "outputs": [], 38 | "source": [ 39 | "from cntk.layers import Dense, Sequential\n", 40 | "model = Sequential([Dense(32),\n", 41 | " Dense(1)])(X)" 42 | ] 43 | }, 44 | { 45 | "cell_type": "code", 46 | "execution_count": 5, 47 | "metadata": {}, 48 | "outputs": [], 49 | "source": [ 50 | "loss = cntk.squared_error(model, y)" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": 6, 56 | "metadata": {}, 57 | "outputs": [], 58 | "source": [ 59 | "learning_rate = 0.001\n", 60 | "trainer = cntk.Trainer(model, (loss),\n", 61 | "cntk.adagrad(model.parameters, learning_rate))" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 7, 67 | "metadata": {}, 68 | "outputs": [], 69 | "source": [ 70 | "for epoch in range(10):\n", 71 | " trainer.train_minibatch({X: x_input, y: y_input})" 72 | ] 73 | } 74 | ], 75 | "metadata": { 76 | "kernelspec": { 77 | "display_name": "Python 3", 78 | "language": "python", 79 | "name": "python3" 80 | }, 81 | "language_info": { 82 | "codemirror_mode": { 83 | "name": "ipython", 84 | "version": 3 85 | }, 86 | "file_extension": ".py", 87 | "mimetype": "text/x-python", 88 | "name": "python", 89 | "nbconvert_exporter": "python", 90 | "pygments_lexer": "ipython3", 91 | "version": "3.5.2" 92 | }, 93 | "widgets": { 94 | "state": {}, 95 | "version": "1.1.2" 96 | } 97 | }, 98 | "nbformat": 4, 99 | "nbformat_minor": 2 100 | } 101 | -------------------------------------------------------------------------------- /Chapter01/Chapter 1 - Intuitively building networks with Keras .ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "name": "stderr", 10 | "output_type": "stream", 11 | "text": [ 12 | "Using TensorFlow backend.\n" 13 | ] 14 | } 15 | ], 16 | "source": [ 17 | "from keras.models import Sequential\n", 18 | "from keras.layers import Dense" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 2, 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "import numpy as np\n", 28 | "x_input = np.array([[1,2,3,4,5]])\n", 29 | "y_input = np.array([[10]])" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": 3, 35 | "metadata": {}, 36 | "outputs": [], 37 | "source": [ 38 | "model = Sequential()\n", 39 | "model.add(Dense(units=32, input_dim=x_input.shape[1]))\n", 40 | "model.add(Dense(units=1))" 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": 4, 46 | "metadata": {}, 47 | "outputs": [], 48 | "source": [ 49 | "model.compile(loss='mse', optimizer='sgd', metrics=['accuracy'])" 50 | ] 51 | }, 52 | { 53 | "cell_type": "code", 54 | "execution_count": 5, 55 | "metadata": {}, 56 | "outputs": [ 57 | { 58 | "name": "stdout", 59 | "output_type": "stream", 60 | "text": [ 61 | "_________________________________________________________________\n", 62 | "Layer (type) Output Shape Param # \n", 63 | "=================================================================\n", 64 | "dense_1 (Dense) (None, 32) 192 \n", 65 | "_________________________________________________________________\n", 66 | "dense_2 (Dense) (None, 1) 33 \n", 67 | "=================================================================\n", 68 | "Total params: 225\n", 69 | "Trainable params: 225\n", 70 | "Non-trainable params: 0\n", 71 | "_________________________________________________________________\n" 72 | ] 73 | } 74 | ], 75 | "source": [ 76 | "model.summary()" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": 6, 82 | "metadata": {}, 83 | "outputs": [ 84 | { 85 | "name": "stdout", 86 | "output_type": "stream", 87 | "text": [ 88 | "Epoch 1/10\n", 89 | "1/1 [==============================] - 0s - loss: 43.3018 - acc: 0.0000e+00\n", 90 | "Epoch 2/10\n", 91 | "1/1 [==============================] - 0s - loss: 1389.6208 - acc: 0.0000e+00\n", 92 | "Epoch 3/10\n", 93 | "1/1 [==============================] - 0s - loss: 108900760.0000 - acc: 0.0000e+00\n", 94 | "Epoch 4/10\n", 95 | "1/1 [==============================] - 0s - loss: 52705562569571402477891223552.0000 - acc: 0.0000e+00\n", 96 | "Epoch 5/10\n", 97 | "1/1 [==============================] - 0s - loss: inf - acc: 0.0000e+00\n", 98 | "Epoch 6/10\n", 99 | "1/1 [==============================] - 0s - loss: inf - acc: 0.0000e+00\n", 100 | "Epoch 7/10\n", 101 | "1/1 [==============================] - 0s - loss: nan - acc: 0.0000e+00\n", 102 | "Epoch 8/10\n", 103 | "1/1 [==============================] - 0s - loss: nan - acc: 0.0000e+00\n", 104 | "Epoch 9/10\n", 105 | "1/1 [==============================] - 0s - loss: nan - acc: 0.0000e+00\n", 106 | "Epoch 10/10\n", 107 | "1/1 [==============================] - 0s - loss: nan - acc: 0.0000e+00\n" 108 | ] 109 | } 110 | ], 111 | "source": [ 112 | "history = model.fit(x_input, y_input, epochs=10, batch_size=32)" 113 | ] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": 7, 118 | "metadata": {}, 119 | "outputs": [], 120 | "source": [ 121 | "pred = model.predict(x_input, batch_size=128)" 122 | ] 123 | } 124 | ], 125 | "metadata": { 126 | "kernelspec": { 127 | "display_name": "Python 3", 128 | "language": "python", 129 | "name": "python3" 130 | }, 131 | "language_info": { 132 | "codemirror_mode": { 133 | "name": "ipython", 134 | "version": 3 135 | }, 136 | "file_extension": ".py", 137 | "mimetype": "text/x-python", 138 | "name": "python", 139 | "nbconvert_exporter": "python", 140 | "pygments_lexer": "ipython3", 141 | "version": "3.5.2" 142 | }, 143 | "widgets": { 144 | "state": {}, 145 | "version": "1.1.2" 146 | } 147 | }, 148 | "nbformat": 4, 149 | "nbformat_minor": 2 150 | } 151 | -------------------------------------------------------------------------------- /Chapter01/Chapter 1 - Using PyTorchGÇÖs dynamic computation graphs for RNNs.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import torch" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 2, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "batch_size = 32\n", 19 | "input_shape = 5\n", 20 | "output_shape = 10" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": 3, 26 | "metadata": {}, 27 | "outputs": [], 28 | "source": [ 29 | "torch.set_default_tensor_type('torch.cuda.FloatTensor')" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": 4, 35 | "metadata": {}, 36 | "outputs": [], 37 | "source": [ 38 | "from torch.autograd import Variable\n", 39 | "X = Variable(torch.randn(batch_size, input_shape))\n", 40 | "y = Variable(torch.randn(batch_size, output_shape), requires_grad=False)" 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": 5, 46 | "metadata": {}, 47 | "outputs": [], 48 | "source": [ 49 | "model = torch.nn.Sequential(\n", 50 | " torch.nn.Linear(input_shape, 32),\n", 51 | " torch.nn.Linear(32, output_shape),\n", 52 | " ).cuda()" 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": 6, 58 | "metadata": {}, 59 | "outputs": [], 60 | "source": [ 61 | "loss_function = torch.nn.MSELoss()" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 7, 67 | "metadata": {}, 68 | "outputs": [ 69 | { 70 | "name": "stdout", 71 | "output_type": "stream", 72 | "text": [ 73 | "1.0649133920669556\n", 74 | "1.0644211769104004\n", 75 | "1.0639302730560303\n", 76 | "1.0634393692016602\n", 77 | "1.0629512071609497\n", 78 | "1.0624619722366333\n", 79 | "1.061974287033081\n", 80 | "1.0614871978759766\n", 81 | "1.0610013008117676\n", 82 | "1.0605162382125854\n" 83 | ] 84 | } 85 | ], 86 | "source": [ 87 | "learning_rate = 0.001\n", 88 | "for i in range(10):\n", 89 | " y_pred = model(X)\n", 90 | " loss = loss_function(y_pred, y)\n", 91 | " print(loss.data[0])\n", 92 | " # Zero gradients\n", 93 | " model.zero_grad()\n", 94 | " loss.backward()\n", 95 | " # Update weights\n", 96 | " for param in model.parameters():\n", 97 | " param.data -= learning_rate * param.grad.data" 98 | ] 99 | } 100 | ], 101 | "metadata": { 102 | "kernelspec": { 103 | "display_name": "Python 3", 104 | "language": "python", 105 | "name": "python3" 106 | }, 107 | "language_info": { 108 | "codemirror_mode": { 109 | "name": "ipython", 110 | "version": 3 111 | }, 112 | "file_extension": ".py", 113 | "mimetype": "text/x-python", 114 | "name": "python", 115 | "nbconvert_exporter": "python", 116 | "pygments_lexer": "ipython3", 117 | "version": "3.5.2" 118 | }, 119 | "widgets": { 120 | "state": {}, 121 | "version": "1.1.2" 122 | } 123 | }, 124 | "nbformat": 4, 125 | "nbformat_minor": 2 126 | } 127 | -------------------------------------------------------------------------------- /Chapter02/Chapter 2 - Adding dropout to prevent overfitting.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np \n", 10 | "import pandas as pd\n", 11 | "from matplotlib import pyplot as plt\n", 12 | "\n", 13 | "from keras.models import Sequential\n", 14 | "from keras.layers import Dense, Dropout\n", 15 | "\n", 16 | "import numpy as np\n", 17 | "from matplotlib import pyplot as plt" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": null, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "# Dataset can be downloaded at https://archive.ics.uci.edu/ml/machine-learning-databases/00275/" 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": null, 32 | "metadata": {}, 33 | "outputs": [], 34 | "source": [ 35 | "data = pd.read_csv('Data/bike-sharing/hour.csv')\n", 36 | "# Feature engineering\n", 37 | "ohe_features = ['season', 'weathersit', 'mnth', 'hr', 'weekday']\n", 38 | "for feature in ohe_features:\n", 39 | " dummies = pd.get_dummies(data[feature], prefix=feature, drop_first=False)\n", 40 | " data = pd.concat([data, dummies], axis=1)\n", 41 | "\n", 42 | "drop_features = ['instant', 'dteday', 'season', 'weathersit', 'weekday', 'atemp', 'mnth', 'workingday', 'hr', 'casual', 'registered']\n", 43 | "data = data.drop(drop_features, axis=1)" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": null, 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "norm_features = ['cnt', 'temp', 'hum', 'windspeed']\n", 53 | "scaled_features = {}\n", 54 | "for feature in norm_features:\n", 55 | " mean, std = data[feature].mean(), data[feature].std()\n", 56 | " scaled_features[feature] = [mean, std]\n", 57 | " data.loc[:, feature] = (data[feature] - mean)/std" 58 | ] 59 | }, 60 | { 61 | "cell_type": "code", 62 | "execution_count": null, 63 | "metadata": {}, 64 | "outputs": [], 65 | "source": [ 66 | "# Save the final month for testing\n", 67 | "test_data = data[-31*24:]\n", 68 | "data = data[:-31*24]\n", 69 | "\n", 70 | "# Extract the target field\n", 71 | "target_fields = ['cnt']\n", 72 | "features, targets = data.drop(target_fields, axis=1), data[target_fields]\n", 73 | "test_features, test_targets = test_data.drop(target_fields, axis=1), test_data[target_fields]\n", 74 | "\n", 75 | "# Create a validation set (based on the last )\n", 76 | "X_train, y_train = features[:-30*24], targets[:-30*24]\n", 77 | "X_val, y_val = features[-30*24:], targets[-30*24:]" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": null, 83 | "metadata": {}, 84 | "outputs": [], 85 | "source": [ 86 | "model = Sequential()\n", 87 | "model.add(Dense(250, input_dim=X_train.shape[1], activation='relu'))\n", 88 | "model.add(Dense(150, activation='relu'))\n", 89 | "model.add(Dense(50, activation='relu'))\n", 90 | "model.add(Dense(25, activation='relu'))\n", 91 | "model.add(Dense(1, activation='linear'))\n", 92 | "\n", 93 | "# Compile model\n", 94 | "model.compile(loss='mse', optimizer='sgd', metrics=['mse'])" 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": null, 100 | "metadata": {}, 101 | "outputs": [], 102 | "source": [ 103 | "n_epochs = 1000\n", 104 | "batch_size = 1024\n", 105 | "\n", 106 | "history = model.fit(X_train.values, y_train['cnt'], \n", 107 | " validation_data=(X_val.values, y_val['cnt']), \n", 108 | " batch_size=batch_size, epochs=n_epochs, verbose=0\n", 109 | " )" 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": null, 115 | "metadata": {}, 116 | "outputs": [], 117 | "source": [ 118 | "plt.plot(np.arange(len(history.history['loss'])), history.history['loss'], label='training')\n", 119 | "plt.plot(np.arange(len(history.history['val_loss'])), history.history['val_loss'], label='validation')\n", 120 | "plt.title('Overfit on Bike Sharing dataset')\n", 121 | "plt.xlabel('epochs')\n", 122 | "plt.ylabel('loss')\n", 123 | "plt.legend(loc=0)\n", 124 | "plt.show()" 125 | ] 126 | }, 127 | { 128 | "cell_type": "code", 129 | "execution_count": null, 130 | "metadata": {}, 131 | "outputs": [], 132 | "source": [ 133 | "print('Minimum loss: ', min(history.history['val_loss']), \n", 134 | " '\\nAfter ', np.argmin(history.history['val_loss']), ' epochs')\n", 135 | "\n", 136 | "# Minimum loss: 0.129907280207 \n", 137 | "# After 980 epochs" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": null, 143 | "metadata": {}, 144 | "outputs": [], 145 | "source": [ 146 | "model_drop = Sequential()\n", 147 | "model_drop.add(Dense(250, input_dim=X_train.shape[1], activation='relu'))\n", 148 | "model_drop.add(Dropout(0.20))\n", 149 | "model_drop.add(Dense(150, activation='relu'))\n", 150 | "model_drop.add(Dropout(0.20))\n", 151 | "model_drop.add(Dense(50, activation='relu'))\n", 152 | "model_drop.add(Dropout(0.20))\n", 153 | "model_drop.add(Dense(25, activation='relu'))\n", 154 | "model_drop.add(Dropout(0.20))\n", 155 | "model_drop.add(Dense(1, activation='linear'))\n", 156 | "\n", 157 | "# Compile model\n", 158 | "model_drop.compile(loss='mse', optimizer='sgd', metrics=['mse'])" 159 | ] 160 | }, 161 | { 162 | "cell_type": "code", 163 | "execution_count": null, 164 | "metadata": {}, 165 | "outputs": [], 166 | "source": [ 167 | "history_drop = model_drop.fit(X_train.values, y_train['cnt'], \n", 168 | " validation_data=(X_val.values, y_val['cnt']), \n", 169 | " batch_size=batch_size, epochs=n_epochs, verbose=0\n", 170 | " )" 171 | ] 172 | }, 173 | { 174 | "cell_type": "code", 175 | "execution_count": null, 176 | "metadata": {}, 177 | "outputs": [], 178 | "source": [ 179 | "plt.plot(np.arange(len(history_drop.history['loss'])), history_drop.history['loss'], label='training')\n", 180 | "plt.plot(np.arange(len(history_drop.history['val_loss'])), history_drop.history['val_loss'], label='validation')\n", 181 | "plt.title('Use dropout for Bike Sharing dataset')\n", 182 | "plt.xlabel('epochs')\n", 183 | "plt.ylabel('loss')\n", 184 | "plt.legend(loc=0)\n", 185 | "plt.show()" 186 | ] 187 | }, 188 | { 189 | "cell_type": "code", 190 | "execution_count": null, 191 | "metadata": {}, 192 | "outputs": [], 193 | "source": [ 194 | "print('Minimum loss: ', min(history_drop.history['val_loss']), \n", 195 | " '\\nAfter ', np.argmin(history_drop.history['val_loss']), ' epochs')\n", 196 | "\n", 197 | "# Minimum loss: 0.126063346863 \n", 198 | "# After 998 epochs" 199 | ] 200 | }, 201 | { 202 | "cell_type": "code", 203 | "execution_count": null, 204 | "metadata": {}, 205 | "outputs": [], 206 | "source": [] 207 | } 208 | ], 209 | "metadata": { 210 | "kernelspec": { 211 | "display_name": "Python 3", 212 | "language": "python", 213 | "name": "python3" 214 | }, 215 | "language_info": { 216 | "codemirror_mode": { 217 | "name": "ipython", 218 | "version": 3 219 | }, 220 | "file_extension": ".py", 221 | "mimetype": "text/x-python", 222 | "name": "python", 223 | "nbconvert_exporter": "python", 224 | "pygments_lexer": "ipython3", 225 | "version": "3.5.2" 226 | }, 227 | "widgets": { 228 | "state": {}, 229 | "version": "1.1.2" 230 | } 231 | }, 232 | "nbformat": 4, 233 | "nbformat_minor": 2 234 | } 235 | -------------------------------------------------------------------------------- /Chapter02/Chapter 2 - Improving generalization with regularization.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np\n", 10 | "import pandas as pd\n", 11 | "from matplotlib import pyplot as plt\n", 12 | "\n", 13 | "from keras.models import Sequential\n", 14 | "from keras.layers import Dense, Dropout\n", 15 | "from keras import regularizers" 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": null, 21 | "metadata": {}, 22 | "outputs": [], 23 | "source": [ 24 | "# Dataset can be downloaded at https://archive.ics.uci.edu/ml/machine-learning-databases/00275/" 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": null, 30 | "metadata": {}, 31 | "outputs": [], 32 | "source": [ 33 | "data = pd.read_csv('Data/bike-sharing/hour.csv')\n", 34 | "# Feature engineering\n", 35 | "ohe_features = ['season', 'weathersit', 'mnth', 'hr', 'weekday']\n", 36 | "for feature in ohe_features:\n", 37 | " dummies = pd.get_dummies(data[feature], prefix=feature, drop_first=False)\n", 38 | " data = pd.concat([data, dummies], axis=1)\n", 39 | "\n", 40 | "drop_features = ['instant', 'dteday', 'season', 'weathersit', \n", 41 | " 'weekday', 'atemp', 'mnth', 'workingday', 'hr', 'casual', 'registered']\n", 42 | "data = data.drop(drop_features, axis=1)" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": null, 48 | "metadata": {}, 49 | "outputs": [], 50 | "source": [ 51 | "norm_features = ['cnt', 'temp', 'hum', 'windspeed']\n", 52 | "scaled_features = {}\n", 53 | "for feature in norm_features:\n", 54 | " mean, std = data[feature].mean(), data[feature].std()\n", 55 | " scaled_features[feature] = [mean, std]\n", 56 | " data.loc[:, feature] = (data[feature] - mean)/std" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": null, 62 | "metadata": {}, 63 | "outputs": [], 64 | "source": [ 65 | "# Save the final month for testing\n", 66 | "test_data = data[-31*24:]\n", 67 | "data = data[:-31*24]\n", 68 | "# Extract the target field\n", 69 | "target_fields = ['cnt']\n", 70 | "features, targets = data.drop(target_fields, axis=1), data[target_fields]\n", 71 | "test_features, test_targets = test_data.drop(target_fields, axis=1), test_data[target_fields]\n", 72 | "# Create a validation set (based on the last )\n", 73 | "X_train, y_train = features[:-30*24], targets[:-30*24]\n", 74 | "X_val, y_val = features[-30*24:], targets[-30*24:]" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": null, 80 | "metadata": {}, 81 | "outputs": [], 82 | "source": [ 83 | "model = Sequential()\n", 84 | "model.add(Dense(250, input_dim=X_train.shape[1], activation='relu'))\n", 85 | "model.add(Dense(150, activation='relu'))\n", 86 | "model.add(Dense(50, activation='relu'))\n", 87 | "model.add(Dense(25, activation='relu'))\n", 88 | "model.add(Dense(1, activation='linear'))\n", 89 | "\n", 90 | "# Compile model\n", 91 | "model.compile(loss='mse', optimizer='sgd', metrics=['mse'])" 92 | ] 93 | }, 94 | { 95 | "cell_type": "code", 96 | "execution_count": null, 97 | "metadata": {}, 98 | "outputs": [], 99 | "source": [ 100 | "n_epochs = 4000\n", 101 | "batch_size = 1024\n", 102 | "\n", 103 | "history = model.fit(X_train.values, y_train['cnt'], \n", 104 | " validation_data=(X_val.values, y_val['cnt']), \n", 105 | " batch_size=batch_size, epochs=n_epochs, verbose=0\n", 106 | " )" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": null, 112 | "metadata": {}, 113 | "outputs": [], 114 | "source": [ 115 | "plt.plot(np.arange(len(history.history['loss'])), history.history['loss'], label='training')\n", 116 | "plt.plot(np.arange(len(history.history['val_loss'])), history.history['val_loss'], label='validation')\n", 117 | "plt.title('Overfit on Bike Sharing dataset')\n", 118 | "plt.xlabel('epochs')\n", 119 | "plt.ylabel('loss')\n", 120 | "plt.legend(loc=0)\n", 121 | "plt.show()" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": null, 127 | "metadata": {}, 128 | "outputs": [], 129 | "source": [ 130 | "print('Minimum loss: ', min(history.history['val_loss']), \n", 131 | " '\\nAfter ', np.argmin(history.history['val_loss']), ' epochs')\n", 132 | "\n", 133 | "# Minimum loss: 0.140975862741 \n", 134 | "# After 730 epochs" 135 | ] 136 | }, 137 | { 138 | "cell_type": "code", 139 | "execution_count": null, 140 | "metadata": {}, 141 | "outputs": [], 142 | "source": [ 143 | "model_reg = Sequential()\n", 144 | "model_reg.add(Dense(250, input_dim=X_train.shape[1], activation='relu',\n", 145 | " kernel_regularizer=regularizers.l2(0.005)))\n", 146 | "model_reg.add(Dense(150, activation='relu'))\n", 147 | "model_reg.add(Dense(50, activation='relu'))\n", 148 | "model_reg.add(Dense(25, activation='relu',\n", 149 | " kernel_regularizer=regularizers.l2(0.005)))\n", 150 | "model_reg.add(Dense(1, activation='linear'))\n", 151 | "\n", 152 | "# Compile model\n", 153 | "model_reg.compile(loss='mse', optimizer='sgd', metrics=['mse'])" 154 | ] 155 | }, 156 | { 157 | "cell_type": "code", 158 | "execution_count": null, 159 | "metadata": {}, 160 | "outputs": [], 161 | "source": [ 162 | "history_reg = model_reg.fit(X_train.values, y_train['cnt'], \n", 163 | "validation_data=(X_val.values, y_val['cnt']), \n", 164 | " batch_size=batch_size, epochs=n_epochs, verbose=0\n", 165 | " )" 166 | ] 167 | }, 168 | { 169 | "cell_type": "code", 170 | "execution_count": null, 171 | "metadata": {}, 172 | "outputs": [], 173 | "source": [ 174 | "plt.plot(np.arange(len(history_reg.history['loss'])), history_reg.history['loss'], label='training')\n", 175 | "plt.plot(np.arange(len(history_reg.history['val_loss'])), history_reg.history['val_loss'], label='validation')\n", 176 | "plt.title('Use regularisation for Bike Sharing dataset')\n", 177 | "plt.xlabel('epochs')\n", 178 | "plt.ylabel('loss')\n", 179 | "plt.legend(loc=0)\n", 180 | "plt.show()" 181 | ] 182 | }, 183 | { 184 | "cell_type": "code", 185 | "execution_count": null, 186 | "metadata": {}, 187 | "outputs": [], 188 | "source": [ 189 | "print('Minimum loss: ', min(history_reg.history['val_loss']), \n", 190 | " '\\nAfter ', np.argmin(history_reg.history['val_loss']), ' epochs')\n", 191 | "\n", 192 | "# Minimum loss: 0.13514482975 \n", 193 | "# After 3647 epochs" 194 | ] 195 | }, 196 | { 197 | "cell_type": "code", 198 | "execution_count": null, 199 | "metadata": {}, 200 | "outputs": [], 201 | "source": [] 202 | } 203 | ], 204 | "metadata": { 205 | "kernelspec": { 206 | "display_name": "Python 3", 207 | "language": "python", 208 | "name": "python3" 209 | }, 210 | "language_info": { 211 | "codemirror_mode": { 212 | "name": "ipython", 213 | "version": 3 214 | }, 215 | "file_extension": ".py", 216 | "mimetype": "text/x-python", 217 | "name": "python", 218 | "nbconvert_exporter": "python", 219 | "pygments_lexer": "ipython3", 220 | "version": "3.5.2" 221 | }, 222 | "widgets": { 223 | "state": {}, 224 | "version": "1.1.2" 225 | } 226 | }, 227 | "nbformat": 4, 228 | "nbformat_minor": 2 229 | } 230 | -------------------------------------------------------------------------------- /Chapter02/Chapter 2 - Tuning the loss function.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "name": "stderr", 10 | "output_type": "stream", 11 | "text": [ 12 | "Using TensorFlow backend.\n" 13 | ] 14 | } 15 | ], 16 | "source": [ 17 | "import numpy as np\n", 18 | "from matplotlib import pyplot as plt\n", 19 | "from sklearn.metrics import confusion_matrix\n", 20 | "\n", 21 | "from keras.datasets import mnist\n", 22 | "from keras.models import Sequential\n", 23 | "from keras.layers import Dense, Dropout\n", 24 | "from keras.optimizers import Adam\n", 25 | "from keras.callbacks import EarlyStopping" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": 2, 31 | "metadata": {}, 32 | "outputs": [], 33 | "source": [ 34 | "(X_train, y_train), (X_test, y_test) = mnist.load_data()\n", 35 | "# Extract all 9s and 100 examples of 4s\n", 36 | "y_train_9 = y_train[y_train == 9]\n", 37 | "y_train_4 = y_train[y_train == 4][:100]\n", 38 | "X_train_9 = X_train[y_train == 9]\n", 39 | "X_train_4 = X_train[y_train == 4][:100]\n", 40 | "X_train = np.concatenate((X_train_9, X_train_4), axis=0)\n", 41 | "y_train = np.concatenate((y_train_9, y_train_4), axis=0)\n", 42 | "\n", 43 | "y_test_9 = y_test[y_test == 9]\n", 44 | "y_test_4 = y_test[y_test == 4]\n", 45 | "X_test_9 = X_test[y_test == 9]\n", 46 | "X_test_4 = X_test[y_test == 4]\n", 47 | "X_test = np.concatenate((X_test_9, X_test_4), axis=0)\n", 48 | "y_test = np.concatenate((y_test_9, y_test_4), axis=0)" 49 | ] 50 | }, 51 | { 52 | "cell_type": "code", 53 | "execution_count": 3, 54 | "metadata": {}, 55 | "outputs": [], 56 | "source": [ 57 | "X_train = X_train.astype('float32')/255.\n", 58 | "X_test = X_test.astype('float32')/255.\n", 59 | "X_train = X_train.reshape(len(X_train), np.prod(X_train.shape[1:]))\n", 60 | "X_test = X_test.reshape(len(X_test), np.prod(X_test.shape[1:]))" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": 4, 66 | "metadata": {}, 67 | "outputs": [ 68 | { 69 | "data": { 70 | "text/plain": [ 71 | "(1991, 784)" 72 | ] 73 | }, 74 | "execution_count": 4, 75 | "metadata": {}, 76 | "output_type": "execute_result" 77 | } 78 | ], 79 | "source": [ 80 | "X_test.shape" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": 5, 86 | "metadata": {}, 87 | "outputs": [ 88 | { 89 | "name": "stdout", 90 | "output_type": "stream", 91 | "text": [ 92 | "(array([False, True], dtype=bool), array([ 100, 5949]))\n" 93 | ] 94 | } 95 | ], 96 | "source": [ 97 | "y_train_binary = y_train == 9\n", 98 | "y_test_binary = y_test == 9\n", 99 | "print(np.unique(y_train_binary, return_counts=True))" 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": 6, 105 | "metadata": {}, 106 | "outputs": [], 107 | "source": [ 108 | "model = Sequential()\n", 109 | "model.add(Dense(512, input_dim=X_train.shape[1], activation='relu'))\n", 110 | "model.add(Dropout(0.75))\n", 111 | "model.add(Dense(512, activation='relu'))\n", 112 | "model.add(Dropout(0.75))\n", 113 | "model.add(Dense(128, activation='relu'))\n", 114 | "model.add(Dropout(0.75))\n", 115 | "model.add(Dense(128, activation='relu'))\n", 116 | "model.add(Dense(1, activation='sigmoid'))\n", 117 | "\n", 118 | "opt = Adam()\n", 119 | "\n", 120 | "model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['binary_accuracy'])" 121 | ] 122 | }, 123 | { 124 | "cell_type": "code", 125 | "execution_count": 7, 126 | "metadata": {}, 127 | "outputs": [], 128 | "source": [ 129 | "callbacks = [EarlyStopping(monitor='val_loss', patience=5)]" 130 | ] 131 | }, 132 | { 133 | "cell_type": "code", 134 | "execution_count": 8, 135 | "metadata": {}, 136 | "outputs": [], 137 | "source": [ 138 | "class_weight_equal = {False : 1., True: 1}\n", 139 | "class_weight_imbalanced = {False : 100, True: 1}" 140 | ] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": 9, 145 | "metadata": {}, 146 | "outputs": [ 147 | { 148 | "data": { 149 | "text/plain": [ 150 | "" 151 | ] 152 | }, 153 | "execution_count": 9, 154 | "metadata": {}, 155 | "output_type": "execute_result" 156 | } 157 | ], 158 | "source": [ 159 | "n_epochs = 1000\n", 160 | "batch_size = 512\n", 161 | "validation_split = 0.01\n", 162 | "\n", 163 | "model.fit(X_train, y_train_binary, epochs=n_epochs, \n", 164 | "batch_size=batch_size, shuffle=True, validation_split=validation_split, class_weight=class_weight_equal,\n", 165 | " callbacks=callbacks, verbose=0\n", 166 | ")" 167 | ] 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": 10, 172 | "metadata": {}, 173 | "outputs": [ 174 | { 175 | "data": { 176 | "text/plain": [ 177 | "array([[1009, 0],\n", 178 | " [ 982, 0]])" 179 | ] 180 | }, 181 | "execution_count": 10, 182 | "metadata": {}, 183 | "output_type": "execute_result" 184 | } 185 | ], 186 | "source": [ 187 | "preds_equal = model.predict(X_test)\n", 188 | "confusion_matrix(y_test_binary, np.round(preds_equal), labels=[True, False])\n", 189 | "\n", 190 | "#array([[1009, 0],\n", 191 | "# [ 982, 0]])" 192 | ] 193 | }, 194 | { 195 | "cell_type": "code", 196 | "execution_count": 11, 197 | "metadata": {}, 198 | "outputs": [ 199 | { 200 | "data": { 201 | "text/plain": [ 202 | "array([[1009, 0],\n", 203 | " [ 508, 474]])" 204 | ] 205 | }, 206 | "execution_count": 11, 207 | "metadata": {}, 208 | "output_type": "execute_result" 209 | } 210 | ], 211 | "source": [ 212 | "model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['binary_accuracy'])\n", 213 | "\n", 214 | "model.fit(X_train, y_train_binary, epochs=n_epochs, \n", 215 | " batch_size=batch_size, shuffle=True, validation_split=validation_split, class_weight=class_weight_imbalanced,\n", 216 | " callbacks=callbacks, verbose=0\n", 217 | " )\n", 218 | "\n", 219 | "preds_imbalanced = model.predict(X_test)\n", 220 | "confusion_matrix(y_test_binary, np.round(preds_imbalanced), labels=[True, False])\n", 221 | "\n", 222 | "#array([[1009, 3],\n", 223 | "# [ 546, 436]])" 224 | ] 225 | } 226 | ], 227 | "metadata": { 228 | "kernelspec": { 229 | "display_name": "Python 3", 230 | "language": "python", 231 | "name": "python3" 232 | }, 233 | "language_info": { 234 | "codemirror_mode": { 235 | "name": "ipython", 236 | "version": 3 237 | }, 238 | "file_extension": ".py", 239 | "mimetype": "text/x-python", 240 | "name": "python", 241 | "nbconvert_exporter": "python", 242 | "pygments_lexer": "ipython3", 243 | "version": "3.5.2" 244 | }, 245 | "widgets": { 246 | "state": {}, 247 | "version": "1.1.2" 248 | } 249 | }, 250 | "nbformat": 4, 251 | "nbformat_minor": 2 252 | } 253 | -------------------------------------------------------------------------------- /Chapter04/Chapter 4 - Adding Long Short-Term Memory (LSTM).ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "name": "stderr", 10 | "output_type": "stream", 11 | "text": [ 12 | "Using TensorFlow backend.\n" 13 | ] 14 | } 15 | ], 16 | "source": [ 17 | "import numpy as np\n", 18 | "\n", 19 | "from keras.preprocessing import sequence\n", 20 | "from keras.models import Sequential\n", 21 | "from keras.layers import Dense, Dropout, Activation\n", 22 | "from keras.layers import Embedding\n", 23 | "from keras.layers import LSTM\n", 24 | "from keras.callbacks import EarlyStopping\n", 25 | "\n", 26 | "from keras.datasets import imdb" 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": 2, 32 | "metadata": {}, 33 | "outputs": [ 34 | { 35 | "name": "stdout", 36 | "output_type": "stream", 37 | "text": [ 38 | "Train seq: 25000\n", 39 | "Test seq: 25000\n" 40 | ] 41 | } 42 | ], 43 | "source": [ 44 | "n_words = 1000\n", 45 | "(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=n_words)\n", 46 | "print('Train seq: {}'.format(len(X_train)))\n", 47 | "print('Test seq: {}'.format(len(X_train)))" 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": 3, 53 | "metadata": {}, 54 | "outputs": [ 55 | { 56 | "name": "stdout", 57 | "output_type": "stream", 58 | "text": [ 59 | "Train example: \n", 60 | "[1, 14, 22, 16, 43, 530, 973, 2, 2, 65, 458, 2, 66, 2, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 2, 2, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2, 19, 14, 22, 4, 2, 2, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 2, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2, 2, 16, 480, 66, 2, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 2, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 2, 15, 256, 4, 2, 7, 2, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 2, 2, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2, 56, 26, 141, 6, 194, 2, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 2, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 2, 88, 12, 16, 283, 5, 16, 2, 113, 103, 32, 15, 16, 2, 19, 178, 32]\n", 61 | "\n", 62 | "Test example: \n", 63 | "[1, 89, 27, 2, 2, 17, 199, 132, 5, 2, 16, 2, 24, 8, 760, 4, 2, 7, 4, 22, 2, 2, 16, 2, 17, 2, 7, 2, 2, 9, 4, 2, 8, 14, 991, 13, 877, 38, 19, 27, 239, 13, 100, 235, 61, 483, 2, 4, 7, 4, 20, 131, 2, 72, 8, 14, 251, 27, 2, 7, 308, 16, 735, 2, 17, 29, 144, 28, 77, 2, 18, 12]\n" 64 | ] 65 | } 66 | ], 67 | "source": [ 68 | "print('Train example: \\n{}'.format(X_train[0]))\n", 69 | "print('\\nTest example: \\n{}'.format(X_test[0]))\n", 70 | "\n", 71 | "# Note: the data is already preprocessed (words are mapped to vectors)" 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": 4, 77 | "metadata": {}, 78 | "outputs": [], 79 | "source": [ 80 | "# Pad sequences with max_len\n", 81 | "max_len = 200\n", 82 | "X_train = sequence.pad_sequences(X_train, maxlen=max_len)\n", 83 | "X_test = sequence.pad_sequences(X_test, maxlen=max_len)" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": 5, 89 | "metadata": {}, 90 | "outputs": [ 91 | { 92 | "name": "stdout", 93 | "output_type": "stream", 94 | "text": [ 95 | "_________________________________________________________________\n", 96 | "Layer (type) Output Shape Param # \n", 97 | "=================================================================\n", 98 | "embedding_1 (Embedding) (None, 200, 50) 50000 \n", 99 | "_________________________________________________________________\n", 100 | "dropout_1 (Dropout) (None, 200, 50) 0 \n", 101 | "_________________________________________________________________\n", 102 | "lstm_1 (LSTM) (None, 100) 60400 \n", 103 | "_________________________________________________________________\n", 104 | "dense_1 (Dense) (None, 250) 25250 \n", 105 | "_________________________________________________________________\n", 106 | "dropout_2 (Dropout) (None, 250) 0 \n", 107 | "_________________________________________________________________\n", 108 | "dense_2 (Dense) (None, 1) 251 \n", 109 | "=================================================================\n", 110 | "Total params: 135,901\n", 111 | "Trainable params: 135,901\n", 112 | "Non-trainable params: 0\n", 113 | "_________________________________________________________________\n" 114 | ] 115 | } 116 | ], 117 | "source": [ 118 | "# Define network architecture and compile\n", 119 | "model = Sequential()\n", 120 | "model.add(Embedding(n_words, 50, input_length=max_len))\n", 121 | "model.add(Dropout(0.2))\n", 122 | "model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2))\n", 123 | "model.add(Dense(250, activation='relu'))\n", 124 | "model.add(Dropout(0.2))\n", 125 | "model.add(Dense(1, activation='sigmoid'))\n", 126 | "\n", 127 | "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n", 128 | "model.summary()" 129 | ] 130 | }, 131 | { 132 | "cell_type": "code", 133 | "execution_count": 6, 134 | "metadata": {}, 135 | "outputs": [], 136 | "source": [ 137 | "callbacks = [EarlyStopping(monitor='val_acc', patience=3)]" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 7, 143 | "metadata": {}, 144 | "outputs": [ 145 | { 146 | "name": "stdout", 147 | "output_type": "stream", 148 | "text": [ 149 | "Train on 20000 samples, validate on 5000 samples\n", 150 | "Epoch 1/100\n", 151 | "20000/20000 [==============================] - 53s - loss: 0.5725 - acc: 0.6879 - val_loss: 0.4544 - val_acc: 0.7848\n", 152 | "Epoch 2/100\n", 153 | "20000/20000 [==============================] - 52s - loss: 0.4408 - acc: 0.8005 - val_loss: 0.4222 - val_acc: 0.8084\n", 154 | "Epoch 3/100\n", 155 | "20000/20000 [==============================] - 52s - loss: 0.4474 - acc: 0.7942 - val_loss: 0.4522 - val_acc: 0.7926\n", 156 | "Epoch 4/100\n", 157 | "20000/20000 [==============================] - 52s - loss: 0.4125 - acc: 0.8211 - val_loss: 0.4134 - val_acc: 0.8144\n", 158 | "Epoch 5/100\n", 159 | "20000/20000 [==============================] - 52s - loss: 0.3986 - acc: 0.8309 - val_loss: 0.4009 - val_acc: 0.8280\n", 160 | "Epoch 6/100\n", 161 | "20000/20000 [==============================] - 51s - loss: 0.4301 - acc: 0.8023 - val_loss: 0.4107 - val_acc: 0.8148\n", 162 | "Epoch 7/100\n", 163 | "20000/20000 [==============================] - 50s - loss: 0.4345 - acc: 0.7989 - val_loss: 0.4431 - val_acc: 0.7896\n", 164 | "Epoch 8/100\n", 165 | "20000/20000 [==============================] - 50s - loss: 0.3933 - acc: 0.8278 - val_loss: 0.4060 - val_acc: 0.8256\n", 166 | "Epoch 9/100\n", 167 | "20000/20000 [==============================] - 50s - loss: 0.3707 - acc: 0.8389 - val_loss: 0.3988 - val_acc: 0.8228\n" 168 | ] 169 | }, 170 | { 171 | "data": { 172 | "text/plain": [ 173 | "" 174 | ] 175 | }, 176 | "execution_count": 7, 177 | "metadata": {}, 178 | "output_type": "execute_result" 179 | } 180 | ], 181 | "source": [ 182 | "batch_size = 128\n", 183 | "n_epochs = 100\n", 184 | "\n", 185 | "model.fit(X_train, y_train, batch_size=batch_size, epochs=n_epochs, validation_split=0.2, callbacks=callbacks)" 186 | ] 187 | }, 188 | { 189 | "cell_type": "code", 190 | "execution_count": 8, 191 | "metadata": {}, 192 | "outputs": [ 193 | { 194 | "name": "stdout", 195 | "output_type": "stream", 196 | "text": [ 197 | "25000/25000 [==============================] - 50s \n", 198 | "Accuracy on test set: 0.82884\n" 199 | ] 200 | } 201 | ], 202 | "source": [ 203 | "print('Accuracy on test set: {}'.format(model.evaluate(X_test, y_test)[1]))\n", 204 | "\n", 205 | "# Accuracy on test set: 0.82884" 206 | ] 207 | } 208 | ], 209 | "metadata": { 210 | "kernelspec": { 211 | "display_name": "Python 3", 212 | "language": "python", 213 | "name": "python3" 214 | }, 215 | "language_info": { 216 | "codemirror_mode": { 217 | "name": "ipython", 218 | "version": 3 219 | }, 220 | "file_extension": ".py", 221 | "mimetype": "text/x-python", 222 | "name": "python", 223 | "nbconvert_exporter": "python", 224 | "pygments_lexer": "ipython3", 225 | "version": "3.5.2" 226 | }, 227 | "widgets": { 228 | "state": {}, 229 | "version": "1.1.2" 230 | } 231 | }, 232 | "nbformat": 4, 233 | "nbformat_minor": 2 234 | } 235 | -------------------------------------------------------------------------------- /Chapter04/Chapter 4 - Implementing a simple RNN.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 2, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "X = []\n", 19 | "X.append([1,0,0,0])\n", 20 | "X.append([0,1,0,0])\n", 21 | "X.append([0,0,1,0])\n", 22 | "X.append([0,0,0,1])\n", 23 | "X.append([0,0,0,1])\n", 24 | "X.append([1,0,0,0])\n", 25 | "X.append([0,1,0,0])\n", 26 | "X.append([0,0,1,0])\n", 27 | "X.append([0,0,0,1])\n", 28 | "\n", 29 | "y = [0.20, 0.30, 0.40, 0.50, 0.05, 0.10, 0.20, 0.30, 0.40]" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": 3, 35 | "metadata": {}, 36 | "outputs": [], 37 | "source": [ 38 | "def sigmoid(x):\n", 39 | " return 1 / (1 + np.exp(-x))\n", 40 | "\n", 41 | "def sigmoid_der(x):\n", 42 | " return 1.0 - x**2" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 4, 48 | "metadata": {}, 49 | "outputs": [], 50 | "source": [ 51 | "layers = []\n", 52 | "# 4 input variables, 10 hidden units and 1 output variable\n", 53 | "n_units = (4, 16, 1)\n", 54 | "n_layers = len(n_units)\n", 55 | "\n", 56 | "layers.append(np.ones(n_units[0]+1+n_units[1]))\n", 57 | "for i in range(1, n_layers):\n", 58 | " layers.append(np.ones(n_units[i]))\n", 59 | "\n", 60 | "weights = []\n", 61 | "for i in range(n_layers-1):\n", 62 | " weights.append(np.zeros((layers[i].size, layers[i+1].size)))\n", 63 | "\n", 64 | "weights_delta = [0,]*len(weights)" 65 | ] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": 5, 70 | "metadata": {}, 71 | "outputs": [], 72 | "source": [ 73 | "def forwards(data):\n", 74 | " layers[0][:n_units[0]] = data\n", 75 | " layers[0][n_units[0]:-1] = layers[1]\n", 76 | "\n", 77 | " # Propagate the data forwards\n", 78 | " for i in range(1, n_layers):\n", 79 | " layers[i][...] = sigmoid(np.dot(layers[i-1], weights[i-1]))\n", 80 | "\n", 81 | " return layers[-1]" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": 6, 87 | "metadata": {}, 88 | "outputs": [], 89 | "source": [ 90 | "def backwards(target, learning_rate=0.1, momentum=0.1):\n", 91 | " deltas = []\n", 92 | " error = target - layers[-1]\n", 93 | " delta = error * sigmoid_der(layers[-1])\n", 94 | " deltas.append(delta)\n", 95 | "\n", 96 | " # Determine error in hidden layers\n", 97 | " for i in range(n_layers-2, 0, -1):\n", 98 | " delta = np.dot(deltas[0], weights[i].T) * sigmoid_der(layers[i])\n", 99 | " deltas.insert(0, delta)\n", 100 | "\n", 101 | " # Update weights\n", 102 | " for i in range(len(weights)):\n", 103 | " layer = np.atleast_2d(layers[i])\n", 104 | " delta = np.atleast_2d(deltas[i])\n", 105 | " weights_delta_temp = np.dot(layer.T, delta)\n", 106 | " weights[i] += learning_rate*weights_delta_temp + momentum*weights_delta[i]\n", 107 | " weights_delta[i] = weights_delta_temp\n", 108 | "\n", 109 | " return (error**2).sum()" 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": 7, 115 | "metadata": {}, 116 | "outputs": [ 117 | { 118 | "name": "stdout", 119 | "output_type": "stream", 120 | "text": [ 121 | "epoch 0 - loss: 0.3116\n", 122 | "epoch 1000 - loss: 0.1660\n", 123 | "epoch 2000 - loss: 0.1801\n", 124 | "epoch 3000 - loss: 0.1877\n", 125 | "epoch 4000 - loss: 0.1914\n", 126 | "epoch 5000 - loss: 0.1922\n", 127 | "epoch 6000 - loss: 0.1920\n", 128 | "epoch 7000 - loss: 0.1916\n", 129 | "epoch 8000 - loss: 0.1913\n", 130 | "epoch 9000 - loss: 0.1912\n" 131 | ] 132 | } 133 | ], 134 | "source": [ 135 | "n_epochs = 10000\n", 136 | "\n", 137 | "for i in range(n_epochs):\n", 138 | " loss = 0\n", 139 | " for j in range(len(X)):\n", 140 | " forwards(X[j])\n", 141 | " backwards(y[j])\n", 142 | " loss += (y[j]-forwards(X[j]))**2\n", 143 | " if i%1000 == 0: print('epoch {} - loss: {:04.4f}'.format(i, loss[0]))" 144 | ] 145 | }, 146 | { 147 | "cell_type": "code", 148 | "execution_count": 8, 149 | "metadata": {}, 150 | "outputs": [ 151 | { 152 | "name": "stdout", 153 | "output_type": "stream", 154 | "text": [ 155 | "X: [1, 0, 0, 0]; y: 0.20; pred: 0.15\n", 156 | "X: [0, 1, 0, 0]; y: 0.30; pred: 0.39\n", 157 | "X: [0, 0, 1, 0]; y: 0.40; pred: 0.29\n", 158 | "X: [0, 0, 0, 1]; y: 0.50; pred: 0.34\n", 159 | "X: [0, 0, 0, 1]; y: 0.05; pred: 0.30\n", 160 | "X: [1, 0, 0, 0]; y: 0.10; pred: 0.16\n", 161 | "X: [0, 1, 0, 0]; y: 0.20; pred: 0.38\n", 162 | "X: [0, 0, 1, 0]; y: 0.30; pred: 0.30\n", 163 | "X: [0, 0, 0, 1]; y: 0.40; pred: 0.33\n" 164 | ] 165 | } 166 | ], 167 | "source": [ 168 | "for i in range(len(X)):\n", 169 | " pred = forwards(X[i])\n", 170 | " loss = (y[i]-pred)**2\n", 171 | " print('X: {}; y: {:04.2f}; pred: {:04.2f}'.format(X[i], y[i], pred[0]))" 172 | ] 173 | } 174 | ], 175 | "metadata": { 176 | "kernelspec": { 177 | "display_name": "Python 3", 178 | "language": "python", 179 | "name": "python3" 180 | }, 181 | "language_info": { 182 | "codemirror_mode": { 183 | "name": "ipython", 184 | "version": 3 185 | }, 186 | "file_extension": ".py", 187 | "mimetype": "text/x-python", 188 | "name": "python", 189 | "nbconvert_exporter": "python", 190 | "pygments_lexer": "ipython3", 191 | "version": "3.5.2" 192 | }, 193 | "widgets": { 194 | "state": {}, 195 | "version": "1.1.2" 196 | } 197 | }, 198 | "nbformat": 4, 199 | "nbformat_minor": 2 200 | } 201 | -------------------------------------------------------------------------------- /Chapter04/Chapter 4 - Using gated recurrent units (GRUs).ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "name": "stderr", 10 | "output_type": "stream", 11 | "text": [ 12 | "Using TensorFlow backend.\n" 13 | ] 14 | } 15 | ], 16 | "source": [ 17 | "import numpy as np\n", 18 | "import pandas as pd\n", 19 | "\n", 20 | "from keras.preprocessing import sequence\n", 21 | "from keras.models import Sequential\n", 22 | "from keras.layers import Dense, Dropout, Activation\n", 23 | "from keras.layers import Embedding\n", 24 | "from keras.layers import GRU\n", 25 | "from keras.callbacks import EarlyStopping\n", 26 | "\n", 27 | "from keras.datasets import imdb\n" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": 2, 33 | "metadata": {}, 34 | "outputs": [ 35 | { 36 | "name": "stdout", 37 | "output_type": "stream", 38 | "text": [ 39 | "Train seq: 25000\n", 40 | "Test seq: 25000\n" 41 | ] 42 | } 43 | ], 44 | "source": [ 45 | "n_words = 1000\n", 46 | "(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=n_words)\n", 47 | "print('Train seq: {}'.format(len(X_train)))\n", 48 | "print('Test seq: {}'.format(len(X_train)))" 49 | ] 50 | }, 51 | { 52 | "cell_type": "code", 53 | "execution_count": 3, 54 | "metadata": {}, 55 | "outputs": [], 56 | "source": [ 57 | "# Pad sequences with max_len\n", 58 | "max_len = 200\n", 59 | "X_train = sequence.pad_sequences(X_train, maxlen=max_len)\n", 60 | "X_test = sequence.pad_sequences(X_test, maxlen=max_len)" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": 4, 66 | "metadata": {}, 67 | "outputs": [ 68 | { 69 | "name": "stdout", 70 | "output_type": "stream", 71 | "text": [ 72 | "_________________________________________________________________\n", 73 | "Layer (type) Output Shape Param # \n", 74 | "=================================================================\n", 75 | "embedding_1 (Embedding) (None, 200, 50) 50000 \n", 76 | "_________________________________________________________________\n", 77 | "dropout_1 (Dropout) (None, 200, 50) 0 \n", 78 | "_________________________________________________________________\n", 79 | "gru_1 (GRU) (None, 100) 45300 \n", 80 | "_________________________________________________________________\n", 81 | "dense_1 (Dense) (None, 250) 25250 \n", 82 | "_________________________________________________________________\n", 83 | "dropout_2 (Dropout) (None, 250) 0 \n", 84 | "_________________________________________________________________\n", 85 | "dense_2 (Dense) (None, 1) 251 \n", 86 | "=================================================================\n", 87 | "Total params: 120,801\n", 88 | "Trainable params: 120,801\n", 89 | "Non-trainable params: 0\n", 90 | "_________________________________________________________________\n" 91 | ] 92 | } 93 | ], 94 | "source": [ 95 | "# Define network architecture and compile\n", 96 | "model = Sequential()\n", 97 | "model.add(Embedding(n_words, 50, input_length=max_len))\n", 98 | "model.add(Dropout(0.2))\n", 99 | "model.add(GRU(100, dropout=0.2, recurrent_dropout=0.2))\n", 100 | "model.add(Dense(250, activation='relu'))\n", 101 | "model.add(Dropout(0.2))\n", 102 | "model.add(Dense(1, activation='sigmoid'))\n", 103 | "\n", 104 | "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n", 105 | "model.summary()" 106 | ] 107 | }, 108 | { 109 | "cell_type": "code", 110 | "execution_count": 5, 111 | "metadata": {}, 112 | "outputs": [], 113 | "source": [ 114 | "callbacks = [EarlyStopping(monitor='val_acc', patience=3)]\n" 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": 6, 120 | "metadata": {}, 121 | "outputs": [ 122 | { 123 | "name": "stdout", 124 | "output_type": "stream", 125 | "text": [ 126 | "Train on 20000 samples, validate on 5000 samples\n", 127 | "Epoch 1/100\n", 128 | "20000/20000 [==============================] - 13s - loss: 0.6693 - acc: 0.5890 - val_loss: 0.5819 - val_acc: 0.6908\n", 129 | "Epoch 2/100\n", 130 | "20000/20000 [==============================] - 12s - loss: 0.5243 - acc: 0.7364 - val_loss: 0.4923 - val_acc: 0.7560\n", 131 | "Epoch 3/100\n", 132 | "20000/20000 [==============================] - 12s - loss: 0.4716 - acc: 0.7816 - val_loss: 0.5005 - val_acc: 0.7526\n", 133 | "Epoch 4/100\n", 134 | "20000/20000 [==============================] - 12s - loss: 0.4516 - acc: 0.7972 - val_loss: 0.4436 - val_acc: 0.7908\n", 135 | "Epoch 5/100\n", 136 | "20000/20000 [==============================] - 12s - loss: 0.4340 - acc: 0.8066 - val_loss: 0.4718 - val_acc: 0.7848\n", 137 | "Epoch 6/100\n", 138 | "20000/20000 [==============================] - 12s - loss: 0.4191 - acc: 0.8184 - val_loss: 0.4514 - val_acc: 0.7876\n", 139 | "Epoch 7/100\n", 140 | "20000/20000 [==============================] - 12s - loss: 0.4242 - acc: 0.8132 - val_loss: 0.4123 - val_acc: 0.8198\n", 141 | "Epoch 8/100\n", 142 | "20000/20000 [==============================] - 12s - loss: 0.3997 - acc: 0.8256 - val_loss: 0.3968 - val_acc: 0.8206\n", 143 | "Epoch 9/100\n", 144 | "20000/20000 [==============================] - 12s - loss: 0.3918 - acc: 0.8325 - val_loss: 0.4049 - val_acc: 0.8188\n", 145 | "Epoch 10/100\n", 146 | "20000/20000 [==============================] - 12s - loss: 0.3745 - acc: 0.8388 - val_loss: 0.3926 - val_acc: 0.8334\n", 147 | "Epoch 11/100\n", 148 | "20000/20000 [==============================] - 12s - loss: 0.3967 - acc: 0.8256 - val_loss: 0.3952 - val_acc: 0.8348\n", 149 | "Epoch 12/100\n", 150 | "20000/20000 [==============================] - 12s - loss: 0.3601 - acc: 0.8469 - val_loss: 0.3854 - val_acc: 0.8302\n", 151 | "Epoch 13/100\n", 152 | "20000/20000 [==============================] - 12s - loss: 0.3500 - acc: 0.8504 - val_loss: 0.3801 - val_acc: 0.8356\n", 153 | "Epoch 14/100\n", 154 | "20000/20000 [==============================] - 12s - loss: 0.3506 - acc: 0.8500 - val_loss: 0.3825 - val_acc: 0.8386\n", 155 | "Epoch 15/100\n", 156 | "20000/20000 [==============================] - 12s - loss: 0.3415 - acc: 0.8555 - val_loss: 0.3783 - val_acc: 0.8352\n", 157 | "Epoch 16/100\n", 158 | "20000/20000 [==============================] - 12s - loss: 0.3299 - acc: 0.8601 - val_loss: 0.3771 - val_acc: 0.8296\n", 159 | "Epoch 17/100\n", 160 | "20000/20000 [==============================] - 12s - loss: 0.3322 - acc: 0.8590 - val_loss: 0.3974 - val_acc: 0.8348\n", 161 | "Epoch 18/100\n", 162 | "20000/20000 [==============================] - 12s - loss: 0.3788 - acc: 0.8356 - val_loss: 0.3966 - val_acc: 0.8226\n" 163 | ] 164 | }, 165 | { 166 | "data": { 167 | "text/plain": [ 168 | "" 169 | ] 170 | }, 171 | "execution_count": 6, 172 | "metadata": {}, 173 | "output_type": "execute_result" 174 | } 175 | ], 176 | "source": [ 177 | "batch_size = 512\n", 178 | "n_epochs = 100\n", 179 | "\n", 180 | "model.fit(X_train, y_train, batch_size=batch_size, epochs=n_epochs, validation_split=0.2, callbacks=callbacks)" 181 | ] 182 | }, 183 | { 184 | "cell_type": "code", 185 | "execution_count": 8, 186 | "metadata": {}, 187 | "outputs": [ 188 | { 189 | "name": "stdout", 190 | "output_type": "stream", 191 | "text": [ 192 | "25000/25000 [==============================] - 40s \n", 193 | "\n", 194 | "Accuracy on test set: 0.83004\n" 195 | ] 196 | } 197 | ], 198 | "source": [ 199 | "print('Accuracy on test set: {}'.format(model.evaluate(X_test, y_test)[1]))\n", 200 | "\n", 201 | "# Accuracy on test set: 0.83004" 202 | ] 203 | } 204 | ], 205 | "metadata": { 206 | "kernelspec": { 207 | "display_name": "Python 3", 208 | "language": "python", 209 | "name": "python3" 210 | }, 211 | "language_info": { 212 | "codemirror_mode": { 213 | "name": "ipython", 214 | "version": 3 215 | }, 216 | "file_extension": ".py", 217 | "mimetype": "text/x-python", 218 | "name": "python", 219 | "nbconvert_exporter": "python", 220 | "pygments_lexer": "ipython3", 221 | "version": "3.5.2" 222 | }, 223 | "widgets": { 224 | "state": {}, 225 | "version": "1.1.2" 226 | } 227 | }, 228 | "nbformat": 4, 229 | "nbformat_minor": 2 230 | } 231 | -------------------------------------------------------------------------------- /Chapter06/Chapter 6 - Understanding GANs.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np\n", 10 | "from keras.models import Sequential, Model\n", 11 | "from keras.layers import Input, Dense, Activation, Flatten, Reshape\n", 12 | "from keras.layers import Conv2D, Conv2DTranspose, UpSampling2D\n", 13 | "from keras.layers import LeakyReLU, Dropout\n", 14 | "from keras.layers import BatchNormalization\n", 15 | "from keras.optimizers import Adam\n", 16 | "from keras import initializers\n", 17 | "\n", 18 | "from keras.datasets import mnist\n", 19 | "\n", 20 | "import matplotlib.pyplot as plt" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": null, 26 | "metadata": {}, 27 | "outputs": [], 28 | "source": [ 29 | "(X_train, y_train), (X_test, y_test) = mnist.load_data()\n", 30 | "\n", 31 | "img_rows, img_cols = X_train.shape[1:]\n", 32 | "\n", 33 | "X_train = (X_train.reshape(-1, img_rows*img_cols, 1).astype(np.float32)-127.5)/127.5" 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": null, 39 | "metadata": {}, 40 | "outputs": [], 41 | "source": [ 42 | "def discriminator_model(dropout=0.5):\n", 43 | " model = Sequential()\n", 44 | " model.add(Dense(1024, input_dim=784, kernel_initializer=initializers.RandomNormal(stddev=0.02)))\n", 45 | " model.add(LeakyReLU(0.2))\n", 46 | " model.add(Dropout(dropout))\n", 47 | " model.add(Dense(512))\n", 48 | " model.add(LeakyReLU(0.2))\n", 49 | " model.add(Dropout(dropout))\n", 50 | " model.add(Dense(256))\n", 51 | " model.add(LeakyReLU(0.2))\n", 52 | " model.add(Dropout(dropout))\n", 53 | " model.add(Dense(1, activation='sigmoid'))\n", 54 | " opt = Adam(lr=0.0001)\n", 55 | " model.compile(loss='binary_crossentropy', optimizer=opt)\n", 56 | " return model" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": null, 62 | "metadata": {}, 63 | "outputs": [], 64 | "source": [ 65 | "def generator_model():\n", 66 | " model = Sequential()\n", 67 | " model.add(Dense(256, input_dim=100, kernel_initializer=initializers.RandomNormal(stddev=0.02)))\n", 68 | " model.add(LeakyReLU(0.2))\n", 69 | " model.add(Dense(512))\n", 70 | " model.add(LeakyReLU(0.2))\n", 71 | " model.add(Dense(1024))\n", 72 | " model.add(LeakyReLU(0.2))\n", 73 | " model.add(Dense(784, activation='tanh'))\n", 74 | " opt = Adam(lr=0.00005)\n", 75 | " model.compile(loss='binary_crossentropy', optimizer=opt)\n", 76 | " return model" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": null, 82 | "metadata": {}, 83 | "outputs": [], 84 | "source": [ 85 | "discriminator = discriminator_model()\n", 86 | "generator = generator_model()\n", 87 | "discriminator.trainable = False\n", 88 | "gan_input = Input(shape=(100,))\n", 89 | "x = generator(gan_input)\n", 90 | "gan_output = discriminator(x)\n", 91 | "gan = Model(inputs=gan_input, outputs=gan_output)\n", 92 | "opt = Adam(lr=0.0001)\n", 93 | "gan.compile(loss='binary_crossentropy', optimizer=opt)" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": null, 99 | "metadata": {}, 100 | "outputs": [], 101 | "source": [ 102 | "def plot_images(samples=16, step=0):\n", 103 | " plt.figure(figsize=(5,5))\n", 104 | " for i in range(samples):\n", 105 | " noise = np.random.uniform(-1, 1, size=[batch_size, 100])#np.random.normal(0, 1, size=[batch_size, 100])\n", 106 | " images = generator.predict(noise)\n", 107 | " plt.subplot(4, 4, i+1)\n", 108 | " image = images[i, :,]\n", 109 | " image = np.reshape(image, [img_rows, img_cols])\n", 110 | " plt.imshow(image, cmap='gray')\n", 111 | " plt.axis('off')\n", 112 | " plt.show()" 113 | ] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": null, 118 | "metadata": {}, 119 | "outputs": [], 120 | "source": [ 121 | "batch_size = 32\n", 122 | "n_steps = 100000\n", 123 | "plot_every = 1000\n", 124 | "\n", 125 | "noise_input = np.random.uniform(-1, 1, size=[16, 100])\n", 126 | "for step in range(n_steps):\n", 127 | " \n", 128 | " noise = np.random.uniform(-1, 1, size=[batch_size, 100])\n", 129 | " batch = X_train[np.random.randint(0, X_train.shape[0], size=batch_size)].reshape(batch_size, 784)\n", 130 | "\n", 131 | " gen_output = generator.predict(noise)\n", 132 | " X = np.concatenate([batch, gen_output])\n", 133 | "\n", 134 | " y_D = np.zeros(2*batch_size)\n", 135 | " y_D[:batch_size] = 1\n", 136 | "\n", 137 | " discriminator.trainable = True\n", 138 | " loss_D = discriminator.train_on_batch(X, y_D)\n", 139 | "\n", 140 | " noise = np.random.uniform(-1, 1, size=[batch_size, 100])\n", 141 | " y_G = np.ones(batch_size)\n", 142 | " discriminator.trainable = False\n", 143 | " loss_G = gan.train_on_batch(noise, y_G)\n", 144 | " \n", 145 | " if step % plot_every == 0:\n", 146 | " print('Step {}'.format(step))\n", 147 | " plot_images(samples=noise_input.shape[0], step=(step+1))\n", 148 | " " 149 | ] 150 | } 151 | ], 152 | "metadata": { 153 | "kernelspec": { 154 | "display_name": "Python 3", 155 | "language": "python", 156 | "name": "python3" 157 | }, 158 | "language_info": { 159 | "codemirror_mode": { 160 | "name": "ipython", 161 | "version": 3 162 | }, 163 | "file_extension": ".py", 164 | "mimetype": "text/x-python", 165 | "name": "python", 166 | "nbconvert_exporter": "python", 167 | "pygments_lexer": "ipython3", 168 | "version": "3.5.2" 169 | }, 170 | "widgets": { 171 | "state": {}, 172 | "version": "1.1.2" 173 | } 174 | }, 175 | "nbformat": 4, 176 | "nbformat_minor": 2 177 | } 178 | -------------------------------------------------------------------------------- /Chapter07/Chapter 7 - Augmenting images with computer vision techniques.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np\n", 10 | "import cv2\n", 11 | "import matplotlib.pyplot as plt\n", 12 | "import glob" 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": null, 18 | "metadata": {}, 19 | "outputs": [], 20 | "source": [ 21 | "DATA_DIR = 'Data/augmentation/'\n", 22 | "images = glob.glob(DATA_DIR + '*')\n", 23 | "\n", 24 | "plt.figure(figsize=(10, 10))\n", 25 | "i = 1\n", 26 | "for image in images:\n", 27 | " img = cv2.imread(image)\n", 28 | " img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n", 29 | " plt.subplot(3, 3, i)\n", 30 | " plt.imshow(img)\n", 31 | " i += 1\n", 32 | " \n", 33 | "plt.show()" 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": null, 39 | "metadata": {}, 40 | "outputs": [], 41 | "source": [ 42 | "def plot_images(image, function, *args):\n", 43 | " plt.figure(figsize=(10, 10))\n", 44 | " n_examples = 3\n", 45 | " for i in range(n_examples):\n", 46 | " img = cv2.imread(image)\n", 47 | " img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n", 48 | " img = function(img, *args)\n", 49 | " plt.subplot(3, 3, i+1)\n", 50 | " plt.imshow(img)\n", 51 | " plt.show()" 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": null, 57 | "metadata": {}, 58 | "outputs": [], 59 | "source": [ 60 | "def rotate_image(image, rotate=20):\n", 61 | " width, height, _ = image.shape\n", 62 | " random_rotation = np.random.uniform(low=-rotate, high=rotate)\n", 63 | " M = cv2.getRotationMatrix2D((width/2, height/2), random_rotation, 1)\n", 64 | " return(cv2.warpAffine(image, M, (width, height)))\n", 65 | "\n", 66 | "plot_images(images[2], rotate_image, 40)" 67 | ] 68 | }, 69 | { 70 | "cell_type": "code", 71 | "execution_count": null, 72 | "metadata": {}, 73 | "outputs": [], 74 | "source": [ 75 | " def adjust_brightness(image, brightness=60):\n", 76 | " rand_brightness = np.random.uniform(low=-brightness, high=brightness)\n", 77 | " return(cv2.add(image, rand_brightness))\n", 78 | "\n", 79 | "plot_images(images[0], adjust_brightness, 85)" 80 | ] 81 | }, 82 | { 83 | "cell_type": "code", 84 | "execution_count": null, 85 | "metadata": {}, 86 | "outputs": [], 87 | "source": [ 88 | "def random_shifts(image, shift_max_x=100, shift_max_y=100):\n", 89 | " width, height, _ = image.shape\n", 90 | " shift_x = np.random.randint(shift_max_x)\n", 91 | " shift_y = np.random.randint(shift_max_y)\n", 92 | " M = np.float32([[1, 0, shift_x],[0, 1, shift_y]])\n", 93 | " return (cv2.warpAffine(image, M, (height, width)))\n", 94 | "\n", 95 | "plot_images(images[1], random_shifts, 100, 20)" 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": null, 101 | "metadata": {}, 102 | "outputs": [], 103 | "source": [ 104 | "def scale_image(image, scale_range=[0.6, 1.4]):\n", 105 | " width, height, _ = image.shape \n", 106 | " \n", 107 | " scale_x = np.random.uniform(low=scale_range[0], high=scale_range[1])\n", 108 | " scale_y = np.random.uniform(low=scale_range[0], high=scale_range[1])\n", 109 | " scale_matrix = np.array([[scale_x, 0., (1. - scale_x) * width / 2.], \n", 110 | " [0., scale_y, (1. - scale_y) * height / 2.]], \n", 111 | " dtype=np.float32)\n", 112 | " return(cv2.warpAffine(image, scale_matrix, (width, height), flags=cv2.INTER_LINEAR, \n", 113 | " borderMode=cv2.BORDER_REFLECT_101))\n", 114 | "\n", 115 | "plot_images(images[2], scale_image, [0.7, 1.3])" 116 | ] 117 | }, 118 | { 119 | "cell_type": "code", 120 | "execution_count": null, 121 | "metadata": {}, 122 | "outputs": [], 123 | "source": [ 124 | "def random_flip(image, p_flip=0.5):\n", 125 | " rand = np.random.random()\n", 126 | " if rand < p_flip:\n", 127 | " image = cv2.flip(image, 1)\n", 128 | " return image\n", 129 | "\n", 130 | "plot_images(images[2], random_flip)" 131 | ] 132 | }, 133 | { 134 | "cell_type": "code", 135 | "execution_count": null, 136 | "metadata": {}, 137 | "outputs": [], 138 | "source": [ 139 | "plt.figure(figsize=(15, 15))\n", 140 | "image = images[1]\n", 141 | "for i in range(32):\n", 142 | " img = cv2.imread(image)\n", 143 | " img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n", 144 | " img = transform_image(img)\n", 145 | " plt.subplot(8, 8, i+1)\n", 146 | " plt.axis('off')\n", 147 | " plt.imshow(img, interpolation=\"nearest\")\n", 148 | "plt.show()" 149 | ] 150 | } 151 | ], 152 | "metadata": { 153 | "kernelspec": { 154 | "display_name": "Python 3", 155 | "language": "python", 156 | "name": "python3" 157 | }, 158 | "language_info": { 159 | "codemirror_mode": { 160 | "name": "ipython", 161 | "version": 3 162 | }, 163 | "file_extension": ".py", 164 | "mimetype": "text/x-python", 165 | "name": "python", 166 | "nbconvert_exporter": "python", 167 | "pygments_lexer": "ipython3", 168 | "version": "3.5.2" 169 | }, 170 | "widgets": { 171 | "state": {}, 172 | "version": "1.1.2" 173 | } 174 | }, 175 | "nbformat": 4, 176 | "nbformat_minor": 2 177 | } 178 | -------------------------------------------------------------------------------- /Chapter07/Chapter 7 - Classifying objects in images.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np\n", 10 | "import glob\n", 11 | "import cv2\n", 12 | "import matplotlib.pyplot as plt\n", 13 | "\n", 14 | "from sklearn.preprocessing import LabelBinarizer\n", 15 | "from sklearn.model_selection import train_test_split\n", 16 | "from sklearn.metrics import accuracy_score\n", 17 | "\n", 18 | "import keras\n", 19 | "from keras.models import Sequential, load_model\n", 20 | "from keras.layers import Dense, Dropout, Activation, Flatten, Conv2D, MaxPooling2D, Lambda, Cropping2D\n", 21 | "from keras.utils import np_utils\n", 22 | "from keras import optimizers\n", 23 | "from keras.callbacks import EarlyStopping\n", 24 | "\n", 25 | "SEED = 2017" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": null, 31 | "metadata": {}, 32 | "outputs": [], 33 | "source": [ 34 | "# Data can be downloaded at http://download.tensorflow.org/example_images/flower_photos.tgz" 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": null, 40 | "metadata": {}, 41 | "outputs": [], 42 | "source": [ 43 | "# Specify data directory and extract all file names\n", 44 | "DATA_DIR = 'Data/'\n", 45 | "images = glob.glob(DATA_DIR + \"flower_photos/*/*.jpg\")\n", 46 | "# Extract labels from file names\n", 47 | "labels = [x.split('/')[3] for x in images]" 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": null, 53 | "metadata": {}, 54 | "outputs": [], 55 | "source": [ 56 | "unique_labels = set(labels)\n", 57 | "plt.figure(figsize=(15, 15))\n", 58 | "i = 1\n", 59 | "for label in unique_labels:\n", 60 | " image = images[labels.index(label)]\n", 61 | " img = cv2.imread(image)\n", 62 | " img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n", 63 | " plt.subplot(5, 5, i)\n", 64 | " plt.title(\"{0} ({1})\".format(label, labels.count(label)))\n", 65 | " i += 1\n", 66 | " _ = plt.imshow(img)\n", 67 | "plt.show()" 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": null, 73 | "metadata": {}, 74 | "outputs": [], 75 | "source": [ 76 | "encoder = LabelBinarizer()\n", 77 | "encoder.fit(labels)\n", 78 | "y = encoder.transform(labels).astype(float)" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": null, 84 | "metadata": {}, 85 | "outputs": [], 86 | "source": [ 87 | "X_train, X_val, y_train , y_val = train_test_split(images, y, test_size=0.2, random_state=SEED)" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": null, 93 | "metadata": {}, 94 | "outputs": [], 95 | "source": [ 96 | "# Define architecture\n", 97 | "model = Sequential()\n", 98 | "model.add(Lambda(lambda x: (x / 255.) - 0.5, input_shape=(100, 100, 3)))\n", 99 | "model.add(Conv2D(16, (5, 5), activation='relu', padding='same'))\n", 100 | "model.add(MaxPooling2D(pool_size=(2,2)))\n", 101 | "model.add(Dropout(0.75))\n", 102 | "model.add(Conv2D(32, (5, 5), activation='relu', padding='same'))\n", 103 | "model.add(MaxPooling2D(pool_size=(2,2)))\n", 104 | "model.add(Dropout(0.75))\n", 105 | "model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))\n", 106 | "model.add(MaxPooling2D(pool_size=(2,2)))\n", 107 | "model.add(Dropout(0.75))\n", 108 | "model.add(Flatten())\n", 109 | "model.add(Dense(256, activation='relu'))\n", 110 | "model.add(Dropout(0.75))\n", 111 | "model.add(Dense(5, activation='softmax'))\n", 112 | "\n", 113 | "# Define optimizer and compile\n", 114 | "opt = optimizers.Adam()\n", 115 | "model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])" 116 | ] 117 | }, 118 | { 119 | "cell_type": "code", 120 | "execution_count": null, 121 | "metadata": {}, 122 | "outputs": [], 123 | "source": [ 124 | "img_rows = img_cols = 100\n", 125 | "img_channels = 3\n", 126 | "\n", 127 | "def batchgen(x, y, batch_size, transform=False):\n", 128 | " # Create empty numpy arrays\n", 129 | " images = np.zeros((batch_size, img_rows, img_cols, img_channels))\n", 130 | " class_id = np.zeros((batch_size, len(y[0])))\n", 131 | "\n", 132 | " while 1:\n", 133 | " for n in range(batch_size):\n", 134 | " i = np.random.randint(len(x))\n", 135 | " x_ = cv2.imread(x[i])\n", 136 | " x_ = cv2.cvtColor(x_, cv2.COLOR_BGR2RGB)\n", 137 | " # The images have different sizes, we transform all to 100x100 pixels\n", 138 | " x_ = cv2.resize(x_, (100, 100)) \n", 139 | " images[n] = x_\n", 140 | " class_id[n] = y[i]\n", 141 | " yield images, class_id" 142 | ] 143 | }, 144 | { 145 | "cell_type": "code", 146 | "execution_count": null, 147 | "metadata": {}, 148 | "outputs": [], 149 | "source": [ 150 | "callbacks = [EarlyStopping(monitor='val_acc', patience=5)]" 151 | ] 152 | }, 153 | { 154 | "cell_type": "code", 155 | "execution_count": null, 156 | "metadata": {}, 157 | "outputs": [], 158 | "source": [ 159 | "len(X_val)" 160 | ] 161 | }, 162 | { 163 | "cell_type": "code", 164 | "execution_count": null, 165 | "metadata": {}, 166 | "outputs": [], 167 | "source": [ 168 | "batch_size = 256\n", 169 | "n_epochs = 100\n", 170 | "steps_per_epoch = len(X_train) // batch_size\n", 171 | "val_steps = len(X_val) // batch_size\n", 172 | "\n", 173 | "train_generator = batchgen(X_train, y_train, batch_size, True)\n", 174 | "val_generator = batchgen(X_val, y_val, batch_size, True)\n", 175 | "\n", 176 | "history = model.fit_generator(train_generator, \n", 177 | " steps_per_epoch=steps_per_epoch, \n", 178 | " epochs=n_epochs, \n", 179 | " validation_data=val_generator,\n", 180 | " validation_steps=val_steps,\n", 181 | " callbacks=callbacks\n", 182 | " )" 183 | ] 184 | }, 185 | { 186 | "cell_type": "code", 187 | "execution_count": null, 188 | "metadata": {}, 189 | "outputs": [], 190 | "source": [ 191 | "plt.plot(history.history['acc'])\n", 192 | "plt.plot(history.history['val_acc'])\n", 193 | "plt.title('Model accuracy')\n", 194 | "plt.ylabel('accuracy')\n", 195 | "plt.xlabel('epochs')\n", 196 | "plt.legend(['train', 'validation'], loc='lower right')\n", 197 | "plt.show()" 198 | ] 199 | }, 200 | { 201 | "cell_type": "code", 202 | "execution_count": null, 203 | "metadata": {}, 204 | "outputs": [], 205 | "source": [ 206 | "test_generator = batchgen(X_val, y_val, 1, False)\n", 207 | "preds = model.predict_generator(test_generator, steps=len(X_val))\n", 208 | "\n", 209 | "y_val_ = [np.argmax(x) for x in y_val]\n", 210 | "y_preds = [np.argmax(x) for x in preds]\n", 211 | "accuracy_score(y_val_, y_preds)" 212 | ] 213 | }, 214 | { 215 | "cell_type": "code", 216 | "execution_count": null, 217 | "metadata": {}, 218 | "outputs": [], 219 | "source": [ 220 | "n_predictions = 5\n", 221 | "plt.figure(figsize=(15, 15))\n", 222 | "for i in range(n_predictions):\n", 223 | " \n", 224 | " plt.subplot(n_predictions, n_predictions, i+1)\n", 225 | " plt.title(\"{0} ({1})\".format(list(set(labels))[np.argmax(preds[i])], \n", 226 | " list(set(labels))[np.argmax(y_val[i])]))\n", 227 | " img = cv2.imread(X_val[i])\n", 228 | " img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n", 229 | " plt.axis('off')\n", 230 | " plt.imshow(img)\n", 231 | " plt.tight_layout() \n", 232 | "plt.show()" 233 | ] 234 | }, 235 | { 236 | "cell_type": "code", 237 | "execution_count": null, 238 | "metadata": {}, 239 | "outputs": [], 240 | "source": [] 241 | } 242 | ], 243 | "metadata": { 244 | "kernelspec": { 245 | "display_name": "Python 3", 246 | "language": "python", 247 | "name": "python3" 248 | }, 249 | "language_info": { 250 | "codemirror_mode": { 251 | "name": "ipython", 252 | "version": 3 253 | }, 254 | "file_extension": ".py", 255 | "mimetype": "text/x-python", 256 | "name": "python", 257 | "nbconvert_exporter": "python", 258 | "pygments_lexer": "ipython3", 259 | "version": "3.5.2" 260 | }, 261 | "widgets": { 262 | "state": {}, 263 | "version": "1.1.2" 264 | } 265 | }, 266 | "nbformat": 4, 267 | "nbformat_minor": 2 268 | } 269 | -------------------------------------------------------------------------------- /Chapter07/Chapter 7 - Localizing an object in images.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np\n", 10 | "import pandas as pd\n", 11 | "import glob\n", 12 | "import cv2\n", 13 | "import matplotlib.pyplot as plt\n", 14 | "\n", 15 | "from sklearn.preprocessing import LabelBinarizer\n", 16 | "from sklearn.model_selection import train_test_split\n", 17 | "from sklearn.metrics import accuracy_score\n", 18 | "\n", 19 | "from keras.models import Sequential, load_model\n", 20 | "from keras.layers import Dense, Dropout, Activation, Flatten, Conv2D, MaxPooling2D, Lambda, Cropping2D\n", 21 | "from keras.utils import np_utils\n", 22 | "from keras import backend as K\n", 23 | "from keras.callbacks import EarlyStopping\n", 24 | "\n", 25 | "from keras import optimizers" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": null, 31 | "metadata": {}, 32 | "outputs": [], 33 | "source": [ 34 | "DATA_DIR = '../Data/object-detection-crowdai/'\n", 35 | "labels = pd.read_csv(DATA_DIR + 'labels.csv', usecols=[0,1,2,3,4,5])\n", 36 | "# We will only localize Trucks\n", 37 | "labels = labels[labels.Label == 'Truck']\n", 38 | "# We only use images were one Truck is annotated\n", 39 | "labels = labels[~labels.Frame.isin(labels.Frame[labels.Frame.duplicated()].values)]\n", 40 | "labels.columns=['xmin', 'ymin', 'xmax', 'ymax', 'Frame', 'Label']\n", 41 | "labels[30:50]" 42 | ] 43 | }, 44 | { 45 | "cell_type": "code", 46 | "execution_count": null, 47 | "metadata": {}, 48 | "outputs": [], 49 | "source": [ 50 | "image_list = ['1479498416965787036.jpg',\n", 51 | " '1479498541974858765.jpg']\n", 52 | "\n", 53 | "plt.figure(figsize=(15,15))\n", 54 | "i=1\n", 55 | "for image in image_list:\n", 56 | " plt.subplot(len(image_list), len(image_list), i)\n", 57 | " img_info = labels[labels.Frame == image]\n", 58 | " img = cv2.imread(DATA_DIR + image)\n", 59 | " img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n", 60 | " print(img.shape)\n", 61 | " cv2.rectangle(img, (img_info.xmin, img_info.ymin),(img_info.xmax, img_info.ymax), (255, 0 , 255), 4)\n", 62 | " print(img_info)\n", 63 | " plt.imshow(img)\n", 64 | " i+=1\n", 65 | "plt.show()" 66 | ] 67 | }, 68 | { 69 | "cell_type": "code", 70 | "execution_count": null, 71 | "metadata": {}, 72 | "outputs": [], 73 | "source": [ 74 | "X_train = labels.iloc[:1970]\n", 75 | "X_val = labels.iloc[2000:]\n", 76 | "print(X_train.shape)\n", 77 | "print(X_val.shape)" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": null, 83 | "metadata": {}, 84 | "outputs": [], 85 | "source": [ 86 | "image_list = ['1479502622732414408.jpg',\n", 87 | " '1479502623247392322.jpg',\n", 88 | " '1479502623755460204.jpg',\n", 89 | " '1479502623247392322.jpg',\n", 90 | " '1479502625253159017.jpg']\n", 91 | "n_images = len(image_list)\n", 92 | "\n", 93 | "plt.figure(figsize=(15,15))\n", 94 | "for i in range(n_images):\n", 95 | " plt.subplot(n_images, n_images, i+1)\n", 96 | " plt.title(\"{0}\".format(image_list[i]))\n", 97 | " img = cv2.imread(DATA_DIR + image_list[i])\n", 98 | " img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n", 99 | " plt.axis('off')\n", 100 | " plt.imshow(img)\n", 101 | " plt.tight_layout() \n", 102 | "plt.show()" 103 | ] 104 | }, 105 | { 106 | "cell_type": "code", 107 | "execution_count": null, 108 | "metadata": {}, 109 | "outputs": [], 110 | "source": [ 111 | "X_train = labels.iloc[:1970] # We've picked this frame because the car makes a right turn\n", 112 | "X_val = labels.iloc[2000:]\n", 113 | "print(X_train.shape)\n", 114 | "print(X_val.shape)" 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": null, 120 | "metadata": {}, 121 | "outputs": [], 122 | "source": [ 123 | "def IOU_calc(y_true, y_pred, smooth=0.9):\n", 124 | " y_true_f = K.flatten(y_true)\n", 125 | " y_pred_f = K.flatten(y_pred)\n", 126 | " intersection = K.sum(y_true_f * y_pred_f)\n", 127 | " return 2*(intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)" 128 | ] 129 | }, 130 | { 131 | "cell_type": "code", 132 | "execution_count": null, 133 | "metadata": {}, 134 | "outputs": [], 135 | "source": [ 136 | "img_rows = 200\n", 137 | "img_cols = 200\n", 138 | "img_channels = 3\n", 139 | "\n", 140 | "model = Sequential()\n", 141 | "model.add(Lambda(lambda x: (x / 255.) - 0.5, input_shape=(img_rows, img_cols, img_channels)))\n", 142 | "model.add(Conv2D(16, (5, 5), activation='relu', padding='same'))\n", 143 | "model.add(MaxPooling2D(pool_size=(2,2)))\n", 144 | "model.add(Conv2D(32, (5, 5), activation='relu', padding='same'))\n", 145 | "model.add(MaxPooling2D(pool_size=(2,2)))\n", 146 | "model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))\n", 147 | "model.add(MaxPooling2D(pool_size=(2,2)))\n", 148 | "model.add(Flatten())\n", 149 | "model.add(Dense(256, activation='relu'))\n", 150 | "model.add(Dropout(0.5))\n", 151 | "model.add(Dense(4, activation='sigmoid'))\n", 152 | "\n", 153 | "# Define optimizer and compile\n", 154 | "opt = optimizers.Adam(lr=1e-8)\n", 155 | "model.compile(optimizer=opt, loss='mse', metrics=[IOU_calc])\n", 156 | "model.summary()" 157 | ] 158 | }, 159 | { 160 | "cell_type": "code", 161 | "execution_count": null, 162 | "metadata": {}, 163 | "outputs": [], 164 | "source": [ 165 | "def batchgen(x, y, batch_size)\n", 166 | " # Create empty numpy arrays\n", 167 | " images = np.zeros((batch_size, img_rows, img_cols, img_channels))\n", 168 | " class_id = np.zeros((batch_size, 4))#len(y[0])))\n", 169 | "\n", 170 | " while 1:\n", 171 | " for n in range(batch_size):\n", 172 | " i = np.random.randint(len(x))\n", 173 | " \n", 174 | " x_ = x.Frame.iloc[i]\n", 175 | " x_ = cv2.imread(DATA_DIR + image)\n", 176 | " x_ = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n", 177 | " \n", 178 | " x_min = (x.iloc[i].xmin * (img_cols/1920)) / img_cols\n", 179 | " x_max = (x.iloc[i].xmax * (img_cols/1920)) / img_cols\n", 180 | " y_min = (x.iloc[i].ymin * (img_rows/1200)) / img_rows\n", 181 | " y_max = (x.iloc[i].ymax * (img_rows/1200)) / img_rows\n", 182 | " y_ = (x_min, y_min, x_max, y_max) \n", 183 | " x_ = cv2.resize(x_, (img_cols, img_rows))\n", 184 | " images[n] = x_\n", 185 | " class_id[n] = y_\n", 186 | " yield images, class_id" 187 | ] 188 | }, 189 | { 190 | "cell_type": "code", 191 | "execution_count": null, 192 | "metadata": {}, 193 | "outputs": [], 194 | "source": [ 195 | "callbacks = [EarlyStopping(monitor='val_IOU_calc', patience=10, verbose=0)]" 196 | ] 197 | }, 198 | { 199 | "cell_type": "code", 200 | "execution_count": null, 201 | "metadata": {}, 202 | "outputs": [], 203 | "source": [ 204 | "batch_size = 64\n", 205 | "n_epochs = 1000\n", 206 | "steps_per_epoch = 512\n", 207 | "val_steps = len(X_val)//batch_size\n", 208 | "\n", 209 | "train_generator = batchgen(X_train, _, batch_size)\n", 210 | "val_generator = batchgen(X_val, _, batch_size)\n", 211 | "\n", 212 | "history = model.fit_generator(train_generator, \n", 213 | " steps_per_epoch=steps_per_epoch, \n", 214 | " epochs=n_epochs, \n", 215 | " validation_data=val_generator,\n", 216 | " validation_steps = val_steps,\n", 217 | " callbacks=callbacks\n", 218 | " )" 219 | ] 220 | } 221 | ], 222 | "metadata": { 223 | "kernelspec": { 224 | "display_name": "Python 3", 225 | "language": "python", 226 | "name": "python3" 227 | }, 228 | "language_info": { 229 | "codemirror_mode": { 230 | "name": "ipython", 231 | "version": 3 232 | }, 233 | "file_extension": ".py", 234 | "mimetype": "text/x-python", 235 | "name": "python", 236 | "nbconvert_exporter": "python", 237 | "pygments_lexer": "ipython3", 238 | "version": "3.5.2" 239 | }, 240 | "widgets": { 241 | "state": {}, 242 | "version": "1.1.2" 243 | } 244 | }, 245 | "nbformat": 4, 246 | "nbformat_minor": 2 247 | } 248 | -------------------------------------------------------------------------------- /Chapter08/Chapter 8 - Analyzing sentiment.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "from nltk.tokenize import word_tokenize \n", 10 | "from nltk.stem import WordNetLemmatizer \n", 11 | "import numpy as np \n", 12 | "import random \n", 13 | "import pickle \n", 14 | "from collections import Counter \n", 15 | "\n", 16 | "import tensorflow as tf \n", 17 | "\n", 18 | "\n", 19 | "lemmatizer = WordNetLemmatizer() \n", 20 | "max_lines = 10000000 \n", 21 | "pos = 'Data/sentences/positive.txt'\n", 22 | "neg = 'Data/sentences/negative.txt'\n", 23 | "\n", 24 | "def create_lexicon(pos, neg): \n", 25 | " lexicon = [] \n", 26 | " for fi in [pos, neg]: \n", 27 | " with open(fi, 'r') as f: \n", 28 | " contents = f.readlines() \n", 29 | " for l in contents[:max_lines]: \n", 30 | " all_words = word_tokenize(l.lower()) \n", 31 | " lexicon += list(all_words) \n", 32 | " \n", 33 | " lexicon = [lemmatizer.lemmatize(i) for i in lexicon] \n", 34 | " w_counts = Counter(lexicon) \n", 35 | " \n", 36 | " l2 =[] \n", 37 | " for w in w_counts: \n", 38 | " if 1000 > w_counts[w] > 50: \n", 39 | " l2.append(w) \n", 40 | " return l2 \n", 41 | " \n", 42 | "def sample_handling(sample,lexicon,classification): \n", 43 | " featureset = [] \n", 44 | " with open(sample,'r') as f: \n", 45 | " contents = f.readlines() \n", 46 | " for l in contents[:max_lines]: \n", 47 | " current_words = word_tokenize(l.lower()) \n", 48 | " current_words = [lemmatizer.lemmatize(i) for i in current_words] \n", 49 | " features = np.zeros(len(lexicon)) \n", 50 | " for word in current_words: \n", 51 | " if word.lower() in lexicon: \n", 52 | " index_value = lexicon.index(word.lower()) \n", 53 | " features[index_value] += 1 \n", 54 | " \n", 55 | " features = list(features) \n", 56 | " featureset.append([features,classification]) \n", 57 | " \n", 58 | " return featureset \n", 59 | " \n", 60 | "lexicon = create_lexicon(pos,neg) \n", 61 | "features = [] \n", 62 | "features += sample_handling(pos, lexicon,[1,0]) \n", 63 | "features += sample_handling(neg, lexicon,[0,1]) \n", 64 | "random.shuffle(features) \n", 65 | "features = np.array(features) \n", 66 | "\n", 67 | "testing_size = int(0.1*len(features)) \n", 68 | "\n", 69 | "X_train = list(features[:,0][:-testing_size]) \n", 70 | "y_train = list(features[:,1][:-testing_size]) \n", 71 | "X_test = list(features[:,0][-testing_size:]) \n", 72 | "y_test = list(features[:,1][-testing_size:]) " 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": null, 78 | "metadata": {}, 79 | "outputs": [], 80 | "source": [ 81 | "n_epochs = 10 \n", 82 | "batch_size = 128 \n", 83 | "h1 = 500 \n", 84 | "h2 = 500 \n", 85 | "n_classes = 2 " 86 | ] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": null, 91 | "metadata": {}, 92 | "outputs": [], 93 | "source": [ 94 | "x_input = tf.placeholder('float') \n", 95 | "y_input = tf.placeholder('float') \n", 96 | " \n", 97 | "hidden_1 = {'weight':tf.Variable(tf.random_normal([len(X_train[0]), h1])), \n", 98 | " 'bias':tf.Variable(tf.random_normal([h1]))} \n", 99 | " \n", 100 | "hidden_2 = {'weight':tf.Variable(tf.random_normal([h1, h2])), \n", 101 | " 'bias':tf.Variable(tf.random_normal([h2]))} \n", 102 | " \n", 103 | "output_layer = {'weight':tf.Variable(tf.random_normal([h2, n_classes])), \n", 104 | " 'bias':tf.Variable(tf.random_normal([n_classes])),} " 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": null, 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "l1 = tf.add(tf.matmul(x_input, hidden_1['weight']), hidden_1['bias']) \n", 114 | "l1 = tf.nn.relu(l1) \n", 115 | "\n", 116 | "l2 = tf.add(tf.matmul(l1, hidden_2['weight']), hidden_2['bias']) \n", 117 | "l2 = tf.nn.relu(l2) \n", 118 | "\n", 119 | "output = tf.matmul(l2, output_layer['weight']) + output_layer['bias'] " 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": null, 125 | "metadata": {}, 126 | "outputs": [], 127 | "source": [ 128 | "loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=output, labels=y_input)) \n", 129 | "opt = tf.train.AdamOptimizer().minimize(loss) \n", 130 | "\n", 131 | "\n", 132 | "with tf.Session() as sess: \n", 133 | " sess.run(tf.global_variables_initializer()) \n", 134 | "\n", 135 | " for epoch in range(n_epochs): \n", 136 | " epoch_loss = 0 \n", 137 | " i = 0 \n", 138 | " while i < len(X_train): \n", 139 | " start = i \n", 140 | " end = i + batch_size \n", 141 | " batch_x = np.array(X_train[start:end]) \n", 142 | " batch_y = np.array(y_train[start:end]) \n", 143 | "\n", 144 | " _, batch_loss = sess.run([opt, loss], feed_dict={x_input: batch_x, y_input: batch_y}) \n", 145 | " epoch_loss += batch_loss\n", 146 | " i += batch_size \n", 147 | "\n", 148 | " print('Epoch {}: loss {}'.format(epoch, epoch_loss))" 149 | ] 150 | } 151 | ], 152 | "metadata": { 153 | "kernelspec": { 154 | "display_name": "Python 3", 155 | "language": "python", 156 | "name": "python3" 157 | }, 158 | "language_info": { 159 | "codemirror_mode": { 160 | "name": "ipython", 161 | "version": 3 162 | }, 163 | "file_extension": ".py", 164 | "mimetype": "text/x-python", 165 | "name": "python", 166 | "nbconvert_exporter": "python", 167 | "pygments_lexer": "ipython3", 168 | "version": "3.5.2" 169 | }, 170 | "widgets": { 171 | "state": {}, 172 | "version": "1.1.2" 173 | } 174 | }, 175 | "nbformat": 4, 176 | "nbformat_minor": 2 177 | } 178 | -------------------------------------------------------------------------------- /Chapter08/Chapter 8 - Summarizing text.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np\n", 10 | "import tensorflow as tf" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": null, 16 | "metadata": {}, 17 | "outputs": [], 18 | "source": [ 19 | "# Data can be downloaded at https://github.com/harvardnlp/sent-summary" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": null, 25 | "metadata": {}, 26 | "outputs": [], 27 | "source": [ 28 | "article_filename = 'Data/sumdata/train/train.article.txt'\n", 29 | "title_filename = 'Data/sumdata/train/train.title.txt'\n", 30 | "\n", 31 | "with open(article_filename) as article_file:\n", 32 | " articles = article_file.readlines()\n", 33 | "with open(title_filename) as title_file:\n", 34 | " titles = title_file.readlines()" 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": null, 40 | "metadata": {}, 41 | "outputs": [], 42 | "source": [ 43 | "def create_lookup_tables(text):\n", 44 | " vocab = set(text.split())\n", 45 | " vocab_to_int = {'': 0, '': 1, '': 2, '': 3 }\n", 46 | "\n", 47 | " for i, v in enumerate(vocab, len(vocab_to_int)):\n", 48 | " vocab_to_int[v] = i\n", 49 | "\n", 50 | " int_to_vocab = {i: v for v, i in vocab_to_int.items()}\n", 51 | "\n", 52 | " return vocab_to_int, int_to_vocab\n", 53 | "\n", 54 | "vocab_to_int_article, int_to_vocab_article = create_lookup_tables([x.lower() for x in articles])\n", 55 | "vocab_to_int_title, int_to_vocab_title = create_lookup_tables([x.lower() for x in titles])" 56 | ] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "execution_count": null, 61 | "metadata": {}, 62 | "outputs": [], 63 | "source": [ 64 | "def text_to_ids(source_text, target_text, source_vocab_to_int, target_vocab_to_int):\n", 65 | " source_id_text = [[source_vocab_to_int[word] for word in sentence.split()] for sentence in source_text.split('\\n')]\n", 66 | " target_id_text = [[target_vocab_to_int[word] for word in sentence.split()]+[target_vocab_to_int['']] for sentence in target_text.split('\\n')]\n", 67 | " \n", 68 | " return source_id_text, target_id_text\n", 69 | "\n", 70 | "X, y = text_to_ids(articles.lower(), titles.lower(), vocab_to_int_articles, vocab_to_int_titles)" 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": null, 76 | "metadata": {}, 77 | "outputs": [], 78 | "source": [ 79 | "learning_rate = 0.001\n", 80 | "hidden_units = 400\n", 81 | "embedding_size = 200\n", 82 | "n_layers = 1\n", 83 | "dropout = 0.5\n", 84 | "n_iters = 40" 85 | ] 86 | }, 87 | { 88 | "cell_type": "code", 89 | "execution_count": null, 90 | "metadata": {}, 91 | "outputs": [], 92 | "source": [ 93 | "encoder_forward_cell = tf.contrib.rnn.GRUCell(state_size)\n", 94 | "encoder_backward_cell = tf.contrib.rnn.GRUCell(state_size)\n", 95 | "decoder_cell = tf.contrib.rnn.GRUCell(state_size)\n", 96 | "\n", 97 | "encoder_forward_cell = tf.contrib.rnn.DropoutWrapper(encoder_forward_cell, output_keep_prob = (1-dropout))\n", 98 | "encoder_backward_cell = tf.contrib.rnn.DropoutWrapper(encoder_backward_cell, output_keep_prob = (1-dropout))\n", 99 | "decoder_cell = tf.contrib.rnn.DropoutWrapper(decoder_cell, output_keep_prob = (1-dropout))" 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": null, 105 | "metadata": {}, 106 | "outputs": [], 107 | "source": [ 108 | "with tf.variable_scope(\"seq2seq\", dtype=dtype):\n", 109 | " with tf.variable_scope(\"encoder\"):\n", 110 | " encoder_embedding = tf.get_variable(\"embedding\", [source_vocab_size, embedding_size],initializer=embedding_init)\n", 111 | " encoder_inputs_embedding = tf.nn.embedding_lookup(encoder_embedding, self.encoder_inputs)\n", 112 | " encoder_outputs, encoder_states = tf.nn.bidirectional_dynamic_rnn(encoder_forward_cell, encoder_backward_cell, encoder_inputs_embedding, sequence_length=self.encoder_len, dtype=dtype)\n", 113 | "\n", 114 | " with tf.variable_scope(\"init_state\"):\n", 115 | " init_state = fc_layer(\n", 116 | " tf.concat(encoder_states, 1), state_size)\n", 117 | " # the shape of bidirectional_dynamic_rnn is weird\n", 118 | " # None for batch_size\n", 119 | " self.init_state = init_state\n", 120 | " self.init_state.set_shape([self.batch_size, state_size])\n", 121 | " self.att_states = tf.concat(encoder_outputs, 2)\n", 122 | " self.att_states.set_shape([self.batch_size, None, state_size*2])\n", 123 | "\n", 124 | " with tf.variable_scope(\"attention\"):\n", 125 | " attention = tf.contrib.seq2seq.BahdanauAttention(\n", 126 | " state_size, self.att_states, self.encoder_len)\n", 127 | " decoder_cell = tf.contrib.seq2seq.DynamicAttentionWrapper(\n", 128 | " decoder_cell, attention, state_size * 2)\n", 129 | " wrapper_state = tf.contrib.seq2seq.DynamicAttentionWrapperState(\n", 130 | " self.init_state, self.prev_att)\n", 131 | "\n", 132 | " with tf.variable_scope(\"decoder\") as scope:\n", 133 | " decoder_emb = tf.get_variable(\"embedding\", [target_vocab_size, embedding_size],initializer=emb_init)\n", 134 | " decoder_cell = tf.contrib.rnn.OutputProjectionWrapper(decoder_cell, target_vocab_size)\n", 135 | " decoder_inputs_emb = tf.nn.embedding_lookup(decoder_emb, self.decoder_inputs)\n", 136 | "\n", 137 | " helper = tf.contrib.seq2seq.TrainingHelper(decoder_inputs_emb, self.decoder_len)\n", 138 | " decoder = tf.contrib.seq2seq.BasicDecoder(decoder_cell, helper, wrapper_state)\n", 139 | "\n", 140 | " outputs, final_state = tf.contrib.seq2seq.dynamic_decode(decoder)\n", 141 | "\n", 142 | " outputs_logits = outputs[0]\n", 143 | " self.outputs = outputs_logits\n", 144 | "\n", 145 | " weights = tf.sequence_mask(self.decoder_len, dtype=tf.float32)\n", 146 | "\n", 147 | " loss_t = tf.contrib.seq2seq.sequence_loss(outputs_logits, self.decoder_targets, weights, average_across_timesteps=False, average_across_batch=False)\n", 148 | " self.loss = tf.reduce_sum(loss_t) / self.batch_size\n", 149 | "\n", 150 | " params = tf.trainable_variables()\n", 151 | " opt = tf.train.AdadeltaOptimizer(self.learning_rate, epsilon=1e-6)\n", 152 | " gradients = tf.gradients(self.loss, params)\n", 153 | " clipped_gradients, norm = tf.clip_by_global_norm(gradients, max_gradient)\n", 154 | " self.updates = opt.apply_gradients(zip(clipped_gradients, params), global_step=self.global_step)" 155 | ] 156 | }, 157 | { 158 | "cell_type": "code", 159 | "execution_count": null, 160 | "metadata": {}, 161 | "outputs": [], 162 | "source": [ 163 | "def get_batches(int_text, batch_size, seq_length):\n", 164 | " n_batches = int(len(int_text) / (batch_size * seq_length))\n", 165 | " inputs = np.array(int_text[: n_batches * batch_size * seq_length])\n", 166 | " outputs = np.array(int_text[1: n_batches * batch_size * seq_length + 1])\n", 167 | "\n", 168 | " x = np.split(inputs.reshape(batch_size, -1), n_batches, 1)\n", 169 | " y = np.split(outputs.reshape(batch_size, -1), n_batches, 1)\n", 170 | "\n", 171 | " return np.array(list(zip(x, y)))" 172 | ] 173 | }, 174 | { 175 | "cell_type": "code", 176 | "execution_count": null, 177 | "metadata": {}, 178 | "outputs": [], 179 | "source": [ 180 | "with tf.Session() as sess:\n", 181 | " model = create_model(sess, False)\n", 182 | " loss = 0.0\n", 183 | " current_step = sess.run(model.global_step)\n", 184 | "\n", 185 | " while current_step <= n_iters:\n", 186 | " rand = np.random.random_sample()\n", 187 | " bucket_id = min([i for i in range(len(train_buckets_scale))\n", 188 | " if train_buckets_scale[i] > rand])\n", 189 | "\n", 190 | " encoder_inputs, decoder_inputs, encoder_len, decoder_len = model.get_batches(train_set, bucket_id)\n", 191 | " step_loss, _ = model.step(sess, encoder_inputs, decoder_inputs, encoder_len, decoder_len, False, train_writer)\n", 192 | " loss += step_loss * batch_size / np.sum(decoder_len)\n", 193 | " current_step += 1" 194 | ] 195 | } 196 | ], 197 | "metadata": { 198 | "kernelspec": { 199 | "display_name": "Python 3", 200 | "language": "python", 201 | "name": "python3" 202 | }, 203 | "language_info": { 204 | "codemirror_mode": { 205 | "name": "ipython", 206 | "version": 3 207 | }, 208 | "file_extension": ".py", 209 | "mimetype": "text/x-python", 210 | "name": "python", 211 | "nbconvert_exporter": "python", 212 | "pygments_lexer": "ipython3", 213 | "version": "3.5.2" 214 | }, 215 | "widgets": { 216 | "state": {}, 217 | "version": "1.1.2" 218 | } 219 | }, 220 | "nbformat": 4, 221 | "nbformat_minor": 2 222 | } 223 | -------------------------------------------------------------------------------- /Chapter09/Chapter 9 - Identifying speakers with voice recognition.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import glob\n", 12 | "import numpy as np\n", 13 | "import random\n", 14 | "import librosa\n", 15 | "from sklearn.model_selection import train_test_split\n", 16 | "from sklearn.preprocessing import LabelBinarizer\n", 17 | "\n", 18 | "import keras\n", 19 | "from keras.layers import LSTM, Dense, Dropout, Flatten\n", 20 | "from keras.models import Sequential\n", 21 | "from keras.optimizers import Adam\n", 22 | "from keras.callbacks import EarlyStopping, ModelCheckpoint" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": null, 28 | "metadata": { 29 | "collapsed": true 30 | }, 31 | "outputs": [], 32 | "source": [ 33 | "# Data can be downloaded at http://pannous.net/spoken_numbers.tar" 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": null, 39 | "metadata": { 40 | "collapsed": true 41 | }, 42 | "outputs": [], 43 | "source": [ 44 | "SEED = 2017\n", 45 | "DATA_DIR = '../Data/spoken_numbers_pcm/' " 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": null, 51 | "metadata": { 52 | "collapsed": true 53 | }, 54 | "outputs": [], 55 | "source": [ 56 | "files = glob.glob(DATA_DIR + \"*.wav\")\n", 57 | "X_train, X_val = train_test_split(files, test_size=0.2, random_state=SEED)\n", 58 | "\n", 59 | "print('# Training examples: {}'.format(len(X_train)))\n", 60 | "print('# Validation examples: {}'.format(len(X_val)))" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": null, 66 | "metadata": { 67 | "collapsed": true 68 | }, 69 | "outputs": [], 70 | "source": [ 71 | "labels = []\n", 72 | "for i in range(len(X_train)):\n", 73 | " label = X_train[i].split('/')[-1].split('_')[1]\n", 74 | " if label not in labels:\n", 75 | " labels.append(label)\n", 76 | "print(labels)" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": null, 82 | "metadata": { 83 | "collapsed": true 84 | }, 85 | "outputs": [], 86 | "source": [ 87 | "label_binarizer = LabelBinarizer()\n", 88 | "label_binarizer.fit(list(set(labels)))\n", 89 | "\n", 90 | "def one_hot_encode(x): return label_binarizer.transform(x)" 91 | ] 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": null, 96 | "metadata": { 97 | "collapsed": true 98 | }, 99 | "outputs": [], 100 | "source": [ 101 | "n_features = 20\n", 102 | "max_length = 80\n", 103 | "n_classes = len(labels)" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": null, 109 | "metadata": { 110 | "collapsed": true 111 | }, 112 | "outputs": [], 113 | "source": [ 114 | "def batch_generator(data, batch_size=16):\n", 115 | " while 1:\n", 116 | " random.shuffle(data)\n", 117 | " X, y = [], []\n", 118 | " for i in range(batch_size):\n", 119 | " wav = data[i]\n", 120 | " wave, sr = librosa.load(wav, mono=True)\n", 121 | " label = wav.split('/')[-1].split('_')[1]\n", 122 | " y.append(label)\n", 123 | " mfcc = librosa.feature.mfcc(wave, sr)\n", 124 | " mfcc = np.pad(mfcc, ((0,0), (0, max_length-len(mfcc[0]))), mode='constant', constant_values=0) \n", 125 | " X.append(np.array(mfcc))\n", 126 | " yield np.array(X), np.array(one_hot_encode(y))" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": null, 132 | "metadata": { 133 | "collapsed": true 134 | }, 135 | "outputs": [], 136 | "source": [ 137 | "learning_rate = 0.001\n", 138 | "batch_size = 64\n", 139 | "n_epochs = 50\n", 140 | "dropout = 0.5\n", 141 | "\n", 142 | "input_shape = (n_features, max_length)\n", 143 | "steps_per_epoch = 50" 144 | ] 145 | }, 146 | { 147 | "cell_type": "code", 148 | "execution_count": null, 149 | "metadata": { 150 | "collapsed": true 151 | }, 152 | "outputs": [], 153 | "source": [ 154 | "model = Sequential()\n", 155 | "model.add(LSTM(256, return_sequences=True, input_shape=input_shape,\n", 156 | "dropout=dropout))\n", 157 | "model.add(Flatten())\n", 158 | "model.add(Dense(128, activation='relu'))\n", 159 | "model.add(Dropout(dropout))\n", 160 | "model.add(Dense(n_classes, activation='softmax'))" 161 | ] 162 | }, 163 | { 164 | "cell_type": "code", 165 | "execution_count": null, 166 | "metadata": { 167 | "collapsed": true 168 | }, 169 | "outputs": [], 170 | "source": [ 171 | "opt = Adam(lr=learning_rate)\n", 172 | "model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])\n", 173 | "model.summary()" 174 | ] 175 | }, 176 | { 177 | "cell_type": "code", 178 | "execution_count": null, 179 | "metadata": { 180 | "collapsed": true 181 | }, 182 | "outputs": [], 183 | "source": [ 184 | "callbacks = [ModelCheckpoint('checkpoints/voice_recognition_best_model_{epoch:02d}.hdf5', save_best_only=True),\n", 185 | " EarlyStopping(monitor='val_acc', patience=2)]" 186 | ] 187 | }, 188 | { 189 | "cell_type": "code", 190 | "execution_count": null, 191 | "metadata": { 192 | "collapsed": true 193 | }, 194 | "outputs": [], 195 | "source": [ 196 | "history = model.fit_generator(\n", 197 | " generator=batch_generator(X_train, batch_size),\n", 198 | " steps_per_epoch=steps_per_epoch,\n", 199 | " epochs=n_epochs,\n", 200 | " verbose=1,\n", 201 | " validation_data=batch_generator(X_val, 32),\n", 202 | " validation_steps=5,\n", 203 | " callbacks=callbacks\n", 204 | " )" 205 | ] 206 | }, 207 | { 208 | "cell_type": "code", 209 | "execution_count": null, 210 | "metadata": { 211 | "collapsed": true 212 | }, 213 | "outputs": [], 214 | "source": [] 215 | } 216 | ], 217 | "metadata": { 218 | "kernelspec": { 219 | "display_name": "Python 3", 220 | "language": "python", 221 | "name": "python3" 222 | }, 223 | "language_info": { 224 | "codemirror_mode": { 225 | "name": "ipython", 226 | "version": 3 227 | }, 228 | "file_extension": ".py", 229 | "mimetype": "text/x-python", 230 | "name": "python", 231 | "nbconvert_exporter": "python", 232 | "pygments_lexer": "ipython3", 233 | "version": "3.6.0" 234 | } 235 | }, 236 | "nbformat": 4, 237 | "nbformat_minor": 2 238 | } 239 | -------------------------------------------------------------------------------- /Chapter09/Chapter 9 - Implementing a speech recognition pipeline from scratch.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 2, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [ 10 | { 11 | "name": "stderr", 12 | "output_type": "stream", 13 | "text": [ 14 | "Using Theano backend.\n" 15 | ] 16 | } 17 | ], 18 | "source": [ 19 | "import glob\n", 20 | "import numpy as np\n", 21 | "import random\n", 22 | "import librosa\n", 23 | "from sklearn.model_selection import train_test_split\n", 24 | "\n", 25 | "import keras\n", 26 | "from keras.layers import LSTM, Dense, Dropout, Flatten\n", 27 | "from keras.models import Sequential\n", 28 | "from keras.optimizers import Adam\n", 29 | "from keras.callbacks import EarlyStopping, ModelCheckpoint" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": null, 35 | "metadata": { 36 | "collapsed": true 37 | }, 38 | "outputs": [], 39 | "source": [ 40 | "# Data can be downloaded at http://pannous.net/spoken_numbers.tar" 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": 3, 46 | "metadata": { 47 | "collapsed": true 48 | }, 49 | "outputs": [], 50 | "source": [ 51 | "SEED = 2017\n", 52 | "DATA_DIR = 'Data/spoken_numbers_pcm/' " 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": 4, 58 | "metadata": { 59 | "collapsed": false 60 | }, 61 | "outputs": [ 62 | { 63 | "name": "stdout", 64 | "output_type": "stream", 65 | "text": [ 66 | "# Training examples: 0\n", 67 | "# Validation examples: 0\n" 68 | ] 69 | } 70 | ], 71 | "source": [ 72 | "files = glob.glob(DATA_DIR + \"*.wav\")\n", 73 | "X_train, X_val = train_test_split(files, test_size=0.2, random_state=SEED)\n", 74 | "\n", 75 | "print('# Training examples: {}'.format(len(X_train)))\n", 76 | "print('# Validation examples: {}'.format(len(X_val)))" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": 5, 82 | "metadata": { 83 | "collapsed": true 84 | }, 85 | "outputs": [], 86 | "source": [ 87 | "n_features = 20\n", 88 | "max_length = 80\n", 89 | "n_classes = 10" 90 | ] 91 | }, 92 | { 93 | "cell_type": "code", 94 | "execution_count": null, 95 | "metadata": { 96 | "collapsed": true 97 | }, 98 | "outputs": [], 99 | "source": [ 100 | "def batch_generator(data, batch_size=16):\n", 101 | " while 1:\n", 102 | " random.shuffle(data)\n", 103 | " X, y = [], []\n", 104 | " for i in range(batch_size):\n", 105 | " wav = data[i]\n", 106 | " wave, sr = librosa.load(wav, mono=True)\n", 107 | " label = one_hot_encode(int(wav.split('/')[-1][0]), n_classes)\n", 108 | " y.append(label)\n", 109 | " mfcc = librosa.feature.mfcc(wave, sr)\n", 110 | " mfcc = np.pad(mfcc, ((0,0), (0, max_length-len(mfcc[0]))), mode='constant', constant_values=0) \n", 111 | " X.append(np.array(mfcc))\n", 112 | " yield np.array(X), np.array(y)" 113 | ] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": null, 118 | "metadata": { 119 | "collapsed": true 120 | }, 121 | "outputs": [], 122 | "source": [ 123 | "def one_hot_encode(labels_dense, n_classes=10):\n", 124 | " return np.eye(n_classes)[labels_dense]" 125 | ] 126 | }, 127 | { 128 | "cell_type": "code", 129 | "execution_count": null, 130 | "metadata": { 131 | "collapsed": true 132 | }, 133 | "outputs": [], 134 | "source": [ 135 | "X_example, y_example = next(batch_generator(X_train, batch_size=1))\n", 136 | "print('Shape of training example: {}'.format(X_example.shape))\n", 137 | "print('Shape of training example label: {}'.format(y_example.shape))" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": null, 143 | "metadata": { 144 | "collapsed": true 145 | }, 146 | "outputs": [], 147 | "source": [ 148 | "learning_rate = 0.001\n", 149 | "batch_size = 64\n", 150 | "n_epochs = 50\n", 151 | "dropout = 0.5\n", 152 | "\n", 153 | "input_shape = X_example.shape[1:]\n", 154 | "steps_per_epoch = 50" 155 | ] 156 | }, 157 | { 158 | "cell_type": "code", 159 | "execution_count": null, 160 | "metadata": { 161 | "collapsed": true 162 | }, 163 | "outputs": [], 164 | "source": [ 165 | "model = Sequential() model.add(LSTM(256, return_sequences=True, input_shape=input_shape, dropout=dropout)) \n", 166 | "model.add(Flatten()) model.add(Dense(128, activation='relu')) model.add(Dropout(dropout)) \n", 167 | "model.add(Dense(n_classes, activation='softmax'))" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": null, 173 | "metadata": { 174 | "collapsed": true 175 | }, 176 | "outputs": [], 177 | "source": [ 178 | "opt = Adam(lr=learning_rate) model.compile(loss='categorical_crossentropy', optimizer=opt, \n", 179 | "metrics=['accuracy']) model.summary()" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": null, 185 | "metadata": { 186 | "collapsed": true 187 | }, 188 | "outputs": [], 189 | "source": [ 190 | "history = model.fit_generator(\n", 191 | " generator=batch_generator(X_train, batch_size),\n", 192 | " steps_per_epoch=steps_per_epoch,\n", 193 | " epochs=n_epochs,\n", 194 | " verbose=1,\n", 195 | " validation_data=batch_generator(X_val, 32),\n", 196 | " validation_steps=5\n", 197 | " )" 198 | ] 199 | } 200 | ], 201 | "metadata": { 202 | "kernelspec": { 203 | "display_name": "Python 3", 204 | "language": "python", 205 | "name": "python3" 206 | }, 207 | "language_info": { 208 | "codemirror_mode": { 209 | "name": "ipython", 210 | "version": 3 211 | }, 212 | "file_extension": ".py", 213 | "mimetype": "text/x-python", 214 | "name": "python", 215 | "nbconvert_exporter": "python", 216 | "pygments_lexer": "ipython3", 217 | "version": "3.6.0" 218 | } 219 | }, 220 | "nbformat": 4, 221 | "nbformat_minor": 2 222 | } 223 | -------------------------------------------------------------------------------- /Chapter09/Chapter 9 - Understanding videos with deep learning.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "from keras.applications.inception_v3 import InceptionV3\n", 12 | "from keras.preprocessing.image import ImageDataGenerator\n", 13 | "from keras.models import Sequential, Model\n", 14 | "from keras.layers import Dense, GlobalAveragePooling2D\n", 15 | "from keras.optimizers import Adam\n", 16 | "from keras.callbacks import EarlyStopping, ModelCheckpoint" 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": null, 22 | "metadata": { 23 | "collapsed": true 24 | }, 25 | "outputs": [], 26 | "source": [ 27 | "# Data can be downloaded at https://research.google.com/youtube8m/" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": null, 33 | "metadata": { 34 | "collapsed": true 35 | }, 36 | "outputs": [], 37 | "source": [ 38 | "batch_size = 32\n", 39 | "n_epochs = 50\n", 40 | "steps_per_epoch = 100" 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": null, 46 | "metadata": { 47 | "collapsed": true 48 | }, 49 | "outputs": [], 50 | "source": [ 51 | "train_datagen = ImageDataGenerator(\n", 52 | " rescale=1./255,\n", 53 | " shear_range=0.2,\n", 54 | " horizontal_flip=True,\n", 55 | " rotation_range=10.,\n", 56 | " width_shift_range=0.2,\n", 57 | " height_shift_range=0.2)\n", 58 | "\n", 59 | "val_datagen = ImageDataGenerator(rescale=1./255)" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": null, 65 | "metadata": { 66 | "collapsed": true 67 | }, 68 | "outputs": [], 69 | "source": [ 70 | "classes = np.loadtxt('Data/videos/classes.txt') \n", 71 | "train_generator = train_datagen.flow_from_directory(\n", 72 | " 'Data/videos/train/',\n", 73 | " target_size=(299, 299),\n", 74 | " batch_size=batch_size,\n", 75 | " classes=classes,\n", 76 | " class_mode='categorical')\n", 77 | "\n", 78 | "validation_generator = val_datagen.flow_from_directory(\n", 79 | " 'Data/videos/test/',\n", 80 | " target_size=(299, 299),\n", 81 | " batch_size=batch_size,\n", 82 | " classes=classes,\n", 83 | " class_mode='categorical')" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": null, 89 | "metadata": { 90 | "collapsed": true 91 | }, 92 | "outputs": [], 93 | "source": [ 94 | "inception_model = InceptionV3(include_top=False)" 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": null, 100 | "metadata": { 101 | "collapsed": true 102 | }, 103 | "outputs": [], 104 | "source": [ 105 | "x = inception_model.output\n", 106 | "x = GlobalAveragePooling2D()(x)\n", 107 | "x = Dense(1024, activation='relu')(x)\n", 108 | "predictions = Dense(len(classes), activation='softmax')(x)\n", 109 | "model = Model(inputs=inception_model.input, outputs=predictions)" 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": null, 115 | "metadata": { 116 | "collapsed": true 117 | }, 118 | "outputs": [], 119 | "source": [ 120 | "for layer in model.layers[:-2]:\n", 121 | " layer.trainable = False" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": null, 127 | "metadata": { 128 | "collapsed": true 129 | }, 130 | "outputs": [], 131 | "source": [ 132 | "model.compile(optimizer=Adam(), loss='categorical_crossentropy', metrics=['accuracy'])" 133 | ] 134 | }, 135 | { 136 | "cell_type": "code", 137 | "execution_count": null, 138 | "metadata": { 139 | "collapsed": true 140 | }, 141 | "outputs": [], 142 | "source": [ 143 | "model.fit_generator(\n", 144 | " train_generator,\n", 145 | " steps_per_epoch=steps_per_epoch,\n", 146 | " validation_data=validation_generator,\n", 147 | " validation_steps=10,\n", 148 | " epochs=n_epochs)" 149 | ] 150 | }, 151 | { 152 | "cell_type": "code", 153 | "execution_count": null, 154 | "metadata": { 155 | "collapsed": true 156 | }, 157 | "outputs": [], 158 | "source": [ 159 | "model.layers.pop()\n", 160 | "model.layers.pop()\n", 161 | "model.outputs = [model.layers[-1].output]\n", 162 | "model.output_layers = [model.layers[-1]]\n", 163 | "model.layers[-1].outbound_nodes = []" 164 | ] 165 | }, 166 | { 167 | "cell_type": "code", 168 | "execution_count": null, 169 | "metadata": { 170 | "collapsed": true 171 | }, 172 | "outputs": [], 173 | "source": [ 174 | "for video in data:\n", 175 | " path = 'Data/videos/' + str(video)\n", 176 | " frames = get_frame(video)\n", 177 | " sequence = []\n", 178 | " for image in frames:\n", 179 | " features = model.predict(image)\n", 180 | " sequence.append(features)\n", 181 | " np.savetxt(path, sequence)" 182 | ] 183 | }, 184 | { 185 | "cell_type": "code", 186 | "execution_count": null, 187 | "metadata": { 188 | "collapsed": true 189 | }, 190 | "outputs": [], 191 | "source": [ 192 | "model_lstm = Sequential()\n", 193 | "model_lstm.add(LSTM(1024, return_sequences=True, input_shape=input_shape, dropout=dropout))\n", 194 | "model_lstm.add(Flatten())\n", 195 | "model_lstm.add(Dense(512, activation='relu'))\n", 196 | "model_lstm.add(Dropout(dropout))\n", 197 | "model_lstm.add(Dense(self.nb_classes, activation='softmax'))" 198 | ] 199 | }, 200 | { 201 | "cell_type": "code", 202 | "execution_count": null, 203 | "metadata": { 204 | "collapsed": true 205 | }, 206 | "outputs": [], 207 | "source": [ 208 | "batch_size_lstm = 32\n", 209 | "n_epochs_lstm = 40" 210 | ] 211 | }, 212 | { 213 | "cell_type": "code", 214 | "execution_count": null, 215 | "metadata": { 216 | "collapsed": true 217 | }, 218 | "outputs": [], 219 | "source": [ 220 | "model_lstm.fit(X_train,\n", 221 | " y_train,\n", 222 | " batch_size=batch_size_lstm,\n", 223 | " validation_data=(X_val, y_val),\n", 224 | " verbose=1,\n", 225 | " epochs=n_epochs_lstm)" 226 | ] 227 | } 228 | ], 229 | "metadata": { 230 | "kernelspec": { 231 | "display_name": "Python 3", 232 | "language": "python", 233 | "name": "python3" 234 | }, 235 | "language_info": { 236 | "codemirror_mode": { 237 | "name": "ipython", 238 | "version": 3 239 | }, 240 | "file_extension": ".py", 241 | "mimetype": "text/x-python", 242 | "name": "python", 243 | "nbconvert_exporter": "python", 244 | "pygments_lexer": "ipython3", 245 | "version": "3.6.0" 246 | } 247 | }, 248 | "nbformat": 4, 249 | "nbformat_minor": 2 250 | } 251 | -------------------------------------------------------------------------------- /Chapter10/Chapter 10 - Predicting bike sharing demand.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "from sklearn import preprocessing\n", 10 | "import pandas as pd\n", 11 | "import numpy as np\n", 12 | "from math import pi, sin, cos\n", 13 | "from datetime import datetime\n", 14 | "\n", 15 | "from keras.models import Sequential\n", 16 | "from keras.layers import Dense, Activation, Dropout\n", 17 | "from keras.optimizers import Adam\n", 18 | "from keras.callbacks import EarlyStopping\n" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "# Data can be downloaded at https://www.kaggle.com/c/bike-sharing-demand/data" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": null, 33 | "metadata": {}, 34 | "outputs": [], 35 | "source": [ 36 | "train = pd.read_csv('../Data/bike-sharing/train.csv')\n", 37 | "test = pd.read_csv('../Data/bike-sharing/test.csv')\n", 38 | "data = pd.concat([train, test])\n", 39 | "test_split = train.shape[0]" 40 | ] 41 | }, 42 | { 43 | "cell_type": "code", 44 | "execution_count": null, 45 | "metadata": {}, 46 | "outputs": [], 47 | "source": [ 48 | "data.head()" 49 | ] 50 | }, 51 | { 52 | "cell_type": "code", 53 | "execution_count": null, 54 | "metadata": {}, 55 | "outputs": [], 56 | "source": [ 57 | "data['hour'] = data.apply(lambda x: datetime.strptime(x['datetime'], '%Y-%m-%d %H:%M:%S').hour, axis=1)\n", 58 | "data['weekday'] = data.apply(lambda x: datetime.strptime(x['datetime'], '%Y-%m-%d %H:%M:%S').weekday(), axis=1)\n", 59 | "data['month'] = data.apply(lambda x: datetime.strptime(x['datetime'], '%Y-%m-%d %H:%M:%S').month, axis=1)\n", 60 | "data['year'] = data.apply(lambda x: datetime.strptime(x['datetime'], '%Y-%m-%d %H:%M:%S').year, axis=1)" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": null, 66 | "metadata": {}, 67 | "outputs": [], 68 | "source": [ 69 | "data['hour_sin'] = data.apply(lambda x: sin(x['hour'] / 24.0 * 2 * pi), axis=1)\n", 70 | "data['hour_cos'] = data.apply(lambda x: cos(x['hour'] / 24.0 * 2 * pi), axis=1)\n", 71 | "data['weekday_sin'] = data.apply(lambda x: sin(x['weekday'] / 7.0 * 2 * pi), axis=1)\n", 72 | "data['weekday_cos'] = data.apply(lambda x: cos(x['weekday'] / 7.0 * 2 * pi), axis=1)\n", 73 | "data['month_sin'] = data.apply(lambda x: sin(((x['month'] - 5) % 12) / 12.0 * 2 * pi), axis=1)\n", 74 | "data['month_cos'] = data.apply(lambda x: cos(((x['month'] - 5) % 12) / 12.0 * 2 * pi), axis=1)\n", 75 | "data['season_sin'] = data.apply(lambda x: sin(((x['season'] - 3) % 4) / 4.0 * 2 * pi), axis=1)\n", 76 | "data['season_cos'] = data.apply(lambda x: cos(((x['season'] - 3) % 4) / 4.0 * 2 * pi), axis=1)" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": null, 82 | "metadata": {}, 83 | "outputs": [], 84 | "source": [ 85 | "X_train = data[:test_split].drop(['datetime', 'casual', 'registered', 'count'], inplace=False, axis=1)\n", 86 | "X_test = data[test_split:].drop(['datetime', 'casual', 'registered', 'count'], inplace=False, axis=1)\n", 87 | "y_train = data['count'][:test_split]\n", 88 | "y_test = data['count'][test_split:]" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": null, 94 | "metadata": {}, 95 | "outputs": [], 96 | "source": [ 97 | "scaler = preprocessing.StandardScaler()\n", 98 | "X_train = scaler.fit_transform(X_train)\n", 99 | "X_test = scaler.transform(X_test)" 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": null, 105 | "metadata": {}, 106 | "outputs": [], 107 | "source": [ 108 | "model = Sequential()\n", 109 | "model.add(Dense(200, input_dim=X_train.shape[1]))\n", 110 | "model.add(Activation('relu'))\n", 111 | "model.add(Dropout(0.1))\n", 112 | "model.add(Dense(200))\n", 113 | "model.add(Activation('relu'))\n", 114 | "model.add(Dropout(0.1))\n", 115 | "model.add(Dense(1))\n", 116 | "\n", 117 | "opt = Adam()\n", 118 | "model.compile(loss='mean_squared_logarithmic_error', optimizer=opt)" 119 | ] 120 | }, 121 | { 122 | "cell_type": "code", 123 | "execution_count": null, 124 | "metadata": {}, 125 | "outputs": [], 126 | "source": [ 127 | "n_epochs = 1000\n", 128 | "batch_size = 128" 129 | ] 130 | }, 131 | { 132 | "cell_type": "code", 133 | "execution_count": null, 134 | "metadata": {}, 135 | "outputs": [], 136 | "source": [ 137 | "callbacks = [EarlyStopping(monitor='val_loss', patience=5)]\n", 138 | "history = model.fit(X_train, y_train, shuffle=True, epochs=n_epochs, batch_size=batch_size, validation_split=0.1, verbose=1)" 139 | ] 140 | }, 141 | { 142 | "cell_type": "code", 143 | "execution_count": null, 144 | "metadata": {}, 145 | "outputs": [], 146 | "source": [] 147 | } 148 | ], 149 | "metadata": { 150 | "kernelspec": { 151 | "display_name": "Python 3", 152 | "language": "python", 153 | "name": "python3" 154 | }, 155 | "language_info": { 156 | "codemirror_mode": { 157 | "name": "ipython", 158 | "version": 3 159 | }, 160 | "file_extension": ".py", 161 | "mimetype": "text/x-python", 162 | "name": "python", 163 | "nbconvert_exporter": "python", 164 | "pygments_lexer": "ipython3", 165 | "version": "3.5.2" 166 | }, 167 | "widgets": { 168 | "state": {}, 169 | "version": "1.1.2" 170 | } 171 | }, 172 | "nbformat": 4, 173 | "nbformat_minor": 2 174 | } 175 | -------------------------------------------------------------------------------- /Chapter10/Chapter 10 - Predicting stock prices with neural networks.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import matplotlib.pyplot as plt\n", 10 | "import numpy as np\n", 11 | "import pandas as pd\n", 12 | "from sklearn.preprocessing import MinMaxScaler\n", 13 | "\n", 14 | "from keras.layers.core import Dense, Activation, Dropout\n", 15 | "from keras.layers.recurrent import LSTM\n", 16 | "from keras.models import Sequential" 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": null, 22 | "metadata": {}, 23 | "outputs": [], 24 | "source": [ 25 | "# Data can be downloaded at https://ca.finance.yahoo.com/quote/%5EIXIC/, you can replace it with any other index or stock" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": null, 31 | "metadata": {}, 32 | "outputs": [], 33 | "source": [ 34 | "data = pd.read_csv('../Data/stock-data-2000-2017.csv')\n", 35 | "# Reorder the columns for convenience\n", 36 | "data = data[['Open', 'High', 'Low', 'Volume', 'Close']]\n", 37 | "data.head()" 38 | ] 39 | }, 40 | { 41 | "cell_type": "code", 42 | "execution_count": null, 43 | "metadata": {}, 44 | "outputs": [], 45 | "source": [ 46 | "sequence_length = 21 # 20 preceeding inputs\n", 47 | "n_features = len(data.columns)\n", 48 | "val_ratio = 0.1\n", 49 | "n_epochs = 300\n", 50 | "batch_size = 512" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": null, 56 | "metadata": {}, 57 | "outputs": [], 58 | "source": [ 59 | "data = data.as_matrix()\n", 60 | "data_processed = []\n", 61 | "for index in range(len(data) - sequence_length):\n", 62 | " data_processed.append(data[index : index + sequence_length])\n", 63 | "data_processed = np.array(data_processed)" 64 | ] 65 | }, 66 | { 67 | "cell_type": "code", 68 | "execution_count": null, 69 | "metadata": {}, 70 | "outputs": [], 71 | "source": [ 72 | "val_split = round((1-val_ratio) * data_processed.shape[0])\n", 73 | "train = data_processed[: int(val_split), :]\n", 74 | "val = data_processed[int(val_split) :, :]\n", 75 | "\n", 76 | "print('Training data: {}'.format(train.shape))\n", 77 | "print('Validation data: {}'.format(val.shape))" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": null, 83 | "metadata": {}, 84 | "outputs": [], 85 | "source": [ 86 | "train_samples, train_nx, train_ny = train.shape\n", 87 | "val_samples, val_nx, val_ny = val.shape\n", 88 | "\n", 89 | "train = train.reshape((train_samples, train_nx * train_ny))\n", 90 | "val = val.reshape((val_samples, val_nx * val_ny))\n", 91 | "\n", 92 | "preprocessor = MinMaxScaler().fit(train)\n", 93 | "train = preprocessor.transform(train)\n", 94 | "val = preprocessor.transform(val)\n", 95 | "\n", 96 | "train = train.reshape((train_samples, train_nx, train_ny))\n", 97 | "val = val.reshape((val_samples, val_nx, val_ny))" 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": null, 103 | "metadata": {}, 104 | "outputs": [], 105 | "source": [ 106 | "X_train = train[:, : -1]\n", 107 | "y_train = train[:, -1][: ,-1]\n", 108 | "X_val = val[:, : -1]\n", 109 | "y_val = val[:, -1][ : ,-1]\n", 110 | "\n", 111 | "X_train = np.reshape(X_train, (X_train.shape[0], X_train.shape[1], n_features))\n", 112 | "X_val = np.reshape(X_val, (X_val.shape[0], X_val.shape[1], n_features))" 113 | ] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": null, 118 | "metadata": {}, 119 | "outputs": [], 120 | "source": [ 121 | "model = Sequential()\n", 122 | "model.add(LSTM(input_shape=(X_train.shape[1:]), units = 128, return_sequences=True))\n", 123 | "model.add(Dropout(0.5))\n", 124 | "model.add(LSTM(128, return_sequences=False))\n", 125 | "model.add(Dropout(0.25))\n", 126 | "model.add(Dense(units=1))\n", 127 | "model.add(Activation(\"linear\"))\n", 128 | "\n", 129 | "model.compile(loss=\"mse\", optimizer=\"adam\")" 130 | ] 131 | }, 132 | { 133 | "cell_type": "code", 134 | "execution_count": null, 135 | "metadata": {}, 136 | "outputs": [], 137 | "source": [ 138 | "history = model.fit(\n", 139 | " X_train,\n", 140 | " y_train,\n", 141 | " batch_size=batch_size,\n", 142 | " epochs=n_epochs,\n", 143 | " verbose=1\n", 144 | " )" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": null, 150 | "metadata": {}, 151 | "outputs": [], 152 | "source": [ 153 | "preds_val = model.predict(X_val)\n", 154 | "diff = []\n", 155 | "for i in range(len(y_val)):\n", 156 | " pred = preds_val[i][0]\n", 157 | " diff.append(y_val[i] - pred)" 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": null, 163 | "metadata": {}, 164 | "outputs": [], 165 | "source": [ 166 | "real_min = preprocessor.data_min_[104]\n", 167 | "real_max = preprocessor.data_max_[104]\n", 168 | "print(preprocessor.data_min_[104])\n", 169 | "print(preprocessor.data_max_[104])\n", 170 | "\n", 171 | "preds_real = preds_val * (real_max - real_min) + real_min\n", 172 | "y_val_real = y_val * (real_max - real_min) + real_min" 173 | ] 174 | }, 175 | { 176 | "cell_type": "code", 177 | "execution_count": null, 178 | "metadata": {}, 179 | "outputs": [], 180 | "source": [ 181 | "plt.plot(preds_real, label='Predictions')\n", 182 | "plt.plot(y_val_real, label='Actual values')\n", 183 | "plt.xlabel('test')\n", 184 | "plt.legend(loc=0)\n", 185 | "plt.show()" 186 | ] 187 | } 188 | ], 189 | "metadata": { 190 | "kernelspec": { 191 | "display_name": "Python 3", 192 | "language": "python", 193 | "name": "python3" 194 | }, 195 | "language_info": { 196 | "codemirror_mode": { 197 | "name": "ipython", 198 | "version": 3 199 | }, 200 | "file_extension": ".py", 201 | "mimetype": "text/x-python", 202 | "name": "python", 203 | "nbconvert_exporter": "python", 204 | "pygments_lexer": "ipython3", 205 | "version": "3.5.2" 206 | }, 207 | "widgets": { 208 | "state": {}, 209 | "version": "1.1.2" 210 | } 211 | }, 212 | "nbformat": 4, 213 | "nbformat_minor": 2 214 | } 215 | -------------------------------------------------------------------------------- /Chapter11/Chapter 11 - Learning to drive a car with end-to-end learning.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import pandas as pd\n", 10 | "import numpy as np\n", 11 | "import cv2\n", 12 | "import math\n", 13 | "\n", 14 | "from keras.models import Sequential\n", 15 | "from keras.layers.core import Dense, Dropout, Activation, Lambda\n", 16 | "from keras.layers import Input, ELU\n", 17 | "from keras.optimizers import SGD, Adam\n", 18 | "from keras.utils import np_utils\n", 19 | "from keras.layers import Conv2D, MaxPooling2D, Flatten\n", 20 | "from keras import initializers\n", 21 | "from keras.callbacks import ModelCheckpoint" 22 | ] 23 | }, 24 | { 25 | "cell_type": "code", 26 | "execution_count": null, 27 | "metadata": {}, 28 | "outputs": [], 29 | "source": [ 30 | "data_path = 'Data/SDC/training.csv'\n", 31 | "data = pd.read_csv(data_path, header=None, skiprows=[0], names=['center', 'left', 'right', 'steering', 'throttle', 'brake', 'speed'])" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": null, 37 | "metadata": {}, 38 | "outputs": [], 39 | "source": [ 40 | "img_cols = 64\n", 41 | "img_rows = 64\n", 42 | "img_channels = 3" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": null, 48 | "metadata": {}, 49 | "outputs": [], 50 | "source": [ 51 | "def augment_image(image):\n", 52 | " image = cv2.cvtColor(image,cv2.COLOR_RGB2HSV)\n", 53 | " random_bright = .25 + np.random.uniform()\n", 54 | " image[:,:,2] = image[:,:,2] * random_bright\n", 55 | " image = cv2.cvtColor(image,cv2.COLOR_HSV2RGB)\n", 56 | " return(image)" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": null, 62 | "metadata": {}, 63 | "outputs": [], 64 | "source": [ 65 | "def preprocess_image(data):\n", 66 | " # Randomly pick left, center or right\n", 67 | " # camera image\n", 68 | " rand = np.random.randint(3)\n", 69 | " if (rand == 0):\n", 70 | " path_file = data['left'][0].strip()\n", 71 | " shift_ang = .25\n", 72 | " if (rand == 1):\n", 73 | " path_file = data['center'][0].strip()\n", 74 | " shift_ang = 0.\n", 75 | " if (rand == 2):\n", 76 | " path_file = data['right'][0].strip()\n", 77 | " shift_ang = -.25\n", 78 | " y = data['steering'][0] + shift_ang\n", 79 | "\n", 80 | " # Read image\n", 81 | " image = cv2.imread(path_file)\n", 82 | " image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n", 83 | "\n", 84 | " # Crop image\n", 85 | " shape = image.shape\n", 86 | " image = image[math.floor(shape[0]/4):\n", 87 | " shape[0]-20, 0:shape[1]]\n", 88 | "\n", 89 | " # Resize image\n", 90 | " image = cv2.resize(image, (img_cols, img_rows),\n", 91 | " interpolation=cv2.INTER_AREA) \n", 92 | "\n", 93 | " # Augment image\n", 94 | " image = augment_image(image)\n", 95 | " image = np.array(image)\n", 96 | "\n", 97 | " if np.random.choice([True, False]):\n", 98 | " image = cv2.flip(image,1)\n", 99 | " y = -y\n", 100 | " \n", 101 | " return(image, y)" 102 | ] 103 | }, 104 | { 105 | "cell_type": "code", 106 | "execution_count": null, 107 | "metadata": {}, 108 | "outputs": [], 109 | "source": [ 110 | "def batch_gen(data, batch_size):\n", 111 | " # Create empty numpy arrays\n", 112 | " batch_images = np.zeros((batch_size, img_rows,\n", 113 | " img_cols, img_channels))\n", 114 | " batch_steering = np.zeros(batch_size)\n", 115 | "\n", 116 | " small_steering_threshold = 0.8\n", 117 | "\n", 118 | " # Custom batch generator \n", 119 | " while 1:\n", 120 | " for n in range(batch_size):\n", 121 | " i = np.random.randint(len(data))\n", 122 | " data_sub = data.iloc[[i]].reset_index()\n", 123 | " # Only keep training data with small \n", 124 | " #steering angles with probablitiy \n", 125 | " keep = False\n", 126 | " while keep == False:\n", 127 | " x, y = preprocess_image(data_sub)\n", 128 | " pr_unif = np.random\n", 129 | " if abs(y) < .01:\n", 130 | " next;\n", 131 | " if abs(y) < .10:\n", 132 | " small_steering_rand = np.random.uniform()\n", 133 | " if small_steering_rand > \n", 134 | " small_steering_threshold:\n", 135 | " keep = True\n", 136 | " else:\n", 137 | " keep = True\n", 138 | "\n", 139 | " batch_images[n] = x\n", 140 | " batch_steering[n] = y\n", 141 | " yield batch_images, batch_steering" 142 | ] 143 | }, 144 | { 145 | "cell_type": "code", 146 | "execution_count": null, 147 | "metadata": {}, 148 | "outputs": [], 149 | "source": [ 150 | "input_shape = (img_rows, img_cols, img_channels)\n", 151 | "model = Sequential()\n", 152 | "model.add(Lambda(lambda x: x/255.-0.5, \n", 153 | "input_shape=input_shape))\n", 154 | "model.add(Conv2D(3, (1, 1), padding='valid', kernel_initializer='he_normal'))\n", 155 | "model.add(ELU())\n", 156 | "model.add(Conv2D(32, (3, 3), padding='valid', kernel_initializer='he_normal'))\n", 157 | "model.add(ELU())\n", 158 | "model.add(Conv2D(32, (3, 3), padding='valid', kernel_initializer='he_normal'))\n", 159 | "model.add(ELU())\n", 160 | "model.add(MaxPooling2D(pool_size=(2, 2)))\n", 161 | "model.add(Dropout(0.5))\n", 162 | "model.add(Conv2D(64, (3, 3), padding='valid', kernel_initializer='he_normal'))\n", 163 | "model.add(ELU())\n", 164 | "model.add(Conv2D(64, (3, 3), padding='valid', kernel_initializer='he_normal'))\n", 165 | "model.add(ELU())\n", 166 | "model.add(MaxPooling2D(pool_size=(2, 2)))\n", 167 | "model.add(Dropout(0.5))\n", 168 | "model.add(Conv2D(128, (3, 3), padding='valid', kernel_initializer='he_normal'))\n", 169 | "model.add(ELU())\n", 170 | "model.add(Conv2D(128, (3, 3), padding='valid', kernel_initializer='he_normal'))\n", 171 | "model.add(ELU())\n", 172 | "model.add(MaxPooling2D(pool_size=(2, 2)))\n", 173 | "model.add(Dropout(0.5))\n", 174 | "model.add(Flatten())\n", 175 | "model.add(Dense(512, kernel_initializer='he_normal', activation='relu'))\n", 176 | "model.add(ELU())\n", 177 | "model.add(Dropout(0.5))\n", 178 | "model.add(Dense(64, kernel_initializer='he_normal'))\n", 179 | "model.add(ELU())\n", 180 | "model.add(Dropout(0.5))\n", 181 | "model.add(Dense(16, kernel_initializer='he_normal'))\n", 182 | "model.add(ELU())\n", 183 | "model.add(Dropout(0.5))\n", 184 | "model.add(Dense(1, kernel_initializer='he_normal'))\n", 185 | "\n", 186 | "opt = Adam(lr=1e-4, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)\n", 187 | "model.compile(optimizer=opt, loss='mse')\n", 188 | "model.summary()" 189 | ] 190 | }, 191 | { 192 | "cell_type": "code", 193 | "execution_count": null, 194 | "metadata": {}, 195 | "outputs": [], 196 | "source": [ 197 | "callbacks = [ModelCheckpoint('checkpoints/sdc.h5', save_best_only=False)]" 198 | ] 199 | }, 200 | { 201 | "cell_type": "code", 202 | "execution_count": null, 203 | "metadata": {}, 204 | "outputs": [], 205 | "source": [ 206 | "train_generator = batchgen(data, batch_size)\n", 207 | "val_generator = batchgen(data, batch_size)" 208 | ] 209 | }, 210 | { 211 | "cell_type": "code", 212 | "execution_count": null, 213 | "metadata": {}, 214 | "outputs": [], 215 | "source": [ 216 | "n_epochs = 8\n", 217 | "batch_size = 64\n", 218 | "steps_per_epoch = 500\n", 219 | "validation_steps = 50" 220 | ] 221 | }, 222 | { 223 | "cell_type": "code", 224 | "execution_count": null, 225 | "metadata": {}, 226 | "outputs": [], 227 | "source": [ 228 | "history = model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=n_epochs, validation_data=val_generator, validation_steps=validation_steps, callbacks=callbacks)" 229 | ] 230 | } 231 | ], 232 | "metadata": { 233 | "kernelspec": { 234 | "display_name": "Python 3", 235 | "language": "python", 236 | "name": "python3" 237 | }, 238 | "language_info": { 239 | "codemirror_mode": { 240 | "name": "ipython", 241 | "version": 3 242 | }, 243 | "file_extension": ".py", 244 | "mimetype": "text/x-python", 245 | "name": "python", 246 | "nbconvert_exporter": "python", 247 | "pygments_lexer": "ipython3", 248 | "version": "3.5.2" 249 | }, 250 | "widgets": { 251 | "state": {}, 252 | "version": "1.1.2" 253 | } 254 | }, 255 | "nbformat": 4, 256 | "nbformat_minor": 2 257 | } 258 | -------------------------------------------------------------------------------- /Chapter12/Chapter 12 - Adding dropouts to prevent overfitting.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "from keras.models import Sequential\n", 10 | "from keras.layers import Dense, Dropout, Activation, Flatten\n", 11 | "from keras.layers import Conv2D, MaxPooling2D\n", 12 | "from keras.optimizers import Adam\n", 13 | "from sklearn.model_selection import train_test_split\n", 14 | "from keras.utils import to_categorical\n", 15 | "from keras.callbacks import EarlyStopping, TensorBoard, ModelCheckpoint\n", 16 | "\n", 17 | "from keras.datasets import cifar10" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": null, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "(X_train, y_train), (X_test, y_test) = cifar10.load_data()\n", 27 | "validation_split = 0.1\n", 28 | "X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=validation_split, random_state=SEED)\n", 29 | "\n", 30 | "X_train = X_train.astype('float32')\n", 31 | "X_train /=255.\n", 32 | "X_val = X_val.astype('float32')\n", 33 | "X_val /=255.\n", 34 | "X_test = X_test.astype('float32')\n", 35 | "X_test /=255.\n", 36 | "\n", 37 | "n_classes = 10\n", 38 | "y_train = to_categorical(y_train, n_classes)\n", 39 | "y_val = to_categorical(y_val, n_classes)\n", 40 | "y_test = to_categorical(y_test, n_classes)" 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": null, 46 | "metadata": {}, 47 | "outputs": [], 48 | "source": [ 49 | "callbacks =[EarlyStopping(monitor='val_acc', patience=5, verbose=2),\n", 50 | " ModelCheckpoint('checkpoints/{epoch:02d}.h5', save_best_only=True),\n", 51 | " TensorBoard('~/notebooks/logs-lrscheduler', write_graph=True, write_grads=True, write_images=True, embeddings_freq=0, embeddings_layer_names=None, embeddings_metadata=None),\n", 52 | " ]" 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": null, 58 | "metadata": {}, 59 | "outputs": [], 60 | "source": [ 61 | "model = Sequential()\n", 62 | "model.add(Conv2D(32, (3, 3), padding='same', input_shape=X_train.shape[1:]))\n", 63 | "model.add(Activation('relu'))\n", 64 | "model.add(Conv2D(32, (3, 3)))\n", 65 | "model.add(Activation('relu'))\n", 66 | "model.add(MaxPooling2D(pool_size=(2, 2)))\n", 67 | "\n", 68 | "model.add(Conv2D(64, (3, 3), padding='same'))\n", 69 | "model.add(Activation('relu'))\n", 70 | "model.add(Conv2D(64, (3, 3)))\n", 71 | "model.add(Activation('relu'))\n", 72 | "model.add(MaxPooling2D(pool_size=(2, 2)))\n", 73 | "\n", 74 | "model.add(Flatten())\n", 75 | "model.add(Dense(512))\n", 76 | "model.add(Activation('relu'))\n", 77 | "model.add(Dense(n_classes))\n", 78 | "model.add(Activation('softmax'))\n", 79 | "optimizer = SGD()\n", 80 | "model.compile(loss='categorical_crossentropy', \n", 81 | "optimizer=optimizer, metrics=['accuracy'])" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "metadata": {}, 88 | "outputs": [], 89 | "source": [ 90 | "n_epochs = 1000\n", 91 | "batch_size = 128\n", 92 | "\n", 93 | "history = model.fit(X_train, y_train, epochs=n_epochs, batch_size=batch_size, \n", 94 | " validation_data=[X_val, y_val],\n", 95 | " verbose = 1, callbacks=callbacks)" 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": null, 101 | "metadata": {}, 102 | "outputs": [], 103 | "source": [ 104 | "model_dropout = Sequential()\n", 105 | "model_dropout.add(Conv2D(32, (3, 3), padding='same', input_shape=X_train.shape[1:]))\n", 106 | "model_dropout.add(Activation('relu'))\n", 107 | "model_dropout.add(Conv2D(32, (3, 3)))\n", 108 | "model_dropout.add(Activation('relu'))\n", 109 | "model_dropout.add(MaxPooling2D(pool_size=(2, 2)))\n", 110 | "model_dropout.add(Dropout(0.25))\n", 111 | "\n", 112 | "model_dropout.add(Conv2D(64, (3, 3), padding='same'))\n", 113 | "model_dropout.add(Activation('relu'))\n", 114 | "model_dropout.add(Conv2D(64, (3, 3)))\n", 115 | "model_dropout.add(Activation('relu'))\n", 116 | "model_dropout.add(MaxPooling2D(pool_size=(2, 2)))\n", 117 | "model_dropout.add(Dropout(0.25))\n", 118 | "\n", 119 | "model_dropout.add(Flatten())\n", 120 | "model_dropout.add(Dense(512))\n", 121 | "model_dropout.add(Activation('relu'))\n", 122 | "model_dropout.add(Dropout(0.5))\n", 123 | "model_dropout.add(Dense(n_classes))\n", 124 | "model_dropout.add(Activation('softmax'))\n", 125 | "optimizer = Adam()\n", 126 | "model_dropout.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": null, 132 | "metadata": {}, 133 | "outputs": [], 134 | "source": [ 135 | "n_epochs = 1000\n", 136 | "batch_size = 128\n", 137 | "\n", 138 | "history_dropout = model_dropout.fit(X_train, y_train, epochs=n_epochs, batch_size=batch_size, \n", 139 | " validation_data=[X_val, y_val],\n", 140 | " verbose = 1, callbacks=callbacks)" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": null, 146 | "metadata": {}, 147 | "outputs": [], 148 | "source": [ 149 | "plt.plot(np.arange(len(history.history['acc'])), history.history['acc'], label='training')\n", 150 | "plt.plot(np.arange(len(history.history['val_acc'])), history.history['val_acc'], label='validation')\n", 151 | "plt.title('Accuracy of model without dropouts')\n", 152 | "plt.xlabel('epochs')\n", 153 | "plt.ylabel('accuracy')\n", 154 | "plt.legend(loc=0)\n", 155 | "plt.show()" 156 | ] 157 | }, 158 | { 159 | "cell_type": "code", 160 | "execution_count": null, 161 | "metadata": {}, 162 | "outputs": [], 163 | "source": [ 164 | "plt.plot(np.arange(len(history_dropout.history['acc'])), history_dropout.history['acc'], label='training')\n", 165 | "plt.plot(np.arange(len(history_dropout.history['val_acc'])), history_dropout.history['val_acc'], label='validation')\n", 166 | "plt.title('Accuracy of model with dropouts')\n", 167 | "plt.xlabel('epochs')\n", 168 | "plt.ylabel('accuracy')\n", 169 | "plt.legend(loc=0)\n", 170 | "plt.show()" 171 | ] 172 | } 173 | ], 174 | "metadata": { 175 | "kernelspec": { 176 | "display_name": "Python 3", 177 | "language": "python", 178 | "name": "python3" 179 | }, 180 | "language_info": { 181 | "codemirror_mode": { 182 | "name": "ipython", 183 | "version": 3 184 | }, 185 | "file_extension": ".py", 186 | "mimetype": "text/x-python", 187 | "name": "python", 188 | "nbconvert_exporter": "python", 189 | "pygments_lexer": "ipython3", 190 | "version": "3.5.2" 191 | }, 192 | "widgets": { 193 | "state": {}, 194 | "version": "1.1.2" 195 | } 196 | }, 197 | "nbformat": 4, 198 | "nbformat_minor": 2 199 | } 200 | -------------------------------------------------------------------------------- /Chapter12/Chapter 12 - Comparing optimizers.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np\n", 10 | "import pandas as pd\n", 11 | "from matplotlib import pyplot as plt\n", 12 | "\n", 13 | "from sklearn.model_selection import train_test_split\n", 14 | "from keras.models import Sequential\n", 15 | "from keras.layers import Dense, Dropout\n", 16 | "from keras.wrappers.scikit_learn import KerasRegressor\n", 17 | "from keras.callbacks import EarlyStopping, ModelCheckpoint\n", 18 | "from keras.optimizers import SGD, Adadelta, Adam, RMSprop, Adagrad, Nadam, Adamax" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "(X_train, y_train), (X_test, y_test) = cifar10.load_data()\n", 28 | "validation_split = 0.1\n", 29 | "X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=validation_split, random_state=SEED)\n", 30 | "\n", 31 | "X_train = X_train.astype('float32')\n", 32 | "X_train /=255.\n", 33 | "X_val = X_val.astype('float32')\n", 34 | "X_val /=255.\n", 35 | "X_test = X_test.astype('float32')\n", 36 | "X_test /=255.\n", 37 | "\n", 38 | "n_classes = 10\n", 39 | "y_train = to_categorical(y_train, n_classes)\n", 40 | "y_val = to_categorical(y_val, n_classes)\n", 41 | "y_test = to_categorical(y_test, n_classes)" 42 | ] 43 | }, 44 | { 45 | "cell_type": "code", 46 | "execution_count": null, 47 | "metadata": {}, 48 | "outputs": [], 49 | "source": [ 50 | "def create_model(opt): \n", 51 | " model = Sequential()\n", 52 | " model.add(Conv2D(32, (3, 3), padding='same', input_shape=x_train.shape[1:]))\n", 53 | " model.add(Activation('relu'))\n", 54 | " model.add(Conv2D(32, (3, 3)))\n", 55 | " model.add(Activation('relu'))\n", 56 | " model.add(MaxPooling2D(pool_size=(2, 2)))\n", 57 | " model.add(Dropout(0.25))\n", 58 | "\n", 59 | " model.add(Conv2D(64, (3, 3), padding='same'))\n", 60 | " model.add(Activation('relu'))\n", 61 | " model.add(Conv2D(64, (3, 3)))\n", 62 | " model.add(Activation('relu'))\n", 63 | " model.add(MaxPooling2D(pool_size=(2, 2)))\n", 64 | " model.add(Dropout(0.25))\n", 65 | "\n", 66 | " model.add(Flatten())\n", 67 | " model.add(Dense(512))\n", 68 | " model.add(Activation('relu'))\n", 69 | " model.add(Dropout(0.5))\n", 70 | " model.add(Dense(num_classes))\n", 71 | " model.add(Activation('softmax'))\n", 72 | " return model" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": null, 78 | "metadata": {}, 79 | "outputs": [], 80 | "source": [ 81 | "def create_callbacks(opt):\n", 82 | " callbacks = [EarlyStopping(monitor='val_acc', patience=5, verbose=2),\n", 83 | " ModelCheckpoint('checkpoints/weights.{epoch:02d}-'+opt+'.h5', save_best_only=False, verbose=True),\n", 84 | " TensorBoard()]\n", 85 | " return callbacks" 86 | ] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": null, 91 | "metadata": {}, 92 | "outputs": [], 93 | "source": [ 94 | "opts = dict({\n", 95 | " 'sgd': SGD(),\n", 96 | " 'sgd-0001': SGD(lr=0.0001, decay=0.00001),\n", 97 | " 'adam': Adam(),\n", 98 | " 'adam': Adam(lr=0.0001),\n", 99 | " 'adadelta': Adadelta(),\n", 100 | " 'rmsprop': RMSprop(),\n", 101 | " 'rmsprop-0001': RMSprop(lr=0.0001),\n", 102 | " 'nadam': Nadam(),\n", 103 | " 'adamax': Adamax()\n", 104 | " })" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": null, 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "n_epochs = 1000\n", 114 | "batch_size = 128\n", 115 | "\n", 116 | "results = []\n", 117 | "# Loop through the optimizers\n", 118 | "for opt in opts:\n", 119 | " model = create_model(opt)\n", 120 | " callbacks = create_callbacks(opt)\n", 121 | " model.compile(loss='categorical_crossentropy', optimizer=opts[opt], metrics=['accuracy'])\n", 122 | " hist = model.fit(X_train, y_train, batch_size=batch_size, epochs=n_epochs,\n", 123 | " validation_data=(X_val, y_val),\n", 124 | " verbose=1,\n", 125 | " callbacks=callbacks)\n", 126 | " best_epoch = np.argmax(hist.history['val_acc'])\n", 127 | " best_acc = hist.history['val_acc'][best_epoch] \n", 128 | " best_model = create_model(opt)\n", 129 | " # Load the model weights with the highest validation accuracy \n", 130 | " best_model.load_weights('checkpoints/weights.{:02d}-{}.h5'.format(best_epoch, opt))\n", 131 | " best_model.compile(loss='mse', optimizer=opts[opt], metrics=['accuracy'])\n", 132 | " score = best_model.evaluate(X_test, y_test, verbose=0)\n", 133 | " results.append([opt, best_epoch, best_acc, score[1]])" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": null, 139 | "metadata": {}, 140 | "outputs": [], 141 | "source": [ 142 | "res = pd.DataFrame(results)\n", 143 | "res.columns = ['optimizer', 'epochs', 'val_accuracy', 'test_last', 'test_accuracy']\n", 144 | "res" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": null, 150 | "metadata": {}, 151 | "outputs": [], 152 | "source": [] 153 | } 154 | ], 155 | "metadata": { 156 | "kernelspec": { 157 | "display_name": "Python 3", 158 | "language": "python", 159 | "name": "python3" 160 | }, 161 | "language_info": { 162 | "codemirror_mode": { 163 | "name": "ipython", 164 | "version": 3 165 | }, 166 | "file_extension": ".py", 167 | "mimetype": "text/x-python", 168 | "name": "python", 169 | "nbconvert_exporter": "python", 170 | "pygments_lexer": "ipython3", 171 | "version": "3.5.2" 172 | }, 173 | "widgets": { 174 | "state": {}, 175 | "version": "1.1.2" 176 | } 177 | }, 178 | "nbformat": 4, 179 | "nbformat_minor": 2 180 | } 181 | -------------------------------------------------------------------------------- /Chapter12/Chapter 12 - Learning rates and learning rate schedulers.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import math\n", 10 | "\n", 11 | "from keras.models import Sequential\n", 12 | "from keras.layers import Dense, Dropout, Activation, Flatten\n", 13 | "from keras.layers import Conv2D, MaxPooling2D\n", 14 | "from keras.optimizers import SGD\n", 15 | "from sklearn.model_selection import train_test_split\n", 16 | "from keras.utils import to_categorical\n", 17 | "from keras.callbacks import EarlyStopping, TensorBoard, ModelCheckpoint, LearningRateScheduler, Callback\n", 18 | "from keras import backend as K\n", 19 | "\n", 20 | "from keras.datasets import cifar10" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": null, 26 | "metadata": {}, 27 | "outputs": [], 28 | "source": [ 29 | "(X_train, y_train), (X_test, y_test) = cifar10.load_data()\n", 30 | "validation_split = 0.1\n", 31 | "X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=validation_split, random_state=SEED)\n", 32 | "\n", 33 | "X_train = X_train.astype('float32')\n", 34 | "X_train /=255.\n", 35 | "X_val = X_val.astype('float32')\n", 36 | "X_val /=255.\n", 37 | "X_test = X_test.astype('float32')\n", 38 | "X_test /=255.\n", 39 | "\n", 40 | "n_classes = 10\n", 41 | "y_train = to_categorical(y_train, n_classes)\n", 42 | "y_val = to_categorical(y_val, n_classes)\n", 43 | "y_test = to_categorical(y_test, n_classes)" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": null, 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "learning_rate_schedule = {0: '0.1', 10: '0.01', 25: '0.0025'}\n", 53 | "\n", 54 | "class get_learning_rate(Callback):\n", 55 | " def on_epoch_end(self, epoch, logs={}):\n", 56 | " optimizer = self.model.optimizer\n", 57 | " if epoch in learning_rate_schedule:\n", 58 | " K.set_value(optimizer.lr, learning_rate_schedule[epoch])\n", 59 | " lr = K.eval(optimizer.lr)\n", 60 | " print('\\nlr: {:.4f}'.format(lr))" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": null, 66 | "metadata": {}, 67 | "outputs": [], 68 | "source": [ 69 | "callbacks =[EarlyStopping(monitor='val_acc', patience=5, verbose=2),\n", 70 | " ModelCheckpoint('checkpoints/{epoch:02d}.h5', \n", 71 | " save_best_only=True),\n", 72 | " TensorBoard('~/notebooks/logs-lrscheduler'),\n", 73 | " get_learning_rate()\n", 74 | " ]" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": null, 80 | "metadata": {}, 81 | "outputs": [], 82 | "source": [ 83 | "model = Sequential()\n", 84 | "model.add(Conv2D(32, (3, 3), padding='same', input_shape=X_train.shape[1:]))\n", 85 | "model.add(Activation('relu'))\n", 86 | "model.add(Conv2D(32, (3, 3)))\n", 87 | "model.add(Activation('relu'))\n", 88 | "model.add(MaxPooling2D(pool_size=(2, 2)))\n", 89 | "model.add(Dropout(0.25))\n", 90 | "\n", 91 | "model.add(Conv2D(64, (3, 3), padding='same'))\n", 92 | "model.add(Activation('relu'))\n", 93 | "model.add(Conv2D(64, (3, 3)))\n", 94 | "model.add(Activation('relu'))\n", 95 | "model.add(MaxPooling2D(pool_size=(2, 2)))\n", 96 | "model.add(Dropout(0.25))\n", 97 | "\n", 98 | "model.add(Flatten())\n", 99 | "model.add(Dense(512))\n", 100 | "model.add(Activation('relu'))\n", 101 | "model.add(Dropout(0.5))\n", 102 | "model.add(Dense(n_classes))\n", 103 | "model.add(Activation('softmax'))\n", 104 | "optimizer = SGD()\n", 105 | "model.compile(loss='categorical_crossentropy',\n", 106 | "optimizer=optimizer, metrics=['accuracy'])" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": null, 112 | "metadata": {}, 113 | "outputs": [], 114 | "source": [ 115 | "n_epochs = 20\n", 116 | "batch_size = 128\n", 117 | "\n", 118 | "history = model.fit(X_train, y_train, epochs=n_epochs, batch_size=batch_size, \n", 119 | " validation_data=[X_val, y_val],\n", 120 | " verbose = 1, callbacks=callbacks)" 121 | ] 122 | } 123 | ], 124 | "metadata": { 125 | "kernelspec": { 126 | "display_name": "Python 3", 127 | "language": "python", 128 | "name": "python3" 129 | }, 130 | "language_info": { 131 | "codemirror_mode": { 132 | "name": "ipython", 133 | "version": 3 134 | }, 135 | "file_extension": ".py", 136 | "mimetype": "text/x-python", 137 | "name": "python", 138 | "nbconvert_exporter": "python", 139 | "pygments_lexer": "ipython3", 140 | "version": "3.5.2" 141 | }, 142 | "widgets": { 143 | "state": {}, 144 | "version": "1.1.2" 145 | } 146 | }, 147 | "nbformat": 4, 148 | "nbformat_minor": 2 149 | } 150 | -------------------------------------------------------------------------------- /Chapter12/Chapter 12 - Making a model more robust with data augmentation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "from keras.preprocessing.image import ImageDataGenerator\n", 10 | "from keras.models import Sequential\n", 11 | "from keras.layers import Dense, Dropout, Activation, Flatten\n", 12 | "from keras.layers import Conv2D, MaxPooling2D\n", 13 | "from keras.optimizers import Adam\n", 14 | "from keras.callbacks import EarlyStopping, TensorBoard, ModelCheckpoint\n", 15 | "\n", 16 | "from keras.datasets import cifar10" 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": null, 22 | "metadata": {}, 23 | "outputs": [], 24 | "source": [ 25 | "(X_train, y_train), (X_val, y_val) = cifar10.load_data()\n", 26 | "\n", 27 | "X_train = X_train.astype('float32')/255.\n", 28 | "X_val = X_val.astype('float32')/255.\n", 29 | "n_classes = 10\n", 30 | "y_train = keras.utils.to_categorical(y_train, n_classes)\n", 31 | "y_val = keras.utils.to_categorical(y_val, n_classes)" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": null, 37 | "metadata": {}, 38 | "outputs": [], 39 | "source": [ 40 | "model = Sequential()\n", 41 | "model.add(Conv2D(32, (3, 3), padding='same', input_shape=X_train.shape[1:]))\n", 42 | "model.add(Activation('relu'))\n", 43 | "model.add(Conv2D(32, (3, 3)))\n", 44 | "model.add(Activation('relu'))\n", 45 | "model.add(MaxPooling2D(pool_size=(2, 2)))\n", 46 | "model.add(Dropout(0.25))\n", 47 | "\n", 48 | "model.add(Conv2D(64, (3, 3), padding='same'))\n", 49 | "model.add(Activation('relu'))\n", 50 | "model.add(Conv2D(64, (3, 3)))\n", 51 | "model.add(Activation('relu'))\n", 52 | "model.add(MaxPooling2D(pool_size=(2, 2)))\n", 53 | "model.add(Dropout(0.25))\n", 54 | "\n", 55 | "model.add(Flatten())\n", 56 | "model.add(Dense(512))\n", 57 | "model.add(Activation('relu'))\n", 58 | "model.add(Dropout(0.5))\n", 59 | "model.add(Dense(n_classes))\n", 60 | "model.add(Activation('softmax'))" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": null, 66 | "metadata": {}, 67 | "outputs": [], 68 | "source": [ 69 | "opt = Adam(lr=0.0001)\n", 70 | "model.compile(loss='categorical_crossentropy',\n", 71 | "optimizer=opt, metrics=['accuracy'])" 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": null, 77 | "metadata": {}, 78 | "outputs": [], 79 | "source": [ 80 | "datagen = ImageDataGenerator(\n", 81 | " rotation_range=15,\n", 82 | " width_shift_range=0.15,\n", 83 | " height_shift_range=0.15,\n", 84 | " horizontal_flip=True,\n", 85 | " vertical_flip=False)\n", 86 | "\n", 87 | "datagen.fit(X_train)" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": null, 93 | "metadata": {}, 94 | "outputs": [], 95 | "source": [ 96 | "callbacks = [EarlyStopping(monitor='val_acc', patience=5, verbose=2), ModelCheckpoint('checkpoints/weights.{epoch:02d}-'+str(batch_size)+'.hdf5', save_best_only=True),\n", 97 | " TensorBoard('~/notebooks/logs-lrscheduler')\n", 98 | " ]" 99 | ] 100 | }, 101 | { 102 | "cell_type": "code", 103 | "execution_count": null, 104 | "metadata": {}, 105 | "outputs": [], 106 | "source": [ 107 | "batch_size = 128\n", 108 | "n_epochs = 1000\n", 109 | "history = model.fit_generator(datagen.flow(X_train, y_train,\n", 110 | " batch_size=batch_size),\n", 111 | " steps_per_epoch=X_train.shape[0] // batch_size,\n", 112 | " epochs=epochs,\n", 113 | " validation_data=(X_val, y_val),\n", 114 | " callbacks=callbacks\n", 115 | " )" 116 | ] 117 | } 118 | ], 119 | "metadata": { 120 | "kernelspec": { 121 | "display_name": "Python 3", 122 | "language": "python", 123 | "name": "python3" 124 | }, 125 | "language_info": { 126 | "codemirror_mode": { 127 | "name": "ipython", 128 | "version": 3 129 | }, 130 | "file_extension": ".py", 131 | "mimetype": "text/x-python", 132 | "name": "python", 133 | "nbconvert_exporter": "python", 134 | "pygments_lexer": "ipython3", 135 | "version": "3.5.2" 136 | }, 137 | "widgets": { 138 | "state": {}, 139 | "version": "1.1.2" 140 | } 141 | }, 142 | "nbformat": 4, 143 | "nbformat_minor": 2 144 | } 145 | -------------------------------------------------------------------------------- /Chapter12/Chapter 12 - Using grid search for parameter tuning.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import sys\n", 10 | "import numpy as np\n", 11 | "\n", 12 | "from hyperopt import fmin, tpe, hp, STATUS_OK, Trials\n", 13 | "\n", 14 | "from keras.models import Sequential\n", 15 | "from keras.layers import Dense, Dropout, Activation, Flatten\n", 16 | "from keras.layers import Conv2D, MaxPooling2D\n", 17 | "from keras.optimizers import Adam\n", 18 | "from sklearn.model_selection import train_test_split\n", 19 | "from keras.utils import to_categorical\n", 20 | "from keras.callbacks import EarlyStopping, TensorBoard, ModelCheckpoint\n", 21 | "\n", 22 | "from keras.datasets import cifar10" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": null, 28 | "metadata": {}, 29 | "outputs": [], 30 | "source": [ 31 | "(X_train, y_train), (X_test, y_test) = cifar10.load_data()\n", 32 | "validation_split = 0.1\n", 33 | "X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=validation_split, random_state=SEED)\n", 34 | "\n", 35 | "X_train = X_train.astype('float32')\n", 36 | "X_train /=255.\n", 37 | "X_val = X_val.astype('float32')\n", 38 | "X_val /=255.\n", 39 | "X_test = X_test.astype('float32')\n", 40 | "X_test /=255.\n", 41 | "\n", 42 | "n_classes = 10\n", 43 | "y_train = to_categorical(y_train, n_classes)\n", 44 | "y_val = to_categorical(y_val, n_classes)\n", 45 | "y_test = to_categorical(y_test, n_classes)" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": null, 51 | "metadata": {}, 52 | "outputs": [], 53 | "source": [ 54 | "space = {'batch_size' : hp.choice('batch_size', [32, 64, 128,256]), 'n_epochs' : 1000}" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": {}, 61 | "outputs": [], 62 | "source": [ 63 | "def get_callbacks(pars):\n", 64 | " callbacks =[EarlyStopping(monitor='val_acc', p atience=5, verbose=2),\n", 65 | " ModelCheckpoint('checkpoints/{}.h5'.format(pars['batch_size']), save_best_only=True),\n", 66 | " TensorBoard('~/notebooks/logs-gridsearch', write_graph=True, write_grads=True, write_images=True, embeddings_freq=0, embeddings_layer_names=None, embeddings_metadata=None)]\n", 67 | " return callbacks" 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": null, 73 | "metadata": {}, 74 | "outputs": [], 75 | "source": [ 76 | "def f_nn(pars): \n", 77 | " print ('Parameters: ', pars)\n", 78 | " model = Sequential()\n", 79 | " model.add(Conv2D(32, (3, 3), padding='same', input_shape=X_train.shape[1:]))\n", 80 | " model.add(Activation('relu'))\n", 81 | " model.add(Conv2D(32, (3, 3)))\n", 82 | " model.add(Activation('relu'))\n", 83 | " model.add(MaxPooling2D(pool_size=(2, 2)))\n", 84 | " model.add(Dropout(0.25))\n", 85 | "\n", 86 | " model.add(Conv2D(64, (3, 3), padding='same'))\n", 87 | " model.add(Activation('relu'))\n", 88 | " model.add(Conv2D(64, (3, 3)))\n", 89 | " model.add(Activation('relu'))\n", 90 | " model.add(MaxPooling2D(pool_size=(2, 2)))\n", 91 | " model.add(Dropout(0.25))\n", 92 | "\n", 93 | " model.add(Flatten())\n", 94 | " model.add(Dense(512))\n", 95 | " model.add(Activation('relu'))\n", 96 | " model.add(Dropout(0.5))\n", 97 | " model.add(Dense(n_classes))\n", 98 | " model.add(Activation('softmax'))\n", 99 | " optimizer = Adam()\n", 100 | " model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n", 101 | " \n", 102 | " history = model.fit(X_train, y_train, epochs=pars['n_epochs'], batch_size=pars['batch_size'], \n", 103 | " validation_data=[X_val, y_val],\n", 104 | " verbose = 1, callbacks=get_callbacks(pars)) \n", 105 | " best_epoch = np.argmax(history.history['val_acc'])\n", 106 | " best_val_acc = np.max(history.history['val_acc'])\n", 107 | " print('Epoch {} - val acc: {}'.format(best_epoch, best_val_acc))\n", 108 | " sys.stdout.flush() \n", 109 | " return {'val_acc': best_val_acc, 'best_epoch': best_epoch, 'eval_time': time.time(), 'status': STATUS_OK}" 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": null, 115 | "metadata": {}, 116 | "outputs": [], 117 | "source": [ 118 | "trials = Trials()\n", 119 | "best = fmin(f_nn, space, algo=tpe.suggest, max_evals=50, trials=trials)\n", 120 | "print(best)" 121 | ] 122 | } 123 | ], 124 | "metadata": { 125 | "kernelspec": { 126 | "display_name": "Python 3", 127 | "language": "python", 128 | "name": "python3" 129 | }, 130 | "language_info": { 131 | "codemirror_mode": { 132 | "name": "ipython", 133 | "version": 3 134 | }, 135 | "file_extension": ".py", 136 | "mimetype": "text/x-python", 137 | "name": "python", 138 | "nbconvert_exporter": "python", 139 | "pygments_lexer": "ipython3", 140 | "version": "3.5.2" 141 | }, 142 | "widgets": { 143 | "state": {}, 144 | "version": "1.1.2" 145 | } 146 | }, 147 | "nbformat": 4, 148 | "nbformat_minor": 2 149 | } 150 | -------------------------------------------------------------------------------- /Chapter12/Chapter 12 - Visualizing training with TensorBoard and Keras.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "from keras.datasets import cifar10\n", 10 | "from keras.preprocessing.image import ImageDataGenerator\n", 11 | "from keras.models import Sequential\n", 12 | "from keras.layers import Dense, Dropout, Activation, Flatten, Conv2D, MaxPooling2D\n", 13 | "from keras.optimizers import Adam\n", 14 | "from keras.callbacks import EarlyStopping, TensorBoard, ModelCheckpoint\n", 15 | "from keras.utils import to_categorical" 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": null, 21 | "metadata": {}, 22 | "outputs": [], 23 | "source": [ 24 | "(X_train, y_train), (X_val, y_val) = cifar10.load_data()\n", 25 | "\n", 26 | "X_train = X_train.astype('float32')\n", 27 | "X_train /=255.\n", 28 | "X_val = X_val.astype('float32')\n", 29 | "X_val /=255.\n", 30 | "\n", 31 | "n_classes = 10\n", 32 | "y_train = to_categorical(y_train, n_classes)\n", 33 | "y_val = to_categorical(y_val, n_classes)" 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": null, 39 | "metadata": {}, 40 | "outputs": [], 41 | "source": [ 42 | "model = Sequential()\n", 43 | "model.add(Conv2D(32, (3, 3), padding='same', input_shape=X_train.shape[1:]))\n", 44 | "model.add(Activation('relu'))\n", 45 | "model.add(Conv2D(32, (3, 3)))\n", 46 | "model.add(Activation('relu'))\n", 47 | "model.add(MaxPooling2D(pool_size=(2, 2)))\n", 48 | "model.add(Dropout(0.25))\n", 49 | "\n", 50 | "model.add(Conv2D(64, (3, 3), padding='same'))\n", 51 | "model.add(Activation('relu'))\n", 52 | "model.add(Conv2D(64, (3, 3)))\n", 53 | "model.add(Activation('relu'))\n", 54 | "model.add(MaxPooling2D(pool_size=(2, 2)))\n", 55 | "model.add(Dropout(0.25))\n", 56 | "\n", 57 | "model.add(Flatten())\n", 58 | "model.add(Dense(512))\n", 59 | "model.add(Activation('relu'))\n", 60 | "model.add(Dropout(0.5))\n", 61 | "model.add(Dense(n_classes))\n", 62 | "model.add(Activation('softmax'))" 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": null, 68 | "metadata": {}, 69 | "outputs": [], 70 | "source": [ 71 | "opt = Adam()\n", 72 | "model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": null, 78 | "metadata": {}, 79 | "outputs": [], 80 | "source": [ 81 | "data_gen = ImageDataGenerator(\n", 82 | " rotation_range=15,\n", 83 | " width_shift_range=0.15,\n", 84 | " height_shift_range=0.15,\n", 85 | " horizontal_flip=True,\n", 86 | " vertical_flip=False)\n", 87 | "\n", 88 | "data_gen.fit(X_train)" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": null, 94 | "metadata": {}, 95 | "outputs": [], 96 | "source": [ 97 | "callbacks = [EarlyStopping(monitor='val_acc', patience=5, verbose=2), ModelCheckpoint('checkpoints/weights.{epoch:02d}.hdf5', save_best_only=True), TensorBoard('~/notebooks/logs', write_graph=True, write_grads=True, write_images=True, embeddings_freq=0, embeddings_layer_names=None, embeddings_metadata=None)]" 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": null, 103 | "metadata": {}, 104 | "outputs": [], 105 | "source": [ 106 | "batch_size = 128\n", 107 | "n_epochs = 1000\n", 108 | "model.fit_generator(data_gen.flow(X_train, y_train, batch_size=batch_size),\n", 109 | " steps_per_epoch=X_train.shape[0] // batch_size,\n", 110 | " epochs=n_epochs,\n", 111 | " validation_data=(X_val, y_val),\n", 112 | " callbacks=callbacks\n", 113 | " )" 114 | ] 115 | } 116 | ], 117 | "metadata": { 118 | "kernelspec": { 119 | "display_name": "Python 3", 120 | "language": "python", 121 | "name": "python3" 122 | }, 123 | "language_info": { 124 | "codemirror_mode": { 125 | "name": "ipython", 126 | "version": 3 127 | }, 128 | "file_extension": ".py", 129 | "mimetype": "text/x-python", 130 | "name": "python", 131 | "nbconvert_exporter": "python", 132 | "pygments_lexer": "ipython3", 133 | "version": "3.5.2" 134 | }, 135 | "widgets": { 136 | "state": {}, 137 | "version": "1.1.2" 138 | } 139 | }, 140 | "nbformat": 4, 141 | "nbformat_minor": 2 142 | } 143 | -------------------------------------------------------------------------------- /Chapter12/Chapter 12 - Working with batches and mini-batches.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "from keras.datasets import cifar10\n", 10 | "from keras.preprocessing.image import ImageDataGenerator\n", 11 | "from keras.models import Sequential\n", 12 | "from keras.layers import Dense, Dropout, Activation, Flatten\n", 13 | "from keras.layers import Conv2D, MaxPooling2D\n", 14 | "from keras.optimizers import Adam\n", 15 | "from keras.callbacks import EarlyStopping, ModelCheckpoint" 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": null, 21 | "metadata": {}, 22 | "outputs": [], 23 | "source": [ 24 | "(X_train, y_train), (X_test, y_test) = cifar10.load_data()\n", 25 | "\n", 26 | "X_train = X_train.astype('float32')/255.\n", 27 | "X_test = X_test.astype('float32')/255.\n", 28 | "n_classes = 10\n", 29 | "y_train = keras.utils.to_categorical(y_train, n_classes)\n", 30 | "y_test = keras.utils.to_categorical(y_test, n_classes)" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": null, 36 | "metadata": {}, 37 | "outputs": [], 38 | "source": [ 39 | "model = Sequential()\n", 40 | "model.add(Conv2D(32, (3, 3), padding='same', input_shape=X_train.shape[1:]))\n", 41 | "model.add(Activation('relu'))\n", 42 | "model.add(Conv2D(32, (3, 3)))\n", 43 | "model.add(Activation('relu'))\n", 44 | "model.add(MaxPooling2D(pool_size=(2, 2)))\n", 45 | "model.add(Dropout(0.25))\n", 46 | "\n", 47 | "model.add(Conv2D(64, (3, 3), padding='same'))\n", 48 | "model.add(Activation('relu'))\n", 49 | "model.add(Conv2D(64, (3, 3)))\n", 50 | "model.add(Activation('relu'))\n", 51 | "model.add(MaxPooling2D(pool_size=(2, 2)))\n", 52 | "model.add(Dropout(0.25))\n", 53 | "\n", 54 | "model.add(Flatten())\n", 55 | "model.add(Dense(512))\n", 56 | "model.add(Activation('relu'))\n", 57 | "model.add(Dropout(0.5))\n", 58 | "model.add(Dense(n_classes))\n", 59 | "model.add(Activation('softmax'))" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": null, 65 | "metadata": {}, 66 | "outputs": [], 67 | "source": [ 68 | "opt = Adam(lr=0.0001)\n", 69 | "model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": null, 75 | "metadata": {}, 76 | "outputs": [], 77 | "source": [ 78 | "datagen = ImageDataGenerator(\n", 79 | " rotation_range=15,\n", 80 | " width_shift_range=0.15,\n", 81 | " height_shift_range=0.15,\n", 82 | " horizontal_flip=True,\n", 83 | " vertical_flip=False)\n", 84 | "\n", 85 | "datagen.fit(x_train)" 86 | ] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": null, 91 | "metadata": {}, 92 | "outputs": [], 93 | "source": [ 94 | "callbacks = [EarlyStopping(monitor='val_acc', patience=5, verbose=2), ModelCheckpoint('checkpoints/weights.{epoch:02d}-'+str(batch_size)+'.hdf5', save_best_only=True)]" 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": null, 100 | "metadata": {}, 101 | "outputs": [], 102 | "source": [ 103 | "batch_size = 32\n", 104 | "n_epochs = 1000\n", 105 | "history_32 = model.fit_generator(datagen.flow(X_train, y_train,\n", 106 | " batch_size=batch_size),\n", 107 | " steps_per_epoch=X_train.shape[0] // batch_size,\n", 108 | " epochs=epochs,\n", 109 | " validation_data=(X_val, y_val),\n", 110 | " callbacks=callbacks\n", 111 | " )" 112 | ] 113 | }, 114 | { 115 | "cell_type": "code", 116 | "execution_count": null, 117 | "metadata": {}, 118 | "outputs": [], 119 | "source": [ 120 | "model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])" 121 | ] 122 | }, 123 | { 124 | "cell_type": "code", 125 | "execution_count": null, 126 | "metadata": {}, 127 | "outputs": [], 128 | "source": [ 129 | "batch_size = 256\n", 130 | "history_32 = model.fit_generator(datagen.flow(X_train, y_train,\n", 131 | " batch_size=batch_size),\n", 132 | " steps_per_epoch=X_train.shape[0] // batch_size,\n", 133 | " epochs=epochs,\n", 134 | " validation_data=(X_val, y_val),\n", 135 | " callbacks=callbacks\n", 136 | " )" 137 | ] 138 | } 139 | ], 140 | "metadata": { 141 | "kernelspec": { 142 | "display_name": "Python 3", 143 | "language": "python", 144 | "name": "python3" 145 | }, 146 | "language_info": { 147 | "codemirror_mode": { 148 | "name": "ipython", 149 | "version": 3 150 | }, 151 | "file_extension": ".py", 152 | "mimetype": "text/x-python", 153 | "name": "python", 154 | "nbconvert_exporter": "python", 155 | "pygments_lexer": "ipython3", 156 | "version": "3.5.2" 157 | }, 158 | "widgets": { 159 | "state": {}, 160 | "version": "1.1.2" 161 | } 162 | }, 163 | "nbformat": 4, 164 | "nbformat_minor": 2 165 | } 166 | -------------------------------------------------------------------------------- /Chapter13/Chapter 13 - Freezing layers.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import tensorflow as tf\n", 10 | "from tensorflow.examples.tutorials.mnist import input_data" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 2, 16 | "metadata": {}, 17 | "outputs": [ 18 | { 19 | "name": "stdout", 20 | "output_type": "stream", 21 | "text": [ 22 | "Extracting Data/mnist/train-images-idx3-ubyte.gz\n", 23 | "Extracting Data/mnist/train-labels-idx1-ubyte.gz\n", 24 | "Extracting Data/mnist/t10k-images-idx3-ubyte.gz\n", 25 | "Extracting Data/mnist/t10k-labels-idx1-ubyte.gz\n" 26 | ] 27 | } 28 | ], 29 | "source": [ 30 | "mnist = input_data.read_data_sets('Data/mnist', one_hot=True)" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": 3, 36 | "metadata": {}, 37 | "outputs": [], 38 | "source": [ 39 | "n_classes = 10\n", 40 | "input_size = 784\n", 41 | "\n", 42 | "x = tf.placeholder(tf.float32, shape=[None, input_size])\n", 43 | "y = tf.placeholder(tf.float32, shape=[None, n_classes])\n", 44 | "keep_prob = tf.placeholder(tf.float32)" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": 4, 50 | "metadata": {}, 51 | "outputs": [], 52 | "source": [ 53 | "def weight_variable(shape, name='undefined'):\n", 54 | " initial = tf.truncated_normal(shape, stddev=0.1)\n", 55 | " return tf.Variable(initial, name=name)\n", 56 | "\n", 57 | "def bias_variable(shape, name='undefined'):\n", 58 | " initial = tf.constant(0.1, shape=shape)\n", 59 | " return tf.Variable(initial, name=name)\n", 60 | "\n", 61 | "def conv2d(x, W):\n", 62 | " return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')\n", 63 | "\n", 64 | "def max_pool_2x2(x):\n", 65 | " return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')" 66 | ] 67 | }, 68 | { 69 | "cell_type": "code", 70 | "execution_count": 5, 71 | "metadata": {}, 72 | "outputs": [], 73 | "source": [ 74 | "x_image = tf.reshape(x, [-1,28,28,1])\n", 75 | "\n", 76 | "W_conv1 = weight_variable([7, 7, 1, 100], name='1st_layer_weights')\n", 77 | "b_conv1 = bias_variable([100], name='1st_layer_bias')\n", 78 | "h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)\n", 79 | "h_pool1 = max_pool_2x2(h_conv1) \n", 80 | "\n", 81 | "W_conv2 = weight_variable([4, 4, 100, 150])\n", 82 | "b_conv2 = bias_variable([150])\n", 83 | "h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)\n", 84 | "h_pool2 = max_pool_2x2(h_conv2)\n", 85 | "\n", 86 | "W_conv3 = weight_variable([4, 4, 150, 250])\n", 87 | "b_conv3 = bias_variable([250])\n", 88 | "h_conv3 = tf.nn.relu(conv2d(h_pool2, W_conv3) + b_conv3)\n", 89 | "h_pool3 = max_pool_2x2(h_conv3)\n", 90 | "\n", 91 | "W_fc1 = weight_variable([4 * 4 * 250, 300])\n", 92 | "b_fc1 = bias_variable([300])\n", 93 | "h_pool3_flat = tf.reshape(h_pool3, [-1, 4*4*250])\n", 94 | "h_fc1 = tf.nn.relu(tf.matmul(h_pool3_flat, W_fc1) + b_fc1)\n", 95 | "h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)\n", 96 | "\n", 97 | "W_fc2 = weight_variable([300, n_classes])\n", 98 | "b_fc2 = bias_variable([n_classes])\n", 99 | "y_pred = tf.matmul(h_fc1_drop, W_fc2) + b_fc2\n", 100 | "\n", 101 | "diff = tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=y_pred)\n", 102 | "cross_entropy = tf.reduce_mean(diff)\n", 103 | "correct_prediction = tf.equal(tf.argmax(y_pred, 1), tf.argmax(y, 1))\n", 104 | "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": 6, 110 | "metadata": {}, 111 | "outputs": [ 112 | { 113 | "name": "stdout", 114 | "output_type": "stream", 115 | "text": [ 116 | "\n", 117 | "\n", 118 | "\n", 119 | "\n", 120 | "\n", 121 | "\n", 122 | "\n", 123 | "\n", 124 | "\n", 125 | "\n" 126 | ] 127 | } 128 | ], 129 | "source": [ 130 | "trainable_vars = tf.trainable_variables()\n", 131 | "\n", 132 | "for i in range(len(trainable_vars)):\n", 133 | " print(trainable_vars[i])" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": 7, 139 | "metadata": {}, 140 | "outputs": [], 141 | "source": [ 142 | "vars_train = [var for var in trainable_vars if '1st_' in var.name]\n", 143 | "\n", 144 | "learning_rate = 0.001\n", 145 | "train_step = tf.train.AdamOptimizer(learning_rate).minimize(cross_entropy, var_list=vars_train)" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": 8, 151 | "metadata": {}, 152 | "outputs": [ 153 | { 154 | "name": "stdout", 155 | "output_type": "stream", 156 | "text": [ 157 | "Step 0000: train_acc: 0.1016\n", 158 | "Step 0001: train_acc: 0.1328\n", 159 | "Step 0002: train_acc: 0.0938\n", 160 | "Step 0003: train_acc: 0.0938\n", 161 | "Step 0004: train_acc: 0.1094\n", 162 | "Step 0005: train_acc: 0.0547\n", 163 | "Step 0006: train_acc: 0.1016\n", 164 | "Step 0007: train_acc: 0.1016\n", 165 | "Step 0008: train_acc: 0.1172\n", 166 | "Step 0009: train_acc: 0.0781\n" 167 | ] 168 | } 169 | ], 170 | "source": [ 171 | "n_steps = 10\n", 172 | "batch_size = 128\n", 173 | "dropout = 0.25\n", 174 | "evaluate_every = 10\n", 175 | "\n", 176 | "sess = tf.InteractiveSession()\n", 177 | "tf.global_variables_initializer().run()\n", 178 | "for i in range(n_steps):\n", 179 | " x_batch, y_batch = mnist.train.next_batch(batch_size)\n", 180 | " _, train_acc = sess.run([train_step, accuracy], feed_dict={x: x_batch, y: y_batch, keep_prob: dropout})\n", 181 | " print('Step {:04.0f}: train_acc: {:.4f}'.format(i, train_acc))" 182 | ] 183 | } 184 | ], 185 | "metadata": { 186 | "kernelspec": { 187 | "display_name": "Python 3", 188 | "language": "python", 189 | "name": "python3" 190 | }, 191 | "language_info": { 192 | "codemirror_mode": { 193 | "name": "ipython", 194 | "version": 3 195 | }, 196 | "file_extension": ".py", 197 | "mimetype": "text/x-python", 198 | "name": "python", 199 | "nbconvert_exporter": "python", 200 | "pygments_lexer": "ipython3", 201 | "version": "3.5.2" 202 | }, 203 | "widgets": { 204 | "state": {}, 205 | "version": "1.1.2" 206 | } 207 | }, 208 | "nbformat": 4, 209 | "nbformat_minor": 2 210 | } 211 | -------------------------------------------------------------------------------- /Chapter13/Chapter 13 - Storing the network topology and trained weights.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import tensorflow as tf\n", 10 | "from tensorflow.examples.tutorials.mnist import input_data" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 2, 16 | "metadata": {}, 17 | "outputs": [ 18 | { 19 | "name": "stdout", 20 | "output_type": "stream", 21 | "text": [ 22 | "Extracting Data/mnist/train-images-idx3-ubyte.gz\n", 23 | "Extracting Data/mnist/train-labels-idx1-ubyte.gz\n", 24 | "Extracting Data/mnist/t10k-images-idx3-ubyte.gz\n", 25 | "Extracting Data/mnist/t10k-labels-idx1-ubyte.gz\n" 26 | ] 27 | } 28 | ], 29 | "source": [ 30 | "mnist = input_data.read_data_sets('Data/mnist', one_hot=True)" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": 3, 36 | "metadata": {}, 37 | "outputs": [], 38 | "source": [ 39 | "n_classes = 10\n", 40 | "input_size = 784\n", 41 | "\n", 42 | "x = tf.placeholder(tf.float32, shape=[None, input_size])\n", 43 | "y = tf.placeholder(tf.float32, shape=[None, n_classes])\n", 44 | "keep_prob = tf.placeholder(tf.float32)" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": 4, 50 | "metadata": {}, 51 | "outputs": [], 52 | "source": [ 53 | "def weight_variable(shape):\n", 54 | " initial = tf.truncated_normal(shape, stddev=0.1)\n", 55 | " return tf.Variable(initial)\n", 56 | "\n", 57 | "def bias_variable(shape):\n", 58 | " initial = tf.constant(0.1, shape=shape)\n", 59 | " return tf.Variable(initial)\n", 60 | "\n", 61 | "def conv2d(x, W):\n", 62 | " return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')\n", 63 | "\n", 64 | "def max_pool_2x2(x):\n", 65 | " return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')" 66 | ] 67 | }, 68 | { 69 | "cell_type": "code", 70 | "execution_count": 5, 71 | "metadata": {}, 72 | "outputs": [], 73 | "source": [ 74 | "x_image = tf.reshape(x, [-1,28,28,1])\n", 75 | "\n", 76 | "W_conv1 = weight_variable([7, 7, 1, 100])\n", 77 | "b_conv1 = bias_variable([100])\n", 78 | "h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)\n", 79 | "h_pool1 = max_pool_2x2(h_conv1) \n", 80 | "\n", 81 | "W_conv2 = weight_variable([4, 4, 100, 150])\n", 82 | "b_conv2 = bias_variable([150])\n", 83 | "h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)\n", 84 | "h_pool2 = max_pool_2x2(h_conv2)\n", 85 | "\n", 86 | "W_conv3 = weight_variable([4, 4, 150, 250])\n", 87 | "b_conv3 = bias_variable([250])\n", 88 | "h_conv3 = tf.nn.relu(conv2d(h_pool2, W_conv3) + b_conv3)\n", 89 | "h_pool3 = max_pool_2x2(h_conv3)\n", 90 | "\n", 91 | "W_fc1 = weight_variable([4 * 4 * 250, 300])\n", 92 | "b_fc1 = bias_variable([300])\n", 93 | "h_pool3_flat = tf.reshape(h_pool3, [-1, 4*4*250])\n", 94 | "h_fc1 = tf.nn.relu(tf.matmul(h_pool3_flat, W_fc1) + b_fc1)\n", 95 | "h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)\n", 96 | "\n", 97 | "W_fc2 = weight_variable([300, n_classes])\n", 98 | "b_fc2 = bias_variable([n_classes])\n", 99 | "y_pred = tf.matmul(h_fc1_drop, W_fc2) + b_fc2" 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": 6, 105 | "metadata": {}, 106 | "outputs": [], 107 | "source": [ 108 | "diff = tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=y_pred)\n", 109 | "cross_entropy = tf.reduce_mean(diff)\n", 110 | "correct_prediction = tf.equal(tf.argmax(y_pred, 1), tf.argmax(y, 1))\n", 111 | "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))" 112 | ] 113 | }, 114 | { 115 | "cell_type": "code", 116 | "execution_count": 7, 117 | "metadata": {}, 118 | "outputs": [], 119 | "source": [ 120 | "learning_rate = 0.001\n", 121 | "train_step = tf.train.AdamOptimizer(learning_rate).minimize(cross_entropy)" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": 15, 127 | "metadata": {}, 128 | "outputs": [], 129 | "source": [ 130 | "n_steps = 25\n", 131 | "batch_size = 32\n", 132 | "dropout = 0.25\n", 133 | "evaluate_every = 10\n", 134 | "n_val_steps = mnist.test.images.shape[0] // batch_size" 135 | ] 136 | }, 137 | { 138 | "cell_type": "code", 139 | "execution_count": 16, 140 | "metadata": {}, 141 | "outputs": [], 142 | "source": [ 143 | "saver = tf.train.Saver(max_to_keep=5)\n", 144 | "save_dir = 'checkpoints/'" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": 21, 150 | "metadata": {}, 151 | "outputs": [ 152 | { 153 | "name": "stdout", 154 | "output_type": "stream", 155 | "text": [ 156 | "Step 0000: train_acc: 0.0625; val_acc: 0.1358\n", 157 | "Model saved\n", 158 | "Step 0010: train_acc: 0.1875; val_acc: 0.3109\n", 159 | "Model saved\n", 160 | "Step 0020: train_acc: 0.1250; val_acc: 0.3996\n" 161 | ] 162 | }, 163 | { 164 | "data": { 165 | "text/plain": [ 166 | "'checkpoints/last-model'" 167 | ] 168 | }, 169 | "execution_count": 21, 170 | "metadata": {}, 171 | "output_type": "execute_result" 172 | } 173 | ], 174 | "source": [ 175 | "sess = tf.InteractiveSession()\n", 176 | "tf.global_variables_initializer().run()\n", 177 | "best_val = 0.0\n", 178 | "for i in range(n_steps):\n", 179 | " x_batch, y_batch = mnist.train.next_batch(batch_size)\n", 180 | " _, train_acc = sess.run([train_step, accuracy], feed_dict={x: x_batch, y: y_batch, keep_prob: dropout})\n", 181 | " \n", 182 | " if i % evaluate_every == 0:\n", 183 | " val_accs = []\n", 184 | " for j in range(n_val_steps):\n", 185 | " x_batch, y_batch = mnist.test.next_batch(batch_size)\n", 186 | " val_acc = sess.run(accuracy, feed_dict={x: x_batch, y: y_batch, keep_prob: 1.0})\n", 187 | " val_accs.append(val_acc)\n", 188 | " print('Step {:04.0f}: train_acc: {:.4f}; val_acc: {:.4f}'.format(i, train_acc, sum(val_accs)/len(val_accs)))\n", 189 | " if val_acc > best_val:\n", 190 | " saver.save(sess, save_dir+'best-model', global_step=i)\n", 191 | " print('Model saved')\n", 192 | " best_val = val_acc\n", 193 | "saver.save(sess, save_dir+'last-model')" 194 | ] 195 | }, 196 | { 197 | "cell_type": "code", 198 | "execution_count": 22, 199 | "metadata": {}, 200 | "outputs": [ 201 | { 202 | "name": "stdout", 203 | "output_type": "stream", 204 | "text": [ 205 | "INFO:tensorflow:Restoring parameters from checkpoints/last-model\n" 206 | ] 207 | } 208 | ], 209 | "source": [ 210 | "with tf.Session() as sess:\n", 211 | " new_saver = tf.train.import_meta_graph(save_dir+'last-model.meta')\n", 212 | " new_saver.restore(sess, save_dir+'last-model')\n", 213 | " for i in range(35):\n", 214 | " x_batch, y_batch = mnist.train.next_batch(batch_size)\n", 215 | " _, train_acc = sess.run([train_step, accuracy], feed_dict={x: x_batch, y: y_batch, keep_prob: dropout})" 216 | ] 217 | } 218 | ], 219 | "metadata": { 220 | "kernelspec": { 221 | "display_name": "Python 3", 222 | "language": "python", 223 | "name": "python3" 224 | }, 225 | "language_info": { 226 | "codemirror_mode": { 227 | "name": "ipython", 228 | "version": 3 229 | }, 230 | "file_extension": ".py", 231 | "mimetype": "text/x-python", 232 | "name": "python", 233 | "nbconvert_exporter": "python", 234 | "pygments_lexer": "ipython3", 235 | "version": "3.5.2" 236 | }, 237 | "widgets": { 238 | "state": {}, 239 | "version": "1.1.2" 240 | } 241 | }, 242 | "nbformat": 4, 243 | "nbformat_minor": 2 244 | } 245 | -------------------------------------------------------------------------------- /Chapter14/Chapter 14 - Extracting bottleneck features with ResNet.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "from keras.models import Model\n", 10 | "from keras.applications.resnet50 import ResNet50\n", 11 | "\n", 12 | "from keras.applications.resnet50 import preprocess_input\n", 13 | "from keras.preprocessing.image import load_img\n", 14 | "from keras.preprocessing.image import img_to_array\n", 15 | "from keras.applications import imagenet_utils" 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": null, 21 | "metadata": {}, 22 | "outputs": [], 23 | "source": [ 24 | "resnet_model = ResNet50(weights='imagenet')" 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": null, 30 | "metadata": {}, 31 | "outputs": [], 32 | "source": [ 33 | "model = Model(inputs=resnet_model.input, outputs=resnet_model.get_layer('avg_pool').output)" 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": null, 39 | "metadata": {}, 40 | "outputs": [], 41 | "source": [ 42 | "input_dim = (224, 224)\n", 43 | "image = load_img('../Data/dog_example.jpg', target_size=input_dim)\n", 44 | "image = img_to_array(image)\n", 45 | "image = image.reshape((1, *image.shape))\n", 46 | "image = preprocess_input(image)" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": null, 52 | "metadata": {}, 53 | "outputs": [], 54 | "source": [ 55 | "avg_pool_features = model.predict(image)" 56 | ] 57 | } 58 | ], 59 | "metadata": { 60 | "kernelspec": { 61 | "display_name": "Python 3", 62 | "language": "python", 63 | "name": "python3" 64 | }, 65 | "language_info": { 66 | "codemirror_mode": { 67 | "name": "ipython", 68 | "version": 3 69 | }, 70 | "file_extension": ".py", 71 | "mimetype": "text/x-python", 72 | "name": "python", 73 | "nbconvert_exporter": "python", 74 | "pygments_lexer": "ipython3", 75 | "version": "3.5.2" 76 | }, 77 | "widgets": { 78 | "state": {}, 79 | "version": "1.1.2" 80 | } 81 | }, 82 | "nbformat": 4, 83 | "nbformat_minor": 2 84 | } 85 | -------------------------------------------------------------------------------- /Chapter14/Chapter 14 - Fine-tuning with Xception.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np\n", 10 | "from keras.models import Model\n", 11 | "from keras.applications import Xception\n", 12 | "from keras.layers import Dense, GlobalAveragePooling2D\n", 13 | "from keras.optimizers import Adam\n", 14 | "\n", 15 | "from keras.applications import imagenet_utils\n", 16 | "from keras.utils import np_utils\n", 17 | "from keras.callbacks import EarlyStopping" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": null, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "from keras.datasets import cifar10\n", 27 | "(X_train, y_train), (X_test, y_test) = cifar10.load_data()" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": null, 33 | "metadata": {}, 34 | "outputs": [], 35 | "source": [ 36 | "n_classes = len(np.unique(y_train))\n", 37 | "y_train = np_utils.to_categorical(y_train, n_classes)\n", 38 | "y_test = np_utils.to_categorical(y_test, n_classes)\n", 39 | "\n", 40 | "X_train = X_train.astype('float32')/255.\n", 41 | "X_test = X_test.astype('float32')/255." 42 | ] 43 | }, 44 | { 45 | "cell_type": "code", 46 | "execution_count": null, 47 | "metadata": {}, 48 | "outputs": [], 49 | "source": [ 50 | "xception_model = Xception(weights='imagenet', include_top=False)" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": null, 56 | "metadata": {}, 57 | "outputs": [], 58 | "source": [ 59 | "x = xception_model.output\n", 60 | "x = GlobalAveragePooling2D()(x)\n", 61 | "x = Dense(512, activation='relu')(x)\n", 62 | "out = Dense(10, activation='softmax')(x)" 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": null, 68 | "metadata": {}, 69 | "outputs": [], 70 | "source": [ 71 | "model = Model(inputs=xception_model.input, outputs=out)" 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": null, 77 | "metadata": {}, 78 | "outputs": [], 79 | "source": [ 80 | "for layer in xception_model.layers:\n", 81 | " layer.trainable = False" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "metadata": {}, 88 | "outputs": [], 89 | "source": [ 90 | "opt = Adam()\n", 91 | "model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n", 92 | "model.summary()" 93 | ] 94 | }, 95 | { 96 | "cell_type": "code", 97 | "execution_count": null, 98 | "metadata": {}, 99 | "outputs": [], 100 | "source": [ 101 | "callbacks = [EarlyStopping(monitor='val_acc', patience=5, verbose=0)]\n" 102 | ] 103 | }, 104 | { 105 | "cell_type": "code", 106 | "execution_count": null, 107 | "metadata": {}, 108 | "outputs": [], 109 | "source": [ 110 | "n_epochs = 100\n", 111 | "batch_size = 512\n", 112 | "history = model.fit(X_train, y_train, epochs=n_epochs, batch_size=batch_size, validation_split=0.2, verbose=1, callbacks=callbacks)" 113 | ] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": null, 118 | "metadata": {}, 119 | "outputs": [], 120 | "source": [ 121 | "for i, layer in enumerate(model.layers):\n", 122 | " print(i, layer.name)" 123 | ] 124 | }, 125 | { 126 | "cell_type": "code", 127 | "execution_count": null, 128 | "metadata": {}, 129 | "outputs": [], 130 | "source": [ 131 | "for layer in model.layers[:115]:\n", 132 | " layer.trainable = False\n", 133 | "for layer in model.layers[115:]:\n", 134 | " layer.trainable = True" 135 | ] 136 | }, 137 | { 138 | "cell_type": "code", 139 | "execution_count": null, 140 | "metadata": {}, 141 | "outputs": [], 142 | "source": [ 143 | "opt_finetune = Adam()\n", 144 | "model.compile(optimizer=opt_finetune, loss='categorical_crossentropy', metrics=['accuracy'])\n", 145 | "model.summary()" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": null, 151 | "metadata": {}, 152 | "outputs": [], 153 | "source": [ 154 | "history_finetune = model.fit(X_train, y_train, epochs=n_epochs, batch_size=batch_size, validation_split=0.2, verbose=1, callbacks=callbacks)" 155 | ] 156 | } 157 | ], 158 | "metadata": { 159 | "kernelspec": { 160 | "display_name": "Python 3", 161 | "language": "python", 162 | "name": "python3" 163 | }, 164 | "language_info": { 165 | "codemirror_mode": { 166 | "name": "ipython", 167 | "version": 3 168 | }, 169 | "file_extension": ".py", 170 | "mimetype": "text/x-python", 171 | "name": "python", 172 | "nbconvert_exporter": "python", 173 | "pygments_lexer": "ipython3", 174 | "version": "3.5.2" 175 | }, 176 | "widgets": { 177 | "state": {}, 178 | "version": "1.1.2" 179 | } 180 | }, 181 | "nbformat": 4, 182 | "nbformat_minor": 2 183 | } 184 | -------------------------------------------------------------------------------- /Chapter14/Chapter 14 - Large-scale visual recognition with GoogLeNet_Inception.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np\n", 10 | "\n", 11 | "from keras.applications.inception_v3 import InceptionV3\n", 12 | "from keras.applications import imagenet_utils\n", 13 | "from keras.preprocessing.image import load_img\n", 14 | "from keras.preprocessing.image import img_to_array" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": null, 20 | "metadata": {}, 21 | "outputs": [], 22 | "source": [ 23 | "pretrained_model = InceptionV3\n", 24 | "model = pretrained_model(weights=\"imagenet\")" 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": null, 30 | "metadata": {}, 31 | "outputs": [], 32 | "source": [ 33 | "input_dim = (299, 299)" 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": null, 39 | "metadata": {}, 40 | "outputs": [], 41 | "source": [ 42 | "preprocess = imagenet_utils.preprocess_input" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": null, 48 | "metadata": {}, 49 | "outputs": [], 50 | "source": [ 51 | "image = load_img('../Data/dog_example_2.jpg', target_size=input_dim)\n", 52 | "image = img_to_array(image)\n", 53 | "image = image.reshape((1, *image.shape))\n", 54 | "image = preprocess(image)" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": {}, 61 | "outputs": [], 62 | "source": [ 63 | "preds = model.predict(image)" 64 | ] 65 | }, 66 | { 67 | "cell_type": "code", 68 | "execution_count": null, 69 | "metadata": {}, 70 | "outputs": [], 71 | "source": [ 72 | "preds_decoded = imagenet_utils.decode_predictions(preds)" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": null, 78 | "metadata": {}, 79 | "outputs": [], 80 | "source": [ 81 | "preds_decoded" 82 | ] 83 | } 84 | ], 85 | "metadata": { 86 | "kernelspec": { 87 | "display_name": "Python 3", 88 | "language": "python", 89 | "name": "python3" 90 | }, 91 | "language_info": { 92 | "codemirror_mode": { 93 | "name": "ipython", 94 | "version": 3 95 | }, 96 | "file_extension": ".py", 97 | "mimetype": "text/x-python", 98 | "name": "python", 99 | "nbconvert_exporter": "python", 100 | "pygments_lexer": "ipython3", 101 | "version": "3.5.2" 102 | }, 103 | "widgets": { 104 | "state": {}, 105 | "version": "1.1.2" 106 | } 107 | }, 108 | "nbformat": 4, 109 | "nbformat_minor": 2 110 | } 111 | -------------------------------------------------------------------------------- /Chapter14/Chapter 14 - Leveraging pretrained VGG models for new classes.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np\n", 10 | "import matplotlib.pyplot as plt\n", 11 | "\n", 12 | "from keras.models import Model\n", 13 | "from keras.applications.vgg16 import VGG16\n", 14 | "from keras.layers import Dense, GlobalAveragePooling2D\n", 15 | "from keras.optimizers import Adam\n", 16 | "\n", 17 | "from keras.applications import imagenet_utils\n", 18 | "from keras.utils import np_utils\n", 19 | "from keras.callbacks import EarlyStopping" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": null, 25 | "metadata": {}, 26 | "outputs": [], 27 | "source": [ 28 | "from keras.datasets import cifar10\n", 29 | "(X_train, y_train), (X_test, y_test) = cifar10.load_data()" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": null, 35 | "metadata": {}, 36 | "outputs": [], 37 | "source": [ 38 | "n_classes = len(np.unique(y_train))\n", 39 | "y_train = np_utils.to_categorical(y_train, n_classes)\n", 40 | "y_test = np_utils.to_categorical(y_test, n_classes)\n", 41 | "\n", 42 | "X_train = X_train.astype('float32')/255.\n", 43 | "X_test = X_test.astype('float32')/255." 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": null, 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "vgg_model = VGG16(weights='imagenet', include_top=False)" 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": null, 58 | "metadata": {}, 59 | "outputs": [], 60 | "source": [ 61 | "x = vgg_model.output\n", 62 | "x = GlobalAveragePooling2D()(x)\n", 63 | "x = Dense(512, activation='relu')(x)\n", 64 | "out = Dense(10, activation='softmax')(x)" 65 | ] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": null, 70 | "metadata": {}, 71 | "outputs": [], 72 | "source": [ 73 | "model = Model(inputs=vgg_model.input, outputs=out)" 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": null, 79 | "metadata": {}, 80 | "outputs": [], 81 | "source": [ 82 | "for layer in vgg_model.layers:\n", 83 | " layer.trainable = False" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": null, 89 | "metadata": {}, 90 | "outputs": [], 91 | "source": [ 92 | "opt = Adam()\n", 93 | "model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n", 94 | "model.summary()" 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": null, 100 | "metadata": {}, 101 | "outputs": [], 102 | "source": [ 103 | "callbacks = [EarlyStopping(monitor='val_acc', patience=5, verbose=0)]" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": null, 109 | "metadata": {}, 110 | "outputs": [], 111 | "source": [ 112 | "n_epochs = 50\n", 113 | "batch_size = 512\n", 114 | "history = model.fit(X_train, y_train, epochs=n_epochs, batch_size=batch_size, validation_split=0.2, verbose=1, callbacks=callbacks)" 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": null, 120 | "metadata": {}, 121 | "outputs": [], 122 | "source": [ 123 | "plt.plot(np.arange(len(history.history['acc'])), history.history['acc'], label='training')\n", 124 | "plt.plot(np.arange(len(history.history['val_acc'])), history.history['val_acc'], label='validation')\n", 125 | "plt.title('Accuracy')\n", 126 | "plt.xlabel('batches')\n", 127 | "plt.ylabel('accuracy ')\n", 128 | "plt.legend(loc=0)\n", 129 | "plt.show()" 130 | ] 131 | } 132 | ], 133 | "metadata": { 134 | "kernelspec": { 135 | "display_name": "Python 3", 136 | "language": "python", 137 | "name": "python3" 138 | }, 139 | "language_info": { 140 | "codemirror_mode": { 141 | "name": "ipython", 142 | "version": 3 143 | }, 144 | "file_extension": ".py", 145 | "mimetype": "text/x-python", 146 | "name": "python", 147 | "nbconvert_exporter": "python", 148 | "pygments_lexer": "ipython3", 149 | "version": "3.5.2" 150 | }, 151 | "widgets": { 152 | "state": {}, 153 | "version": "1.1.2" 154 | } 155 | }, 156 | "nbformat": 4, 157 | "nbformat_minor": 2 158 | } 159 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Packt 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | # Python Deep Learning Cookbook 5 | This is the code repository for [Python Deep Learning Cookbook](https://www.packtpub.com/big-data-and-business-intelligence/python-deep-learning-cookbook?utm_source=github&utm_medium=repository&utm_campaign=9781787125193), published by [Packt](https://www.packtpub.com/?utm_source=github). It contains all the supporting project files necessary to work through the book from start to finish. 6 | ## About the Book 7 | Deep Learning is revolutionizing a wide range of industries. For many applications, deep learning has proven to outperform humans by making faster and more accurate predictions. This book provides a top-down and bottom-up approach to demonstrate deep learning solutions to real-world problems in different areas. These applications include Computer Vision, Natural Language Processing, Time Series, and Robotics. 8 | 9 | The Python Deep Learning Cookbook presents technical solutions to the issues presented, along with a detailed explanation of the solutions. Furthermore, a discussion on corresponding pros and cons of implementing the proposed solution using one of the popular frameworks like TensorFlow, PyTorch, Keras and CNTK is provided. The book includes recipes that are related to the basic concepts of neural networks. All techniques s, as well as classical networks topologies. The main purpose of this book is to provide Python programmers a detailed list of recipes to apply deep learning to common and not-so-common scenarios. 10 | 11 | ## Instructions and Navigation 12 | All of the code is organized into folders. Each folder starts with a number followed by the application name. For example, Chapter02. 13 | 14 | 15 | 16 | The code will look like the following: 17 | ``` 18 | import numpy as np 19 | x_input = np.array([[1,2,3,4,5]]) 20 | y_input = np.array([[10]]) 21 | ``` 22 | 23 | This book is focused on AI in Python, as opposed to Python itself. We have used Python 3 24 | to build various applications. We focus on how to utilize various Python libraries in the best 25 | possible way to build real-world applications. In that spirit, we have tried to keep all of the 26 | code as friendly and readable as possible. We feel that this will enable our readers to easily 27 | understand the code and readily use it in different scenarios. 28 | 29 | ## Related Products 30 | * [Eder Santana's Deep Learning with Python](https://www.packtpub.com/application-development/eder-santanas-deep-learning-python?utm_source=github&utm_medium=repository&utm_campaign=9781787280465) 31 | 32 | * [Deep Learning By Example](https://www.packtpub.com/big-data-and-business-intelligence/deep-learning-example?utm_source=github&utm_medium=repository&utm_campaign=9781788399906) 33 | 34 | * [Python: End-to-end Data Analysis](https://www.packtpub.com/big-data-and-business-intelligence/python-end-end-data-analysis?utm_source=github&utm_medium=repository&utm_campaign=9781788394697) 35 | ### Download a free PDF 36 | 37 | If you have already purchased a print or Kindle version of this book, you can get a DRM-free PDF version at no cost.
Simply click on the link to claim your free PDF. 38 |

https://packt.link/free-ebook/9781787125193

--------------------------------------------------------------------------------