├── .gitignore ├── README.md └── tutorials ├── 01_basics ├── .ipynb_checkpoints │ ├── Sequential_and_Model-checkpoint.ipynb │ └── BasicStyle-checkpoint.ipynb ├── Sequential_and_Model.ipynb └── BasicStyle.ipynb ├── 03_advanced └── graph_convnet.ipynb ├── 99_others └── BN_bug.ipynb ├── 02_intermediate ├── .ipynb_checkpoints │ ├── Convolutinal_Neural_Network-checkpoint.ipynb │ ├── Recurrent_Neural_Network-checkpoint.ipynb │ ├── residual_network-checkpoint.ipynb │ ├── Bidrectional_Recurrent_Neural_Network_using_KerasAPI-checkpoint.ipynb │ ├── residual_network_using_KerasAPI-checkpoint.ipynb │ └── Convolutinal_Neural_Network_using_KerasAPI-checkpoint.ipynb ├── Convolutinal_Neural_Network_using_KerasAPI.ipynb └── Recurrent_Neural_Network.ipynb └── 00_lowlevel └── TF_eager_basics.ipynb /.gitignore: -------------------------------------------------------------------------------- 1 | /.ipynb_checkpoints/ 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # TensorFlow2.0_Eager_Execution_Tutorials 2 | 3 | This repository provides tutorial code of TensorFlow2.0 . This tutorials refer to the PyTorch tutorials 4 | 5 | https://github.com/yunjey/pytorch-tutorial 6 | 7 | 8 | 9 | ## Table of Contents 10 | ### 0. Low Level 11 | * [Using_tf.Variable](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/00_lowlevel/LinearModel_using_tf_Variable.ipynb) 12 | * [TF_Eager Basics](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/01_basics/TF_eager_basics.ipynb) 13 | * [eager_vs_graph](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/00_lowlevel/eager_vs_graph.ipynb) 14 | 15 | ### 1. Basics 16 | * [KerasAPI Sequential_and_Model](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/01_basics/Sequential_and_Model.ipynb) 17 | * [Linear Regression](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/01_basics/linear_regression.ipynb) 18 | * [Logistic Regression](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/01_basics/logistic_regression.ipynb) 19 | * [Feedforward Neural Network](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/01_basics/feed_forward_neural_network.ipynb) 20 | 21 | ### 2. Intermediate 22 | * [Convolutional Neural Network](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/02_intermediate/Convolutinal_Neural_Network.ipynb) 23 | * [Convolutional Neural Network using Keras API](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/02_intermediate/Convolutinal_Neural_Network_using_KerasAPI.ipynb) 24 | * [Deep Residual Network](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/02_intermediate/residual_network.ipynb) 25 | * [Deep Residual Network using Keras API](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/02_interimediate/residual_network.ipynb) 26 | * [Recurrent Neural Network using Keras API](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/02_intermediate/Recurrent_Neural_Network.ipynb) 27 | * [Bidirectional Recurrent Neural Network using Keras API](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/02_intermediate/Bidrectional_Recurrent_Neural_Network.ipynb) 28 | * Language Model (RNN-LM) 29 | 30 | ### 3. Advanced 31 | * [Generative Adversarial Network](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/03_advanced/Generative_Adversarial_Network.ipynb) 32 | * [Variational Auto-Encoder](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/03_advanced/Variational_Auto_Encoder.ipynb) 33 | * [Graph Convolution](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/03_advanced/graph_convnet.ipynb) 34 | * Neural Style Transfer 35 | * Image Captioning (CNN-RNN) 36 | 37 | ### 4. Probability 38 | * [MCMC Regression with JointDistribution](https://github.com/hellocybernetics/TensorFlow2.0_Eager_Execution_Tutorials/blob/master/tutorials/04_probability/Regiression_with_tfd.JointDistributionCoroutine.ipynb) 39 | * [Variational Regression with JointDistribution](https://github.com/hellocybernetics/TensorFlow2.0_Eager_Execution_Tutorials/blob/master/tutorials/04_probability/VariationalRegression_JointDistribution.ipynb) 40 | * [basic modeling with tfp.layers](https://github.com/hellocybernetics/TensorFlow2.0_Eager_Execution_Tutorials/blob/master/tutorials/04_probability/probabilistic_model_by_tfp_layers.ipynb) 41 | 42 | ### X. Others 43 | * [Hyper Parameter Optmization](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/99_others/hyper_parameter_optimization_using_optuna.ipynb) 44 | * [eager vs pytorch speed](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/99_others/Eager_vs_PyTorch_speed.ipynb) 45 | * [learning discontinuity](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/99_others/learninig_discontinuity.ipynb) 46 | * [dropout bayes neural network](https://github.com/hellocybernetics/TensorFlow_Eager_Execution_Tutorials/blob/master/tutorials/99_others/dropout_bayes.ipynb) 47 | 48 | ## Dependencies 49 | My environment is Google Colab or 50 | 51 | Python 3.7 52 | 53 | TensorFlow 2.0 54 | TensorFlow Probability 0.9.0 (nightly) 55 | 56 | -------------------------------------------------------------------------------- /tutorials/01_basics/.ipynb_checkpoints/Sequential_and_Model-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "colab_type": "text", 7 | "id": "view-in-github" 8 | }, 9 | "source": [ 10 | "\"Open" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 0, 16 | "metadata": { 17 | "colab": {}, 18 | "colab_type": "code", 19 | "id": "uk1oMHkAyQ2s" 20 | }, 21 | "outputs": [], 22 | "source": [ 23 | "import tensorflow as tf\n", 24 | "import tensorflow.keras.layers as L\n", 25 | "\n", 26 | "tf.enable_eager_execution()" 27 | ] 28 | }, 29 | { 30 | "cell_type": "markdown", 31 | "metadata": { 32 | "colab_type": "text", 33 | "id": "7gRtOWU70Z-q" 34 | }, 35 | "source": [ 36 | "## tf.keras.Sequential\n", 37 | "\n", 38 | "`tf.keras.Sequential` recieves a list of layers." 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": 0, 44 | "metadata": { 45 | "colab": {}, 46 | "colab_type": "code", 47 | "id": "kcwWxgaiyUiI" 48 | }, 49 | "outputs": [], 50 | "source": [ 51 | "model = tf.keras.Sequential([\n", 52 | " L.Dense(10),\n", 53 | " L.Dense(5),\n", 54 | " L.Dense(3)\n", 55 | "])" 56 | ] 57 | }, 58 | { 59 | "cell_type": "markdown", 60 | "metadata": { 61 | "colab_type": "text", 62 | "id": "LjvFRCiY0nfj" 63 | }, 64 | "source": [ 65 | "### Note\n", 66 | "'tf.keras.layers' don't need input_dimension, only output_dimension as above. Input_dimension is determined by first execution, so you have to write `model(inputs_data)` code before writing `model.summary()`." 67 | ] 68 | }, 69 | { 70 | "cell_type": "code", 71 | "execution_count": 0, 72 | "metadata": { 73 | "colab": {}, 74 | "colab_type": "code", 75 | "id": "Xm0kVQ4PynHK" 76 | }, 77 | "outputs": [], 78 | "source": [ 79 | "model.compile(optimizer=tf.train.AdamOptimizer(),\n", 80 | " loss=tf.losses.mean_squared_error)\n", 81 | "_ = model(tf.random_normal(shape=[100, 20]))" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": 5, 87 | "metadata": { 88 | "colab": { 89 | "base_uri": "https://localhost:8080/", 90 | "height": 251 91 | }, 92 | "colab_type": "code", 93 | "id": "OIsZnSTvy0sf", 94 | "outputId": "9672563d-ab62-4ea4-e6e9-d4eab2177e65" 95 | }, 96 | "outputs": [ 97 | { 98 | "name": "stdout", 99 | "output_type": "stream", 100 | "text": [ 101 | "_________________________________________________________________\n", 102 | "Layer (type) Output Shape Param # \n", 103 | "=================================================================\n", 104 | "dense (Dense) multiple 210 \n", 105 | "_________________________________________________________________\n", 106 | "dense_1 (Dense) multiple 55 \n", 107 | "_________________________________________________________________\n", 108 | "dense_2 (Dense) multiple 18 \n", 109 | "=================================================================\n", 110 | "Total params: 283\n", 111 | "Trainable params: 283\n", 112 | "Non-trainable params: 0\n", 113 | "_________________________________________________________________\n" 114 | ] 115 | } 116 | ], 117 | "source": [ 118 | "model.summary()" 119 | ] 120 | }, 121 | { 122 | "cell_type": "markdown", 123 | "metadata": { 124 | "colab_type": "text", 125 | "id": "-IkJwYAl1l5N" 126 | }, 127 | "source": [ 128 | "## tf.keras.Model\n", 129 | "`tf.keras.Model` is used as below. This is like PyTorch `torch.nn.Module` and `chainer.Chain`. When Inheriting `tf.keras.Model`, call method must be implimated.\n" 130 | ] 131 | }, 132 | { 133 | "cell_type": "code", 134 | "execution_count": 0, 135 | "metadata": { 136 | "colab": {}, 137 | "colab_type": "code", 138 | "id": "f9cYPCt_zUUJ" 139 | }, 140 | "outputs": [], 141 | "source": [ 142 | "class Model(tf.keras.Model):\n", 143 | " def __init__(self):\n", 144 | " super(Model, self).__init__()\n", 145 | " \n", 146 | " self.dense1 = L.Dense(10)\n", 147 | " self.dense2 = L.Dense(5)\n", 148 | " self.dense3 = L.Dense(3)\n", 149 | " \n", 150 | " def call(self, x):\n", 151 | " \n", 152 | " x = self.dense1(x)\n", 153 | " x = self.dense2(x)\n", 154 | " x = self.dense3(x)\n", 155 | " \n", 156 | " return x" 157 | ] 158 | }, 159 | { 160 | "cell_type": "code", 161 | "execution_count": 0, 162 | "metadata": { 163 | "colab": {}, 164 | "colab_type": "code", 165 | "id": "9Xwy3Mf3zqyb" 166 | }, 167 | "outputs": [], 168 | "source": [ 169 | "model = Model()" 170 | ] 171 | }, 172 | { 173 | "cell_type": "code", 174 | "execution_count": 0, 175 | "metadata": { 176 | "colab": {}, 177 | "colab_type": "code", 178 | "id": "rkfLpKsizr7Z" 179 | }, 180 | "outputs": [], 181 | "source": [ 182 | "model.compile(optimizer=tf.train.AdamOptimizer(),\n", 183 | " loss=tf.losses.mean_squared_error)\n", 184 | "_ = model(tf.random_normal(shape=[100, 20]))" 185 | ] 186 | }, 187 | { 188 | "cell_type": "code", 189 | "execution_count": 15, 190 | "metadata": { 191 | "colab": { 192 | "base_uri": "https://localhost:8080/", 193 | "height": 251 194 | }, 195 | "colab_type": "code", 196 | "id": "-7rHSADzzxX_", 197 | "outputId": "ad69842d-6471-4beb-d7ef-f50be5ac7d35" 198 | }, 199 | "outputs": [ 200 | { 201 | "name": "stdout", 202 | "output_type": "stream", 203 | "text": [ 204 | "_________________________________________________________________\n", 205 | "Layer (type) Output Shape Param # \n", 206 | "=================================================================\n", 207 | "dense_6 (Dense) multiple 210 \n", 208 | "_________________________________________________________________\n", 209 | "dense_7 (Dense) multiple 55 \n", 210 | "_________________________________________________________________\n", 211 | "dense_8 (Dense) multiple 18 \n", 212 | "=================================================================\n", 213 | "Total params: 283\n", 214 | "Trainable params: 283\n", 215 | "Non-trainable params: 0\n", 216 | "_________________________________________________________________\n" 217 | ] 218 | } 219 | ], 220 | "source": [ 221 | "model.summary()" 222 | ] 223 | }, 224 | { 225 | "cell_type": "code", 226 | "execution_count": 0, 227 | "metadata": { 228 | "colab": {}, 229 | "colab_type": "code", 230 | "id": "lvSm_AS_zzQP" 231 | }, 232 | "outputs": [], 233 | "source": [] 234 | } 235 | ], 236 | "metadata": { 237 | "colab": { 238 | "include_colab_link": true, 239 | "name": "Sequential_and_Model.ipynb", 240 | "provenance": [], 241 | "version": "0.3.2" 242 | }, 243 | "kernelspec": { 244 | "display_name": "Python 3", 245 | "language": "python", 246 | "name": "python3" 247 | }, 248 | "language_info": { 249 | "codemirror_mode": { 250 | "name": "ipython", 251 | "version": 3 252 | }, 253 | "file_extension": ".py", 254 | "mimetype": "text/x-python", 255 | "name": "python", 256 | "nbconvert_exporter": "python", 257 | "pygments_lexer": "ipython3", 258 | "version": "3.5.3" 259 | } 260 | }, 261 | "nbformat": 4, 262 | "nbformat_minor": 1 263 | } 264 | -------------------------------------------------------------------------------- /tutorials/03_advanced/graph_convnet.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "Untitled14.ipynb", 7 | "version": "0.3.2", 8 | "provenance": [], 9 | "include_colab_link": true 10 | }, 11 | "kernelspec": { 12 | "name": "python3", 13 | "display_name": "Python 3" 14 | } 15 | }, 16 | "cells": [ 17 | { 18 | "cell_type": "markdown", 19 | "metadata": { 20 | "id": "view-in-github", 21 | "colab_type": "text" 22 | }, 23 | "source": [ 24 | "\"Open" 25 | ] 26 | }, 27 | { 28 | "metadata": { 29 | "id": "kiZexuBZWmZl", 30 | "colab_type": "code", 31 | "outputId": "cba073aa-aab9-4bb5-e141-d8edbcabbc68", 32 | "colab": { 33 | "base_uri": "https://localhost:8080/", 34 | "height": 359 35 | } 36 | }, 37 | "cell_type": "code", 38 | "source": [ 39 | "!pip install tf-nightly-2.0-preview" 40 | ], 41 | "execution_count": 1, 42 | "outputs": [ 43 | { 44 | "output_type": "stream", 45 | "text": [ 46 | "Requirement already satisfied: tf-nightly-2.0-preview in /usr/local/lib/python3.6/dist-packages (2.0.0.dev20190126)\n", 47 | "Requirement already satisfied: absl-py>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (0.7.0)\n", 48 | "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (0.7.1)\n", 49 | "Requirement already satisfied: tensorflow-estimator-2.0-preview in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.13.0.dev2019012600)\n", 50 | "Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (0.1.1)\n", 51 | "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (3.6.1)\n", 52 | "Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (0.2.2)\n", 53 | "Requirement already satisfied: tb-nightly<1.14.0a0,>=1.13.0a0 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.13.0a20190126)\n", 54 | "Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.0.6)\n", 55 | "Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.11.0)\n", 56 | "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (0.32.3)\n", 57 | "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.15.0)\n", 58 | "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.0.5)\n", 59 | "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.14.6)\n", 60 | "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.1.0)\n", 61 | "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.6.1->tf-nightly-2.0-preview) (40.6.3)\n", 62 | "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tb-nightly<1.14.0a0,>=1.13.0a0->tf-nightly-2.0-preview) (0.14.1)\n", 63 | "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tb-nightly<1.14.0a0,>=1.13.0a0->tf-nightly-2.0-preview) (3.0.1)\n", 64 | "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.6->tf-nightly-2.0-preview) (2.8.0)\n" 65 | ], 66 | "name": "stdout" 67 | } 68 | ] 69 | }, 70 | { 71 | "metadata": { 72 | "id": "fExBv4RaW1lL", 73 | "colab_type": "code", 74 | "colab": {} 75 | }, 76 | "cell_type": "code", 77 | "source": [ 78 | "import math\n", 79 | "import tensorflow as tf" 80 | ], 81 | "execution_count": 0, 82 | "outputs": [] 83 | }, 84 | { 85 | "metadata": { 86 | "id": "x7QuA8JpNGIq", 87 | "colab_type": "code", 88 | "colab": { 89 | "base_uri": "https://localhost:8080/", 90 | "height": 35 91 | }, 92 | "outputId": "0138f535-d7ca-421c-ae67-3d0f55f6c2cd" 93 | }, 94 | "cell_type": "code", 95 | "source": [ 96 | "class GraphConvolution(tf.keras.layers.Layer):\n", 97 | " \"\"\"\n", 98 | " Simple GCN layer, similar to https://arxiv.org/abs/1609.02907\n", 99 | " \"\"\"\n", 100 | "\n", 101 | " def __init__(self, out_features, use_bias=True):\n", 102 | " super(GraphConvolution, self).__init__()\n", 103 | " \n", 104 | " self.out_features = out_features \n", 105 | " self.use_bias = use_bias\n", 106 | " \n", 107 | " def build(self, input_shape):\n", 108 | " \n", 109 | " stddv = 1 / math.sqrt(self.out_features)\n", 110 | " initializer = tf.keras.initializers.RandomUniform(\n", 111 | " minval=-stddv,\n", 112 | " maxval=stddv\n", 113 | " )\n", 114 | " \n", 115 | " self.kernel = self.add_variable(\n", 116 | " name=\"kernel\", \n", 117 | " shape=[int(input_shape[-1]), self.out_features],\n", 118 | " initializer=initializer\n", 119 | " )\n", 120 | " \n", 121 | " if self.use_bias:\n", 122 | " self.bias = self.add_variable(\n", 123 | " name=\"bias\",\n", 124 | " shape=[self.out_features],\n", 125 | " initializer=initializer\n", 126 | " )\n", 127 | " def call(self, input, adj):\n", 128 | " support = tf.matmul(input, self.kernel)\n", 129 | " output = tf.sparse.sparse_dense_matmul(adj, support)\n", 130 | " if self.bias:\n", 131 | " return output + self.bias\n", 132 | " else:\n", 133 | " return output\n", 134 | "\n", 135 | " def __repr__(self):\n", 136 | "\n", 137 | " \n", 138 | " return self.__class__.__name__ + ' (' \\\n", 139 | " + \"output features is : \" \\\n", 140 | " + str(self.out_features) + ')'\n", 141 | "\n", 142 | "layer = GraphConvolution(10)\n", 143 | "layer" 144 | ], 145 | "execution_count": 11, 146 | "outputs": [ 147 | { 148 | "output_type": "execute_result", 149 | "data": { 150 | "text/plain": [ 151 | "GraphConvolution (output features is : 10)" 152 | ] 153 | }, 154 | "metadata": { 155 | "tags": [] 156 | }, 157 | "execution_count": 11 158 | } 159 | ] 160 | }, 161 | { 162 | "metadata": { 163 | "id": "eb0rUP4eaH7l", 164 | "colab_type": "code", 165 | "colab": {} 166 | }, 167 | "cell_type": "code", 168 | "source": [ 169 | "layer = GraphConvolution(5)" 170 | ], 171 | "execution_count": 0, 172 | "outputs": [] 173 | }, 174 | { 175 | "metadata": { 176 | "id": "F8xZHx4paTbB", 177 | "colab_type": "code", 178 | "colab": {} 179 | }, 180 | "cell_type": "code", 181 | "source": [ 182 | "class GCN(tf.keras.Model):\n", 183 | " def __init__(self, nhid, nclass, dropout):\n", 184 | " super(GCN, self).__init__()\n", 185 | "\n", 186 | " self.gc1 = GraphConvolution(nhid)\n", 187 | " self.gc2 = GraphConvolution(nclass)\n", 188 | " self.dropout = tf.keras.layers.Dropout(dropout)\n", 189 | "\n", 190 | " def forward(self, x, adj, training=False):\n", 191 | " x = tf.relu(self.gc1(x, adj))\n", 192 | " x = self.dropout(x, self.dropout, training=training)\n", 193 | " x = self.gc2(x, adj)\n", 194 | " return tf.nn.log_softmax(x, axis=1)" 195 | ], 196 | "execution_count": 0, 197 | "outputs": [] 198 | }, 199 | { 200 | "metadata": { 201 | "id": "eU_7dIXqafFJ", 202 | "colab_type": "code", 203 | "colab": {} 204 | }, 205 | "cell_type": "code", 206 | "source": [ 207 | "model = GCN(50, 2, 0.3)" 208 | ], 209 | "execution_count": 0, 210 | "outputs": [] 211 | }, 212 | { 213 | "metadata": { 214 | "id": "0CQZr1KRbi61", 215 | "colab_type": "code", 216 | "outputId": "aceefb89-2fb3-400c-e213-5d0b71bf2876", 217 | "colab": { 218 | "base_uri": "https://localhost:8080/", 219 | "height": 35 220 | } 221 | }, 222 | "cell_type": "code", 223 | "source": [ 224 | "model" 225 | ], 226 | "execution_count": 15, 227 | "outputs": [ 228 | { 229 | "output_type": "execute_result", 230 | "data": { 231 | "text/plain": [ 232 | "<__main__.GCN at 0x7fd8d7d0d8d0>" 233 | ] 234 | }, 235 | "metadata": { 236 | "tags": [] 237 | }, 238 | "execution_count": 15 239 | } 240 | ] 241 | }, 242 | { 243 | "metadata": { 244 | "id": "Wpk4IkDaSSyp", 245 | "colab_type": "code", 246 | "colab": {} 247 | }, 248 | "cell_type": "code", 249 | "source": [ 250 | "" 251 | ], 252 | "execution_count": 0, 253 | "outputs": [] 254 | } 255 | ] 256 | } -------------------------------------------------------------------------------- /tutorials/99_others/BN_bug.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "BN_bug.ipynb", 7 | "version": "0.3.2", 8 | "provenance": [], 9 | "include_colab_link": true 10 | }, 11 | "kernelspec": { 12 | "name": "python3", 13 | "display_name": "Python 3" 14 | } 15 | }, 16 | "cells": [ 17 | { 18 | "cell_type": "markdown", 19 | "metadata": { 20 | "id": "view-in-github", 21 | "colab_type": "text" 22 | }, 23 | "source": [ 24 | "\"Open" 25 | ] 26 | }, 27 | { 28 | "metadata": { 29 | "id": "JR1iortXoU4D", 30 | "colab_type": "code", 31 | "colab": { 32 | "base_uri": "https://localhost:8080/", 33 | "height": 667 34 | }, 35 | "outputId": "4791c2d6-6eb0-4b28-f40d-1a1c8794af5a" 36 | }, 37 | "cell_type": "code", 38 | "source": [ 39 | "!pip install tf-nightly-2.0-preview" 40 | ], 41 | "execution_count": 1, 42 | "outputs": [ 43 | { 44 | "output_type": "stream", 45 | "text": [ 46 | "Collecting tf-nightly-2.0-preview\n", 47 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/48/0f/ce6e382f12ee04c5f5f11f97242da093534596981ba14f6a2318bed8b508/tf_nightly_2.0_preview-1.13.0.dev20190116-cp36-cp36m-manylinux1_x86_64.whl (75.0MB)\n", 48 | "\u001b[K 100% |████████████████████████████████| 75.0MB 325kB/s \n", 49 | "\u001b[?25hRequirement already satisfied: gast>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (0.2.1.post0)\n", 50 | "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.0.5)\n", 51 | "Collecting tb-nightly<1.14.0a0,>=1.13.0a0 (from tf-nightly-2.0-preview)\n", 52 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/5f/97/202716b08532d905b995d2a3d8dfbbe834ce6f56cbc66cbfbf9928373b79/tb_nightly-1.13.0a20190116-py3-none-any.whl (3.2MB)\n", 53 | "\u001b[K 100% |████████████████████████████████| 3.2MB 9.1MB/s \n", 54 | "\u001b[?25hRequirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.11.0)\n", 55 | "Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.0.6)\n", 56 | "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.14.6)\n", 57 | "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.1.0)\n", 58 | "Requirement already satisfied: absl-py>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (0.6.1)\n", 59 | "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.15.0)\n", 60 | "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (3.6.1)\n", 61 | "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (0.7.1)\n", 62 | "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (0.32.3)\n", 63 | "Collecting google-pasta>=0.1.1 (from tf-nightly-2.0-preview)\n", 64 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/86/6c/9eabce1c1cdaa657751a802f94d71ca29b8f82e10cac97c3fd5f8c82736c/google_pasta-0.1.1-py3-none-any.whl (51kB)\n", 65 | "\u001b[K 100% |████████████████████████████████| 61kB 26.6MB/s \n", 66 | "\u001b[?25hCollecting tensorflow-estimator-2.0-preview (from tf-nightly-2.0-preview)\n", 67 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/66/57/755322cd96bcd1d9122c057de9774322545cc724d4ece536ed5ca4c23b56/tensorflow_estimator_2.0_preview-1.13.0.dev2019011600-py2.py3-none-any.whl (243kB)\n", 68 | "\u001b[K 100% |████████████████████████████████| 245kB 20.0MB/s \n", 69 | "\u001b[?25hRequirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tb-nightly<1.14.0a0,>=1.13.0a0->tf-nightly-2.0-preview) (3.0.1)\n", 70 | "Requirement already satisfied: werkzeug>=0.11.10 in /usr/local/lib/python3.6/dist-packages (from tb-nightly<1.14.0a0,>=1.13.0a0->tf-nightly-2.0-preview) (0.14.1)\n", 71 | "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.6->tf-nightly-2.0-preview) (2.8.0)\n", 72 | "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.6.1->tf-nightly-2.0-preview) (40.6.3)\n", 73 | "Collecting mock>=2.0.0 (from tensorflow-estimator-2.0-preview->tf-nightly-2.0-preview)\n", 74 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl (56kB)\n", 75 | "\u001b[K 100% |████████████████████████████████| 61kB 26.2MB/s \n", 76 | "\u001b[?25hCollecting pbr>=0.11 (from mock>=2.0.0->tensorflow-estimator-2.0-preview->tf-nightly-2.0-preview)\n", 77 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/f3/04/fddc1c2dd75b256eda4d360024692231a2c19a0c61ad7f4a162407c1ab58/pbr-5.1.1-py2.py3-none-any.whl (106kB)\n", 78 | "\u001b[K 100% |████████████████████████████████| 112kB 20.9MB/s \n", 79 | "\u001b[?25hInstalling collected packages: tb-nightly, google-pasta, pbr, mock, tensorflow-estimator-2.0-preview, tf-nightly-2.0-preview\n", 80 | "Successfully installed google-pasta-0.1.1 mock-2.0.0 pbr-5.1.1 tb-nightly-1.13.0a20190116 tensorflow-estimator-2.0-preview-1.13.0.dev2019011600 tf-nightly-2.0-preview-1.13.0.dev20190116\n" 81 | ], 82 | "name": "stdout" 83 | } 84 | ] 85 | }, 86 | { 87 | "metadata": { 88 | "id": "ijIoaP0voajK", 89 | "colab_type": "code", 90 | "colab": {} 91 | }, 92 | "cell_type": "code", 93 | "source": [ 94 | "import tensorflow as tf" 95 | ], 96 | "execution_count": 0, 97 | "outputs": [] 98 | }, 99 | { 100 | "metadata": { 101 | "id": "w1hi8CfeocCi", 102 | "colab_type": "code", 103 | "colab": {} 104 | }, 105 | "cell_type": "code", 106 | "source": [ 107 | "X = tf.random.normal([100, 10])\n", 108 | "y = tf.random.normal([100, 1])\n", 109 | "\n", 110 | "model = tf.keras.Sequential([\n", 111 | " tf.keras.layers.Dense(10),\n", 112 | " tf.keras.layers.ReLU(),\n", 113 | " tf.keras.layers.BatchNormalization(-1),\n", 114 | " tf.keras.layers.Dense(1),\n", 115 | "])" 116 | ], 117 | "execution_count": 0, 118 | "outputs": [] 119 | }, 120 | { 121 | "metadata": { 122 | "id": "YX8vCb3Nouva", 123 | "colab_type": "code", 124 | "colab": {} 125 | }, 126 | "cell_type": "code", 127 | "source": [ 128 | "loss_fn = lambda y_true, y_pred: tf.reduce_sum(\n", 129 | " tf.keras.losses.mse(y_true, y_pred)\n", 130 | ")\n", 131 | "\n", 132 | "optimizer = tf.keras.optimizers.Adam()" 133 | ], 134 | "execution_count": 0, 135 | "outputs": [] 136 | }, 137 | { 138 | "metadata": { 139 | "id": "t9FD54mipIaW", 140 | "colab_type": "code", 141 | "colab": { 142 | "base_uri": "https://localhost:8080/", 143 | "height": 397 144 | }, 145 | "outputId": "ed20d170-6de1-459f-fa77-e85cae30959e" 146 | }, 147 | "cell_type": "code", 148 | "source": [ 149 | "for _ in range(10):\n", 150 | " \n", 151 | " with tf.GradientTape() as g:\n", 152 | " \n", 153 | " y_pred = model(X)\n", 154 | " loss = loss_fn(y, y_pred)\n", 155 | " \n", 156 | " grads = g.gradient(loss, model.variables)\n", 157 | " optimizer.apply_gradients(zip(grads, model.variables))\n", 158 | " \n", 159 | " print(loss.numpy())" 160 | ], 161 | "execution_count": 15, 162 | "outputs": [ 163 | { 164 | "output_type": "stream", 165 | "text": [ 166 | "WARNING:tensorflow:Gradients does not exist for variables ['sequential/batch_normalization_v2/moving_mean:0', 'sequential/batch_normalization_v2/moving_variance:0'] when minimizing the loss.\n", 167 | "131.86316\n", 168 | "WARNING:tensorflow:Gradients does not exist for variables ['sequential/batch_normalization_v2/moving_mean:0', 'sequential/batch_normalization_v2/moving_variance:0'] when minimizing the loss.\n", 169 | "130.33951\n", 170 | "WARNING:tensorflow:Gradients does not exist for variables ['sequential/batch_normalization_v2/moving_mean:0', 'sequential/batch_normalization_v2/moving_variance:0'] when minimizing the loss.\n", 171 | "128.8607\n", 172 | "WARNING:tensorflow:Gradients does not exist for variables ['sequential/batch_normalization_v2/moving_mean:0', 'sequential/batch_normalization_v2/moving_variance:0'] when minimizing the loss.\n", 173 | "127.42684\n", 174 | "WARNING:tensorflow:Gradients does not exist for variables ['sequential/batch_normalization_v2/moving_mean:0', 'sequential/batch_normalization_v2/moving_variance:0'] when minimizing the loss.\n", 175 | "126.03555\n", 176 | "WARNING:tensorflow:Gradients does not exist for variables ['sequential/batch_normalization_v2/moving_mean:0', 'sequential/batch_normalization_v2/moving_variance:0'] when minimizing the loss.\n", 177 | "124.68613\n", 178 | "WARNING:tensorflow:Gradients does not exist for variables ['sequential/batch_normalization_v2/moving_mean:0', 'sequential/batch_normalization_v2/moving_variance:0'] when minimizing the loss.\n", 179 | "123.37666\n", 180 | "WARNING:tensorflow:Gradients does not exist for variables ['sequential/batch_normalization_v2/moving_mean:0', 'sequential/batch_normalization_v2/moving_variance:0'] when minimizing the loss.\n", 181 | "122.11482\n", 182 | "WARNING:tensorflow:Gradients does not exist for variables ['sequential/batch_normalization_v2/moving_mean:0', 'sequential/batch_normalization_v2/moving_variance:0'] when minimizing the loss.\n", 183 | "120.906\n", 184 | "WARNING:tensorflow:Gradients does not exist for variables ['sequential/batch_normalization_v2/moving_mean:0', 'sequential/batch_normalization_v2/moving_variance:0'] when minimizing the loss.\n", 185 | "119.73827\n" 186 | ], 187 | "name": "stdout" 188 | } 189 | ] 190 | }, 191 | { 192 | "metadata": { 193 | "id": "6l8V7UU4qBsi", 194 | "colab_type": "code", 195 | "colab": {} 196 | }, 197 | "cell_type": "code", 198 | "source": [ 199 | "" 200 | ], 201 | "execution_count": 0, 202 | "outputs": [] 203 | } 204 | ] 205 | } -------------------------------------------------------------------------------- /tutorials/01_basics/Sequential_and_Model.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "Sequential_and_Model.ipynb", 7 | "version": "0.3.2", 8 | "provenance": [], 9 | "include_colab_link": true 10 | }, 11 | "kernelspec": { 12 | "display_name": "Python 3", 13 | "language": "python", 14 | "name": "python3" 15 | } 16 | }, 17 | "cells": [ 18 | { 19 | "cell_type": "markdown", 20 | "metadata": { 21 | "id": "view-in-github", 22 | "colab_type": "text" 23 | }, 24 | "source": [ 25 | "\"Open" 26 | ] 27 | }, 28 | { 29 | "metadata": { 30 | "id": "LpkzTa7HUju0", 31 | "colab_type": "code", 32 | "colab": { 33 | "base_uri": "https://localhost:8080/", 34 | "height": 667 35 | }, 36 | "outputId": "3882b60a-5863-416f-e390-87d30f94ef95" 37 | }, 38 | "cell_type": "code", 39 | "source": [ 40 | "!pip install tf-nightly-2.0-preview" 41 | ], 42 | "execution_count": 1, 43 | "outputs": [ 44 | { 45 | "output_type": "stream", 46 | "text": [ 47 | "Collecting tf-nightly-2.0-preview\n", 48 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/14/f4/caced52bfdded13e1913c6c6700de847773297d53664666d54cd8f3c06d9/tf_nightly_2.0_preview-1.13.0.dev20190110-cp36-cp36m-manylinux1_x86_64.whl (74.3MB)\n", 49 | "\u001b[K 100% |████████████████████████████████| 74.3MB 419kB/s \n", 50 | "\u001b[?25hRequirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.11.0)\n", 51 | "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (0.7.1)\n", 52 | "Collecting google-pasta>=0.1.0 (from tf-nightly-2.0-preview)\n", 53 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/a5/e2/7d191b4613b20fa149e9ebc952f954650fc1dbcabb39e9387f6f1cd5d313/google_pasta-0.1-py3-none-any.whl (51kB)\n", 54 | "\u001b[K 100% |████████████████████████████████| 61kB 23.2MB/s \n", 55 | "\u001b[?25hRequirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.15.0)\n", 56 | "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.14.6)\n", 57 | "Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.0.6)\n", 58 | "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (0.32.3)\n", 59 | "Requirement already satisfied: absl-py>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (0.6.1)\n", 60 | "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.0.5)\n", 61 | "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (3.6.1)\n", 62 | "Collecting tb-nightly<1.14.0a0,>=1.13.0a0 (from tf-nightly-2.0-preview)\n", 63 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/55/4b/780b9caf8c58b3c1c664781de52f36ff482e30ca0760b1ff4f02ff757cec/tb_nightly-1.13.0a20190107-py3-none-any.whl (3.2MB)\n", 64 | "\u001b[K 100% |████████████████████████████████| 3.2MB 8.9MB/s \n", 65 | "\u001b[?25hRequirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (1.1.0)\n", 66 | "Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from tf-nightly-2.0-preview) (0.2.1.post0)\n", 67 | "Collecting tensorflow-estimator-2.0-preview (from tf-nightly-2.0-preview)\n", 68 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/e2/c4/e821cf2999928616840b24ae3b25d2713d418b713510822a0ecd829e0002/tensorflow_estimator_2.0_preview-1.13.0.dev2019011100-py2.py3-none-any.whl (243kB)\n", 69 | "\u001b[K 100% |████████████████████████████████| 245kB 18.0MB/s \n", 70 | "\u001b[?25hRequirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.6->tf-nightly-2.0-preview) (2.8.0)\n", 71 | "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.6.1->tf-nightly-2.0-preview) (40.6.3)\n", 72 | "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tb-nightly<1.14.0a0,>=1.13.0a0->tf-nightly-2.0-preview) (3.0.1)\n", 73 | "Requirement already satisfied: werkzeug>=0.11.10 in /usr/local/lib/python3.6/dist-packages (from tb-nightly<1.14.0a0,>=1.13.0a0->tf-nightly-2.0-preview) (0.14.1)\n", 74 | "Collecting mock>=2.0.0 (from tensorflow-estimator-2.0-preview->tf-nightly-2.0-preview)\n", 75 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl (56kB)\n", 76 | "\u001b[K 100% |████████████████████████████████| 61kB 22.4MB/s \n", 77 | "\u001b[?25hCollecting pbr>=0.11 (from mock>=2.0.0->tensorflow-estimator-2.0-preview->tf-nightly-2.0-preview)\n", 78 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/f3/04/fddc1c2dd75b256eda4d360024692231a2c19a0c61ad7f4a162407c1ab58/pbr-5.1.1-py2.py3-none-any.whl (106kB)\n", 79 | "\u001b[K 100% |████████████████████████████████| 112kB 19.5MB/s \n", 80 | "\u001b[?25hInstalling collected packages: google-pasta, tb-nightly, pbr, mock, tensorflow-estimator-2.0-preview, tf-nightly-2.0-preview\n", 81 | "Successfully installed google-pasta-0.1 mock-2.0.0 pbr-5.1.1 tb-nightly-1.13.0a20190107 tensorflow-estimator-2.0-preview-1.13.0.dev2019011100 tf-nightly-2.0-preview-1.13.0.dev20190110\n" 82 | ], 83 | "name": "stdout" 84 | } 85 | ] 86 | }, 87 | { 88 | "metadata": { 89 | "colab_type": "code", 90 | "id": "uk1oMHkAyQ2s", 91 | "colab": {} 92 | }, 93 | "cell_type": "code", 94 | "source": [ 95 | "import tensorflow as tf\n", 96 | "import tensorflow.keras.layers as L" 97 | ], 98 | "execution_count": 0, 99 | "outputs": [] 100 | }, 101 | { 102 | "metadata": { 103 | "colab_type": "text", 104 | "id": "7gRtOWU70Z-q" 105 | }, 106 | "cell_type": "markdown", 107 | "source": [ 108 | "## tf.keras.Sequential\n", 109 | "\n", 110 | "`tf.keras.Sequential` recieves a list of layers." 111 | ] 112 | }, 113 | { 114 | "metadata": { 115 | "colab_type": "code", 116 | "id": "kcwWxgaiyUiI", 117 | "colab": {} 118 | }, 119 | "cell_type": "code", 120 | "source": [ 121 | "model = tf.keras.Sequential([\n", 122 | " L.InputLayer([20]),\n", 123 | " L.Dense(10),\n", 124 | " L.Dense(5),\n", 125 | " L.Dense(3)\n", 126 | "])" 127 | ], 128 | "execution_count": 0, 129 | "outputs": [] 130 | }, 131 | { 132 | "metadata": { 133 | "colab_type": "text", 134 | "id": "LjvFRCiY0nfj" 135 | }, 136 | "cell_type": "markdown", 137 | "source": [ 138 | "### Note\n", 139 | "'tf.keras.layers' don't need input_dimension, only output_dimension as above. Input_dimension is determined by first execution, so you have to write `model(inputs_data)` code before writing `model.summary()`." 140 | ] 141 | }, 142 | { 143 | "metadata": { 144 | "colab_type": "code", 145 | "id": "Xm0kVQ4PynHK", 146 | "colab": {} 147 | }, 148 | "cell_type": "code", 149 | "source": [ 150 | "model.compile(optimizer=tf.keras.optimizers.Adam(),\n", 151 | " loss=tf.keras.losses.mean_squared_error)" 152 | ], 153 | "execution_count": 0, 154 | "outputs": [] 155 | }, 156 | { 157 | "metadata": { 158 | "colab_type": "code", 159 | "id": "OIsZnSTvy0sf", 160 | "outputId": "f8e55075-c3ea-4e45-c0f9-047a31ac31b8", 161 | "colab": { 162 | "base_uri": "https://localhost:8080/", 163 | "height": 251 164 | } 165 | }, 166 | "cell_type": "code", 167 | "source": [ 168 | "model.summary()" 169 | ], 170 | "execution_count": 12, 171 | "outputs": [ 172 | { 173 | "output_type": "stream", 174 | "text": [ 175 | "_________________________________________________________________\n", 176 | "Layer (type) Output Shape Param # \n", 177 | "=================================================================\n", 178 | "dense_6 (Dense) (None, 10) 210 \n", 179 | "_________________________________________________________________\n", 180 | "dense_7 (Dense) (None, 5) 55 \n", 181 | "_________________________________________________________________\n", 182 | "dense_8 (Dense) (None, 3) 18 \n", 183 | "=================================================================\n", 184 | "Total params: 283\n", 185 | "Trainable params: 283\n", 186 | "Non-trainable params: 0\n", 187 | "_________________________________________________________________\n" 188 | ], 189 | "name": "stdout" 190 | } 191 | ] 192 | }, 193 | { 194 | "metadata": { 195 | "colab_type": "text", 196 | "id": "-IkJwYAl1l5N" 197 | }, 198 | "cell_type": "markdown", 199 | "source": [ 200 | "## tf.keras.Model\n", 201 | "`tf.keras.Model` is used as below. This is like PyTorch `torch.nn.Module` and `chainer.Chain`. When Inheriting `tf.keras.Model`, call method must be implimated.\n" 202 | ] 203 | }, 204 | { 205 | "metadata": { 206 | "colab_type": "code", 207 | "id": "f9cYPCt_zUUJ", 208 | "colab": {} 209 | }, 210 | "cell_type": "code", 211 | "source": [ 212 | "class Model(tf.keras.Model):\n", 213 | " def __init__(self):\n", 214 | " super(Model, self).__init__()\n", 215 | " \n", 216 | " self.dense1 = L.Dense(10)\n", 217 | " self.dense2 = L.Dense(5)\n", 218 | " self.dense3 = L.Dense(3)\n", 219 | " \n", 220 | " def call(self, x):\n", 221 | " \n", 222 | " x = self.dense1(x)\n", 223 | " x = self.dense2(x)\n", 224 | " x = self.dense3(x)\n", 225 | " \n", 226 | " return x" 227 | ], 228 | "execution_count": 0, 229 | "outputs": [] 230 | }, 231 | { 232 | "metadata": { 233 | "colab_type": "code", 234 | "id": "9Xwy3Mf3zqyb", 235 | "colab": {} 236 | }, 237 | "cell_type": "code", 238 | "source": [ 239 | "model = Model()" 240 | ], 241 | "execution_count": 0, 242 | "outputs": [] 243 | }, 244 | { 245 | "metadata": { 246 | "colab_type": "code", 247 | "id": "rkfLpKsizr7Z", 248 | "colab": {} 249 | }, 250 | "cell_type": "code", 251 | "source": [ 252 | "model.compile(optimizer=tf.keras.optimizers.Adam(),\n", 253 | " loss=tf.keras.losses.mean_squared_error)\n", 254 | "\n", 255 | "# need execution before get summary tf.keras.Model\n", 256 | "_ = model(tf.random.normal(shape=[100, 20]))" 257 | ], 258 | "execution_count": 0, 259 | "outputs": [] 260 | }, 261 | { 262 | "metadata": { 263 | "colab_type": "code", 264 | "id": "-7rHSADzzxX_", 265 | "outputId": "1f8ec6ba-137d-4fb9-e7e1-702ea8b4fef6", 266 | "colab": { 267 | "base_uri": "https://localhost:8080/", 268 | "height": 251 269 | } 270 | }, 271 | "cell_type": "code", 272 | "source": [ 273 | "model.summary()" 274 | ], 275 | "execution_count": 24, 276 | "outputs": [ 277 | { 278 | "output_type": "stream", 279 | "text": [ 280 | "_________________________________________________________________\n", 281 | "Layer (type) Output Shape Param # \n", 282 | "=================================================================\n", 283 | "dense_12 (Dense) multiple 210 \n", 284 | "_________________________________________________________________\n", 285 | "dense_13 (Dense) multiple 55 \n", 286 | "_________________________________________________________________\n", 287 | "dense_14 (Dense) multiple 18 \n", 288 | "=================================================================\n", 289 | "Total params: 283\n", 290 | "Trainable params: 283\n", 291 | "Non-trainable params: 0\n", 292 | "_________________________________________________________________\n" 293 | ], 294 | "name": "stdout" 295 | } 296 | ] 297 | }, 298 | { 299 | "metadata": { 300 | "colab_type": "code", 301 | "id": "lvSm_AS_zzQP", 302 | "colab": {} 303 | }, 304 | "cell_type": "code", 305 | "source": [ 306 | "" 307 | ], 308 | "execution_count": 0, 309 | "outputs": [] 310 | } 311 | ] 312 | } -------------------------------------------------------------------------------- /tutorials/02_intermediate/.ipynb_checkpoints/Convolutinal_Neural_Network-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "colab_type": "text", 7 | "id": "view-in-github" 8 | }, 9 | "source": [ 10 | "\"Open" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 0, 16 | "metadata": { 17 | "colab": {}, 18 | "colab_type": "code", 19 | "id": "Wzbzil3EquvD" 20 | }, 21 | "outputs": [], 22 | "source": [ 23 | "import tensorflow as tf\n", 24 | "import numpy as np\n", 25 | "\n", 26 | "tf.enable_eager_execution()\n", 27 | "tfe = tf.contrib.eager\n", 28 | "L = tf.keras.layers" 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": 10, 34 | "metadata": { 35 | "colab": { 36 | "base_uri": "https://localhost:8080/", 37 | "height": 161 38 | }, 39 | "colab_type": "code", 40 | "id": "_b3wl5DyrGbv", 41 | "outputId": "d220cab9-7499-4529-d376-c9f521fe3f02" 42 | }, 43 | "outputs": [ 44 | { 45 | "name": "stdout", 46 | "output_type": "stream", 47 | "text": [ 48 | "training_data\n", 49 | " (50000, 32, 32, 3)\n", 50 | "test_data\n", 51 | " (10000, 32, 32, 3)\n", 52 | "training_label\n", 53 | " (50000, 1)\n", 54 | "test_label\n", 55 | " (10000, 1)\n" 56 | ] 57 | } 58 | ], 59 | "source": [ 60 | "# Hyper parameters\n", 61 | "num_epochs = 10\n", 62 | "num_classes = 10\n", 63 | "batch_size = 1024\n", 64 | "learning_rate = 0.001\n", 65 | "\n", 66 | "(x_train, y_train), (x_test, y_test) = tfk.datasets.cifar10.load_data()\n", 67 | "\n", 68 | "print(\"training_data\\n\", x_train.shape)\n", 69 | "print(\"test_data\\n\", x_test.shape)\n", 70 | "print(\"training_label\\n\", y_train.shape)\n", 71 | "print(\"test_label\\n\", y_test.shape)" 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": 19, 77 | "metadata": { 78 | "colab": { 79 | "base_uri": "https://localhost:8080/", 80 | "height": 53 81 | }, 82 | "colab_type": "code", 83 | "id": "zQ9dPufBDY_e", 84 | "outputId": "aefb4022-f220-4bec-a09e-2eadf72d110e" 85 | }, 86 | "outputs": [ 87 | { 88 | "name": "stdout", 89 | "output_type": "stream", 90 | "text": [ 91 | "(50000, 3, 32, 32)\n", 92 | "(50000, 10)\n" 93 | ] 94 | } 95 | ], 96 | "source": [ 97 | "x_train_ = tf.transpose(tf.convert_to_tensor(x_train, dtype=tf.float32), \n", 98 | " [0, 3, 1, 2])\n", 99 | "y_train_ = tf.reshape(tf.one_hot(y_train, 10), (-1, 10))\n", 100 | "\n", 101 | "print(x_train_.shape)\n", 102 | "print(y_train_.shape)" 103 | ] 104 | }, 105 | { 106 | "cell_type": "code", 107 | "execution_count": 20, 108 | "metadata": { 109 | "colab": { 110 | "base_uri": "https://localhost:8080/", 111 | "height": 35 112 | }, 113 | "colab_type": "code", 114 | "id": "EIFr75qmBuM-", 115 | "outputId": "79a90c99-eb94-4118-a0de-e527a429441b" 116 | }, 117 | "outputs": [ 118 | { 119 | "name": "stdout", 120 | "output_type": "stream", 121 | "text": [ 122 | "\n" 123 | ] 124 | } 125 | ], 126 | "source": [ 127 | "train_dataset = (\n", 128 | " tf.data.Dataset.from_tensor_slices((x_train, y_train))\n", 129 | " .batch(batch_size)\n", 130 | " .shuffle(10000)\n", 131 | ")\n", 132 | "\n", 133 | "train_dataset = (\n", 134 | " train_dataset.map(lambda x, y: \n", 135 | " (tf.div(tf.cast(\n", 136 | " tf.transpose(x, [0, 3, 1, 2]), tf.float32), 255.0), \n", 137 | " tf.reshape(tf.one_hot(y, 10), (-1, 10))))\n", 138 | ")\n", 139 | "\n", 140 | "print(train_dataset)" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": 21, 146 | "metadata": { 147 | "colab": { 148 | "base_uri": "https://localhost:8080/", 149 | "height": 35 150 | }, 151 | "colab_type": "code", 152 | "id": "jNdtaRRB7d5O", 153 | "outputId": "1d3eecd2-65a7-4c63-e947-eb476b0dcc29" 154 | }, 155 | "outputs": [ 156 | { 157 | "name": "stdout", 158 | "output_type": "stream", 159 | "text": [ 160 | "\n" 161 | ] 162 | } 163 | ], 164 | "source": [ 165 | "test_dataset = (\n", 166 | " tf.data.Dataset.from_tensor_slices((x_test, y_test))\n", 167 | " .batch(1000)\n", 168 | " .shuffle(10000)\n", 169 | ")\n", 170 | "test_dataset = (\n", 171 | " test_dataset.map(lambda x, y: \n", 172 | " (tf.div(tf.cast(\n", 173 | " tf.transpose(x, [0, 3, 1, 2]), tf.float32), 255.0), \n", 174 | " tf.reshape(tf.one_hot(y, 10), (-1, 10))))\n", 175 | ")\n", 176 | "print(test_dataset)" 177 | ] 178 | }, 179 | { 180 | "cell_type": "code", 181 | "execution_count": 0, 182 | "metadata": { 183 | "colab": {}, 184 | "colab_type": "code", 185 | "id": "P5PNiSQWrfAF" 186 | }, 187 | "outputs": [], 188 | "source": [] 189 | }, 190 | { 191 | "cell_type": "code", 192 | "execution_count": 0, 193 | "metadata": { 194 | "colab": {}, 195 | "colab_type": "code", 196 | "id": "S2mYcv4D2UGO" 197 | }, 198 | "outputs": [], 199 | "source": [ 200 | "class Cifar10Model(L.Model):\n", 201 | " def __init__(self):\n", 202 | " super(Cifar10Model, self).__init__(name='cifar_cnn')\n", 203 | " \n", 204 | " self.conv_block1 = L.Sequential([\n", 205 | " L.layers.Conv2D(\n", 206 | " 8, \n", 207 | " 5,\n", 208 | " padding='same',\n", 209 | " activation=tf.nn.relu,\n", 210 | " kernel_initializer=tf.initializers.variance_scaling,\n", 211 | " kernel_regularizer=L.regularizers.l2(l=0.001),\n", 212 | " data_format=\"channels_first\"\n", 213 | " ),\n", 214 | " L.layers.MaxPooling2D(\n", 215 | " (3, 3), \n", 216 | " (2, 2), \n", 217 | " padding='same',\n", 218 | " data_format=\"channels_first\"\n", 219 | " ),\n", 220 | " L.layers.BatchNormalization(axis=1),\n", 221 | " ])\n", 222 | "\n", 223 | " self.conv_block2 = L.Sequential([\n", 224 | " L.layers.Conv2D(\n", 225 | " 16, \n", 226 | " 5,\n", 227 | " padding='same',\n", 228 | " activation=tf.nn.relu,\n", 229 | " kernel_initializer=tf.initializers.variance_scaling,\n", 230 | " kernel_regularizer=L.regularizers.l2(l=0.001),\n", 231 | " data_format=\"channels_first\"\n", 232 | " ),\n", 233 | " L.layers.MaxPooling2D(\n", 234 | " (3, 3), \n", 235 | " (2, 2), \n", 236 | " padding='same',\n", 237 | " data_format=\"channels_first\"\n", 238 | " ),\n", 239 | " L.layers.BatchNormalization(axis=1),\n", 240 | " ])\n", 241 | " \n", 242 | " self.conv_block3 = L.Sequential([\n", 243 | " L.layers.Conv2D(\n", 244 | " 32, \n", 245 | " 5,\n", 246 | " padding='same',\n", 247 | " activation=tf.nn.relu,\n", 248 | " kernel_initializer=tf.initializers.variance_scaling,\n", 249 | " kernel_regularizer=L.regularizers.l2(l=0.001),\n", 250 | " data_format=\"channels_first\"\n", 251 | " ),\n", 252 | " L.layers.MaxPooling2D(\n", 253 | " (3, 3), \n", 254 | " (2, 2), \n", 255 | " padding='same',\n", 256 | " data_format=\"channels_first\"\n", 257 | " ),\n", 258 | " L.layers.BatchNormalization(axis=1),\n", 259 | " ])\n", 260 | " \n", 261 | " self.flatten = L.layers.Flatten()\n", 262 | " self.fc1 = L.layers.Dense(\n", 263 | " 256, \n", 264 | " activation=tf.nn.relu,\n", 265 | " kernel_initializer=tf.initializers.variance_scaling,\n", 266 | " kernel_regularizer=L.regularizers.l2(l=0.001))\n", 267 | " self.dropout = L.layers.Dropout(0.8)\n", 268 | " self.fc2 = L.layers.Dense(10)\n", 269 | " self.softmax = L.layers.Softmax()\n", 270 | "\n", 271 | " def call(self, x):\n", 272 | " x = self.conv_block1(x)\n", 273 | " x = self.conv_block2(x)\n", 274 | " x = self.conv_block3(x)\n", 275 | " x = self.flatten(x)\n", 276 | " x = self.dropout(self.fc1(x))\n", 277 | " x = self.fc2(x)\n", 278 | " return self.softmax(x)" 279 | ] 280 | }, 281 | { 282 | "cell_type": "code", 283 | "execution_count": 0, 284 | "metadata": { 285 | "colab": {}, 286 | "colab_type": "code", 287 | "id": "6VyzP7OTtDGf" 288 | }, 289 | "outputs": [], 290 | "source": [ 291 | "model = Cifar10Model()\n", 292 | "\n", 293 | "def loss_fn(model, x, y):\n", 294 | " y_pre = model(x)\n", 295 | " return tf.keras.losses.categorical_crossentropy(y, y_pre)\n", 296 | "\n", 297 | "def accuracy(model, x, y):\n", 298 | " y_pre = model(x)\n", 299 | " return tf.keras.metrics.categorical_accuracy(y, y_pre)\n", 300 | "\n", 301 | "optimizer = tf.train.AdamOptimizer(learning_rate)" 302 | ] 303 | }, 304 | { 305 | "cell_type": "code", 306 | "execution_count": 45, 307 | "metadata": { 308 | "colab": { 309 | "base_uri": "https://localhost:8080/", 310 | "height": 557 311 | }, 312 | "colab_type": "code", 313 | "id": "7zrxUvWPu6PK", 314 | "outputId": "e3b65c6e-5951-4ca6-9a0e-05125c65e466" 315 | }, 316 | "outputs": [ 317 | { 318 | "name": "stdout", 319 | "output_type": "stream", 320 | "text": [ 321 | "-----epoch 1 -----\n", 322 | "loss: 1.9387904575892858\n", 323 | "acc: 0.29365350762192083\n", 324 | "-----epoch 2 -----\n", 325 | "loss: 1.5621210215043049\n", 326 | "acc: 0.43229566301618305\n", 327 | "-----epoch 3 -----\n", 328 | "loss: 1.4413411665935905\n", 329 | "acc: 0.48328862871442524\n", 330 | "-----epoch 4 -----\n", 331 | "loss: 1.3570086420798788\n", 332 | "acc: 0.5132680231211136\n", 333 | "-----epoch 5 -----\n", 334 | "loss: 1.300194720832669\n", 335 | "acc: 0.5373571746203364\n", 336 | "-----epoch 6 -----\n", 337 | "loss: 1.2398374907824459\n", 338 | "acc: 0.5596067856769172\n", 339 | "-----epoch 7 -----\n", 340 | "loss: 1.197290459457709\n", 341 | "acc: 0.5755784949477838\n", 342 | "-----epoch 8 -----\n", 343 | "loss: 1.154668379803093\n", 344 | "acc: 0.5920901785091478\n", 345 | "-----epoch 9 -----\n", 346 | "loss: 1.130118311667929\n", 347 | "acc: 0.600405790367905\n", 348 | "-----epoch 10 -----\n", 349 | "loss: 1.1001540981993383\n", 350 | "acc: 0.6116759631098533\n" 351 | ] 352 | } 353 | ], 354 | "source": [ 355 | "for j in range(num_epochs):\n", 356 | " \n", 357 | " running_loss = 0\n", 358 | " running_acc = 0\n", 359 | "\n", 360 | " for i, (x_, y_) in enumerate(train_dataset):\n", 361 | " \n", 362 | " with tf.device(\"/gpu:0\"):\n", 363 | " with tf.GradientTape() as tape:\n", 364 | " loss = loss_fn(model, x_, y_)\n", 365 | " acc = accuracy(model, x_, y_)\n", 366 | " grads = tape.gradient(loss, model.variables)\n", 367 | " optimizer.apply_gradients(zip(grads, model.variables))\n", 368 | " running_loss += tf.reduce_mean(loss)\n", 369 | " running_acc += tf.reduce_mean(acc)\n", 370 | " \n", 371 | " print(\"-----epoch {} -----\".format(j + 1))\n", 372 | " print(\"loss: \", running_loss.numpy()/(i + 1))\n", 373 | " print(\"acc: \", running_acc.numpy()/(i + 1)) " 374 | ] 375 | }, 376 | { 377 | "cell_type": "code", 378 | "execution_count": 47, 379 | "metadata": { 380 | "colab": { 381 | "base_uri": "https://localhost:8080/", 382 | "height": 35 383 | }, 384 | "colab_type": "code", 385 | "id": "8m2XzQv3G-na", 386 | "outputId": "ebdcd658-7392-4c43-f688-be07ea468c1d" 387 | }, 388 | "outputs": [ 389 | { 390 | "name": "stdout", 391 | "output_type": "stream", 392 | "text": [ 393 | "test accuracy 0.605\n" 394 | ] 395 | } 396 | ], 397 | "source": [ 398 | "test_accuracy = 0\n", 399 | "for i, (x_, y_) in enumerate(test_dataset):\n", 400 | " test_accuracy += tf.reduce_mean(accuracy(model, x_, y_))\n", 401 | "test_accuracy /= i + 1\n", 402 | "\n", 403 | "print(\"test accuracy {:0.3f}\".format(test_accuracy.numpy()))" 404 | ] 405 | }, 406 | { 407 | "cell_type": "code", 408 | "execution_count": 0, 409 | "metadata": { 410 | "colab": {}, 411 | "colab_type": "code", 412 | "id": "dnuY_yuOLrO-" 413 | }, 414 | "outputs": [], 415 | "source": [] 416 | }, 417 | { 418 | "cell_type": "code", 419 | "execution_count": 0, 420 | "metadata": { 421 | "colab": {}, 422 | "colab_type": "code", 423 | "id": "Rzz2FNtOqzdC" 424 | }, 425 | "outputs": [], 426 | "source": [] 427 | } 428 | ], 429 | "metadata": { 430 | "accelerator": "GPU", 431 | "colab": { 432 | "collapsed_sections": [], 433 | "include_colab_link": true, 434 | "name": "Convolutinal_Neural_Network", 435 | "provenance": [], 436 | "version": "0.3.2" 437 | }, 438 | "kernelspec": { 439 | "display_name": "Python 3", 440 | "language": "python", 441 | "name": "python3" 442 | }, 443 | "language_info": { 444 | "codemirror_mode": { 445 | "name": "ipython", 446 | "version": 3 447 | }, 448 | "file_extension": ".py", 449 | "mimetype": "text/x-python", 450 | "name": "python", 451 | "nbconvert_exporter": "python", 452 | "pygments_lexer": "ipython3", 453 | "version": "3.5.3" 454 | } 455 | }, 456 | "nbformat": 4, 457 | "nbformat_minor": 1 458 | } 459 | -------------------------------------------------------------------------------- /tutorials/00_lowlevel/TF_eager_basics.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "TF_eager_basics", 7 | "version": "0.3.2", 8 | "provenance": [], 9 | "collapsed_sections": [], 10 | "include_colab_link": true 11 | }, 12 | "kernelspec": { 13 | "name": "python3", 14 | "display_name": "Python 3" 15 | } 16 | }, 17 | "cells": [ 18 | { 19 | "cell_type": "markdown", 20 | "metadata": { 21 | "id": "view-in-github", 22 | "colab_type": "text" 23 | }, 24 | "source": [ 25 | "\"Open" 26 | ] 27 | }, 28 | { 29 | "metadata": { 30 | "id": "7Ps6A2NfW4cP", 31 | "colab_type": "code", 32 | "colab": {} 33 | }, 34 | "cell_type": "code", 35 | "source": [ 36 | "import tensorflow as tf\n", 37 | "import numpy as np\n", 38 | "import pandas as pd\n", 39 | "\n", 40 | "# To start eager execution (this must be top of code)\n", 41 | "tf.enable_eager_execution()" 42 | ], 43 | "execution_count": 0, 44 | "outputs": [] 45 | }, 46 | { 47 | "metadata": { 48 | "id": "4dihJR-BDjW3", 49 | "colab_type": "text" 50 | }, 51 | "cell_type": "markdown", 52 | "source": [ 53 | "### Create Tensors" 54 | ] 55 | }, 56 | { 57 | "metadata": { 58 | "id": "a0fFhIF28qZn", 59 | "colab_type": "code", 60 | "colab": { 61 | "base_uri": "https://localhost:8080/", 62 | "height": 53 63 | }, 64 | "outputId": "fbd2f5bb-4743-4085-f0dc-7990d64660c9" 65 | }, 66 | "cell_type": "code", 67 | "source": [ 68 | "x = tf.convert_to_tensor(1.)\n", 69 | "w = tf.convert_to_tensor(2.)\n", 70 | "b = tf.convert_to_tensor(3.)\n", 71 | "\n", 72 | "print(type(x))\n", 73 | "print(x)" 74 | ], 75 | "execution_count": 30, 76 | "outputs": [ 77 | { 78 | "output_type": "stream", 79 | "text": [ 80 | "\n", 81 | "tf.Tensor(1.0, shape=(), dtype=float32)\n" 82 | ], 83 | "name": "stdout" 84 | } 85 | ] 86 | }, 87 | { 88 | "metadata": { 89 | "id": "mgpWIN4QDquZ", 90 | "colab_type": "text" 91 | }, 92 | "cell_type": "markdown", 93 | "source": [ 94 | "### Build a computational graph for Automatic differentiation.\n", 95 | "When you focus the $x$ of two functions, \n", 96 | "\n", 97 | "$$\n", 98 | "y(x) = w x + b\n", 99 | "$$ \n", 100 | "and\n", 101 | "$$ \n", 102 | "z(x) = w x^2 + b x\n", 103 | "$$\n", 104 | "you can write the code of \"build a computational graph for automatic differentiation\" as below." 105 | ] 106 | }, 107 | { 108 | "metadata": { 109 | "id": "32aRX9ZW9Ne3", 110 | "colab_type": "code", 111 | "colab": { 112 | "base_uri": "https://localhost:8080/", 113 | "height": 53 114 | }, 115 | "outputId": "dbf429e3-3f69-492c-b62c-8b04d363e688" 116 | }, 117 | "cell_type": "code", 118 | "source": [ 119 | "with tf.GradientTape(persistent=True) as g:\n", 120 | " g.watch(x)\n", 121 | " y = w * x + b\n", 122 | " z = w * x**2 + b * x\n", 123 | "\n", 124 | "# dy/dx = 2\n", 125 | "# dz/dx = 4 * x + 3 (now x=1 so dz/dx = 7)\n", 126 | "dy_dx = g.gradient(y, x)\n", 127 | "dz_dx = g.gradient(z, x)\n", 128 | " \n", 129 | "print(dy_dx)\n", 130 | "print(dz_dx)" 131 | ], 132 | "execution_count": 31, 133 | "outputs": [ 134 | { 135 | "output_type": "stream", 136 | "text": [ 137 | "tf.Tensor(2.0, shape=(), dtype=float32)\n", 138 | "tf.Tensor(7.0, shape=(), dtype=float32)\n" 139 | ], 140 | "name": "stdout" 141 | } 142 | ] 143 | }, 144 | { 145 | "metadata": { 146 | "id": "xKw6MgH2ITVj", 147 | "colab_type": "text" 148 | }, 149 | "cell_type": "markdown", 150 | "source": [ 151 | "### linear model\n", 152 | "Linear model is as below.\n", 153 | "$$\n", 154 | " y_i = Wx_i + b\n", 155 | "$$" 156 | ] 157 | }, 158 | { 159 | "metadata": { 160 | "id": "EtLcSHQgFULq", 161 | "colab_type": "code", 162 | "colab": { 163 | "base_uri": "https://localhost:8080/", 164 | "height": 197 165 | }, 166 | "outputId": "455e4ad8-1f5c-4eea-a316-ee6b17d1db24" 167 | }, 168 | "cell_type": "code", 169 | "source": [ 170 | "x = tf.random_normal(shape=[10, 3])\n", 171 | "y = tf.random_normal(shape=[10, 2])\n", 172 | "\n", 173 | "# tf.keras.layers.Dense needs only output dimension.\n", 174 | "# When tf.keras.layers get input to calculate output at the first time,\n", 175 | "# the input dimension is determined.\n", 176 | "linear = tf.keras.layers.Dense(units=2)\n", 177 | "predict_y = linear(x)\n", 178 | "\n", 179 | "print(\"weight: \\n\", linear.weights[0])\n", 180 | "print(\"bias:\\n\", linear.weights[1], end=\"\\n\\n\")\n", 181 | "print(\"output shape:\\n\", y.shape)" 182 | ], 183 | "execution_count": 63, 184 | "outputs": [ 185 | { 186 | "output_type": "stream", 187 | "text": [ 188 | "weight: \n", 189 | " \n", 193 | "bias:\n", 194 | " \n", 195 | "\n", 196 | "output shape:\n", 197 | " (10, 2)\n" 198 | ], 199 | "name": "stdout" 200 | } 201 | ] 202 | }, 203 | { 204 | "metadata": { 205 | "id": "Ga7dNc0MIc_N", 206 | "colab_type": "text" 207 | }, 208 | "cell_type": "markdown", 209 | "source": [ 210 | "#### loss function\n", 211 | "TensorFlow eager execution has similar API to PyTorch, however the implementation of \"Build a computational graph for Automatic differentiation\" is a little diferrent.\n", 212 | "At PyTorch, Tensor itself holds calculation graph, and have the method for automatic differentiation. On the other hand, at TensorFlow eager execution, computational graph is keeped by some functions (for example, `tf.GradientTape()`). \n", 213 | "\n", 214 | "When training neural network, we can use `tf.contrib.eager.implicit_value_and_gradients()`. This function recognizes the trainable parameters of NN, holds related computational graph, and return the loss value, parameter instances, and that grads." 215 | ] 216 | }, 217 | { 218 | "metadata": { 219 | "id": "x4tHSe-DHd8P", 220 | "colab_type": "code", 221 | "colab": { 222 | "base_uri": "https://localhost:8080/", 223 | "height": 431 224 | }, 225 | "outputId": "a0ab3d89-f646-4532-962d-0b4361715753" 226 | }, 227 | "cell_type": "code", 228 | "source": [ 229 | "def loss_fn(model, x, y):\n", 230 | " predict_y = model(x)\n", 231 | " return tf.keras.losses.mean_squared_error(predict_y, y)\n", 232 | "\n", 233 | "value_and_grads = tf.contrib.eager.implicit_value_and_gradients(loss_fn)\n", 234 | "optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01)\n", 235 | "\n", 236 | "\n", 237 | "loss, grads = value_and_grads(model=linear, x=x, y=y)\n", 238 | "\n", 239 | "print(\"loss: \\n\", loss, end=\"\\n\\n\")\n", 240 | "print(\"weight grads: \\n\", grads[0][0], end=\"\\n\\n\")\n", 241 | "print(\"weight instances: \\n\", grads[0][1], end=\"\\n\\n\")\n", 242 | "print(\"bias grads: \\n\", grads[1][0], end=\"\\n\\n\")\n", 243 | "print(\"bias instances: \\n\", grads[1][1], end=\"\\n\\n\")" 244 | ], 245 | "execution_count": 64, 246 | "outputs": [ 247 | { 248 | "output_type": "stream", 249 | "text": [ 250 | "loss: \n", 251 | " tf.Tensor(\n", 252 | "[0.04972696 1.6548975 0.33088797 0.23095478 0.6001038 1.9319891\n", 253 | " 2.4406717 0.98343223 0.9945836 0.40538147], shape=(10,), dtype=float32)\n", 254 | "\n", 255 | "weight grads: \n", 256 | " tf.Tensor(\n", 257 | "[[ 2.0563087 -3.6274142]\n", 258 | " [ 4.147823 -0.8882311]\n", 259 | " [-1.2567902 2.3985233]], shape=(3, 2), dtype=float32)\n", 260 | "\n", 261 | "weight instances: \n", 262 | " \n", 266 | "\n", 267 | "bias grads: \n", 268 | " tf.Tensor([-4.62727 1.6120654], shape=(2,), dtype=float32)\n", 269 | "\n", 270 | "bias instances: \n", 271 | " \n", 272 | "\n" 273 | ], 274 | "name": "stdout" 275 | } 276 | ] 277 | }, 278 | { 279 | "metadata": { 280 | "id": "dtX0s9fFN8Pi", 281 | "colab_type": "text" 282 | }, 283 | "cell_type": "markdown", 284 | "source": [ 285 | "#### Optimizing\n", 286 | "We aim to decrese loss value with update parameters as below. \n", 287 | "$$\n", 288 | "\\begin{align}\n", 289 | "W & \\leftarrow W - \\epsilon \\frac{dLoss(W)}{dW}\\\\\\\n", 290 | "b & \\leftarrow b - \\epsilon \\frac{dLoss(W)}{db}\n", 291 | "\\end{align}\n", 292 | "$$\n", 293 | "\n", 294 | "where $\\epsilon$ is learning rate.\n", 295 | "\n", 296 | "After understanding this code, you are able to write training loop code." 297 | ] 298 | }, 299 | { 300 | "metadata": { 301 | "id": "u0b2W_z4Ndw6", 302 | "colab_type": "code", 303 | "colab": { 304 | "base_uri": "https://localhost:8080/", 305 | "height": 53 306 | }, 307 | "outputId": "732a24c7-6a78-49aa-e1fc-b51cf8028acf" 308 | }, 309 | "cell_type": "code", 310 | "source": [ 311 | "# initial loss value of sum of all data.\n", 312 | "loss, grad = value_and_grads(model=linear, x=x, y=y)\n", 313 | "print(\"loss: \", tf.reduce_sum(loss))\n", 314 | "\n", 315 | "# update prameters using grads\n", 316 | "optimizer.apply_gradients(grads)\n", 317 | "\n", 318 | "# loss value after update (may be less than before update)\n", 319 | "loss, grad = value_and_grads(model=linear, x=x, y=y)\n", 320 | "print(\"loss: \", tf.reduce_sum(loss))" 321 | ], 322 | "execution_count": 65, 323 | "outputs": [ 324 | { 325 | "output_type": "stream", 326 | "text": [ 327 | "loss: tf.Tensor(9.622629, shape=(), dtype=float32)\n", 328 | "loss: tf.Tensor(9.004152, shape=(), dtype=float32)\n" 329 | ], 330 | "name": "stdout" 331 | } 332 | ] 333 | }, 334 | { 335 | "metadata": { 336 | "id": "H-LY5x9o6gO2", 337 | "colab_type": "text" 338 | }, 339 | "cell_type": "markdown", 340 | "source": [ 341 | "### Data\n", 342 | "#### Convert to tf.Tensor from numpy.ndarray" 343 | ] 344 | }, 345 | { 346 | "metadata": { 347 | "id": "U_sEm-NbSEU3", 348 | "colab_type": "code", 349 | "colab": { 350 | "base_uri": "https://localhost:8080/", 351 | "height": 179 352 | }, 353 | "outputId": "cf263a9d-fa98-44d6-87dc-4bc2f0d7ee3c" 354 | }, 355 | "cell_type": "code", 356 | "source": [ 357 | "X_numpy = np.random.randn(3, 3)\n", 358 | "print(type(X_numpy))\n", 359 | "print(X_numpy)\n", 360 | "\n", 361 | "X_tensor = tf.convert_to_tensor(X_numpy)\n", 362 | "print(type(X_tensor))\n", 363 | "print(X_tensor)" 364 | ], 365 | "execution_count": 76, 366 | "outputs": [ 367 | { 368 | "output_type": "stream", 369 | "text": [ 370 | "\n", 371 | "[[-0.58254555 0.31973299 -1.05691421]\n", 372 | " [-0.50315322 0.52309492 -0.38714436]\n", 373 | " [-0.20711872 0.55952568 0.17786334]]\n", 374 | "\n", 375 | "tf.Tensor(\n", 376 | "[[-0.58254555 0.31973299 -1.05691421]\n", 377 | " [-0.50315322 0.52309492 -0.38714436]\n", 378 | " [-0.20711872 0.55952568 0.17786334]], shape=(3, 3), dtype=float64)\n" 379 | ], 380 | "name": "stdout" 381 | } 382 | ] 383 | }, 384 | { 385 | "metadata": { 386 | "id": "x9wD2JR1TIIC", 387 | "colab_type": "text" 388 | }, 389 | "cell_type": "markdown", 390 | "source": [ 391 | "#### conver to numpy.array from tf.Tensor" 392 | ] 393 | }, 394 | { 395 | "metadata": { 396 | "id": "KFpSvjf0TRvt", 397 | "colab_type": "code", 398 | "colab": { 399 | "base_uri": "https://localhost:8080/", 400 | "height": 179 401 | }, 402 | "outputId": "b804e443-233c-4df5-ff55-2f440113789f" 403 | }, 404 | "cell_type": "code", 405 | "source": [ 406 | "X_tensor = tf.random_normal(shape=[3, 3])\n", 407 | "print(type(X_tensor))\n", 408 | "print(X_tensor)\n", 409 | "\n", 410 | "X_numpy = X_tensor.numpy()\n", 411 | "print(type(X_numpy))\n", 412 | "print(X_numpy)" 413 | ], 414 | "execution_count": 77, 415 | "outputs": [ 416 | { 417 | "output_type": "stream", 418 | "text": [ 419 | "\n", 420 | "tf.Tensor(\n", 421 | "[[-0.91925687 -0.6136823 -1.4136612 ]\n", 422 | " [-0.5081144 -1.202485 1.4589684 ]\n", 423 | " [ 0.24295777 -0.2634425 -0.9960576 ]], shape=(3, 3), dtype=float32)\n", 424 | "\n", 425 | "[[-0.91925687 -0.6136823 -1.4136612 ]\n", 426 | " [-0.5081144 -1.202485 1.4589684 ]\n", 427 | " [ 0.24295777 -0.2634425 -0.9960576 ]]\n" 428 | ], 429 | "name": "stdout" 430 | } 431 | ] 432 | }, 433 | { 434 | "metadata": { 435 | "id": "wKuM5lTRTYD3", 436 | "colab_type": "text" 437 | }, 438 | "cell_type": "markdown", 439 | "source": [ 440 | "### tf.Dataset pipline" 441 | ] 442 | }, 443 | { 444 | "metadata": { 445 | "id": "sTrzI8uZUUeL", 446 | "colab_type": "code", 447 | "colab": {} 448 | }, 449 | "cell_type": "code", 450 | "source": [ 451 | "" 452 | ], 453 | "execution_count": 0, 454 | "outputs": [] 455 | } 456 | ] 457 | } -------------------------------------------------------------------------------- /tutorials/02_intermediate/.ipynb_checkpoints/Recurrent_Neural_Network-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "colab_type": "text", 7 | "id": "view-in-github" 8 | }, 9 | "source": [ 10 | "\"Open" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 0, 16 | "metadata": { 17 | "colab": {}, 18 | "colab_type": "code", 19 | "id": "VkJooysd9saV" 20 | }, 21 | "outputs": [], 22 | "source": [ 23 | "import numpy as np\n", 24 | "import tensorflow as tf\n", 25 | "import matplotlib.pyplot as plt\n", 26 | "%matplotlib inline\n", 27 | "\n", 28 | "tf.enable_eager_execution()\n", 29 | "L = tf.keras.layers\n", 30 | "tfe = tf.contrib.eager" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": 32, 36 | "metadata": { 37 | "colab": { 38 | "base_uri": "https://localhost:8080/", 39 | "height": 89 40 | }, 41 | "colab_type": "code", 42 | "id": "mvWroGCq-AGM", 43 | "outputId": "9eed5404-af6c-4c0a-b0a7-450b13ebdf45" 44 | }, 45 | "outputs": [ 46 | { 47 | "name": "stdout", 48 | "output_type": "stream", 49 | "text": [ 50 | "training_data: (60000, 28, 28)\n", 51 | "test_data: (10000, 28, 28)\n", 52 | "training_label: (60000,)\n", 53 | "test_label: (10000,)\n" 54 | ] 55 | } 56 | ], 57 | "source": [ 58 | "# Hyper parameters\n", 59 | "num_epochs = 25\n", 60 | "num_classes = 10\n", 61 | "batch_size = 512\n", 62 | "learning_rate = 0.001\n", 63 | "\n", 64 | "(x_train, y_train), (x_test, y_test) = tfk.datasets.mnist.load_data()\n", 65 | "\n", 66 | "print(\"training_data: \", x_train.shape)\n", 67 | "print(\"test_data: \", x_test.shape)\n", 68 | "print(\"training_label: \", y_train.shape)\n", 69 | "print(\"test_label: \", y_test.shape)" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 33, 75 | "metadata": { 76 | "colab": { 77 | "base_uri": "https://localhost:8080/", 78 | "height": 89 79 | }, 80 | "colab_type": "code", 81 | "id": "int-eal1-RBm", 82 | "outputId": "79f4ab5e-1944-4ba3-f2c6-14f586b89980" 83 | }, 84 | "outputs": [ 85 | { 86 | "name": "stdout", 87 | "output_type": "stream", 88 | "text": [ 89 | "training_data: (60000, 28, 28)\n", 90 | "test_data: (10000, 28, 28)\n", 91 | "training_label: (60000, 10)\n", 92 | "test_label: (10000, 10)\n" 93 | ] 94 | } 95 | ], 96 | "source": [ 97 | "x_train_eager = tf.convert_to_tensor(x_train, dtype=tf.float32)\n", 98 | "x_test_eager = tf.convert_to_tensor(x_test, dtype=tf.float32)\n", 99 | "y_train_eager = tf.reshape(tf.one_hot(y_train, 10), (-1, 10))\n", 100 | "y_test_eager = tf.reshape(tf.one_hot(y_test, 10), (-1, 10))\n", 101 | "\n", 102 | "print(\"training_data: \", x_train_eager.shape)\n", 103 | "print(\"test_data: \", x_test_eager.shape)\n", 104 | "print(\"training_label: \", y_train_eager.shape)\n", 105 | "print(\"test_label: \", y_test_eager.shape)" 106 | ] 107 | }, 108 | { 109 | "cell_type": "markdown", 110 | "metadata": { 111 | "colab_type": "text", 112 | "id": "jwcKjSDNGD5W" 113 | }, 114 | "source": [ 115 | "### DataSet\n", 116 | "You make Dataset using `tf.data.Dataset` Class but Keras API doesn't need this dataset. If you write training loop code manually, `Dataset` class is very useful. And using keras API, you need numpy.array inputs instead of tf.Tensor. I don't know why...so you only need numpy preprocessing (or get numpy.array from tf.Tensor using numpy() method after preprocessing using function of tf).\n", 117 | "\n", 118 | "### NOTE\n", 119 | "This notebook we don't need 'tf.data.Dataset'. This code only just for reference." 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": 0, 125 | "metadata": { 126 | "colab": {}, 127 | "colab_type": "code", 128 | "id": "YNU_cq4L-u10" 129 | }, 130 | "outputs": [], 131 | "source": [ 132 | "train_dataset = (\n", 133 | " tf.data.Dataset.from_tensor_slices((x_train_eager, y_train_eager))\n", 134 | " .batch(batch_size)\n", 135 | " .shuffle(10000)\n", 136 | ")\n", 137 | "train_dataset = train_dataset.repeat()" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 0, 143 | "metadata": { 144 | "colab": {}, 145 | "colab_type": "code", 146 | "id": "_npuvGwv-588" 147 | }, 148 | "outputs": [], 149 | "source": [ 150 | "test_dataset = (\n", 151 | " tf.data.Dataset.from_tensor_slices((x_test_eager, y_test_eager))\n", 152 | " .batch(1000)\n", 153 | " .shuffle(10000)\n", 154 | ")\n", 155 | "test_dataset = test_dataset.repeat()" 156 | ] 157 | }, 158 | { 159 | "cell_type": "markdown", 160 | "metadata": { 161 | "colab_type": "text", 162 | "id": "e-Z06hTWOx4P" 163 | }, 164 | "source": [ 165 | "### RNN using LSTM\n", 166 | "In keras API, LSTM recives inputs tensor whose shape is (batch_size, seqence_length, feature_dim), and output tensor whose shape is (batch_size, fearure_dim).When you need all time sequence data, you have to give `return_sequences=True` to LSTM's constractor. Generally, when you stack LSTM's, you need all sequence data." 167 | ] 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": 0, 172 | "metadata": { 173 | "colab": {}, 174 | "colab_type": "code", 175 | "id": "gz5RrFnm_HzM" 176 | }, 177 | "outputs": [], 178 | "source": [ 179 | "class RNN(tf.keras.Model):\n", 180 | " def __init__(self, hidden_size=10, num_layers=2, num_classes=10):\n", 181 | " super(RNN, self).__init__(name='mnist_rnn')\n", 182 | " self.hidden_size = hidden_size\n", 183 | " self.num_layers = num_layers\n", 184 | " \n", 185 | " self.lstm = self.get_lstm_layers(hidden_size, num_layers) \n", 186 | " self.fc = L.Dense(num_classes, activation=\"softmax\")\n", 187 | " \n", 188 | " @staticmethod\n", 189 | " def get_lstm_layers(hidden_size, num_layers):\n", 190 | " lstm_layers = []\n", 191 | " # we need all sequence data. write return_sequences=True! \n", 192 | " for i in range(num_layers-1):\n", 193 | " lstm_layers.append(\n", 194 | " L.CuDNNLSTM(units=hidden_size, return_sequences=True)\n", 195 | " )\n", 196 | " # the final layer return only final sequence\n", 197 | " # if you need all sequences, you have to write return_sequences=True.\n", 198 | " lstm_layers.append(L.CuDNNLSTM(units=hidden_size))\n", 199 | " return tf.keras.Sequential(lstm_layers)\n", 200 | " \n", 201 | " def call(self, x): \n", 202 | " # Forward propagate LSTM\n", 203 | " out = self.lstm(x)\n", 204 | " out = self.fc(out)\n", 205 | " return out" 206 | ] 207 | }, 208 | { 209 | "cell_type": "code", 210 | "execution_count": 0, 211 | "metadata": { 212 | "colab": {}, 213 | "colab_type": "code", 214 | "id": "oMARKqbaBjIN" 215 | }, 216 | "outputs": [], 217 | "source": [ 218 | "model = RNN()" 219 | ] 220 | }, 221 | { 222 | "cell_type": "code", 223 | "execution_count": 38, 224 | "metadata": { 225 | "colab": { 226 | "base_uri": "https://localhost:8080/", 227 | "height": 215 228 | }, 229 | "colab_type": "code", 230 | "id": "_NPkLp-rBzCp", 231 | "outputId": "adbc4090-6f0b-42ab-c8f3-33cc2c96065a" 232 | }, 233 | "outputs": [ 234 | { 235 | "name": "stdout", 236 | "output_type": "stream", 237 | "text": [ 238 | "_________________________________________________________________\n", 239 | "Layer (type) Output Shape Param # \n", 240 | "=================================================================\n", 241 | "sequential_3 (Sequential) multiple 2480 \n", 242 | "_________________________________________________________________\n", 243 | "dense_3 (Dense) multiple 110 \n", 244 | "=================================================================\n", 245 | "Total params: 2,590\n", 246 | "Trainable params: 2,590\n", 247 | "Non-trainable params: 0\n", 248 | "_________________________________________________________________\n" 249 | ] 250 | } 251 | ], 252 | "source": [ 253 | "optimizer = tf.train.AdamOptimizer(learning_rate)\n", 254 | "model.compile(optimizer=optimizer,\n", 255 | " loss='categorical_crossentropy',\n", 256 | " metrics=[\"accuracy\"])\n", 257 | "\n", 258 | "# Eager Execution initialize parameters when using model.call()\n", 259 | "model(x_train_eager[:50])\n", 260 | "\n", 261 | "model.summary()" 262 | ] 263 | }, 264 | { 265 | "cell_type": "code", 266 | "execution_count": 39, 267 | "metadata": { 268 | "colab": { 269 | "base_uri": "https://localhost:8080/", 270 | "height": 935 271 | }, 272 | "colab_type": "code", 273 | "id": "wDEyoIMnDKEy", 274 | "outputId": "8e5a9c44-cd00-4999-f450-27e45a4f4030" 275 | }, 276 | "outputs": [ 277 | { 278 | "name": "stdout", 279 | "output_type": "stream", 280 | "text": [ 281 | "Epoch 1/25\n", 282 | "94/94 [==============================] - 4s 47ms/step - loss: 2.1856 - acc: 0.2238 - val_loss: 1.9467 - val_acc: 0.3433\n", 283 | "Epoch 2/25\n", 284 | "94/94 [==============================] - 4s 45ms/step - loss: 1.6989 - acc: 0.4228 - val_loss: 1.4462 - val_acc: 0.5258\n", 285 | "Epoch 3/25\n", 286 | "94/94 [==============================] - 4s 46ms/step - loss: 1.2604 - acc: 0.6018 - val_loss: 1.0583 - val_acc: 0.6913\n", 287 | "Epoch 4/25\n", 288 | "94/94 [==============================] - 4s 46ms/step - loss: 0.9625 - acc: 0.7154 - val_loss: 0.8442 - val_acc: 0.7549\n", 289 | "Epoch 5/25\n", 290 | "94/94 [==============================] - 4s 45ms/step - loss: 0.8116 - acc: 0.7575 - val_loss: 0.7295 - val_acc: 0.7807\n", 291 | "Epoch 6/25\n", 292 | "94/94 [==============================] - 4s 45ms/step - loss: 0.7145 - acc: 0.7822 - val_loss: 0.6535 - val_acc: 0.8047\n", 293 | "Epoch 7/25\n", 294 | "94/94 [==============================] - 4s 45ms/step - loss: 0.6454 - acc: 0.8027 - val_loss: 0.5999 - val_acc: 0.8229\n", 295 | "Epoch 8/25\n", 296 | "94/94 [==============================] - 4s 45ms/step - loss: 0.5950 - acc: 0.8199 - val_loss: 0.5494 - val_acc: 0.8375\n", 297 | "Epoch 9/25\n", 298 | "94/94 [==============================] - 4s 45ms/step - loss: 0.5501 - acc: 0.8353 - val_loss: 0.5157 - val_acc: 0.8486\n", 299 | "Epoch 10/25\n", 300 | "94/94 [==============================] - 5s 49ms/step - loss: 0.5161 - acc: 0.8470 - val_loss: 0.4818 - val_acc: 0.8593\n", 301 | "Epoch 11/25\n", 302 | "94/94 [==============================] - 4s 48ms/step - loss: 0.4854 - acc: 0.8573 - val_loss: 0.4609 - val_acc: 0.8699\n", 303 | "Epoch 12/25\n", 304 | "94/94 [==============================] - 4s 46ms/step - loss: 0.4591 - acc: 0.8662 - val_loss: 0.4296 - val_acc: 0.8768\n", 305 | "Epoch 13/25\n", 306 | "94/94 [==============================] - 4s 45ms/step - loss: 0.4379 - acc: 0.8727 - val_loss: 0.4198 - val_acc: 0.8808\n", 307 | "Epoch 14/25\n", 308 | "94/94 [==============================] - 4s 45ms/step - loss: 0.4195 - acc: 0.8788 - val_loss: 0.3983 - val_acc: 0.8877\n", 309 | "Epoch 15/25\n", 310 | "94/94 [==============================] - 4s 45ms/step - loss: 0.3984 - acc: 0.8844 - val_loss: 0.3806 - val_acc: 0.8923\n", 311 | "Epoch 16/25\n", 312 | "94/94 [==============================] - 4s 45ms/step - loss: 0.3837 - acc: 0.8886 - val_loss: 0.3670 - val_acc: 0.8953\n", 313 | "Epoch 17/25\n", 314 | "94/94 [==============================] - 4s 45ms/step - loss: 0.3731 - acc: 0.8910 - val_loss: 0.3555 - val_acc: 0.9021\n", 315 | "Epoch 18/25\n", 316 | "94/94 [==============================] - 4s 45ms/step - loss: 0.3582 - acc: 0.8972 - val_loss: 0.3418 - val_acc: 0.9040\n", 317 | "Epoch 19/25\n", 318 | "94/94 [==============================] - 4s 46ms/step - loss: 0.3489 - acc: 0.8992 - val_loss: 0.3324 - val_acc: 0.9060\n", 319 | "Epoch 20/25\n", 320 | "94/94 [==============================] - 4s 46ms/step - loss: 0.3364 - acc: 0.9036 - val_loss: 0.3311 - val_acc: 0.9048\n", 321 | "Epoch 21/25\n", 322 | "94/94 [==============================] - 4s 45ms/step - loss: 0.3278 - acc: 0.9050 - val_loss: 0.3148 - val_acc: 0.9112\n", 323 | "Epoch 22/25\n", 324 | "94/94 [==============================] - 4s 45ms/step - loss: 0.3206 - acc: 0.9071 - val_loss: 0.3051 - val_acc: 0.9127\n", 325 | "Epoch 23/25\n", 326 | "94/94 [==============================] - 4s 45ms/step - loss: 0.3112 - acc: 0.9095 - val_loss: 0.3022 - val_acc: 0.9134\n", 327 | "Epoch 24/25\n", 328 | "94/94 [==============================] - 4s 45ms/step - loss: 0.3028 - acc: 0.9124 - val_loss: 0.2974 - val_acc: 0.9141\n", 329 | "Epoch 25/25\n", 330 | "94/94 [==============================] - 4s 45ms/step - loss: 0.2994 - acc: 0.9126 - val_loss: 0.2902 - val_acc: 0.9162\n" 331 | ] 332 | }, 333 | { 334 | "data": { 335 | "text/plain": [ 336 | "" 337 | ] 338 | }, 339 | "execution_count": 39, 340 | "metadata": { 341 | "tags": [] 342 | }, 343 | "output_type": "execute_result" 344 | } 345 | ], 346 | "source": [ 347 | "model.fit(x=x_train_eager.numpy(), \n", 348 | " y=y_train_eager.numpy(), \n", 349 | " validation_split=0.2, \n", 350 | " epochs=num_epochs,\n", 351 | " batch_size=batch_size)" 352 | ] 353 | }, 354 | { 355 | "cell_type": "code", 356 | "execution_count": 41, 357 | "metadata": { 358 | "colab": { 359 | "base_uri": "https://localhost:8080/", 360 | "height": 53 361 | }, 362 | "colab_type": "code", 363 | "id": "oDuYrgFIG850", 364 | "outputId": "afd19d09-2dd7-4575-9655-6227943ad81b" 365 | }, 366 | "outputs": [ 367 | { 368 | "name": "stdout", 369 | "output_type": "stream", 370 | "text": [ 371 | "313/313 [==============================] - 5s 17ms/step\n", 372 | "test_accracy: 0.9114\n" 373 | ] 374 | } 375 | ], 376 | "source": [ 377 | "test_loss, test_acc = model.evaluate(x=x_test_eager.numpy(), \n", 378 | " y=y_test_eager.numpy())\n", 379 | "\n", 380 | "print(\"test_accracy: \", test_acc)" 381 | ] 382 | }, 383 | { 384 | "cell_type": "code", 385 | "execution_count": 0, 386 | "metadata": { 387 | "colab": {}, 388 | "colab_type": "code", 389 | "id": "Yeayjx_nKXaW" 390 | }, 391 | "outputs": [], 392 | "source": [] 393 | } 394 | ], 395 | "metadata": { 396 | "accelerator": "GPU", 397 | "colab": { 398 | "collapsed_sections": [], 399 | "include_colab_link": true, 400 | "name": "Recurrent_Neural_Network", 401 | "provenance": [], 402 | "version": "0.3.2" 403 | }, 404 | "kernelspec": { 405 | "display_name": "Python 3", 406 | "language": "python", 407 | "name": "python3" 408 | }, 409 | "language_info": { 410 | "codemirror_mode": { 411 | "name": "ipython", 412 | "version": 3 413 | }, 414 | "file_extension": ".py", 415 | "mimetype": "text/x-python", 416 | "name": "python", 417 | "nbconvert_exporter": "python", 418 | "pygments_lexer": "ipython3", 419 | "version": "3.5.3" 420 | } 421 | }, 422 | "nbformat": 4, 423 | "nbformat_minor": 1 424 | } 425 | -------------------------------------------------------------------------------- /tutorials/02_intermediate/.ipynb_checkpoints/residual_network-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "colab_type": "text", 7 | "id": "view-in-github" 8 | }, 9 | "source": [ 10 | "\"Open" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 0, 16 | "metadata": { 17 | "colab": {}, 18 | "colab_type": "code", 19 | "id": "W-2MApJumG9i" 20 | }, 21 | "outputs": [], 22 | "source": [ 23 | "import numpy as np\n", 24 | "import tensorflow as tf\n", 25 | "import matplotlib.pyplot as plt\n", 26 | "%matplotlib inline" 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": 0, 32 | "metadata": { 33 | "colab": {}, 34 | "colab_type": "code", 35 | "id": "YY0lj_PWmNKz" 36 | }, 37 | "outputs": [], 38 | "source": [ 39 | "L = tf.keras.layers\n", 40 | "tfe = tf.contrib.eager\n", 41 | "\n", 42 | "tf.enable_eager_execution()" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 3, 48 | "metadata": { 49 | "colab": { 50 | "base_uri": "https://localhost:8080/", 51 | "height": 197 52 | }, 53 | "colab_type": "code", 54 | "id": "Az2CgxXFmilT", 55 | "outputId": "af93c658-6067-4d2d-96a7-2c95806b3421" 56 | }, 57 | "outputs": [ 58 | { 59 | "name": "stdout", 60 | "output_type": "stream", 61 | "text": [ 62 | "Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\n", 63 | "170500096/170498071 [==============================] - 11s 0us/step\n", 64 | "training_data\n", 65 | " (50000, 32, 32, 3)\n", 66 | "test_data\n", 67 | " (10000, 32, 32, 3)\n", 68 | "training_label\n", 69 | " (50000, 1)\n", 70 | "test_label\n", 71 | " (10000, 1)\n" 72 | ] 73 | } 74 | ], 75 | "source": [ 76 | "# Hyper parameters\n", 77 | "num_epochs = 10\n", 78 | "num_classes = 10\n", 79 | "batch_size = 256\n", 80 | "learning_rate = 0.001\n", 81 | "\n", 82 | "(x_train, y_train), (x_test, y_test) = tfk.datasets.cifar10.load_data()\n", 83 | "\n", 84 | "print(\"training_data\\n\", x_train.shape)\n", 85 | "print(\"test_data\\n\", x_test.shape)\n", 86 | "print(\"training_label\\n\", y_train.shape)\n", 87 | "print(\"test_label\\n\", y_test.shape)" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": 4, 93 | "metadata": { 94 | "colab": { 95 | "base_uri": "https://localhost:8080/", 96 | "height": 53 97 | }, 98 | "colab_type": "code", 99 | "id": "U6MRu-QDok4N", 100 | "outputId": "0fdd32cf-0dd4-4473-85b3-711893e1d4a4" 101 | }, 102 | "outputs": [ 103 | { 104 | "name": "stdout", 105 | "output_type": "stream", 106 | "text": [ 107 | "(50000, 32, 32, 3)\n", 108 | "(50000, 10)\n" 109 | ] 110 | } 111 | ], 112 | "source": [ 113 | "x_train_ = tf.convert_to_tensor(x_train, dtype=tf.float32)\n", 114 | "y_train_ = tf.reshape(tf.one_hot(y_train, 10), (-1, 10))\n", 115 | "\n", 116 | "\n", 117 | "print(x_train_.shape)\n", 118 | "print(y_train_.shape)" 119 | ] 120 | }, 121 | { 122 | "cell_type": "code", 123 | "execution_count": 18, 124 | "metadata": { 125 | "colab": { 126 | "base_uri": "https://localhost:8080/", 127 | "height": 35 128 | }, 129 | "colab_type": "code", 130 | "id": "6f15_qjHtZl8", 131 | "outputId": "c6d0117e-3ee9-4fa8-d7d2-d0ca4b49d8d1" 132 | }, 133 | "outputs": [ 134 | { 135 | "name": "stdout", 136 | "output_type": "stream", 137 | "text": [ 138 | "\n" 139 | ] 140 | } 141 | ], 142 | "source": [ 143 | "train_dataset = (\n", 144 | " tf.data.Dataset.from_tensor_slices((x_train, y_train))\n", 145 | " .batch(batch_size)\n", 146 | " .shuffle(10000)\n", 147 | ")\n", 148 | "\n", 149 | "train_dataset = (\n", 150 | " train_dataset.map(lambda x, y: \n", 151 | " (tf.div(tf.cast(x, tf.float32), 255.0), \n", 152 | " tf.reshape(tf.one_hot(y, 10), (-1, 10))))\n", 153 | ")\n", 154 | "\n", 155 | "print(train_dataset)" 156 | ] 157 | }, 158 | { 159 | "cell_type": "code", 160 | "execution_count": 19, 161 | "metadata": { 162 | "colab": { 163 | "base_uri": "https://localhost:8080/", 164 | "height": 35 165 | }, 166 | "colab_type": "code", 167 | "id": "hL_0Ca4mvJ1m", 168 | "outputId": "dcd2e3c2-1cba-4db4-d061-332a6337c3ca" 169 | }, 170 | "outputs": [ 171 | { 172 | "name": "stdout", 173 | "output_type": "stream", 174 | "text": [ 175 | "\n" 176 | ] 177 | } 178 | ], 179 | "source": [ 180 | "test_dataset = (\n", 181 | " tf.data.Dataset.from_tensor_slices((x_test, y_test))\n", 182 | " .batch(1000)\n", 183 | " .shuffle(10000)\n", 184 | ")\n", 185 | "test_dataset = (\n", 186 | " test_dataset.map(lambda x, y: \n", 187 | " (tf.div(tf.cast(x, tf.float32), 255.0), \n", 188 | " tf.reshape(tf.one_hot(y, 10), (-1, 10))))\n", 189 | ")\n", 190 | "\n", 191 | "print(test_dataset)" 192 | ] 193 | }, 194 | { 195 | "cell_type": "code", 196 | "execution_count": 0, 197 | "metadata": { 198 | "colab": {}, 199 | "colab_type": "code", 200 | "id": "jUvyjTFXm_0d" 201 | }, 202 | "outputs": [], 203 | "source": [ 204 | "def conv3x3(out_channels, strides=1):\n", 205 | " return L.Conv2D(out_channels, kernel_size=3, \n", 206 | " strides=strides, padding='same', use_bias=False)" 207 | ] 208 | }, 209 | { 210 | "cell_type": "markdown", 211 | "metadata": { 212 | "colab_type": "text", 213 | "id": "LoUv_nFf_jvM" 214 | }, 215 | "source": [ 216 | "### training flag\n", 217 | "`call` method of `L.BatchNormalization` need to have `traininig` flag because this method have different behavior between traning and evaluation. " 218 | ] 219 | }, 220 | { 221 | "cell_type": "code", 222 | "execution_count": 0, 223 | "metadata": { 224 | "colab": {}, 225 | "colab_type": "code", 226 | "id": "ppETst-0nQtu" 227 | }, 228 | "outputs": [], 229 | "source": [ 230 | "# Residual block\n", 231 | "class ResidualBlock(tf.keras.Model):\n", 232 | " def __init__(self, out_channels, strides=1, downsample=None):\n", 233 | " super(ResidualBlock, self).__init__(name='ResidualBlock')\n", 234 | " self.conv1 = conv3x3(out_channels, strides)\n", 235 | " self.bn1 = L.BatchNormalization(axis=-1)\n", 236 | " self.relu = L.ReLU()\n", 237 | " self.conv2 = conv3x3(out_channels)\n", 238 | " self.bn2 = L.BatchNormalization(axis=-1)\n", 239 | " self.downsample = downsample\n", 240 | " \n", 241 | "\n", 242 | " def call(self, x, training=False):\n", 243 | " residual = x\n", 244 | " out = self.conv1(x)\n", 245 | " out = self.bn1(out, training=training)\n", 246 | " out = self.relu(out)\n", 247 | " out = self.conv2(out)\n", 248 | " out = self.bn2(out, training=training)\n", 249 | " if self.downsample:\n", 250 | " residual = self.downsample(x)\n", 251 | " out += residual\n", 252 | " out = self.relu(out)\n", 253 | " return out" 254 | ] 255 | }, 256 | { 257 | "cell_type": "markdown", 258 | "metadata": { 259 | "colab_type": "text", 260 | "id": "ME6MBqumAR35" 261 | }, 262 | "source": [ 263 | "### tf.keras.Sequential\n", 264 | "`call` method of `tf.keras.Sequential` have `training` flag. This flag affects all layers included by the `tf.keras.Sequential` instance." 265 | ] 266 | }, 267 | { 268 | "cell_type": "code", 269 | "execution_count": 0, 270 | "metadata": { 271 | "colab": {}, 272 | "colab_type": "code", 273 | "id": "c4dcZi3boLkG" 274 | }, 275 | "outputs": [], 276 | "source": [ 277 | "class ResNet(tf.keras.Model):\n", 278 | " def __init__(self, block, layers, num_classes=10):\n", 279 | " super(ResNet, self).__init__(name='ResNet')\n", 280 | " self.in_channels = 16\n", 281 | " self.conv = conv3x3(16)\n", 282 | " self.bn = L.BatchNormalization(axis=-1)\n", 283 | " self.relu = L.ReLU()\n", 284 | " self.layer1 = self.make_layer(block, 16, layers[0])\n", 285 | " self.layer2 = self.make_layer(block, 32, layers[1], 2)\n", 286 | " self.layer3 = self.make_layer(block, 64, layers[2], 2)\n", 287 | " self.avg_pool = L.AvgPool2D(8)\n", 288 | " self.flatten = L.Flatten()\n", 289 | " self.fc = L.Dense(num_classes)\n", 290 | " \n", 291 | " def make_layer(self, block, out_channels, blocks, strides=1):\n", 292 | " downsample = None\n", 293 | " if (strides != 1) or (self.in_channels != out_channels):\n", 294 | " downsample = tf.keras.Sequential([\n", 295 | " conv3x3(out_channels, strides=strides),\n", 296 | " L.BatchNormalization(axis=-1)])\n", 297 | " layers = []\n", 298 | " layers.append(block(out_channels, strides, downsample))\n", 299 | " self.in_channels = out_channels\n", 300 | " for i in range(1, blocks):\n", 301 | " layers.append(block(out_channels))\n", 302 | " return tf.keras.Sequential(layers)\n", 303 | " \n", 304 | " def call(self, x, training=False):\n", 305 | " out = self.conv(x)\n", 306 | " out = self.bn(out, training=training)\n", 307 | " out = self.relu(out)\n", 308 | " out = self.layer1(out, training=training)\n", 309 | " out = self.layer2(out, training=training)\n", 310 | " out = self.layer3(out, training=training)\n", 311 | " out = self.avg_pool(out)\n", 312 | " out = self.flatten(out)\n", 313 | " out = self.fc(out)\n", 314 | " return out" 315 | ] 316 | }, 317 | { 318 | "cell_type": "code", 319 | "execution_count": 0, 320 | "metadata": { 321 | "colab": {}, 322 | "colab_type": "code", 323 | "id": "E6vmD9PdoQxW" 324 | }, 325 | "outputs": [], 326 | "source": [ 327 | "model = ResNet(ResidualBlock, [2, 2, 2])" 328 | ] 329 | }, 330 | { 331 | "cell_type": "code", 332 | "execution_count": 0, 333 | "metadata": { 334 | "colab": {}, 335 | "colab_type": "code", 336 | "id": "Qx0zHRhaqUL3" 337 | }, 338 | "outputs": [], 339 | "source": [ 340 | "def loss_fn(y, y_pre):\n", 341 | " return tf.losses.softmax_cross_entropy(y, y_pre)\n", 342 | "\n", 343 | "def accuracy(y, y_pre):\n", 344 | " return tf.keras.metrics.categorical_accuracy(y, y_pre)\n", 345 | "\n", 346 | "optimizer = tf.train.AdamOptimizer(learning_rate)" 347 | ] 348 | }, 349 | { 350 | "cell_type": "code", 351 | "execution_count": 34, 352 | "metadata": { 353 | "colab": { 354 | "base_uri": "https://localhost:8080/", 355 | "height": 557 356 | }, 357 | "colab_type": "code", 358 | "id": "EGEcgjposP9I", 359 | "outputId": "3bb6ebf5-3483-4808-ff18-7c6d403e1650" 360 | }, 361 | "outputs": [ 362 | { 363 | "name": "stdout", 364 | "output_type": "stream", 365 | "text": [ 366 | "-----epoch 1 -----\n", 367 | "loss: 1.4503074178890305\n", 368 | "acc: 0.4708187336824378\n", 369 | "-----epoch 2 -----\n", 370 | "loss: 1.030255531778141\n", 371 | "acc: 0.6327248008883729\n", 372 | "-----epoch 3 -----\n", 373 | "loss: 0.8536564573949698\n", 374 | "acc: 0.6990194515306123\n", 375 | "-----epoch 4 -----\n", 376 | "loss: 0.7369005631427376\n", 377 | "acc: 0.741896493094308\n", 378 | "-----epoch 5 -----\n", 379 | "loss: 0.6444133836395887\n", 380 | "acc: 0.7760403691505899\n", 381 | "-----epoch 6 -----\n", 382 | "loss: 0.5748496152916733\n", 383 | "acc: 0.7999720670738999\n", 384 | "-----epoch 7 -----\n", 385 | "loss: 0.5158630682497608\n", 386 | "acc: 0.8208665653150908\n", 387 | "-----epoch 8 -----\n", 388 | "loss: 0.46397267555703925\n", 389 | "acc: 0.8399194989885602\n", 390 | "-----epoch 9 -----\n", 391 | "loss: 0.405785541145169\n", 392 | "acc: 0.8601921237244898\n", 393 | "-----epoch 10 -----\n", 394 | "loss: 0.36217514349489793\n", 395 | "acc: 0.8758011642767458\n" 396 | ] 397 | } 398 | ], 399 | "source": [ 400 | "for j in range(num_epochs):\n", 401 | " \n", 402 | " running_loss = 0\n", 403 | " running_acc = 0\n", 404 | "\n", 405 | " for i, (x_, y_) in enumerate(train_dataset):\n", 406 | " \n", 407 | " with tf.device(\"/gpu:0\"):\n", 408 | " with tf.GradientTape() as tape:\n", 409 | " y_pre = model(x_, training=True)\n", 410 | " loss = loss_fn(y_, y_pre)\n", 411 | " acc = accuracy(y_, y_pre)\n", 412 | " grads = tape.gradient(loss, model.variables)\n", 413 | " optimizer.apply_gradients(zip(grads, model.variables))\n", 414 | " running_loss += loss\n", 415 | " running_acc += tf.reduce_mean(acc)\n", 416 | " \n", 417 | " print(\"-----epoch {} -----\".format(j + 1))\n", 418 | " print(\"loss: \", running_loss.numpy()/(i + 1))\n", 419 | " print(\"acc: \", running_acc.numpy()/(i + 1)) " 420 | ] 421 | }, 422 | { 423 | "cell_type": "code", 424 | "execution_count": 42, 425 | "metadata": { 426 | "colab": { 427 | "base_uri": "https://localhost:8080/", 428 | "height": 35 429 | }, 430 | "colab_type": "code", 431 | "id": "8M37rwRAt0kk", 432 | "outputId": "df534c9f-923c-47ab-f21d-ddc07fbc21a0" 433 | }, 434 | "outputs": [ 435 | { 436 | "name": "stdout", 437 | "output_type": "stream", 438 | "text": [ 439 | "test accuracy 0.723\n" 440 | ] 441 | } 442 | ], 443 | "source": [ 444 | "test_accuracy = 0\n", 445 | "for i, (x_, y_) in enumerate(test_dataset):\n", 446 | " y_pre = model(x_, training=False)\n", 447 | " test_accuracy += tf.reduce_mean(accuracy(y_, y_pre))\n", 448 | "test_accuracy /= i + 1\n", 449 | "\n", 450 | "print(\"test accuracy {:0.3f}\".format(test_accuracy.numpy()))" 451 | ] 452 | }, 453 | { 454 | "cell_type": "code", 455 | "execution_count": 0, 456 | "metadata": { 457 | "colab": {}, 458 | "colab_type": "code", 459 | "id": "p1EN7x6nhkxu" 460 | }, 461 | "outputs": [], 462 | "source": [] 463 | }, 464 | { 465 | "cell_type": "code", 466 | "execution_count": 0, 467 | "metadata": { 468 | "colab": {}, 469 | "colab_type": "code", 470 | "id": "oq81ZIfC9SZD" 471 | }, 472 | "outputs": [], 473 | "source": [] 474 | } 475 | ], 476 | "metadata": { 477 | "accelerator": "GPU", 478 | "colab": { 479 | "collapsed_sections": [], 480 | "include_colab_link": true, 481 | "name": "residual_network.ipynb", 482 | "provenance": [], 483 | "version": "0.3.2" 484 | }, 485 | "kernelspec": { 486 | "display_name": "Python 3", 487 | "language": "python", 488 | "name": "python3" 489 | }, 490 | "language_info": { 491 | "codemirror_mode": { 492 | "name": "ipython", 493 | "version": 3 494 | }, 495 | "file_extension": ".py", 496 | "mimetype": "text/x-python", 497 | "name": "python", 498 | "nbconvert_exporter": "python", 499 | "pygments_lexer": "ipython3", 500 | "version": "3.5.3" 501 | } 502 | }, 503 | "nbformat": 4, 504 | "nbformat_minor": 1 505 | } 506 | -------------------------------------------------------------------------------- /tutorials/01_basics/BasicStyle.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "colab_type": "text", 7 | "id": "view-in-github" 8 | }, 9 | "source": [ 10 | "\"Open" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 1, 16 | "metadata": { 17 | "colab": { 18 | "base_uri": "https://localhost:8080/", 19 | "height": 89 20 | }, 21 | "colab_type": "code", 22 | "id": "2wXVpo2lhf2A", 23 | "outputId": "f53997e0-5ebd-417f-8a81-da3657a5780f" 24 | }, 25 | "outputs": [], 26 | "source": [ 27 | "# !pip install -q --upgrade tf-nightly-2.0-preview" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": 2, 33 | "metadata": { 34 | "colab": {}, 35 | "colab_type": "code", 36 | "id": "S3kHX4Vqhl5O" 37 | }, 38 | "outputs": [], 39 | "source": [ 40 | "import tensorflow as tf" 41 | ] 42 | }, 43 | { 44 | "cell_type": "markdown", 45 | "metadata": {}, 46 | "source": [ 47 | "## TensorFlow 2.0" 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": 3, 53 | "metadata": {}, 54 | "outputs": [ 55 | { 56 | "data": { 57 | "text/plain": [ 58 | "'2.0.0'" 59 | ] 60 | }, 61 | "execution_count": 3, 62 | "metadata": {}, 63 | "output_type": "execute_result" 64 | } 65 | ], 66 | "source": [ 67 | "tf.__version__" 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "metadata": { 73 | "colab_type": "text", 74 | "id": "cdgYCKIShn9M" 75 | }, 76 | "source": [ 77 | "## Make DataSet" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": 4, 83 | "metadata": { 84 | "colab": {}, 85 | "colab_type": "code", 86 | "id": "lcVt1_Zth3_6" 87 | }, 88 | "outputs": [], 89 | "source": [ 90 | "'''\n", 91 | "If you have numpy data, you can use\n", 92 | "tensor_data = tf.convert_to_tensor(numpy_data, dtype=tf.float32)\n", 93 | "for translation into tf.Tensor.\n", 94 | "'''\n", 95 | "# example training data\n", 96 | "feature = tf.random.normal(shape=[50000, 1000])\n", 97 | "target = tf.random.normal(shape=[50000, 10])\n", 98 | "\n", 99 | "# example validation data\n", 100 | "val_feature = tf.random.normal(shape=[10000, 1000])\n", 101 | "val_target = tf.random.normal(shape=[10000, 10])\n", 102 | "\n", 103 | "# example test data\n", 104 | "test_feature = tf.random.normal(shape=[5000, 1000])\n", 105 | "test_target = tf.random.normal(shape=[5000, 10])\n", 106 | "\n", 107 | "\n", 108 | "# make dataset\n", 109 | "dataset = tf.data.Dataset.from_tensor_slices((feature, target))\n", 110 | "val_dataset = tf.data.Dataset.from_tensor_slices((val_feature, val_target))\n", 111 | "test_dataset = tf.data.Dataset.from_tensor_slices((test_feature, test_target))" 112 | ] 113 | }, 114 | { 115 | "cell_type": "code", 116 | "execution_count": 5, 117 | "metadata": { 118 | "colab": { 119 | "base_uri": "https://localhost:8080/", 120 | "height": 35 121 | }, 122 | "colab_type": "code", 123 | "id": "zFAMHrm7SL5M", 124 | "outputId": "071d0426-317b-49d0-ffcd-6ec59a91c07a" 125 | }, 126 | "outputs": [ 127 | { 128 | "data": { 129 | "text/plain": [ 130 | "" 131 | ] 132 | }, 133 | "execution_count": 5, 134 | "metadata": {}, 135 | "output_type": "execute_result" 136 | } 137 | ], 138 | "source": [ 139 | "# A dataset have shape information except batchsize and data type.\n", 140 | "dataset" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": 6, 146 | "metadata": { 147 | "colab": {}, 148 | "colab_type": "code", 149 | "id": "s5wtoZOsSzft" 150 | }, 151 | "outputs": [], 152 | "source": [ 153 | "# Training data should be shuffled every epoch.\n", 154 | "# 10000 is buffer size.\n", 155 | "dataset = dataset.shuffle(10000)\n", 156 | "\n", 157 | "# For mini-batch training.\n", 158 | "# 256 is batch size.\n", 159 | "dataset = dataset.batch(256)\n", 160 | "\n", 161 | "# Of course we can write same code as follows\n", 162 | "# dataset = dataset.shuffle(10000).batch(256)\n", 163 | "\n", 164 | "# validation data and test data do NOT need shuffle.\n", 165 | "# batch size is as big as possible.\n", 166 | "val_dataset = val_dataset.batch(10000)\n", 167 | "test_dataset = test_dataset.batch(5000)" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": 7, 173 | "metadata": { 174 | "colab": { 175 | "base_uri": "https://localhost:8080/", 176 | "height": 35 177 | }, 178 | "colab_type": "code", 179 | "id": "X3JLZxT0UsnB", 180 | "outputId": "fbf1cfd8-1347-4372-a3f2-bb15db0e434d" 181 | }, 182 | "outputs": [ 183 | { 184 | "data": { 185 | "text/plain": [ 186 | "" 187 | ] 188 | }, 189 | "execution_count": 7, 190 | "metadata": {}, 191 | "output_type": "execute_result" 192 | } 193 | ], 194 | "source": [ 195 | "# dataset is set for batch training.\n", 196 | "dataset" 197 | ] 198 | }, 199 | { 200 | "cell_type": "markdown", 201 | "metadata": { 202 | "colab_type": "text", 203 | "id": "6vL6rFO6VFBo" 204 | }, 205 | "source": [ 206 | "## Make Network" 207 | ] 208 | }, 209 | { 210 | "cell_type": "code", 211 | "execution_count": 8, 212 | "metadata": { 213 | "colab": {}, 214 | "colab_type": "code", 215 | "id": "wBzNdkCyiFGs" 216 | }, 217 | "outputs": [], 218 | "source": [ 219 | "class MyNet(tf.keras.Model):\n", 220 | " '''\n", 221 | " We use basically tf.keras.Model for making network.\n", 222 | " This class will manage layers and that's trainable parameters.\n", 223 | " '''\n", 224 | " def __init__(self):\n", 225 | " super(MyNet, self).__init__()\n", 226 | " \n", 227 | " \n", 228 | " # We can use tf.keras.Sequential \n", 229 | " # which has a role of putting together some layers.\n", 230 | " # This class inherits tf.keras.Model, so this can manege parameters too.\n", 231 | " # This class only receive layers.Layer class.\n", 232 | " # (Note that tf.keras.Sequential receive tf.keras.layers.ReLU())\n", 233 | " \n", 234 | " self.layer1 = tf.keras.Sequential([\n", 235 | " tf.keras.layers.Dense(1024),\n", 236 | " tf.keras.layers.ReLU(),\n", 237 | " tf.keras.layers.BatchNormalization(axis=-1),\n", 238 | " tf.keras.layers.Dropout(rate=0.2),\n", 239 | " ])\n", 240 | " \n", 241 | " # Of course we can write some layers separately.\n", 242 | " \n", 243 | " self.dense = tf.keras.layers.Dense(256)\n", 244 | " self.bn = tf.keras.layers.BatchNormalization(axis=-1)\n", 245 | " self.do = tf.keras.layers.Dropout(rate=0.2)\n", 246 | " \n", 247 | " self.dense_output = tf.keras.layers.Dense(10)\n", 248 | " \n", 249 | " # tf.function is jit compiler which translate python code into TF graph.\n", 250 | " @tf.function\n", 251 | " def call(self, x, training=False):\n", 252 | " # tf.keras.Sequential class have training propaty\n", 253 | " # which manege behavior of dropout and batchnormalization etc.\n", 254 | " h = self.layer1(x, training=training)\n", 255 | " \n", 256 | " h = self.dense(h)\n", 257 | " # we can use tf.nn.relu function instead of tf.keras.layers.ReLU()\n", 258 | " h = tf.nn.relu(h)\n", 259 | "\n", 260 | " # BatchNormalization and Dropout class also have training property.\n", 261 | " h = self.bn(h, training=training)\n", 262 | " h = self.do(h, training=training)\n", 263 | " \n", 264 | " return self.dense_output(h)" 265 | ] 266 | }, 267 | { 268 | "cell_type": "code", 269 | "execution_count": 9, 270 | "metadata": { 271 | "colab": {}, 272 | "colab_type": "code", 273 | "id": "60kaVa6BiJzz" 274 | }, 275 | "outputs": [], 276 | "source": [ 277 | "model = MyNet()" 278 | ] 279 | }, 280 | { 281 | "cell_type": "code", 282 | "execution_count": 10, 283 | "metadata": { 284 | "colab": { 285 | "base_uri": "https://localhost:8080/", 286 | "height": 89 287 | }, 288 | "colab_type": "code", 289 | "id": "PTEFbRxqizpf", 290 | "outputId": "a784eb57-7caf-4333-c869-b20958f32606" 291 | }, 292 | "outputs": [ 293 | { 294 | "data": { 295 | "text/plain": [ 296 | "" 300 | ] 301 | }, 302 | "execution_count": 10, 303 | "metadata": {}, 304 | "output_type": "execute_result" 305 | } 306 | ], 307 | "source": [ 308 | "# test execution.\n", 309 | "model(tf.random.normal(shape=[1, 1000]))" 310 | ] 311 | }, 312 | { 313 | "cell_type": "code", 314 | "execution_count": 11, 315 | "metadata": { 316 | "colab": { 317 | "base_uri": "https://localhost:8080/", 318 | "height": 341 319 | }, 320 | "colab_type": "code", 321 | "id": "mjM1vGKSj5Ri", 322 | "outputId": "50f551b4-c586-48eb-94ca-f1640b88dc92" 323 | }, 324 | "outputs": [ 325 | { 326 | "name": "stdout", 327 | "output_type": "stream", 328 | "text": [ 329 | "Model: \"my_net\"\n", 330 | "_________________________________________________________________\n", 331 | "Layer (type) Output Shape Param # \n", 332 | "=================================================================\n", 333 | "sequential (Sequential) multiple 1029120 \n", 334 | "_________________________________________________________________\n", 335 | "dense_1 (Dense) multiple 262400 \n", 336 | "_________________________________________________________________\n", 337 | "batch_normalization_1 (Batch multiple 1024 \n", 338 | "_________________________________________________________________\n", 339 | "dropout_1 (Dropout) multiple 0 \n", 340 | "_________________________________________________________________\n", 341 | "dense_2 (Dense) multiple 2570 \n", 342 | "=================================================================\n", 343 | "Total params: 1,295,114\n", 344 | "Trainable params: 1,292,554\n", 345 | "Non-trainable params: 2,560\n", 346 | "_________________________________________________________________\n" 347 | ] 348 | } 349 | ], 350 | "source": [ 351 | "# We can check model compose with model.summary() after first execution.\n", 352 | "model.summary()" 353 | ] 354 | }, 355 | { 356 | "cell_type": "markdown", 357 | "metadata": { 358 | "colab_type": "text", 359 | "id": "xySVCRU9j7l2" 360 | }, 361 | "source": [ 362 | "## Training by hand" 363 | ] 364 | }, 365 | { 366 | "cell_type": "code", 367 | "execution_count": 12, 368 | "metadata": { 369 | "colab": {}, 370 | "colab_type": "code", 371 | "id": "1eQ8DRWTbFjO" 372 | }, 373 | "outputs": [], 374 | "source": [ 375 | "optimizer = tf.optimizers.Adam()\n", 376 | "# for loss execution\n", 377 | "loss_fn = tf.losses.MeanSquaredError()\n", 378 | "\n", 379 | "# for iter loop\n", 380 | "# This class can have property of loss each iter\n", 381 | "train_loss = tf.keras.metrics.Mean() \n", 382 | "val_loss = tf.keras.metrics.Mean()" 383 | ] 384 | }, 385 | { 386 | "cell_type": "code", 387 | "execution_count": 13, 388 | "metadata": { 389 | "colab": {}, 390 | "colab_type": "code", 391 | "id": "pHTjWBcVbvo6" 392 | }, 393 | "outputs": [], 394 | "source": [ 395 | "@tf.function\n", 396 | "def train_step(feature, target):\n", 397 | "\n", 398 | " with tf.GradientTape() as tape:\n", 399 | " y_pred = model(feature, training=True)\n", 400 | " loss = loss_fn(target, y_pred)\n", 401 | " \n", 402 | " grads = tape.gradient(loss, model.variables)\n", 403 | " optimizer.apply_gradients(zip(grads, model.variables))\n", 404 | " \n", 405 | " train_loss.update_state(loss)\n", 406 | "\n", 407 | "@tf.function\n", 408 | "def val_step(feature, target):\n", 409 | " \n", 410 | " y_pred = model(feature)\n", 411 | " loss = loss_fn(target, y_pred)\n", 412 | " \n", 413 | " val_loss.update_state(loss)" 414 | ] 415 | }, 416 | { 417 | "cell_type": "code", 418 | "execution_count": 16, 419 | "metadata": { 420 | "colab": { 421 | "base_uri": "https://localhost:8080/", 422 | "height": 451 423 | }, 424 | "colab_type": "code", 425 | "id": "vzeQsq7DdRmu", 426 | "outputId": "39c7e56c-8dda-41de-9e66-b2022bd50310" 427 | }, 428 | "outputs": [ 429 | { 430 | "name": "stdout", 431 | "output_type": "stream", 432 | "text": [ 433 | "epoch: 1 tr_loss: 0.9433 val_loss: 1.0346\n", 434 | "epoch: 2 tr_loss: 0.9121 val_loss: 1.0497\n", 435 | "epoch: 3 tr_loss: 0.8755 val_loss: 1.0695\n", 436 | "epoch: 4 tr_loss: 0.8345 val_loss: 1.0914\n", 437 | "epoch: 5 tr_loss: 0.7954 val_loss: 1.1103\n", 438 | "epoch: 6 tr_loss: 0.7531 val_loss: 1.1291\n", 439 | "epoch: 7 tr_loss: 0.7167 val_loss: 1.1449\n", 440 | "epoch: 8 tr_loss: 0.6786 val_loss: 1.1546\n", 441 | "epoch: 9 tr_loss: 0.6426 val_loss: 1.1676\n", 442 | "epoch: 10 tr_loss: 0.6140 val_loss: 1.1774\n" 443 | ] 444 | } 445 | ], 446 | "source": [ 447 | "for epoch in range(10):\n", 448 | " \n", 449 | " for _, (batch_feature, batch_target) in enumerate(dataset):\n", 450 | " with tf.device(\"/gpu:0\"):\n", 451 | " train_step(batch_feature, batch_target)\n", 452 | " \n", 453 | " for _, (batch_feature, batch_target) in enumerate(val_dataset):\n", 454 | " with tf.device(\"/gpu:0\"):\n", 455 | " loss_ = val_step(batch_feature, batch_target)\n", 456 | "\n", 457 | " epoch_loss = train_loss.result()\n", 458 | " epoch_loss_val = val_loss.result()\n", 459 | "\n", 460 | " print('epoch: {:} tr_loss: {:.4f} val_loss: {:.4f}'.format(\n", 461 | " epoch + 1, epoch_loss, epoch_loss_val))\n", 462 | "\n", 463 | " train_loss.reset_states()\n", 464 | " val_loss.reset_states()" 465 | ] 466 | }, 467 | { 468 | "cell_type": "code", 469 | "execution_count": null, 470 | "metadata": { 471 | "colab": {}, 472 | "colab_type": "code", 473 | "id": "74I_3WOkejci" 474 | }, 475 | "outputs": [], 476 | "source": [] 477 | } 478 | ], 479 | "metadata": { 480 | "colab": { 481 | "collapsed_sections": [], 482 | "include_colab_link": true, 483 | "name": "BasicStyle.ipynb", 484 | "provenance": [], 485 | "version": "0.3.2" 486 | }, 487 | "kernelspec": { 488 | "display_name": "Python 3", 489 | "language": "python", 490 | "name": "python3" 491 | }, 492 | "language_info": { 493 | "codemirror_mode": { 494 | "name": "ipython", 495 | "version": 3 496 | }, 497 | "file_extension": ".py", 498 | "mimetype": "text/x-python", 499 | "name": "python", 500 | "nbconvert_exporter": "python", 501 | "pygments_lexer": "ipython3", 502 | "version": "3.6.8" 503 | } 504 | }, 505 | "nbformat": 4, 506 | "nbformat_minor": 4 507 | } 508 | -------------------------------------------------------------------------------- /tutorials/01_basics/.ipynb_checkpoints/BasicStyle-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "colab_type": "text", 7 | "id": "view-in-github" 8 | }, 9 | "source": [ 10 | "\"Open" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 1, 16 | "metadata": { 17 | "colab": { 18 | "base_uri": "https://localhost:8080/", 19 | "height": 89 20 | }, 21 | "colab_type": "code", 22 | "id": "2wXVpo2lhf2A", 23 | "outputId": "f53997e0-5ebd-417f-8a81-da3657a5780f" 24 | }, 25 | "outputs": [], 26 | "source": [ 27 | "# !pip install -q --upgrade tf-nightly-2.0-preview" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": 2, 33 | "metadata": { 34 | "colab": {}, 35 | "colab_type": "code", 36 | "id": "S3kHX4Vqhl5O" 37 | }, 38 | "outputs": [], 39 | "source": [ 40 | "import tensorflow as tf" 41 | ] 42 | }, 43 | { 44 | "cell_type": "markdown", 45 | "metadata": {}, 46 | "source": [ 47 | "## TensorFlow 2.0" 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": 3, 53 | "metadata": {}, 54 | "outputs": [ 55 | { 56 | "data": { 57 | "text/plain": [ 58 | "'2.0.0'" 59 | ] 60 | }, 61 | "execution_count": 3, 62 | "metadata": {}, 63 | "output_type": "execute_result" 64 | } 65 | ], 66 | "source": [ 67 | "tf.__version__" 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "metadata": { 73 | "colab_type": "text", 74 | "id": "cdgYCKIShn9M" 75 | }, 76 | "source": [ 77 | "## Make DataSet" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": 4, 83 | "metadata": { 84 | "colab": {}, 85 | "colab_type": "code", 86 | "id": "lcVt1_Zth3_6" 87 | }, 88 | "outputs": [], 89 | "source": [ 90 | "'''\n", 91 | "If you have numpy data, you can use\n", 92 | "tensor_data = tf.convert_to_tensor(numpy_data, dtype=tf.float32)\n", 93 | "for translation into tf.Tensor.\n", 94 | "'''\n", 95 | "# example training data\n", 96 | "feature = tf.random.normal(shape=[50000, 1000])\n", 97 | "target = tf.random.normal(shape=[50000, 10])\n", 98 | "\n", 99 | "# example validation data\n", 100 | "val_feature = tf.random.normal(shape=[10000, 1000])\n", 101 | "val_target = tf.random.normal(shape=[10000, 10])\n", 102 | "\n", 103 | "# example test data\n", 104 | "test_feature = tf.random.normal(shape=[5000, 1000])\n", 105 | "test_target = tf.random.normal(shape=[5000, 10])\n", 106 | "\n", 107 | "\n", 108 | "# make dataset\n", 109 | "dataset = tf.data.Dataset.from_tensor_slices((feature, target))\n", 110 | "val_dataset = tf.data.Dataset.from_tensor_slices((val_feature, val_target))\n", 111 | "test_dataset = tf.data.Dataset.from_tensor_slices((test_feature, test_target))" 112 | ] 113 | }, 114 | { 115 | "cell_type": "code", 116 | "execution_count": 5, 117 | "metadata": { 118 | "colab": { 119 | "base_uri": "https://localhost:8080/", 120 | "height": 35 121 | }, 122 | "colab_type": "code", 123 | "id": "zFAMHrm7SL5M", 124 | "outputId": "071d0426-317b-49d0-ffcd-6ec59a91c07a" 125 | }, 126 | "outputs": [ 127 | { 128 | "data": { 129 | "text/plain": [ 130 | "" 131 | ] 132 | }, 133 | "execution_count": 5, 134 | "metadata": {}, 135 | "output_type": "execute_result" 136 | } 137 | ], 138 | "source": [ 139 | "# A dataset have shape information except batchsize and data type.\n", 140 | "dataset" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": 6, 146 | "metadata": { 147 | "colab": {}, 148 | "colab_type": "code", 149 | "id": "s5wtoZOsSzft" 150 | }, 151 | "outputs": [], 152 | "source": [ 153 | "# Training data should be shuffled every epoch.\n", 154 | "# 10000 is buffer size.\n", 155 | "dataset = dataset.shuffle(10000)\n", 156 | "\n", 157 | "# For mini-batch training.\n", 158 | "# 256 is batch size.\n", 159 | "dataset = dataset.batch(256)\n", 160 | "\n", 161 | "# Of course we can write same code as follows\n", 162 | "# dataset = dataset.shuffle(10000).batch(256)\n", 163 | "\n", 164 | "# validation data and test data do NOT need shuffle.\n", 165 | "# batch size is as big as possible.\n", 166 | "val_dataset = val_dataset.batch(10000)\n", 167 | "test_dataset = test_dataset.batch(5000)" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": 7, 173 | "metadata": { 174 | "colab": { 175 | "base_uri": "https://localhost:8080/", 176 | "height": 35 177 | }, 178 | "colab_type": "code", 179 | "id": "X3JLZxT0UsnB", 180 | "outputId": "fbf1cfd8-1347-4372-a3f2-bb15db0e434d" 181 | }, 182 | "outputs": [ 183 | { 184 | "data": { 185 | "text/plain": [ 186 | "" 187 | ] 188 | }, 189 | "execution_count": 7, 190 | "metadata": {}, 191 | "output_type": "execute_result" 192 | } 193 | ], 194 | "source": [ 195 | "# dataset is set for batch training.\n", 196 | "dataset" 197 | ] 198 | }, 199 | { 200 | "cell_type": "markdown", 201 | "metadata": { 202 | "colab_type": "text", 203 | "id": "6vL6rFO6VFBo" 204 | }, 205 | "source": [ 206 | "## Make Network" 207 | ] 208 | }, 209 | { 210 | "cell_type": "code", 211 | "execution_count": 8, 212 | "metadata": { 213 | "colab": {}, 214 | "colab_type": "code", 215 | "id": "wBzNdkCyiFGs" 216 | }, 217 | "outputs": [], 218 | "source": [ 219 | "class MyNet(tf.keras.Model):\n", 220 | " '''\n", 221 | " We use basically tf.keras.Model for making network.\n", 222 | " This class will manage layers and that's trainable parameters.\n", 223 | " '''\n", 224 | " def __init__(self):\n", 225 | " super(MyNet, self).__init__()\n", 226 | " \n", 227 | " \n", 228 | " # We can use tf.keras.Sequential \n", 229 | " # which has a role of putting together some layers.\n", 230 | " # This class inherits tf.keras.Model, so this can manege parameters too.\n", 231 | " # This class only receive layers.Layer class.\n", 232 | " # (Note that tf.keras.Sequential receive tf.keras.layers.ReLU())\n", 233 | " \n", 234 | " self.layer1 = tf.keras.Sequential([\n", 235 | " tf.keras.layers.Dense(1024),\n", 236 | " tf.keras.layers.ReLU(),\n", 237 | " tf.keras.layers.BatchNormalization(axis=-1),\n", 238 | " tf.keras.layers.Dropout(rate=0.2),\n", 239 | " ])\n", 240 | " \n", 241 | " # Of course we can write some layers separately.\n", 242 | " \n", 243 | " self.dense = tf.keras.layers.Dense(256)\n", 244 | " self.bn = tf.keras.layers.BatchNormalization(axis=-1)\n", 245 | " self.do = tf.keras.layers.Dropout(rate=0.2)\n", 246 | " \n", 247 | " self.dense_output = tf.keras.layers.Dense(10)\n", 248 | " \n", 249 | " # tf.function is jit compiler which translate python code into TF graph.\n", 250 | " @tf.function\n", 251 | " def call(self, x, training=False):\n", 252 | " # tf.keras.Sequential class have training propaty\n", 253 | " # which manege behavior of dropout and batchnormalization etc.\n", 254 | " h = self.layer1(x, training=training)\n", 255 | " \n", 256 | " h = self.dense(h)\n", 257 | " # we can use tf.nn.relu function instead of tf.keras.layers.ReLU()\n", 258 | " h = tf.nn.relu(h)\n", 259 | "\n", 260 | " # BatchNormalization and Dropout class also have training property.\n", 261 | " h = self.bn(h, training=training)\n", 262 | " h = self.do(h, training=training)\n", 263 | " \n", 264 | " return self.dense_output(h)" 265 | ] 266 | }, 267 | { 268 | "cell_type": "code", 269 | "execution_count": 9, 270 | "metadata": { 271 | "colab": {}, 272 | "colab_type": "code", 273 | "id": "60kaVa6BiJzz" 274 | }, 275 | "outputs": [], 276 | "source": [ 277 | "model = MyNet()" 278 | ] 279 | }, 280 | { 281 | "cell_type": "code", 282 | "execution_count": 10, 283 | "metadata": { 284 | "colab": { 285 | "base_uri": "https://localhost:8080/", 286 | "height": 89 287 | }, 288 | "colab_type": "code", 289 | "id": "PTEFbRxqizpf", 290 | "outputId": "a784eb57-7caf-4333-c869-b20958f32606" 291 | }, 292 | "outputs": [ 293 | { 294 | "data": { 295 | "text/plain": [ 296 | "" 300 | ] 301 | }, 302 | "execution_count": 10, 303 | "metadata": {}, 304 | "output_type": "execute_result" 305 | } 306 | ], 307 | "source": [ 308 | "# test execution.\n", 309 | "model(tf.random.normal(shape=[1, 1000]))" 310 | ] 311 | }, 312 | { 313 | "cell_type": "code", 314 | "execution_count": 11, 315 | "metadata": { 316 | "colab": { 317 | "base_uri": "https://localhost:8080/", 318 | "height": 341 319 | }, 320 | "colab_type": "code", 321 | "id": "mjM1vGKSj5Ri", 322 | "outputId": "50f551b4-c586-48eb-94ca-f1640b88dc92" 323 | }, 324 | "outputs": [ 325 | { 326 | "name": "stdout", 327 | "output_type": "stream", 328 | "text": [ 329 | "Model: \"my_net\"\n", 330 | "_________________________________________________________________\n", 331 | "Layer (type) Output Shape Param # \n", 332 | "=================================================================\n", 333 | "sequential (Sequential) multiple 1029120 \n", 334 | "_________________________________________________________________\n", 335 | "dense_1 (Dense) multiple 262400 \n", 336 | "_________________________________________________________________\n", 337 | "batch_normalization_1 (Batch multiple 1024 \n", 338 | "_________________________________________________________________\n", 339 | "dropout_1 (Dropout) multiple 0 \n", 340 | "_________________________________________________________________\n", 341 | "dense_2 (Dense) multiple 2570 \n", 342 | "=================================================================\n", 343 | "Total params: 1,295,114\n", 344 | "Trainable params: 1,292,554\n", 345 | "Non-trainable params: 2,560\n", 346 | "_________________________________________________________________\n" 347 | ] 348 | } 349 | ], 350 | "source": [ 351 | "# We can check model compose with model.summary() after first execution.\n", 352 | "model.summary()" 353 | ] 354 | }, 355 | { 356 | "cell_type": "markdown", 357 | "metadata": { 358 | "colab_type": "text", 359 | "id": "xySVCRU9j7l2" 360 | }, 361 | "source": [ 362 | "## Training by hand" 363 | ] 364 | }, 365 | { 366 | "cell_type": "code", 367 | "execution_count": 12, 368 | "metadata": { 369 | "colab": {}, 370 | "colab_type": "code", 371 | "id": "1eQ8DRWTbFjO" 372 | }, 373 | "outputs": [], 374 | "source": [ 375 | "optimizer = tf.optimizers.Adam()\n", 376 | "# for loss execution\n", 377 | "loss_fn = tf.losses.MeanSquaredError()\n", 378 | "\n", 379 | "# for iter loop\n", 380 | "# This class can have property of loss each iter\n", 381 | "train_loss = tf.keras.metrics.Mean() \n", 382 | "val_loss = tf.keras.metrics.Mean()" 383 | ] 384 | }, 385 | { 386 | "cell_type": "code", 387 | "execution_count": 13, 388 | "metadata": { 389 | "colab": {}, 390 | "colab_type": "code", 391 | "id": "pHTjWBcVbvo6" 392 | }, 393 | "outputs": [], 394 | "source": [ 395 | "@tf.function\n", 396 | "def train_step(feature, target):\n", 397 | "\n", 398 | " with tf.GradientTape() as tape:\n", 399 | " y_pred = model(feature, training=True)\n", 400 | " loss = loss_fn(target, y_pred)\n", 401 | " \n", 402 | " grads = tape.gradient(loss, model.variables)\n", 403 | " optimizer.apply_gradients(zip(grads, model.variables))\n", 404 | " \n", 405 | " train_loss.update_state(loss)\n", 406 | "\n", 407 | "@tf.function\n", 408 | "def val_step(feature, target):\n", 409 | " \n", 410 | " y_pred = model(feature)\n", 411 | " loss = loss_fn(target, y_pred)\n", 412 | " \n", 413 | " val_loss.update_state(loss)" 414 | ] 415 | }, 416 | { 417 | "cell_type": "code", 418 | "execution_count": 16, 419 | "metadata": { 420 | "colab": { 421 | "base_uri": "https://localhost:8080/", 422 | "height": 451 423 | }, 424 | "colab_type": "code", 425 | "id": "vzeQsq7DdRmu", 426 | "outputId": "39c7e56c-8dda-41de-9e66-b2022bd50310" 427 | }, 428 | "outputs": [ 429 | { 430 | "name": "stdout", 431 | "output_type": "stream", 432 | "text": [ 433 | "epoch: 1 tr_loss: 0.9433 val_loss: 1.0346\n", 434 | "epoch: 2 tr_loss: 0.9121 val_loss: 1.0497\n", 435 | "epoch: 3 tr_loss: 0.8755 val_loss: 1.0695\n", 436 | "epoch: 4 tr_loss: 0.8345 val_loss: 1.0914\n", 437 | "epoch: 5 tr_loss: 0.7954 val_loss: 1.1103\n", 438 | "epoch: 6 tr_loss: 0.7531 val_loss: 1.1291\n", 439 | "epoch: 7 tr_loss: 0.7167 val_loss: 1.1449\n", 440 | "epoch: 8 tr_loss: 0.6786 val_loss: 1.1546\n", 441 | "epoch: 9 tr_loss: 0.6426 val_loss: 1.1676\n", 442 | "epoch: 10 tr_loss: 0.6140 val_loss: 1.1774\n" 443 | ] 444 | } 445 | ], 446 | "source": [ 447 | "for epoch in range(10):\n", 448 | " \n", 449 | " for _, (batch_feature, batch_target) in enumerate(dataset):\n", 450 | " with tf.device(\"/gpu:0\"):\n", 451 | " train_step(batch_feature, batch_target)\n", 452 | " \n", 453 | " for _, (batch_feature, batch_target) in enumerate(val_dataset):\n", 454 | " with tf.device(\"/gpu:0\"):\n", 455 | " loss_ = val_step(batch_feature, batch_target)\n", 456 | "\n", 457 | " epoch_loss = train_loss.result()\n", 458 | " epoch_loss_val = val_loss.result()\n", 459 | "\n", 460 | " print('epoch: {:} tr_loss: {:.4f} val_loss: {:.4f}'.format(\n", 461 | " epoch + 1, epoch_loss, epoch_loss_val))\n", 462 | "\n", 463 | " train_loss.reset_states()\n", 464 | " val_loss.reset_states()" 465 | ] 466 | }, 467 | { 468 | "cell_type": "code", 469 | "execution_count": null, 470 | "metadata": { 471 | "colab": {}, 472 | "colab_type": "code", 473 | "id": "74I_3WOkejci" 474 | }, 475 | "outputs": [], 476 | "source": [] 477 | } 478 | ], 479 | "metadata": { 480 | "colab": { 481 | "collapsed_sections": [], 482 | "include_colab_link": true, 483 | "name": "BasicStyle.ipynb", 484 | "provenance": [], 485 | "version": "0.3.2" 486 | }, 487 | "kernelspec": { 488 | "display_name": "Python 3", 489 | "language": "python", 490 | "name": "python3" 491 | }, 492 | "language_info": { 493 | "codemirror_mode": { 494 | "name": "ipython", 495 | "version": 3 496 | }, 497 | "file_extension": ".py", 498 | "mimetype": "text/x-python", 499 | "name": "python", 500 | "nbconvert_exporter": "python", 501 | "pygments_lexer": "ipython3", 502 | "version": "3.6.8" 503 | } 504 | }, 505 | "nbformat": 4, 506 | "nbformat_minor": 4 507 | } 508 | -------------------------------------------------------------------------------- /tutorials/02_intermediate/.ipynb_checkpoints/Bidrectional_Recurrent_Neural_Network_using_KerasAPI-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "colab_type": "text", 7 | "id": "view-in-github" 8 | }, 9 | "source": [ 10 | "\"Open" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 0, 16 | "metadata": { 17 | "colab": {}, 18 | "colab_type": "code", 19 | "id": "VkJooysd9saV" 20 | }, 21 | "outputs": [], 22 | "source": [ 23 | "import numpy as np\n", 24 | "import tensorflow as tf\n", 25 | "import matplotlib.pyplot as plt\n", 26 | "%matplotlib inline\n", 27 | "\n", 28 | "tf.enable_eager_execution()\n", 29 | "L = tf.keras.layers\n", 30 | "tfe = tf.contrib.eager" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": 43, 36 | "metadata": { 37 | "colab": { 38 | "base_uri": "https://localhost:8080/", 39 | "height": 89 40 | }, 41 | "colab_type": "code", 42 | "id": "mvWroGCq-AGM", 43 | "outputId": "67c5137d-eeb3-457b-a170-3033de5674ab" 44 | }, 45 | "outputs": [ 46 | { 47 | "name": "stdout", 48 | "output_type": "stream", 49 | "text": [ 50 | "training_data: (60000, 28, 28)\n", 51 | "test_data: (10000, 28, 28)\n", 52 | "training_label: (60000,)\n", 53 | "test_label: (10000,)\n" 54 | ] 55 | } 56 | ], 57 | "source": [ 58 | "# Hyper parameters\n", 59 | "num_epochs = 25\n", 60 | "num_classes = 10\n", 61 | "batch_size = 512\n", 62 | "learning_rate = 0.001\n", 63 | "\n", 64 | "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n", 65 | "\n", 66 | "print(\"training_data: \", x_train.shape)\n", 67 | "print(\"test_data: \", x_test.shape)\n", 68 | "print(\"training_label: \", y_train.shape)\n", 69 | "print(\"test_label: \", y_test.shape)" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 44, 75 | "metadata": { 76 | "colab": { 77 | "base_uri": "https://localhost:8080/", 78 | "height": 89 79 | }, 80 | "colab_type": "code", 81 | "id": "int-eal1-RBm", 82 | "outputId": "483e4246-19e2-46a2-b713-e2c2979461f4" 83 | }, 84 | "outputs": [ 85 | { 86 | "name": "stdout", 87 | "output_type": "stream", 88 | "text": [ 89 | "training_data: (60000, 28, 28)\n", 90 | "test_data: (10000, 28, 28)\n", 91 | "training_label: (60000, 10)\n", 92 | "test_label: (10000, 10)\n" 93 | ] 94 | } 95 | ], 96 | "source": [ 97 | "x_train_eager = tf.convert_to_tensor(x_train, dtype=tf.float32)\n", 98 | "x_test_eager = tf.convert_to_tensor(x_test, dtype=tf.float32)\n", 99 | "y_train_eager = tf.reshape(tf.one_hot(y_train, 10), (-1, 10))\n", 100 | "y_test_eager = tf.reshape(tf.one_hot(y_test, 10), (-1, 10))\n", 101 | "\n", 102 | "print(\"training_data: \", x_train_eager.shape)\n", 103 | "print(\"test_data: \", x_test_eager.shape)\n", 104 | "print(\"training_label: \", y_train_eager.shape)\n", 105 | "print(\"test_label: \", y_test_eager.shape)" 106 | ] 107 | }, 108 | { 109 | "cell_type": "markdown", 110 | "metadata": { 111 | "colab_type": "text", 112 | "id": "jwcKjSDNGD5W" 113 | }, 114 | "source": [ 115 | "### DataSet\n", 116 | "You make Dataset using `tf.data.Dataset` Class but Keras API doesn't need this dataset. If you write training loop code manually, `Dataset` class is very useful. And using keras API, you need numpy.array inputs instead of tf.Tensor. I don't know why...so you only need numpy preprocessing (or get numpy.array from tf.Tensor using numpy() method after preprocessing using function of tf).\n", 117 | "\n", 118 | "### NOTE\n", 119 | "This notebook we don't need 'tf.data.Dataset'. This code only just for reference." 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": 0, 125 | "metadata": { 126 | "colab": {}, 127 | "colab_type": "code", 128 | "id": "YNU_cq4L-u10" 129 | }, 130 | "outputs": [], 131 | "source": [ 132 | "train_dataset = (\n", 133 | " tf.data.Dataset.from_tensor_slices((x_train_eager, y_train_eager))\n", 134 | " .batch(batch_size)\n", 135 | " .shuffle(10000)\n", 136 | ")\n", 137 | "train_dataset = train_dataset.repeat()" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 0, 143 | "metadata": { 144 | "colab": {}, 145 | "colab_type": "code", 146 | "id": "_npuvGwv-588" 147 | }, 148 | "outputs": [], 149 | "source": [ 150 | "test_dataset = (\n", 151 | " tf.data.Dataset.from_tensor_slices((x_test_eager, y_test_eager))\n", 152 | " .batch(1000)\n", 153 | " .shuffle(10000)\n", 154 | ")\n", 155 | "test_dataset = test_dataset.repeat()" 156 | ] 157 | }, 158 | { 159 | "cell_type": "markdown", 160 | "metadata": { 161 | "colab_type": "text", 162 | "id": "e-Z06hTWOx4P" 163 | }, 164 | "source": [ 165 | "### RNN using LSTM\n", 166 | "In keras API, LSTM recives inputs tensor whose shape is (batch_size, seqence_length, feature_dim), and output tensor whose shape is (batch_size, fearure_dim).When you need all time sequence data, you have to give `return_sequences=True` to LSTM's constractor. Generally, when you stack LSTM's, you need all sequence data.\n", 167 | "\n", 168 | "We use just only `tf.keras.layers.Bidirectional` for using Bidrectional LSTM." 169 | ] 170 | }, 171 | { 172 | "cell_type": "code", 173 | "execution_count": 0, 174 | "metadata": { 175 | "colab": {}, 176 | "colab_type": "code", 177 | "id": "gz5RrFnm_HzM" 178 | }, 179 | "outputs": [], 180 | "source": [ 181 | "class RNN(tf.keras.Model):\n", 182 | " def __init__(self, hidden_size=10, num_layers=2, num_classes=10):\n", 183 | " super(RNN, self).__init__(name='mnist_rnn')\n", 184 | " self.hidden_size = hidden_size\n", 185 | " self.num_layers = num_layers\n", 186 | " \n", 187 | " self.lstm = self.get_lstm_layers(hidden_size, num_layers) \n", 188 | " self.fc = L.Dense(num_classes, activation=\"softmax\")\n", 189 | " \n", 190 | " @staticmethod\n", 191 | " def get_lstm_layers(hidden_size, num_layers):\n", 192 | " lstm_layers = []\n", 193 | " # we need all sequence data. write return_sequences=True! \n", 194 | " for i in range(num_layers-1):\n", 195 | " lstm_layers.append(\n", 196 | " L.Bidirectional(\n", 197 | " L.CuDNNLSTM(units=hidden_size, \n", 198 | " return_sequences=True)\n", 199 | " )\n", 200 | " )\n", 201 | " # the final layer return only final sequence\n", 202 | " # if you need all sequences, you have to write return_sequences=True.\n", 203 | " lstm_layers.append(\n", 204 | " L.Bidirectional(\n", 205 | " L.CuDNNLSTM(units=hidden_size)\n", 206 | " )\n", 207 | " )\n", 208 | " return tf.keras.Sequential(lstm_layers)\n", 209 | " \n", 210 | " def call(self, x): \n", 211 | " # Forward propagate LSTM\n", 212 | " out = self.lstm(x)\n", 213 | " out = self.fc(out)\n", 214 | " return out" 215 | ] 216 | }, 217 | { 218 | "cell_type": "code", 219 | "execution_count": 0, 220 | "metadata": { 221 | "colab": {}, 222 | "colab_type": "code", 223 | "id": "oMARKqbaBjIN" 224 | }, 225 | "outputs": [], 226 | "source": [ 227 | "model = RNN()" 228 | ] 229 | }, 230 | { 231 | "cell_type": "code", 232 | "execution_count": 49, 233 | "metadata": { 234 | "colab": { 235 | "base_uri": "https://localhost:8080/", 236 | "height": 215 237 | }, 238 | "colab_type": "code", 239 | "id": "_NPkLp-rBzCp", 240 | "outputId": "b520e388-52fe-4bc8-e53a-7131b1788ae6" 241 | }, 242 | "outputs": [ 243 | { 244 | "name": "stdout", 245 | "output_type": "stream", 246 | "text": [ 247 | "_________________________________________________________________\n", 248 | "Layer (type) Output Shape Param # \n", 249 | "=================================================================\n", 250 | "sequential_4 (Sequential) multiple 5760 \n", 251 | "_________________________________________________________________\n", 252 | "dense_4 (Dense) multiple 210 \n", 253 | "=================================================================\n", 254 | "Total params: 5,970\n", 255 | "Trainable params: 5,970\n", 256 | "Non-trainable params: 0\n", 257 | "_________________________________________________________________\n" 258 | ] 259 | } 260 | ], 261 | "source": [ 262 | "optimizer = tf.train.AdamOptimizer(learning_rate)\n", 263 | "model.compile(optimizer=optimizer,\n", 264 | " loss='categorical_crossentropy',\n", 265 | " metrics=[\"accuracy\"])\n", 266 | "\n", 267 | "# Eager Execution initialize parameters when using model.call()\n", 268 | "model(x_train_eager[:50])\n", 269 | "\n", 270 | "model.summary()" 271 | ] 272 | }, 273 | { 274 | "cell_type": "code", 275 | "execution_count": 50, 276 | "metadata": { 277 | "colab": { 278 | "base_uri": "https://localhost:8080/", 279 | "height": 935 280 | }, 281 | "colab_type": "code", 282 | "id": "wDEyoIMnDKEy", 283 | "outputId": "3f9c5d25-6472-4b7e-b3cf-28274c0d38ac" 284 | }, 285 | "outputs": [ 286 | { 287 | "name": "stdout", 288 | "output_type": "stream", 289 | "text": [ 290 | "Epoch 1/25\n", 291 | "94/94 [==============================] - 8s 81ms/step - loss: 2.0717 - acc: 0.3138 - val_loss: 1.6623 - val_acc: 0.5032\n", 292 | "Epoch 2/25\n", 293 | "94/94 [==============================] - 8s 80ms/step - loss: 1.3120 - acc: 0.6062 - val_loss: 0.9671 - val_acc: 0.7200\n", 294 | "Epoch 3/25\n", 295 | "94/94 [==============================] - 7s 80ms/step - loss: 0.8209 - acc: 0.7607 - val_loss: 0.6587 - val_acc: 0.8092\n", 296 | "Epoch 4/25\n", 297 | "94/94 [==============================] - 7s 79ms/step - loss: 0.6089 - acc: 0.8215 - val_loss: 0.5239 - val_acc: 0.8412\n", 298 | "Epoch 5/25\n", 299 | "94/94 [==============================] - 7s 80ms/step - loss: 0.4958 - acc: 0.8532 - val_loss: 0.4411 - val_acc: 0.8674\n", 300 | "Epoch 6/25\n", 301 | "94/94 [==============================] - 7s 79ms/step - loss: 0.4225 - acc: 0.8759 - val_loss: 0.3824 - val_acc: 0.8862\n", 302 | "Epoch 7/25\n", 303 | "94/94 [==============================] - 7s 79ms/step - loss: 0.3707 - acc: 0.8913 - val_loss: 0.3413 - val_acc: 0.8983\n", 304 | "Epoch 8/25\n", 305 | "94/94 [==============================] - 7s 80ms/step - loss: 0.3359 - acc: 0.9007 - val_loss: 0.3150 - val_acc: 0.9071\n", 306 | "Epoch 9/25\n", 307 | "94/94 [==============================] - 8s 81ms/step - loss: 0.3081 - acc: 0.9083 - val_loss: 0.2964 - val_acc: 0.9110\n", 308 | "Epoch 10/25\n", 309 | "94/94 [==============================] - 7s 80ms/step - loss: 0.2853 - acc: 0.9161 - val_loss: 0.2761 - val_acc: 0.9169\n", 310 | "Epoch 11/25\n", 311 | "94/94 [==============================] - 7s 79ms/step - loss: 0.2647 - acc: 0.9215 - val_loss: 0.2632 - val_acc: 0.9203\n", 312 | "Epoch 12/25\n", 313 | "94/94 [==============================] - 7s 79ms/step - loss: 0.2480 - acc: 0.9272 - val_loss: 0.2520 - val_acc: 0.9262\n", 314 | "Epoch 13/25\n", 315 | "94/94 [==============================] - 7s 79ms/step - loss: 0.2341 - acc: 0.9309 - val_loss: 0.2367 - val_acc: 0.9265\n", 316 | "Epoch 14/25\n", 317 | "94/94 [==============================] - 7s 79ms/step - loss: 0.2200 - acc: 0.9354 - val_loss: 0.2283 - val_acc: 0.9326\n", 318 | "Epoch 15/25\n", 319 | "94/94 [==============================] - 7s 79ms/step - loss: 0.2098 - acc: 0.9385 - val_loss: 0.2229 - val_acc: 0.9332\n", 320 | "Epoch 16/25\n", 321 | "94/94 [==============================] - 8s 80ms/step - loss: 0.2012 - acc: 0.9406 - val_loss: 0.2198 - val_acc: 0.9327\n", 322 | "Epoch 17/25\n", 323 | "94/94 [==============================] - 7s 79ms/step - loss: 0.1921 - acc: 0.9432 - val_loss: 0.2064 - val_acc: 0.9387\n", 324 | "Epoch 18/25\n", 325 | "94/94 [==============================] - 8s 84ms/step - loss: 0.1827 - acc: 0.9464 - val_loss: 0.2035 - val_acc: 0.9383\n", 326 | "Epoch 19/25\n", 327 | "94/94 [==============================] - 8s 80ms/step - loss: 0.1758 - acc: 0.9493 - val_loss: 0.1989 - val_acc: 0.9406\n", 328 | "Epoch 20/25\n", 329 | "94/94 [==============================] - 7s 79ms/step - loss: 0.1713 - acc: 0.9495 - val_loss: 0.1928 - val_acc: 0.9406\n", 330 | "Epoch 21/25\n", 331 | "94/94 [==============================] - 7s 79ms/step - loss: 0.1653 - acc: 0.9517 - val_loss: 0.1874 - val_acc: 0.9427\n", 332 | "Epoch 22/25\n", 333 | "94/94 [==============================] - 7s 79ms/step - loss: 0.1585 - acc: 0.9535 - val_loss: 0.1809 - val_acc: 0.9455\n", 334 | "Epoch 23/25\n", 335 | "94/94 [==============================] - 8s 80ms/step - loss: 0.1541 - acc: 0.9550 - val_loss: 0.1765 - val_acc: 0.9464\n", 336 | "Epoch 24/25\n", 337 | "94/94 [==============================] - 7s 79ms/step - loss: 0.1489 - acc: 0.9560 - val_loss: 0.1768 - val_acc: 0.9457\n", 338 | "Epoch 25/25\n", 339 | "94/94 [==============================] - 7s 80ms/step - loss: 0.1457 - acc: 0.9577 - val_loss: 0.1712 - val_acc: 0.9473\n" 340 | ] 341 | }, 342 | { 343 | "data": { 344 | "text/plain": [ 345 | "" 346 | ] 347 | }, 348 | "execution_count": 50, 349 | "metadata": { 350 | "tags": [] 351 | }, 352 | "output_type": "execute_result" 353 | } 354 | ], 355 | "source": [ 356 | "model.fit(x=x_train_eager.numpy(), \n", 357 | " y=y_train_eager.numpy(), \n", 358 | " validation_split=0.2, \n", 359 | " epochs=num_epochs,\n", 360 | " batch_size=batch_size)" 361 | ] 362 | }, 363 | { 364 | "cell_type": "code", 365 | "execution_count": 51, 366 | "metadata": { 367 | "colab": { 368 | "base_uri": "https://localhost:8080/", 369 | "height": 53 370 | }, 371 | "colab_type": "code", 372 | "id": "oDuYrgFIG850", 373 | "outputId": "66b7b552-046c-4811-b58e-2465dcb72676" 374 | }, 375 | "outputs": [ 376 | { 377 | "name": "stdout", 378 | "output_type": "stream", 379 | "text": [ 380 | "313/313 [==============================] - 9s 30ms/step\n", 381 | "test_accracy: 0.95\n" 382 | ] 383 | } 384 | ], 385 | "source": [ 386 | "test_loss, test_acc = model.evaluate(x=x_test_eager.numpy(), \n", 387 | " y=y_test_eager.numpy())\n", 388 | "\n", 389 | "print(\"test_accracy: \", test_acc)" 390 | ] 391 | }, 392 | { 393 | "cell_type": "code", 394 | "execution_count": 0, 395 | "metadata": { 396 | "colab": {}, 397 | "colab_type": "code", 398 | "id": "Yeayjx_nKXaW" 399 | }, 400 | "outputs": [], 401 | "source": [] 402 | } 403 | ], 404 | "metadata": { 405 | "accelerator": "GPU", 406 | "colab": { 407 | "collapsed_sections": [], 408 | "include_colab_link": true, 409 | "name": "Recurrent_Neural_Network", 410 | "provenance": [], 411 | "version": "0.3.2" 412 | }, 413 | "kernelspec": { 414 | "display_name": "Python 3", 415 | "language": "python", 416 | "name": "python3" 417 | }, 418 | "language_info": { 419 | "codemirror_mode": { 420 | "name": "ipython", 421 | "version": 3 422 | }, 423 | "file_extension": ".py", 424 | "mimetype": "text/x-python", 425 | "name": "python", 426 | "nbconvert_exporter": "python", 427 | "pygments_lexer": "ipython3", 428 | "version": "3.5.3" 429 | } 430 | }, 431 | "nbformat": 4, 432 | "nbformat_minor": 1 433 | } 434 | -------------------------------------------------------------------------------- /tutorials/02_intermediate/.ipynb_checkpoints/residual_network_using_KerasAPI-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "colab_type": "text", 7 | "id": "view-in-github" 8 | }, 9 | "source": [ 10 | "\"Open" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 0, 16 | "metadata": { 17 | "colab": {}, 18 | "colab_type": "code", 19 | "id": "W-2MApJumG9i" 20 | }, 21 | "outputs": [], 22 | "source": [ 23 | "import numpy as np\n", 24 | "import tensorflow as tf\n", 25 | "import matplotlib.pyplot as plt\n", 26 | "%matplotlib inline" 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": 0, 32 | "metadata": { 33 | "colab": {}, 34 | "colab_type": "code", 35 | "id": "YY0lj_PWmNKz" 36 | }, 37 | "outputs": [], 38 | "source": [ 39 | "tfe = tf.contrib.eager\n", 40 | "L = tf.keras.layers\n", 41 | "tf.enable_eager_execution()" 42 | ] 43 | }, 44 | { 45 | "cell_type": "code", 46 | "execution_count": 0, 47 | "metadata": { 48 | "colab": {}, 49 | "colab_type": "code", 50 | "id": "_0M-QOolbYug" 51 | }, 52 | "outputs": [], 53 | "source": [] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": 0, 58 | "metadata": { 59 | "colab": { 60 | "base_uri": "https://localhost:8080/", 61 | "height": 197 62 | }, 63 | "colab_type": "code", 64 | "id": "Az2CgxXFmilT", 65 | "outputId": "596303c3-38bc-40db-bd1b-33bbf5f6bcbd" 66 | }, 67 | "outputs": [ 68 | { 69 | "name": "stdout", 70 | "output_type": "stream", 71 | "text": [ 72 | "Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\n", 73 | "170500096/170498071 [==============================] - 46s 0us/step\n", 74 | "training_data\n", 75 | " (50000, 32, 32, 3)\n", 76 | "test_data\n", 77 | " (10000, 32, 32, 3)\n", 78 | "training_label\n", 79 | " (50000, 1)\n", 80 | "test_label\n", 81 | " (10000, 1)\n" 82 | ] 83 | } 84 | ], 85 | "source": [ 86 | "# Hyper parameters\n", 87 | "num_epochs = 10\n", 88 | "num_classes = 10\n", 89 | "batch_size = 256\n", 90 | "learning_rate = 0.001\n", 91 | "\n", 92 | "(x_train, y_train), (x_test, y_test) = tfk.datasets.cifar10.load_data()\n", 93 | "\n", 94 | "print(\"training_data\\n\", x_train.shape)\n", 95 | "print(\"test_data\\n\", x_test.shape)\n", 96 | "print(\"training_label\\n\", y_train.shape)\n", 97 | "print(\"test_label\\n\", y_test.shape)" 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": 0, 103 | "metadata": { 104 | "colab": { 105 | "base_uri": "https://localhost:8080/", 106 | "height": 53 107 | }, 108 | "colab_type": "code", 109 | "id": "U6MRu-QDok4N", 110 | "outputId": "7f92a3df-bda0-4291-be16-3ce4b1c2607e" 111 | }, 112 | "outputs": [ 113 | { 114 | "name": "stdout", 115 | "output_type": "stream", 116 | "text": [ 117 | "(50000, 32, 32, 3)\n", 118 | "(50000, 10)\n" 119 | ] 120 | } 121 | ], 122 | "source": [ 123 | "x_train_ = tf.convert_to_tensor(x_train, dtype=tf.float32)\n", 124 | "y_train_ = tf.reshape(tf.one_hot(y_train, 10), (-1, 10))\n", 125 | "\n", 126 | "\n", 127 | "print(x_train_.shape)\n", 128 | "print(y_train_.shape)" 129 | ] 130 | }, 131 | { 132 | "cell_type": "code", 133 | "execution_count": 0, 134 | "metadata": { 135 | "colab": { 136 | "base_uri": "https://localhost:8080/", 137 | "height": 35 138 | }, 139 | "colab_type": "code", 140 | "id": "6f15_qjHtZl8", 141 | "outputId": "2f368c49-9ef3-4021-9484-0aa53e195db3" 142 | }, 143 | "outputs": [ 144 | { 145 | "name": "stdout", 146 | "output_type": "stream", 147 | "text": [ 148 | "\n" 149 | ] 150 | } 151 | ], 152 | "source": [ 153 | "train_dataset = (\n", 154 | " tf.data.Dataset.from_tensor_slices((x_train, y_train))\n", 155 | " .batch(batch_size)\n", 156 | " .shuffle(10000)\n", 157 | ")\n", 158 | "\n", 159 | "train_dataset = (\n", 160 | " train_dataset.map(lambda x, y: \n", 161 | " (tf.div(tf.cast(x, tf.float32), 255.0), \n", 162 | " tf.reshape(tf.one_hot(y, 10), (-1, 10))))\n", 163 | ")\n", 164 | "\n", 165 | "train_dataset = train_dataset.repeat()\n", 166 | "print(train_dataset)" 167 | ] 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": 0, 172 | "metadata": { 173 | "colab": { 174 | "base_uri": "https://localhost:8080/", 175 | "height": 35 176 | }, 177 | "colab_type": "code", 178 | "id": "hL_0Ca4mvJ1m", 179 | "outputId": "eca6085e-f2cc-4f30-f627-370eb6ecf6e5" 180 | }, 181 | "outputs": [ 182 | { 183 | "name": "stdout", 184 | "output_type": "stream", 185 | "text": [ 186 | "\n" 187 | ] 188 | } 189 | ], 190 | "source": [ 191 | "test_dataset = (\n", 192 | " tf.data.Dataset.from_tensor_slices((x_test, y_test))\n", 193 | " .batch(1000)\n", 194 | " .shuffle(10000)\n", 195 | ")\n", 196 | "test_dataset = (\n", 197 | " test_dataset.map(lambda x, y: \n", 198 | " (tf.div(tf.cast(x, tf.float32), 255.0), \n", 199 | " tf.reshape(tf.one_hot(y, 10), (-1, 10))))\n", 200 | ")\n", 201 | "test_dataset = test_dataset.repeat()\n", 202 | "print(test_dataset)" 203 | ] 204 | }, 205 | { 206 | "cell_type": "code", 207 | "execution_count": 0, 208 | "metadata": { 209 | "colab": {}, 210 | "colab_type": "code", 211 | "id": "jUvyjTFXm_0d" 212 | }, 213 | "outputs": [], 214 | "source": [ 215 | "def conv3x3(out_channels, strides=1):\n", 216 | " return L.Conv2D(out_channels, kernel_size=3, \n", 217 | " strides=strides, padding='same', use_bias=False)" 218 | ] 219 | }, 220 | { 221 | "cell_type": "code", 222 | "execution_count": 0, 223 | "metadata": { 224 | "colab": {}, 225 | "colab_type": "code", 226 | "id": "ppETst-0nQtu" 227 | }, 228 | "outputs": [], 229 | "source": [ 230 | "# Residual block\n", 231 | "class ResidualBlock(tf.keras.Model):\n", 232 | " def __init__(self, out_channels, strides=1, downsample=None):\n", 233 | " super(ResidualBlock, self).__init__(name='ResidualBlock')\n", 234 | " self.conv1 = conv3x3(out_channels, strides)\n", 235 | " self.bn1 = L.BatchNormalization(axis=-1)\n", 236 | " self.relu = L.ReLU()\n", 237 | " self.conv2 = conv3x3(out_channels)\n", 238 | " self.bn2 = L.BatchNormalization(axis=-1)\n", 239 | " self.downsample = downsample\n", 240 | " \n", 241 | " def call(self, x):\n", 242 | " residual = x\n", 243 | " out = self.conv1(x)\n", 244 | " out = self.bn1(out)\n", 245 | " out = self.relu(out)\n", 246 | " out = self.conv2(out)\n", 247 | " out = self.bn2(out)\n", 248 | " if self.downsample:\n", 249 | " residual = self.downsample(x)\n", 250 | " out += residual\n", 251 | " out = self.relu(out)\n", 252 | " return out" 253 | ] 254 | }, 255 | { 256 | "cell_type": "code", 257 | "execution_count": 0, 258 | "metadata": { 259 | "colab": {}, 260 | "colab_type": "code", 261 | "id": "c4dcZi3boLkG" 262 | }, 263 | "outputs": [], 264 | "source": [ 265 | "class ResNet(tf.keras.Model):\n", 266 | " def __init__(self, block, layers, num_classes=10):\n", 267 | " super(ResNet, self).__init__(name='ResNet')\n", 268 | " self.in_channels = 16\n", 269 | " self.conv = conv3x3(16)\n", 270 | " self.bn = L.BatchNormalization(axis=-1)\n", 271 | " self.relu = L.ReLU()\n", 272 | " self.layer1 = self.make_layer(block, 16, layers[0])\n", 273 | " self.layer2 = self.make_layer(block, 32, layers[1], 2)\n", 274 | " self.layer3 = self.make_layer(block, 64, layers[2], 2)\n", 275 | " self.avg_pool = L.AvgPool2D(8)\n", 276 | " self.flatten = L.Flatten()\n", 277 | " self.fc = L.Dense(num_classes)\n", 278 | " \n", 279 | " def make_layer(self, block, out_channels, blocks, strides=1):\n", 280 | " downsample = None\n", 281 | " if (strides != 1) or (self.in_channels != out_channels):\n", 282 | " downsample = tf.keras.Sequential([\n", 283 | " conv3x3(out_channels, strides=strides),\n", 284 | " L.BatchNormalization(axis=-1)])\n", 285 | " layers = []\n", 286 | " layers.append(block(out_channels, strides, downsample))\n", 287 | " self.in_channels = out_channels\n", 288 | " for i in range(1, blocks):\n", 289 | " layers.append(block(out_channels))\n", 290 | " return tf.keras.Sequential(layers)\n", 291 | " \n", 292 | " def call(self, x):\n", 293 | " out = self.conv(x)\n", 294 | " out = self.bn(out)\n", 295 | " out = self.relu(out)\n", 296 | " out = self.layer1(out)\n", 297 | " out = self.layer2(out)\n", 298 | " out = self.layer3(out)\n", 299 | " out = self.avg_pool(out)\n", 300 | " out = self.flatten(out)\n", 301 | " out = self.fc(out)\n", 302 | " return out" 303 | ] 304 | }, 305 | { 306 | "cell_type": "code", 307 | "execution_count": 0, 308 | "metadata": { 309 | "colab": {}, 310 | "colab_type": "code", 311 | "id": "E6vmD9PdoQxW" 312 | }, 313 | "outputs": [], 314 | "source": [ 315 | "model = ResNet(ResidualBlock, [2, 2, 2])" 316 | ] 317 | }, 318 | { 319 | "cell_type": "code", 320 | "execution_count": 0, 321 | "metadata": { 322 | "colab": { 323 | "base_uri": "https://localhost:8080/", 324 | "height": 467 325 | }, 326 | "colab_type": "code", 327 | "id": "Qx0zHRhaqUL3", 328 | "outputId": "3a83572e-953e-4183-a45d-dcd4f4be6b90" 329 | }, 330 | "outputs": [ 331 | { 332 | "name": "stdout", 333 | "output_type": "stream", 334 | "text": [ 335 | "_________________________________________________________________\n", 336 | "Layer (type) Output Shape Param # \n", 337 | "=================================================================\n", 338 | "conv2d (Conv2D) multiple 432 \n", 339 | "_________________________________________________________________\n", 340 | "batch_normalization (BatchNo multiple 64 \n", 341 | "_________________________________________________________________\n", 342 | "re_lu (ReLU) multiple 0 \n", 343 | "_________________________________________________________________\n", 344 | "sequential (Sequential) multiple 9472 \n", 345 | "_________________________________________________________________\n", 346 | "sequential_2 (Sequential) multiple 37504 \n", 347 | "_________________________________________________________________\n", 348 | "sequential_4 (Sequential) multiple 148736 \n", 349 | "_________________________________________________________________\n", 350 | "average_pooling2d (AveragePo multiple 0 \n", 351 | "_________________________________________________________________\n", 352 | "flatten (Flatten) multiple 0 \n", 353 | "_________________________________________________________________\n", 354 | "dense (Dense) multiple 650 \n", 355 | "=================================================================\n", 356 | "Total params: 196,858\n", 357 | "Trainable params: 195,738\n", 358 | "Non-trainable params: 1,120\n", 359 | "_________________________________________________________________\n" 360 | ] 361 | } 362 | ], 363 | "source": [ 364 | "model.compile(optimizer=tf.train.AdamOptimizer(learning_rate),\n", 365 | " loss=tf.losses.softmax_cross_entropy,\n", 366 | " metrics=[\"accuracy\"])\n", 367 | "model(x_train_[:50]).shape\n", 368 | "model.summary()" 369 | ] 370 | }, 371 | { 372 | "cell_type": "code", 373 | "execution_count": 0, 374 | "metadata": { 375 | "colab": { 376 | "base_uri": "https://localhost:8080/", 377 | "height": 395 378 | }, 379 | "colab_type": "code", 380 | "id": "EGEcgjposP9I", 381 | "outputId": "ab97dd47-1254-4763-ce64-bcf6ca1bc814" 382 | }, 383 | "outputs": [ 384 | { 385 | "name": "stdout", 386 | "output_type": "stream", 387 | "text": [ 388 | "Epoch 1/10\n", 389 | "195/195 [==============================] - 34s 176ms/step - loss: 1.4596 - acc: 0.4686\n", 390 | "Epoch 2/10\n", 391 | "195/195 [==============================] - 34s 172ms/step - loss: 1.0619 - acc: 0.6191\n", 392 | "Epoch 3/10\n", 393 | "195/195 [==============================] - 33s 172ms/step - loss: 0.8769 - acc: 0.6908\n", 394 | "Epoch 4/10\n", 395 | "195/195 [==============================] - 34s 172ms/step - loss: 0.7570 - acc: 0.7326\n", 396 | "Epoch 5/10\n", 397 | "195/195 [==============================] - 33s 171ms/step - loss: 0.6612 - acc: 0.7683\n", 398 | "Epoch 6/10\n", 399 | "195/195 [==============================] - 33s 172ms/step - loss: 0.5843 - acc: 0.7966\n", 400 | "Epoch 7/10\n", 401 | "195/195 [==============================] - 33s 171ms/step - loss: 0.5179 - acc: 0.8201\n", 402 | "Epoch 8/10\n", 403 | "195/195 [==============================] - 34s 173ms/step - loss: 0.4656 - acc: 0.8364\n", 404 | "Epoch 9/10\n", 405 | "195/195 [==============================] - 33s 172ms/step - loss: 0.4131 - acc: 0.8549\n", 406 | "Epoch 10/10\n", 407 | "195/195 [==============================] - 33s 172ms/step - loss: 0.3726 - acc: 0.8712\n" 408 | ] 409 | }, 410 | { 411 | "data": { 412 | "text/plain": [ 413 | "" 414 | ] 415 | }, 416 | "execution_count": 12, 417 | "metadata": { 418 | "tags": [] 419 | }, 420 | "output_type": "execute_result" 421 | } 422 | ], 423 | "source": [ 424 | "model.fit(train_dataset,\n", 425 | " batch_size=batch_size,\n", 426 | " epochs=num_epochs,\n", 427 | " steps_per_epoch=int(x_train.shape[0]/batch_size),\n", 428 | ")" 429 | ] 430 | }, 431 | { 432 | "cell_type": "code", 433 | "execution_count": 0, 434 | "metadata": { 435 | "colab": { 436 | "base_uri": "https://localhost:8080/", 437 | "height": 53 438 | }, 439 | "colab_type": "code", 440 | "id": "8M37rwRAt0kk", 441 | "outputId": "fb3c6bc2-4c1c-4df8-a241-7217dcb2f336" 442 | }, 443 | "outputs": [ 444 | { 445 | "name": "stdout", 446 | "output_type": "stream", 447 | "text": [ 448 | "100/100 [==============================] - 14s 142ms/step\n" 449 | ] 450 | }, 451 | { 452 | "data": { 453 | "text/plain": [ 454 | "[0.7618808209896087, 0.7497999966144562]" 455 | ] 456 | }, 457 | "execution_count": 17, 458 | "metadata": { 459 | "tags": [] 460 | }, 461 | "output_type": "execute_result" 462 | } 463 | ], 464 | "source": [ 465 | "model.evaluate(test_dataset, steps=100)" 466 | ] 467 | } 468 | ], 469 | "metadata": { 470 | "accelerator": "GPU", 471 | "colab": { 472 | "include_colab_link": true, 473 | "name": "residual_network_using_KerasAPI", 474 | "provenance": [], 475 | "version": "0.3.2" 476 | }, 477 | "kernelspec": { 478 | "display_name": "Python 3", 479 | "language": "python", 480 | "name": "python3" 481 | }, 482 | "language_info": { 483 | "codemirror_mode": { 484 | "name": "ipython", 485 | "version": 3 486 | }, 487 | "file_extension": ".py", 488 | "mimetype": "text/x-python", 489 | "name": "python", 490 | "nbconvert_exporter": "python", 491 | "pygments_lexer": "ipython3", 492 | "version": "3.5.3" 493 | } 494 | }, 495 | "nbformat": 4, 496 | "nbformat_minor": 1 497 | } 498 | -------------------------------------------------------------------------------- /tutorials/02_intermediate/Convolutinal_Neural_Network_using_KerasAPI.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "colab_type": "text", 7 | "id": "view-in-github" 8 | }, 9 | "source": [ 10 | "\"Open" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 0, 16 | "metadata": { 17 | "colab": {}, 18 | "colab_type": "code", 19 | "id": "Wzbzil3EquvD" 20 | }, 21 | "outputs": [], 22 | "source": [ 23 | "import tensorflow as tf\n", 24 | "import numpy as np\n", 25 | "\n", 26 | "tf.enable_eager_execution()\n", 27 | "tfe = tf.contrib.eager\n", 28 | "L = tf.keras.layers" 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": 134, 34 | "metadata": { 35 | "colab": { 36 | "base_uri": "https://localhost:8080/", 37 | "height": 161 38 | }, 39 | "colab_type": "code", 40 | "id": "_b3wl5DyrGbv", 41 | "outputId": "fa8b9ed5-5caf-46bf-8cb6-1941aaa7e67d" 42 | }, 43 | "outputs": [ 44 | { 45 | "name": "stdout", 46 | "output_type": "stream", 47 | "text": [ 48 | "training_data\n", 49 | " (50000, 32, 32, 3)\n", 50 | "test_data\n", 51 | " (10000, 32, 32, 3)\n", 52 | "training_label\n", 53 | " (50000, 1)\n", 54 | "test_label\n", 55 | " (10000, 1)\n" 56 | ] 57 | } 58 | ], 59 | "source": [ 60 | "# Hyper parameters\n", 61 | "num_epochs = 10\n", 62 | "num_classes = 10\n", 63 | "batch_size = 32\n", 64 | "learning_rate = 0.001\n", 65 | "\n", 66 | "(x_train, y_train), (x_test, y_test) = tfk.datasets.cifar10.load_data()\n", 67 | "\n", 68 | "print(\"training_data\\n\", x_train.shape)\n", 69 | "print(\"test_data\\n\", x_test.shape)\n", 70 | "print(\"training_label\\n\", y_train.shape)\n", 71 | "print(\"test_label\\n\", y_test.shape)" 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": 135, 77 | "metadata": { 78 | "colab": { 79 | "base_uri": "https://localhost:8080/", 80 | "height": 53 81 | }, 82 | "colab_type": "code", 83 | "id": "zQ9dPufBDY_e", 84 | "outputId": "31cbe65f-3882-46cd-e5c0-c74682b8b065" 85 | }, 86 | "outputs": [ 87 | { 88 | "name": "stdout", 89 | "output_type": "stream", 90 | "text": [ 91 | "(50000, 3, 32, 32)\n", 92 | "(50000, 10)\n" 93 | ] 94 | } 95 | ], 96 | "source": [ 97 | "x_train_ = tf.transpose(tf.convert_to_tensor(x_train, dtype=tf.float32), \n", 98 | " [0, 3, 1, 2])\n", 99 | "y_train_ = tf.reshape(tf.one_hot(y_train, 10), (-1, 10))\n", 100 | "\n", 101 | "\n", 102 | "print(x_train_.shape)\n", 103 | "print(y_train_.shape)" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": 136, 109 | "metadata": { 110 | "colab": { 111 | "base_uri": "https://localhost:8080/", 112 | "height": 35 113 | }, 114 | "colab_type": "code", 115 | "id": "EIFr75qmBuM-", 116 | "outputId": "af256cc5-31f2-4062-a75d-8a088bdbd6b8" 117 | }, 118 | "outputs": [ 119 | { 120 | "name": "stdout", 121 | "output_type": "stream", 122 | "text": [ 123 | "\n" 124 | ] 125 | } 126 | ], 127 | "source": [ 128 | "train_dataset = (\n", 129 | " tf.data.Dataset.from_tensor_slices((x_train, y_train))\n", 130 | " .batch(batch_size)\n", 131 | " .shuffle(10000)\n", 132 | ")\n", 133 | "\n", 134 | "train_dataset = (\n", 135 | " train_dataset.map(lambda x, y: \n", 136 | " (tf.div(tf.cast(\n", 137 | " tf.transpose(x, [0, 3, 1, 2]), tf.float32), 255.0), \n", 138 | " tf.reshape(tf.one_hot(y, 10), (-1, 10))))\n", 139 | ")\n", 140 | "\n", 141 | "train_dataset = train_dataset.repeat()\n", 142 | "print(train_dataset)" 143 | ] 144 | }, 145 | { 146 | "cell_type": "code", 147 | "execution_count": 137, 148 | "metadata": { 149 | "colab": { 150 | "base_uri": "https://localhost:8080/", 151 | "height": 35 152 | }, 153 | "colab_type": "code", 154 | "id": "jNdtaRRB7d5O", 155 | "outputId": "6016ab3e-f921-4c62-abff-842253f3679e" 156 | }, 157 | "outputs": [ 158 | { 159 | "name": "stdout", 160 | "output_type": "stream", 161 | "text": [ 162 | "\n" 163 | ] 164 | } 165 | ], 166 | "source": [ 167 | "test_dataset = (\n", 168 | " tf.data.Dataset.from_tensor_slices((x_test, y_test))\n", 169 | " .batch(1000)\n", 170 | " .shuffle(10000)\n", 171 | ")\n", 172 | "test_dataset = (\n", 173 | " test_dataset.map(lambda x, y: \n", 174 | " (tf.div(tf.cast(\n", 175 | " tf.transpose(x, [0, 3, 1, 2]), tf.float32), 255.0), \n", 176 | " tf.reshape(tf.one_hot(y, 10), (-1, 10))))\n", 177 | ")\n", 178 | "test_dataset = test_dataset.repeat()\n", 179 | "print(test_dataset)" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": 0, 185 | "metadata": { 186 | "colab": {}, 187 | "colab_type": "code", 188 | "id": "P5PNiSQWrfAF" 189 | }, 190 | "outputs": [], 191 | "source": [] 192 | }, 193 | { 194 | "cell_type": "code", 195 | "execution_count": 0, 196 | "metadata": { 197 | "colab": {}, 198 | "colab_type": "code", 199 | "id": "S2mYcv4D2UGO" 200 | }, 201 | "outputs": [], 202 | "source": [ 203 | "class Cifar10Model(tf.keras.Model):\n", 204 | " def __init__(self):\n", 205 | " super(Cifar10Model, self).__init__(name='cifar_cnn')\n", 206 | " \n", 207 | " self.conv_block1 = tf.keras.Sequential([\n", 208 | " L.Conv2D(\n", 209 | " 8, \n", 210 | " 5,\n", 211 | " padding='same',\n", 212 | " activation=tf.nn.relu,\n", 213 | " kernel_initializer=tf.initializers.variance_scaling,\n", 214 | " kernel_regularizer=tf.keras.regularizers.l2(l=0.001),\n", 215 | " data_format=\"channels_first\"\n", 216 | " ),\n", 217 | " L.MaxPooling2D(\n", 218 | " (3, 3), \n", 219 | " (2, 2), \n", 220 | " padding='same',\n", 221 | " data_format=\"channels_first\"\n", 222 | " ),\n", 223 | " L.BatchNormalization(axis=1),\n", 224 | " ])\n", 225 | "\n", 226 | " self.conv_block2 = tf.keras.Sequential([\n", 227 | " L.Conv2D(\n", 228 | " 16, \n", 229 | " 5,\n", 230 | " padding='same',\n", 231 | " activation=tf.nn.relu,\n", 232 | " kernel_initializer=tf.initializers.variance_scaling,\n", 233 | " kernel_regularizer=tf.keras.regularizers.l2(l=0.001),\n", 234 | " data_format=\"channels_first\"\n", 235 | " ),\n", 236 | " L.MaxPooling2D(\n", 237 | " (3, 3), \n", 238 | " (2, 2), \n", 239 | " padding='same',\n", 240 | " data_format=\"channels_first\"\n", 241 | " ),\n", 242 | " L.BatchNormalization(axis=1),\n", 243 | " ])\n", 244 | " \n", 245 | " self.conv_block3 = tf.keras.Sequential([\n", 246 | " L.Conv2D(\n", 247 | " 32, \n", 248 | " 5,\n", 249 | " padding='same',\n", 250 | " activation=tf.nn.relu,\n", 251 | " kernel_initializer=tf.initializers.variance_scaling,\n", 252 | " kernel_regularizer=tf.keras.regularizers.l2(l=0.001),\n", 253 | " data_format=\"channels_first\"\n", 254 | " ),\n", 255 | " L.MaxPooling2D(\n", 256 | " (3, 3), \n", 257 | " (2, 2), \n", 258 | " padding='same',\n", 259 | " data_format=\"channels_first\"\n", 260 | " ),\n", 261 | " L.BatchNormalization(axis=1),\n", 262 | " ])\n", 263 | " \n", 264 | " self.flatten = L.Flatten()\n", 265 | " self.fc1 = L.Dense(\n", 266 | " 256, \n", 267 | " activation=tf.nn.relu,\n", 268 | " kernel_initializer=tf.initializers.variance_scaling,\n", 269 | " kernel_regularizer=tf.keras.regularizers.l2(l=0.001))\n", 270 | " self.dropout = L.Dropout(0.8)\n", 271 | " self.fc2 = L.Dense(10)\n", 272 | " self.softmax = L.Softmax()\n", 273 | "\n", 274 | " def call(self, x):\n", 275 | " x = self.conv_block1(x)\n", 276 | " x = self.conv_block2(x)\n", 277 | " x = self.conv_block3(x)\n", 278 | " x = self.flatten(x)\n", 279 | " x = self.dropout(self.fc1(x))\n", 280 | " x = self.fc2(x)\n", 281 | " return self.softmax(x)" 282 | ] 283 | }, 284 | { 285 | "cell_type": "code", 286 | "execution_count": 140, 287 | "metadata": { 288 | "colab": { 289 | "base_uri": "https://localhost:8080/", 290 | "height": 35 291 | }, 292 | "colab_type": "code", 293 | "id": "6VyzP7OTtDGf", 294 | "outputId": "d9f4d85c-ffe5-42a7-ab20-110876907e38" 295 | }, 296 | "outputs": [ 297 | { 298 | "data": { 299 | "text/plain": [ 300 | "TensorShape([Dimension(100), Dimension(10)])" 301 | ] 302 | }, 303 | "execution_count": 140, 304 | "metadata": { 305 | "tags": [] 306 | }, 307 | "output_type": "execute_result" 308 | } 309 | ], 310 | "source": [ 311 | "model = Cifar10Model()\n", 312 | "model.compile(optimizer=tf.train.AdamOptimizer(learning_rate),\n", 313 | " loss='categorical_crossentropy',\n", 314 | " metrics=['accuracy'])\n", 315 | "y_init = model(x_train_[:100])\n", 316 | "y_init.shape" 317 | ] 318 | }, 319 | { 320 | "cell_type": "code", 321 | "execution_count": 141, 322 | "metadata": { 323 | "colab": { 324 | "base_uri": "https://localhost:8080/", 325 | "height": 431 326 | }, 327 | "colab_type": "code", 328 | "id": "7zrxUvWPu6PK", 329 | "outputId": "80d0b9b4-2b46-4d55-cc90-331367db911e" 330 | }, 331 | "outputs": [ 332 | { 333 | "name": "stdout", 334 | "output_type": "stream", 335 | "text": [ 336 | "_________________________________________________________________\n", 337 | "Layer (type) Output Shape Param # \n", 338 | "=================================================================\n", 339 | "sequential_31 (Sequential) multiple 640 \n", 340 | "_________________________________________________________________\n", 341 | "sequential_32 (Sequential) multiple 3280 \n", 342 | "_________________________________________________________________\n", 343 | "sequential_33 (Sequential) multiple 12960 \n", 344 | "_________________________________________________________________\n", 345 | "flatten_14 (Flatten) multiple 0 \n", 346 | "_________________________________________________________________\n", 347 | "dense_24 (Dense) multiple 131328 \n", 348 | "_________________________________________________________________\n", 349 | "dropout_10 (Dropout) multiple 0 \n", 350 | "_________________________________________________________________\n", 351 | "dense_25 (Dense) multiple 2570 \n", 352 | "_________________________________________________________________\n", 353 | "softmax_10 (Softmax) multiple 0 \n", 354 | "=================================================================\n", 355 | "Total params: 150,778\n", 356 | "Trainable params: 150,666\n", 357 | "Non-trainable params: 112\n", 358 | "_________________________________________________________________\n" 359 | ] 360 | } 361 | ], 362 | "source": [ 363 | "model.summary()" 364 | ] 365 | }, 366 | { 367 | "cell_type": "code", 368 | "execution_count": 142, 369 | "metadata": { 370 | "colab": { 371 | "base_uri": "https://localhost:8080/", 372 | "height": 395 373 | }, 374 | "colab_type": "code", 375 | "id": "8m2XzQv3G-na", 376 | "outputId": "5b822073-7f3c-43f7-e77b-614da749d074" 377 | }, 378 | "outputs": [ 379 | { 380 | "name": "stdout", 381 | "output_type": "stream", 382 | "text": [ 383 | "Epoch 1/10\n", 384 | "1562/1562 [==============================] - 94s 60ms/step - loss: 2.0714 - acc: 0.3507\n", 385 | "Epoch 2/10\n", 386 | "1562/1562 [==============================] - 93s 59ms/step - loss: 1.6550 - acc: 0.4786\n", 387 | "Epoch 3/10\n", 388 | "1562/1562 [==============================] - 99s 64ms/step - loss: 1.4775 - acc: 0.5425\n", 389 | "Epoch 4/10\n", 390 | "1562/1562 [==============================] - 98s 63ms/step - loss: 1.3834 - acc: 0.5780\n", 391 | "Epoch 5/10\n", 392 | "1562/1562 [==============================] - 94s 60ms/step - loss: 1.3222 - acc: 0.6098\n", 393 | "Epoch 6/10\n", 394 | "1562/1562 [==============================] - 94s 60ms/step - loss: 1.2834 - acc: 0.6273\n", 395 | "Epoch 7/10\n", 396 | "1562/1562 [==============================] - 95s 61ms/step - loss: 1.2596 - acc: 0.6392\n", 397 | "Epoch 8/10\n", 398 | "1562/1562 [==============================] - 95s 61ms/step - loss: 1.2430 - acc: 0.6477\n", 399 | "Epoch 9/10\n", 400 | "1562/1562 [==============================] - 92s 59ms/step - loss: 1.2210 - acc: 0.6587\n", 401 | "Epoch 10/10\n", 402 | "1562/1562 [==============================] - 95s 61ms/step - loss: 1.2108 - acc: 0.6642\n" 403 | ] 404 | }, 405 | { 406 | "data": { 407 | "text/plain": [ 408 | "" 409 | ] 410 | }, 411 | "execution_count": 142, 412 | "metadata": { 413 | "tags": [] 414 | }, 415 | "output_type": "execute_result" 416 | } 417 | ], 418 | "source": [ 419 | "steps_per_epoch = int(x_train.shape[0]/batch_size)\n", 420 | "\n", 421 | "model.fit(\n", 422 | " train_dataset,\n", 423 | " batch_size=batch_size,\n", 424 | " epochs=10,\n", 425 | " steps_per_epoch=steps_per_epoch,\n", 426 | " verbose=1,\n", 427 | " shuffle=True,\n", 428 | " workers=5\n", 429 | ")" 430 | ] 431 | }, 432 | { 433 | "cell_type": "code", 434 | "execution_count": 144, 435 | "metadata": { 436 | "colab": { 437 | "base_uri": "https://localhost:8080/", 438 | "height": 89 439 | }, 440 | "colab_type": "code", 441 | "id": "dnuY_yuOLrO-", 442 | "outputId": "ca7a1bae-9fb6-4a00-d567-1efca72a7d93" 443 | }, 444 | "outputs": [ 445 | { 446 | "name": "stdout", 447 | "output_type": "stream", 448 | "text": [ 449 | "1562/1562 [==============================] - 36s 23ms/step\n", 450 | "train_acc: 0.720\n", 451 | "10/10 [==============================] - 1s 136ms/step\n", 452 | "test_acc: 0.688\n" 453 | ] 454 | } 455 | ], 456 | "source": [ 457 | "train_loss, train_acc = model.evaluate(train_dataset, \n", 458 | " steps=int(y_train.shape[0]/batch_size))\n", 459 | "print(\"train_acc: {:0.3f}\".format(train_acc))\n", 460 | "\n", 461 | "test_loss, test_acc = model.evaluate(test_dataset, \n", 462 | " steps=int(y_test.shape[0]/1000))\n", 463 | "print(\"test_acc: {:0.3f}\".format(test_acc))" 464 | ] 465 | }, 466 | { 467 | "cell_type": "code", 468 | "execution_count": 0, 469 | "metadata": { 470 | "colab": {}, 471 | "colab_type": "code", 472 | "id": "Rzz2FNtOqzdC" 473 | }, 474 | "outputs": [], 475 | "source": [] 476 | } 477 | ], 478 | "metadata": { 479 | "accelerator": "GPU", 480 | "colab": { 481 | "collapsed_sections": [], 482 | "include_colab_link": true, 483 | "name": "Convolutinal_Neural_Network_using_KerasAPI", 484 | "provenance": [], 485 | "version": "0.3.2" 486 | }, 487 | "kernelspec": { 488 | "display_name": "Python 3", 489 | "language": "python", 490 | "name": "python3" 491 | }, 492 | "language_info": { 493 | "codemirror_mode": { 494 | "name": "ipython", 495 | "version": 3 496 | }, 497 | "file_extension": ".py", 498 | "mimetype": "text/x-python", 499 | "name": "python", 500 | "nbconvert_exporter": "python", 501 | "pygments_lexer": "ipython3", 502 | "version": "3.5.3" 503 | } 504 | }, 505 | "nbformat": 4, 506 | "nbformat_minor": 1 507 | } 508 | -------------------------------------------------------------------------------- /tutorials/02_intermediate/.ipynb_checkpoints/Convolutinal_Neural_Network_using_KerasAPI-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "colab_type": "text", 7 | "id": "view-in-github" 8 | }, 9 | "source": [ 10 | "\"Open" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 0, 16 | "metadata": { 17 | "colab": {}, 18 | "colab_type": "code", 19 | "id": "Wzbzil3EquvD" 20 | }, 21 | "outputs": [], 22 | "source": [ 23 | "import tensorflow as tf\n", 24 | "import numpy as np\n", 25 | "\n", 26 | "tf.enable_eager_execution()\n", 27 | "tfe = tf.contrib.eager\n", 28 | "L = tf.keras.layers" 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": 134, 34 | "metadata": { 35 | "colab": { 36 | "base_uri": "https://localhost:8080/", 37 | "height": 161 38 | }, 39 | "colab_type": "code", 40 | "id": "_b3wl5DyrGbv", 41 | "outputId": "fa8b9ed5-5caf-46bf-8cb6-1941aaa7e67d" 42 | }, 43 | "outputs": [ 44 | { 45 | "name": "stdout", 46 | "output_type": "stream", 47 | "text": [ 48 | "training_data\n", 49 | " (50000, 32, 32, 3)\n", 50 | "test_data\n", 51 | " (10000, 32, 32, 3)\n", 52 | "training_label\n", 53 | " (50000, 1)\n", 54 | "test_label\n", 55 | " (10000, 1)\n" 56 | ] 57 | } 58 | ], 59 | "source": [ 60 | "# Hyper parameters\n", 61 | "num_epochs = 10\n", 62 | "num_classes = 10\n", 63 | "batch_size = 32\n", 64 | "learning_rate = 0.001\n", 65 | "\n", 66 | "(x_train, y_train), (x_test, y_test) = tfk.datasets.cifar10.load_data()\n", 67 | "\n", 68 | "print(\"training_data\\n\", x_train.shape)\n", 69 | "print(\"test_data\\n\", x_test.shape)\n", 70 | "print(\"training_label\\n\", y_train.shape)\n", 71 | "print(\"test_label\\n\", y_test.shape)" 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": 135, 77 | "metadata": { 78 | "colab": { 79 | "base_uri": "https://localhost:8080/", 80 | "height": 53 81 | }, 82 | "colab_type": "code", 83 | "id": "zQ9dPufBDY_e", 84 | "outputId": "31cbe65f-3882-46cd-e5c0-c74682b8b065" 85 | }, 86 | "outputs": [ 87 | { 88 | "name": "stdout", 89 | "output_type": "stream", 90 | "text": [ 91 | "(50000, 3, 32, 32)\n", 92 | "(50000, 10)\n" 93 | ] 94 | } 95 | ], 96 | "source": [ 97 | "x_train_ = tf.transpose(tf.convert_to_tensor(x_train, dtype=tf.float32), \n", 98 | " [0, 3, 1, 2])\n", 99 | "y_train_ = tf.reshape(tf.one_hot(y_train, 10), (-1, 10))\n", 100 | "\n", 101 | "\n", 102 | "print(x_train_.shape)\n", 103 | "print(y_train_.shape)" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": 136, 109 | "metadata": { 110 | "colab": { 111 | "base_uri": "https://localhost:8080/", 112 | "height": 35 113 | }, 114 | "colab_type": "code", 115 | "id": "EIFr75qmBuM-", 116 | "outputId": "af256cc5-31f2-4062-a75d-8a088bdbd6b8" 117 | }, 118 | "outputs": [ 119 | { 120 | "name": "stdout", 121 | "output_type": "stream", 122 | "text": [ 123 | "\n" 124 | ] 125 | } 126 | ], 127 | "source": [ 128 | "train_dataset = (\n", 129 | " tf.data.Dataset.from_tensor_slices((x_train, y_train))\n", 130 | " .batch(batch_size)\n", 131 | " .shuffle(10000)\n", 132 | ")\n", 133 | "\n", 134 | "train_dataset = (\n", 135 | " train_dataset.map(lambda x, y: \n", 136 | " (tf.div(tf.cast(\n", 137 | " tf.transpose(x, [0, 3, 1, 2]), tf.float32), 255.0), \n", 138 | " tf.reshape(tf.one_hot(y, 10), (-1, 10))))\n", 139 | ")\n", 140 | "\n", 141 | "train_dataset = train_dataset.repeat()\n", 142 | "print(train_dataset)" 143 | ] 144 | }, 145 | { 146 | "cell_type": "code", 147 | "execution_count": 137, 148 | "metadata": { 149 | "colab": { 150 | "base_uri": "https://localhost:8080/", 151 | "height": 35 152 | }, 153 | "colab_type": "code", 154 | "id": "jNdtaRRB7d5O", 155 | "outputId": "6016ab3e-f921-4c62-abff-842253f3679e" 156 | }, 157 | "outputs": [ 158 | { 159 | "name": "stdout", 160 | "output_type": "stream", 161 | "text": [ 162 | "\n" 163 | ] 164 | } 165 | ], 166 | "source": [ 167 | "test_dataset = (\n", 168 | " tf.data.Dataset.from_tensor_slices((x_test, y_test))\n", 169 | " .batch(1000)\n", 170 | " .shuffle(10000)\n", 171 | ")\n", 172 | "test_dataset = (\n", 173 | " test_dataset.map(lambda x, y: \n", 174 | " (tf.div(tf.cast(\n", 175 | " tf.transpose(x, [0, 3, 1, 2]), tf.float32), 255.0), \n", 176 | " tf.reshape(tf.one_hot(y, 10), (-1, 10))))\n", 177 | ")\n", 178 | "test_dataset = test_dataset.repeat()\n", 179 | "print(test_dataset)" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": 0, 185 | "metadata": { 186 | "colab": {}, 187 | "colab_type": "code", 188 | "id": "P5PNiSQWrfAF" 189 | }, 190 | "outputs": [], 191 | "source": [] 192 | }, 193 | { 194 | "cell_type": "code", 195 | "execution_count": 0, 196 | "metadata": { 197 | "colab": {}, 198 | "colab_type": "code", 199 | "id": "S2mYcv4D2UGO" 200 | }, 201 | "outputs": [], 202 | "source": [ 203 | "class Cifar10Model(tf.keras.Model):\n", 204 | " def __init__(self):\n", 205 | " super(Cifar10Model, self).__init__(name='cifar_cnn')\n", 206 | " \n", 207 | " self.conv_block1 = tf.keras.Sequential([\n", 208 | " L.Conv2D(\n", 209 | " 8, \n", 210 | " 5,\n", 211 | " padding='same',\n", 212 | " activation=tf.nn.relu,\n", 213 | " kernel_initializer=tf.initializers.variance_scaling,\n", 214 | " kernel_regularizer=tf.keras.regularizers.l2(l=0.001),\n", 215 | " data_format=\"channels_first\"\n", 216 | " ),\n", 217 | " L.MaxPooling2D(\n", 218 | " (3, 3), \n", 219 | " (2, 2), \n", 220 | " padding='same',\n", 221 | " data_format=\"channels_first\"\n", 222 | " ),\n", 223 | " L.BatchNormalization(axis=1),\n", 224 | " ])\n", 225 | "\n", 226 | " self.conv_block2 = tf.keras.Sequential([\n", 227 | " L.Conv2D(\n", 228 | " 16, \n", 229 | " 5,\n", 230 | " padding='same',\n", 231 | " activation=tf.nn.relu,\n", 232 | " kernel_initializer=tf.initializers.variance_scaling,\n", 233 | " kernel_regularizer=tf.keras.regularizers.l2(l=0.001),\n", 234 | " data_format=\"channels_first\"\n", 235 | " ),\n", 236 | " L.MaxPooling2D(\n", 237 | " (3, 3), \n", 238 | " (2, 2), \n", 239 | " padding='same',\n", 240 | " data_format=\"channels_first\"\n", 241 | " ),\n", 242 | " L.BatchNormalization(axis=1),\n", 243 | " ])\n", 244 | " \n", 245 | " self.conv_block3 = tf.keras.Sequential([\n", 246 | " L.Conv2D(\n", 247 | " 32, \n", 248 | " 5,\n", 249 | " padding='same',\n", 250 | " activation=tf.nn.relu,\n", 251 | " kernel_initializer=tf.initializers.variance_scaling,\n", 252 | " kernel_regularizer=tf.keras.regularizers.l2(l=0.001),\n", 253 | " data_format=\"channels_first\"\n", 254 | " ),\n", 255 | " L.MaxPooling2D(\n", 256 | " (3, 3), \n", 257 | " (2, 2), \n", 258 | " padding='same',\n", 259 | " data_format=\"channels_first\"\n", 260 | " ),\n", 261 | " L.BatchNormalization(axis=1),\n", 262 | " ])\n", 263 | " \n", 264 | " self.flatten = L.Flatten()\n", 265 | " self.fc1 = L.Dense(\n", 266 | " 256, \n", 267 | " activation=tf.nn.relu,\n", 268 | " kernel_initializer=tf.initializers.variance_scaling,\n", 269 | " kernel_regularizer=tf.keras.regularizers.l2(l=0.001))\n", 270 | " self.dropout = L.Dropout(0.8)\n", 271 | " self.fc2 = L.Dense(10)\n", 272 | " self.softmax = L.Softmax()\n", 273 | "\n", 274 | " def call(self, x):\n", 275 | " x = self.conv_block1(x)\n", 276 | " x = self.conv_block2(x)\n", 277 | " x = self.conv_block3(x)\n", 278 | " x = self.flatten(x)\n", 279 | " x = self.dropout(self.fc1(x))\n", 280 | " x = self.fc2(x)\n", 281 | " return self.softmax(x)" 282 | ] 283 | }, 284 | { 285 | "cell_type": "code", 286 | "execution_count": 140, 287 | "metadata": { 288 | "colab": { 289 | "base_uri": "https://localhost:8080/", 290 | "height": 35 291 | }, 292 | "colab_type": "code", 293 | "id": "6VyzP7OTtDGf", 294 | "outputId": "d9f4d85c-ffe5-42a7-ab20-110876907e38" 295 | }, 296 | "outputs": [ 297 | { 298 | "data": { 299 | "text/plain": [ 300 | "TensorShape([Dimension(100), Dimension(10)])" 301 | ] 302 | }, 303 | "execution_count": 140, 304 | "metadata": { 305 | "tags": [] 306 | }, 307 | "output_type": "execute_result" 308 | } 309 | ], 310 | "source": [ 311 | "model = Cifar10Model()\n", 312 | "model.compile(optimizer=tf.train.AdamOptimizer(learning_rate),\n", 313 | " loss='categorical_crossentropy',\n", 314 | " metrics=['accuracy'])\n", 315 | "y_init = model(x_train_[:100])\n", 316 | "y_init.shape" 317 | ] 318 | }, 319 | { 320 | "cell_type": "code", 321 | "execution_count": 141, 322 | "metadata": { 323 | "colab": { 324 | "base_uri": "https://localhost:8080/", 325 | "height": 431 326 | }, 327 | "colab_type": "code", 328 | "id": "7zrxUvWPu6PK", 329 | "outputId": "80d0b9b4-2b46-4d55-cc90-331367db911e" 330 | }, 331 | "outputs": [ 332 | { 333 | "name": "stdout", 334 | "output_type": "stream", 335 | "text": [ 336 | "_________________________________________________________________\n", 337 | "Layer (type) Output Shape Param # \n", 338 | "=================================================================\n", 339 | "sequential_31 (Sequential) multiple 640 \n", 340 | "_________________________________________________________________\n", 341 | "sequential_32 (Sequential) multiple 3280 \n", 342 | "_________________________________________________________________\n", 343 | "sequential_33 (Sequential) multiple 12960 \n", 344 | "_________________________________________________________________\n", 345 | "flatten_14 (Flatten) multiple 0 \n", 346 | "_________________________________________________________________\n", 347 | "dense_24 (Dense) multiple 131328 \n", 348 | "_________________________________________________________________\n", 349 | "dropout_10 (Dropout) multiple 0 \n", 350 | "_________________________________________________________________\n", 351 | "dense_25 (Dense) multiple 2570 \n", 352 | "_________________________________________________________________\n", 353 | "softmax_10 (Softmax) multiple 0 \n", 354 | "=================================================================\n", 355 | "Total params: 150,778\n", 356 | "Trainable params: 150,666\n", 357 | "Non-trainable params: 112\n", 358 | "_________________________________________________________________\n" 359 | ] 360 | } 361 | ], 362 | "source": [ 363 | "model.summary()" 364 | ] 365 | }, 366 | { 367 | "cell_type": "code", 368 | "execution_count": 142, 369 | "metadata": { 370 | "colab": { 371 | "base_uri": "https://localhost:8080/", 372 | "height": 395 373 | }, 374 | "colab_type": "code", 375 | "id": "8m2XzQv3G-na", 376 | "outputId": "5b822073-7f3c-43f7-e77b-614da749d074" 377 | }, 378 | "outputs": [ 379 | { 380 | "name": "stdout", 381 | "output_type": "stream", 382 | "text": [ 383 | "Epoch 1/10\n", 384 | "1562/1562 [==============================] - 94s 60ms/step - loss: 2.0714 - acc: 0.3507\n", 385 | "Epoch 2/10\n", 386 | "1562/1562 [==============================] - 93s 59ms/step - loss: 1.6550 - acc: 0.4786\n", 387 | "Epoch 3/10\n", 388 | "1562/1562 [==============================] - 99s 64ms/step - loss: 1.4775 - acc: 0.5425\n", 389 | "Epoch 4/10\n", 390 | "1562/1562 [==============================] - 98s 63ms/step - loss: 1.3834 - acc: 0.5780\n", 391 | "Epoch 5/10\n", 392 | "1562/1562 [==============================] - 94s 60ms/step - loss: 1.3222 - acc: 0.6098\n", 393 | "Epoch 6/10\n", 394 | "1562/1562 [==============================] - 94s 60ms/step - loss: 1.2834 - acc: 0.6273\n", 395 | "Epoch 7/10\n", 396 | "1562/1562 [==============================] - 95s 61ms/step - loss: 1.2596 - acc: 0.6392\n", 397 | "Epoch 8/10\n", 398 | "1562/1562 [==============================] - 95s 61ms/step - loss: 1.2430 - acc: 0.6477\n", 399 | "Epoch 9/10\n", 400 | "1562/1562 [==============================] - 92s 59ms/step - loss: 1.2210 - acc: 0.6587\n", 401 | "Epoch 10/10\n", 402 | "1562/1562 [==============================] - 95s 61ms/step - loss: 1.2108 - acc: 0.6642\n" 403 | ] 404 | }, 405 | { 406 | "data": { 407 | "text/plain": [ 408 | "" 409 | ] 410 | }, 411 | "execution_count": 142, 412 | "metadata": { 413 | "tags": [] 414 | }, 415 | "output_type": "execute_result" 416 | } 417 | ], 418 | "source": [ 419 | "steps_per_epoch = int(x_train.shape[0]/batch_size)\n", 420 | "\n", 421 | "model.fit(\n", 422 | " train_dataset,\n", 423 | " batch_size=batch_size,\n", 424 | " epochs=10,\n", 425 | " steps_per_epoch=steps_per_epoch,\n", 426 | " verbose=1,\n", 427 | " shuffle=True,\n", 428 | " workers=5\n", 429 | ")" 430 | ] 431 | }, 432 | { 433 | "cell_type": "code", 434 | "execution_count": 144, 435 | "metadata": { 436 | "colab": { 437 | "base_uri": "https://localhost:8080/", 438 | "height": 89 439 | }, 440 | "colab_type": "code", 441 | "id": "dnuY_yuOLrO-", 442 | "outputId": "ca7a1bae-9fb6-4a00-d567-1efca72a7d93" 443 | }, 444 | "outputs": [ 445 | { 446 | "name": "stdout", 447 | "output_type": "stream", 448 | "text": [ 449 | "1562/1562 [==============================] - 36s 23ms/step\n", 450 | "train_acc: 0.720\n", 451 | "10/10 [==============================] - 1s 136ms/step\n", 452 | "test_acc: 0.688\n" 453 | ] 454 | } 455 | ], 456 | "source": [ 457 | "train_loss, train_acc = model.evaluate(train_dataset, \n", 458 | " steps=int(y_train.shape[0]/batch_size))\n", 459 | "print(\"train_acc: {:0.3f}\".format(train_acc))\n", 460 | "\n", 461 | "test_loss, test_acc = model.evaluate(test_dataset, \n", 462 | " steps=int(y_test.shape[0]/1000))\n", 463 | "print(\"test_acc: {:0.3f}\".format(test_acc))" 464 | ] 465 | }, 466 | { 467 | "cell_type": "code", 468 | "execution_count": 0, 469 | "metadata": { 470 | "colab": {}, 471 | "colab_type": "code", 472 | "id": "Rzz2FNtOqzdC" 473 | }, 474 | "outputs": [], 475 | "source": [] 476 | } 477 | ], 478 | "metadata": { 479 | "accelerator": "GPU", 480 | "colab": { 481 | "collapsed_sections": [], 482 | "include_colab_link": true, 483 | "name": "Convolutinal_Neural_Network_using_KerasAPI", 484 | "provenance": [], 485 | "version": "0.3.2" 486 | }, 487 | "kernelspec": { 488 | "display_name": "Python 3", 489 | "language": "python", 490 | "name": "python3" 491 | }, 492 | "language_info": { 493 | "codemirror_mode": { 494 | "name": "ipython", 495 | "version": 3 496 | }, 497 | "file_extension": ".py", 498 | "mimetype": "text/x-python", 499 | "name": "python", 500 | "nbconvert_exporter": "python", 501 | "pygments_lexer": "ipython3", 502 | "version": "3.5.3" 503 | } 504 | }, 505 | "nbformat": 4, 506 | "nbformat_minor": 1 507 | } 508 | -------------------------------------------------------------------------------- /tutorials/02_intermediate/Recurrent_Neural_Network.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "Recurrent_Neural_Network", 7 | "version": "0.3.2", 8 | "provenance": [], 9 | "collapsed_sections": [], 10 | "include_colab_link": true 11 | }, 12 | "kernelspec": { 13 | "display_name": "Python 3", 14 | "language": "python", 15 | "name": "python3" 16 | }, 17 | "accelerator": "GPU" 18 | }, 19 | "cells": [ 20 | { 21 | "cell_type": "markdown", 22 | "metadata": { 23 | "id": "view-in-github", 24 | "colab_type": "text" 25 | }, 26 | "source": [ 27 | "\"Open" 28 | ] 29 | }, 30 | { 31 | "metadata": { 32 | "colab_type": "code", 33 | "id": "VkJooysd9saV", 34 | "colab": {} 35 | }, 36 | "cell_type": "code", 37 | "source": [ 38 | "import numpy as np\n", 39 | "import tensorflow as tf\n", 40 | "import matplotlib.pyplot as plt\n", 41 | "%matplotlib inline\n", 42 | "\n", 43 | "tf.enable_eager_execution()\n", 44 | "L = tf.keras.layers\n", 45 | "tfe = tf.contrib.eager" 46 | ], 47 | "execution_count": 0, 48 | "outputs": [] 49 | }, 50 | { 51 | "metadata": { 52 | "colab_type": "code", 53 | "id": "mvWroGCq-AGM", 54 | "outputId": "98fd627d-826f-45aa-98e1-254abddb00f0", 55 | "colab": { 56 | "base_uri": "https://localhost:8080/", 57 | "height": 125 58 | } 59 | }, 60 | "cell_type": "code", 61 | "source": [ 62 | "# Hyper parameters\n", 63 | "num_epochs = 25\n", 64 | "num_classes = 10\n", 65 | "batch_size = 512\n", 66 | "learning_rate = 0.001\n", 67 | "\n", 68 | "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n", 69 | "\n", 70 | "print(\"training_data: \", x_train.shape)\n", 71 | "print(\"test_data: \", x_test.shape)\n", 72 | "print(\"training_label: \", y_train.shape)\n", 73 | "print(\"test_label: \", y_test.shape)" 74 | ], 75 | "execution_count": 3, 76 | "outputs": [ 77 | { 78 | "output_type": "stream", 79 | "text": [ 80 | "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n", 81 | "11493376/11490434 [==============================] - 0s 0us/step\n", 82 | "training_data: (60000, 28, 28)\n", 83 | "test_data: (10000, 28, 28)\n", 84 | "training_label: (60000,)\n", 85 | "test_label: (10000,)\n" 86 | ], 87 | "name": "stdout" 88 | } 89 | ] 90 | }, 91 | { 92 | "metadata": { 93 | "colab_type": "code", 94 | "id": "int-eal1-RBm", 95 | "outputId": "676f6ce0-fc7a-471a-ec1e-eb9246e883c2", 96 | "colab": { 97 | "base_uri": "https://localhost:8080/", 98 | "height": 89 99 | } 100 | }, 101 | "cell_type": "code", 102 | "source": [ 103 | "x_train_eager = tf.convert_to_tensor(x_train, dtype=tf.float32)\n", 104 | "x_test_eager = tf.convert_to_tensor(x_test, dtype=tf.float32)\n", 105 | "y_train_eager = tf.reshape(tf.one_hot(y_train, 10), (-1, 10))\n", 106 | "y_test_eager = tf.reshape(tf.one_hot(y_test, 10), (-1, 10))\n", 107 | "\n", 108 | "print(\"training_data: \", x_train_eager.shape)\n", 109 | "print(\"test_data: \", x_test_eager.shape)\n", 110 | "print(\"training_label: \", y_train_eager.shape)\n", 111 | "print(\"test_label: \", y_test_eager.shape)" 112 | ], 113 | "execution_count": 4, 114 | "outputs": [ 115 | { 116 | "output_type": "stream", 117 | "text": [ 118 | "training_data: (60000, 28, 28)\n", 119 | "test_data: (10000, 28, 28)\n", 120 | "training_label: (60000, 10)\n", 121 | "test_label: (10000, 10)\n" 122 | ], 123 | "name": "stdout" 124 | } 125 | ] 126 | }, 127 | { 128 | "metadata": { 129 | "colab_type": "text", 130 | "id": "jwcKjSDNGD5W" 131 | }, 132 | "cell_type": "markdown", 133 | "source": [ 134 | "### DataSet\n", 135 | "You make Dataset using `tf.data.Dataset` Class but Keras API doesn't need this dataset. If you write training loop code manually, `Dataset` class is very useful. And using keras API, you need numpy.array inputs instead of tf.Tensor. I don't know why...so you only need numpy preprocessing (or get numpy.array from tf.Tensor using numpy() method after preprocessing using function of tf).\n", 136 | "\n", 137 | "### NOTE\n", 138 | "This notebook we don't need 'tf.data.Dataset'. This code only just for reference." 139 | ] 140 | }, 141 | { 142 | "metadata": { 143 | "colab_type": "code", 144 | "id": "YNU_cq4L-u10", 145 | "colab": {} 146 | }, 147 | "cell_type": "code", 148 | "source": [ 149 | "train_dataset = (\n", 150 | " tf.data.Dataset.from_tensor_slices((x_train_eager, y_train_eager))\n", 151 | " .batch(batch_size)\n", 152 | " .shuffle(10000)\n", 153 | ")\n", 154 | "train_dataset = train_dataset.repeat()" 155 | ], 156 | "execution_count": 0, 157 | "outputs": [] 158 | }, 159 | { 160 | "metadata": { 161 | "colab_type": "code", 162 | "id": "_npuvGwv-588", 163 | "colab": {} 164 | }, 165 | "cell_type": "code", 166 | "source": [ 167 | "test_dataset = (\n", 168 | " tf.data.Dataset.from_tensor_slices((x_test_eager, y_test_eager))\n", 169 | " .batch(1000)\n", 170 | " .shuffle(10000)\n", 171 | ")\n", 172 | "test_dataset = test_dataset.repeat()" 173 | ], 174 | "execution_count": 0, 175 | "outputs": [] 176 | }, 177 | { 178 | "metadata": { 179 | "colab_type": "text", 180 | "id": "e-Z06hTWOx4P" 181 | }, 182 | "cell_type": "markdown", 183 | "source": [ 184 | "### RNN using LSTM\n", 185 | "In keras API, LSTM recives inputs tensor whose shape is (batch_size, seqence_length, feature_dim), and output tensor whose shape is (batch_size, fearure_dim).When you need all time sequence data, you have to give `return_sequences=True` to LSTM's constractor. Generally, when you stack LSTM's, you need all sequence data." 186 | ] 187 | }, 188 | { 189 | "metadata": { 190 | "colab_type": "code", 191 | "id": "gz5RrFnm_HzM", 192 | "colab": {} 193 | }, 194 | "cell_type": "code", 195 | "source": [ 196 | "class RNN(tf.keras.Model):\n", 197 | " def __init__(self, hidden_size=10, num_layers=2, num_classes=10):\n", 198 | " super(RNN, self).__init__(name='mnist_rnn')\n", 199 | " self.hidden_size = hidden_size\n", 200 | " self.num_layers = num_layers\n", 201 | " \n", 202 | " self.lstm = self.get_lstm_layers(hidden_size, num_layers) \n", 203 | " self.fc = L.Dense(num_classes, activation=\"softmax\")\n", 204 | " \n", 205 | " @staticmethod\n", 206 | " def get_lstm_layers(hidden_size, num_layers):\n", 207 | " lstm_layers = []\n", 208 | " # we need all sequence data. write return_sequences=True! \n", 209 | " for i in range(num_layers-1):\n", 210 | " lstm_layers.append(\n", 211 | " L.CuDNNLSTM(units=hidden_size, return_sequences=True)\n", 212 | " )\n", 213 | " # the final layer return only final sequence\n", 214 | " # if you need all sequences, you have to write return_sequences=True.\n", 215 | " lstm_layers.append(L.CuDNNLSTM(units=hidden_size))\n", 216 | " return tf.keras.Sequential(lstm_layers)\n", 217 | " \n", 218 | " def call(self, x): \n", 219 | " # Forward propagate LSTM\n", 220 | " out = self.lstm(x)\n", 221 | " out = self.fc(out)\n", 222 | " return out" 223 | ], 224 | "execution_count": 0, 225 | "outputs": [] 226 | }, 227 | { 228 | "metadata": { 229 | "colab_type": "code", 230 | "id": "oMARKqbaBjIN", 231 | "colab": {} 232 | }, 233 | "cell_type": "code", 234 | "source": [ 235 | "model = RNN()" 236 | ], 237 | "execution_count": 0, 238 | "outputs": [] 239 | }, 240 | { 241 | "metadata": { 242 | "colab_type": "code", 243 | "id": "_NPkLp-rBzCp", 244 | "outputId": "d97b975e-0e84-45cd-d9a0-f55eee4e62d4", 245 | "colab": { 246 | "base_uri": "https://localhost:8080/", 247 | "height": 215 248 | } 249 | }, 250 | "cell_type": "code", 251 | "source": [ 252 | "optimizer = tf.train.AdamOptimizer(learning_rate)\n", 253 | "model.compile(optimizer=optimizer,\n", 254 | " loss='categorical_crossentropy',\n", 255 | " metrics=[\"accuracy\"])\n", 256 | "\n", 257 | "# Eager Execution initialize parameters when using model.call()\n", 258 | "model(x_train_eager[:50])\n", 259 | "\n", 260 | "model.summary()" 261 | ], 262 | "execution_count": 9, 263 | "outputs": [ 264 | { 265 | "output_type": "stream", 266 | "text": [ 267 | "_________________________________________________________________\n", 268 | "Layer (type) Output Shape Param # \n", 269 | "=================================================================\n", 270 | "sequential (Sequential) multiple 2480 \n", 271 | "_________________________________________________________________\n", 272 | "dense (Dense) multiple 110 \n", 273 | "=================================================================\n", 274 | "Total params: 2,590\n", 275 | "Trainable params: 2,590\n", 276 | "Non-trainable params: 0\n", 277 | "_________________________________________________________________\n" 278 | ], 279 | "name": "stdout" 280 | } 281 | ] 282 | }, 283 | { 284 | "metadata": { 285 | "colab_type": "code", 286 | "id": "wDEyoIMnDKEy", 287 | "outputId": "384d38ce-8cde-474a-e4b8-e4c2b24a68ef", 288 | "colab": { 289 | "base_uri": "https://localhost:8080/", 290 | "height": 935 291 | } 292 | }, 293 | "cell_type": "code", 294 | "source": [ 295 | "model.fit(x=x_train_eager.numpy(), \n", 296 | " y=y_train_eager.numpy(), \n", 297 | " validation_split=0.2, \n", 298 | " epochs=num_epochs,\n", 299 | " batch_size=batch_size)" 300 | ], 301 | "execution_count": 10, 302 | "outputs": [ 303 | { 304 | "output_type": "stream", 305 | "text": [ 306 | "Epoch 1/25\n", 307 | "94/94 [==============================] - 5s 50ms/step - loss: 2.1497 - acc: 0.2485 - val_loss: 1.8979 - val_acc: 0.3624\n", 308 | "Epoch 2/25\n", 309 | "94/94 [==============================] - 5s 49ms/step - loss: 1.6330 - acc: 0.4648 - val_loss: 1.3772 - val_acc: 0.5950\n", 310 | "Epoch 3/25\n", 311 | "94/94 [==============================] - 5s 48ms/step - loss: 1.2279 - acc: 0.6295 - val_loss: 1.0835 - val_acc: 0.6795\n", 312 | "Epoch 4/25\n", 313 | "94/94 [==============================] - 4s 47ms/step - loss: 1.0226 - acc: 0.6914 - val_loss: 0.9252 - val_acc: 0.7232\n", 314 | "Epoch 5/25\n", 315 | "94/94 [==============================] - 4s 47ms/step - loss: 0.8952 - acc: 0.7288 - val_loss: 0.8316 - val_acc: 0.7501\n", 316 | "Epoch 6/25\n", 317 | "94/94 [==============================] - 4s 46ms/step - loss: 0.8089 - acc: 0.7559 - val_loss: 0.7370 - val_acc: 0.7786\n", 318 | "Epoch 7/25\n", 319 | "94/94 [==============================] - 4s 47ms/step - loss: 0.7348 - acc: 0.7785 - val_loss: 0.6799 - val_acc: 0.7974\n", 320 | "Epoch 8/25\n", 321 | "94/94 [==============================] - 4s 47ms/step - loss: 0.6806 - acc: 0.7951 - val_loss: 0.6307 - val_acc: 0.8109\n", 322 | "Epoch 9/25\n", 323 | "94/94 [==============================] - 4s 47ms/step - loss: 0.6391 - acc: 0.8079 - val_loss: 0.5920 - val_acc: 0.8205\n", 324 | "Epoch 10/25\n", 325 | "94/94 [==============================] - 4s 47ms/step - loss: 0.6045 - acc: 0.8187 - val_loss: 0.5606 - val_acc: 0.8320\n", 326 | "Epoch 11/25\n", 327 | "94/94 [==============================] - 4s 47ms/step - loss: 0.5754 - acc: 0.8267 - val_loss: 0.5413 - val_acc: 0.8360\n", 328 | "Epoch 12/25\n", 329 | "94/94 [==============================] - 4s 47ms/step - loss: 0.5520 - acc: 0.8333 - val_loss: 0.5160 - val_acc: 0.8440\n", 330 | "Epoch 13/25\n", 331 | "94/94 [==============================] - 4s 47ms/step - loss: 0.5264 - acc: 0.8415 - val_loss: 0.4991 - val_acc: 0.8471\n", 332 | "Epoch 14/25\n", 333 | "94/94 [==============================] - 4s 47ms/step - loss: 0.5066 - acc: 0.8462 - val_loss: 0.4780 - val_acc: 0.8513\n", 334 | "Epoch 15/25\n", 335 | "94/94 [==============================] - 4s 47ms/step - loss: 0.4885 - acc: 0.8536 - val_loss: 0.4647 - val_acc: 0.8592\n", 336 | "Epoch 16/25\n", 337 | "94/94 [==============================] - 4s 47ms/step - loss: 0.4745 - acc: 0.8559 - val_loss: 0.4501 - val_acc: 0.8623\n", 338 | "Epoch 17/25\n", 339 | "94/94 [==============================] - 4s 47ms/step - loss: 0.4613 - acc: 0.8615 - val_loss: 0.4345 - val_acc: 0.8691\n", 340 | "Epoch 18/25\n", 341 | "94/94 [==============================] - 4s 47ms/step - loss: 0.4442 - acc: 0.8650 - val_loss: 0.4273 - val_acc: 0.8692\n", 342 | "Epoch 19/25\n", 343 | "94/94 [==============================] - 4s 47ms/step - loss: 0.4336 - acc: 0.8676 - val_loss: 0.4206 - val_acc: 0.8737\n", 344 | "Epoch 20/25\n", 345 | "94/94 [==============================] - 4s 46ms/step - loss: 0.4243 - acc: 0.8717 - val_loss: 0.4075 - val_acc: 0.8764\n", 346 | "Epoch 21/25\n", 347 | "94/94 [==============================] - 4s 46ms/step - loss: 0.4118 - acc: 0.8765 - val_loss: 0.3999 - val_acc: 0.8792\n", 348 | "Epoch 22/25\n", 349 | "94/94 [==============================] - 4s 46ms/step - loss: 0.4061 - acc: 0.8778 - val_loss: 0.3906 - val_acc: 0.8827\n", 350 | "Epoch 23/25\n", 351 | "94/94 [==============================] - 4s 47ms/step - loss: 0.3961 - acc: 0.8803 - val_loss: 0.3857 - val_acc: 0.8859\n", 352 | "Epoch 24/25\n", 353 | "94/94 [==============================] - 4s 47ms/step - loss: 0.3859 - acc: 0.8835 - val_loss: 0.3763 - val_acc: 0.8860\n", 354 | "Epoch 25/25\n", 355 | "94/94 [==============================] - 4s 47ms/step - loss: 0.3771 - acc: 0.8864 - val_loss: 0.3725 - val_acc: 0.8889\n" 356 | ], 357 | "name": "stdout" 358 | }, 359 | { 360 | "output_type": "execute_result", 361 | "data": { 362 | "text/plain": [ 363 | "" 364 | ] 365 | }, 366 | "metadata": { 367 | "tags": [] 368 | }, 369 | "execution_count": 10 370 | } 371 | ] 372 | }, 373 | { 374 | "metadata": { 375 | "colab_type": "code", 376 | "id": "oDuYrgFIG850", 377 | "outputId": "9000e274-233a-4aaa-8937-7df672519f9f", 378 | "colab": { 379 | "base_uri": "https://localhost:8080/", 380 | "height": 53 381 | } 382 | }, 383 | "cell_type": "code", 384 | "source": [ 385 | "test_loss, test_acc = model.evaluate(x=x_test_eager.numpy(), \n", 386 | " y=y_test_eager.numpy())\n", 387 | "\n", 388 | "print(\"test_accracy: \", test_acc)" 389 | ], 390 | "execution_count": 11, 391 | "outputs": [ 392 | { 393 | "output_type": "stream", 394 | "text": [ 395 | "313/313 [==============================] - 6s 18ms/step\n", 396 | "test_accracy: 0.8839\n" 397 | ], 398 | "name": "stdout" 399 | } 400 | ] 401 | }, 402 | { 403 | "metadata": { 404 | "colab_type": "code", 405 | "id": "Yeayjx_nKXaW", 406 | "colab": {} 407 | }, 408 | "cell_type": "code", 409 | "source": [ 410 | "" 411 | ], 412 | "execution_count": 0, 413 | "outputs": [] 414 | } 415 | ] 416 | } --------------------------------------------------------------------------------