├── .gitignore ├── Chapter03 ├── 1. 图，张量及会话.ipynb ├── 2. 三层简单神经网络的前向传播算法.ipynb └── 3. 完整神经网络样例程序.ipynb ├── Chapter04 ├── 1. 自定义损失函数.ipynb ├── 2. 学习率的设置.ipynb ├── 3. 正则化.ipynb └── 4. 滑动平均模型.ipynb ├── Chapter05 ├── 1. MNIST读取数据.ipynb ├── 2. TensorFlow训练神经网络 │ ├── 1. 全模型.ipynb │ ├── 2. 不使用正则化.ipynb │ ├── 3. 不使用指数衰减的学习率.ipynb │ ├── 4. 不使用激活函数.ipynb │ ├── 5. 不使用隐藏层.ipynb │ └── 6. 不使用滑动平均.ipynb ├── 3. 变量管理.ipynb ├── 4.1. ckpt文件保存方法.ipynb ├── 4.2.1 滑动平均类的保存.ipynb ├── 4.2.2 variables_to_restore函数的使用样例.ipynb ├── 4.3. pb文件保存方法.ipynb ├── 4.4. 恢复变量以及模型.ipynb ├── 5. MNIST最佳实践 │ ├── mnist_eval.ipynb │ ├── mnist_inference.ipynb │ ├── mnist_inference.py │ ├── mnist_train.ipynb │ └── mnist_train.py └── Saved_Graph │ └── events.out.tfevents.1505460689.cookeemMac.local ├── Chapter06 ├── 1. 卷积层、池化层样例.ipynb ├── 1.1. 卷积和池化基础.ipynb ├── 1.2. MNIST卷积和池化.ipynb ├── 2. 迁移学习.ipynb └── LeNet-5 │ ├── LeNet5_infernece.ipynb │ ├── LeNet5_infernece.py │ └── LeNet5_train.ipynb ├── Chapter07 ├── 1. TFRecord样例程序.ipynb ├── 2.1. TensorFlow图像处理函数.ipynb ├── 2.2. 图像预处理完整样例.ipynb ├── 3. 队列操作.ipynb ├── 4. 多线程队列操作.ipynb ├── 5. 输入文件队列.ipynb ├── 6. 输入数据处理框架.ipynb └── Records │ └── README ├── Chapter08 ├── 1. 循环神经网络前向传播.ipynb ├── 2. PTB数据集介绍.ipynb ├── 3. 使用循环神经网络实现语言模型.ipynb ├── 4. SKlearn封装例子.ipynb ├── 5. 预测正弦函数.ipynb ├── Models │ └── README └── reader.py ├── Chapter09 ├── 1. 命名空间.ipynb ├── 2. 改造后的mnist_train.ipynb ├── 3. 监控指标可视化.ipynb └── mnist_inference.py ├── Chapter10 ├── 1. GPU基本操作.py ├── 2. 多GPU并行.py ├── 3. 分布式TensorFlow.py ├── 4. 异步更新模式样例程序.py ├── 5. 同步更新模式样例程序.py └── mnist_inference.py ├── Python技巧（01.数据结构和算法）.ipynb ├── Python技巧（02.字符串和文本）.ipynb ├── Python技巧（03.数字日期和时间）.ipynb ├── Python技巧（04.迭代器与生成器）.ipynb ├── README.md ├── Scala试题格式转换.ipynb └── images └── cover.JPEG /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | .idea/ 3 | datasets/ 4 | Saved_model/ 5 | .ipynb_checkpoints 6 | -------------------------------------------------------------------------------- /Chapter03/1. 图，张量及会话.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "#### 1. 定义两个不同的图" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": { 14 | "code_folding": [] 15 | }, 16 | "outputs": [ 17 | { 18 | "name": "stdout", 19 | "output_type": "stream", 20 | "text": [ 21 | "[ 0.]\n", 22 | "[ 1.]\n" 23 | ] 24 | } 25 | ], 26 | "source": [ 27 | "import tensorflow as tf\n", 28 | "\n", 29 | "g1 = tf.Graph()\n", 30 | "with g1.as_default():\n", 31 | " v = tf.get_variable(\"v\", [1], initializer = tf.zeros_initializer()) # 设置初始值为0\n", 32 | "\n", 33 | "g2 = tf.Graph()\n", 34 | "with g2.as_default():\n", 35 | " v = tf.get_variable(\"v\", [1], initializer = tf.ones_initializer()) # 设置初始值为1\n", 36 | " \n", 37 | "with tf.Session(graph = g1) as sess:\n", 38 | " tf.global_variables_initializer().run()\n", 39 | " with tf.variable_scope(\"\", reuse=True):\n", 40 | " print(sess.run(tf.get_variable(\"v\")))\n", 41 | "\n", 42 | "with tf.Session(graph = g2) as sess:\n", 43 | " tf.global_variables_initializer().run()\n", 44 | " with tf.variable_scope(\"\", reuse=True):\n", 45 | " print(sess.run(tf.get_variable(\"v\")))\n", 46 | " " 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": 2, 52 | "metadata": {}, 53 | "outputs": [ 54 | { 55 | "name": "stdout", 56 | "output_type": "stream", 57 | "text": [ 58 | "[ 1. 2.]\n" 59 | ] 60 | } 61 | ], 62 | "source": [ 63 | "sess = tf.InteractiveSession() \n", 64 | "v = tf.Variable([1,2], dtype=tf.float32)\n", 65 | "sess.run(tf.global_variables_initializer())\n", 66 | "print(v.eval())\n", 67 | "sess.close()" 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "metadata": { 73 | "collapsed": true 74 | }, 75 | "source": [ 76 | "#### 2. 张量的概念" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": 3, 82 | "metadata": {}, 83 | "outputs": [ 84 | { 85 | "name": "stdout", 86 | "output_type": "stream", 87 | "text": [ 88 | "result: Tensor(\"add:0\", shape=(2,), dtype=float32)\n", 89 | "result.eval() [ 3. 5.]\n" 90 | ] 91 | } 92 | ], 93 | "source": [ 94 | "import tensorflow as tf\n", 95 | "a = tf.constant([1.0, 2.0], name=\"a\")\n", 96 | "b = tf.constant([2.0, 3.0], name=\"b\")\n", 97 | "result = a + b\n", 98 | "print(\"result:\", result)\n", 99 | "\n", 100 | "sess = tf.InteractiveSession ()\n", 101 | "print(\"result.eval()\", result.eval())\n", 102 | "sess.close()" 103 | ] 104 | }, 105 | { 106 | "cell_type": "markdown", 107 | "metadata": {}, 108 | "source": [ 109 | "#### 3. 会话的使用" 110 | ] 111 | }, 112 | { 113 | "cell_type": "markdown", 114 | "metadata": {}, 115 | "source": [ 116 | "3.1 创建和关闭会话" 117 | ] 118 | }, 119 | { 120 | "cell_type": "code", 121 | "execution_count": 4, 122 | "metadata": {}, 123 | "outputs": [ 124 | { 125 | "name": "stdout", 126 | "output_type": "stream", 127 | "text": [ 128 | "[ 3. 5.]\n" 129 | ] 130 | } 131 | ], 132 | "source": [ 133 | "# 创建一个会话。\n", 134 | "sess = tf.Session()\n", 135 | "\n", 136 | "# 使用会话得到之前计算的结果。\n", 137 | "print(sess.run(result))\n", 138 | "\n", 139 | "# 关闭会话使得本次运行中使用到的资源可以被释放。\n", 140 | "sess.close()" 141 | ] 142 | }, 143 | { 144 | "cell_type": "markdown", 145 | "metadata": {}, 146 | "source": [ 147 | "3.2 使用with statement 来创建会话，当程序段结束的时候session自动关闭，无需使用sess.close()" 148 | ] 149 | }, 150 | { 151 | "cell_type": "code", 152 | "execution_count": 5, 153 | "metadata": {}, 154 | "outputs": [ 155 | { 156 | "name": "stdout", 157 | "output_type": "stream", 158 | "text": [ 159 | "[ 3. 5.]\n" 160 | ] 161 | } 162 | ], 163 | "source": [ 164 | "with tf.Session() as sess:\n", 165 | " print(sess.run(result))" 166 | ] 167 | }, 168 | { 169 | "cell_type": "markdown", 170 | "metadata": {}, 171 | "source": [ 172 | "3.3 指定默认会话" 173 | ] 174 | }, 175 | { 176 | "cell_type": "code", 177 | "execution_count": 6, 178 | "metadata": {}, 179 | "outputs": [ 180 | { 181 | "name": "stdout", 182 | "output_type": "stream", 183 | "text": [ 184 | "[ 3. 5.]\n" 185 | ] 186 | } 187 | ], 188 | "source": [ 189 | "sess = tf.Session()\n", 190 | "\n", 191 | "with sess.as_default():\n", 192 | " print(result.eval())\n", 193 | "\n" 194 | ] 195 | }, 196 | { 197 | "cell_type": "code", 198 | "execution_count": 7, 199 | "metadata": {}, 200 | "outputs": [ 201 | { 202 | "name": "stdout", 203 | "output_type": "stream", 204 | "text": [ 205 | "[ 3. 5.]\n", 206 | "[ 3. 5.]\n" 207 | ] 208 | } 209 | ], 210 | "source": [ 211 | "sess = tf.Session()\n", 212 | "\n", 213 | "# 下面的两个命令有相同的功能。\n", 214 | "print(sess.run(result))\n", 215 | "print(result.eval(session=sess))" 216 | ] 217 | }, 218 | { 219 | "cell_type": "markdown", 220 | "metadata": {}, 221 | "source": [ 222 | "4. 使用tf.InteractiveSession构建会话" 223 | ] 224 | }, 225 | { 226 | "cell_type": "code", 227 | "execution_count": 8, 228 | "metadata": {}, 229 | "outputs": [ 230 | { 231 | "name": "stdout", 232 | "output_type": "stream", 233 | "text": [ 234 | "[ 3. 5.]\n" 235 | ] 236 | } 237 | ], 238 | "source": [ 239 | "sess = tf.InteractiveSession ()\n", 240 | "print(result.eval())\n", 241 | "sess.close()\n" 242 | ] 243 | }, 244 | { 245 | "cell_type": "markdown", 246 | "metadata": {}, 247 | "source": [ 248 | "#### 5. 通过ConfigProto配置会话" 249 | ] 250 | }, 251 | { 252 | "cell_type": "code", 253 | "execution_count": 9, 254 | "metadata": { 255 | "collapsed": true 256 | }, 257 | "outputs": [], 258 | "source": [ 259 | "config=tf.ConfigProto(allow_soft_placement=True, log_device_placement=True)\n", 260 | "sess1 = tf.InteractiveSession(config=config)\n", 261 | "sess2 = tf.Session(config=config)" 262 | ] 263 | } 264 | ], 265 | "metadata": { 266 | "kernelspec": { 267 | "display_name": "Python 3", 268 | "language": "python", 269 | "name": "python3" 270 | }, 271 | "language_info": { 272 | "codemirror_mode": { 273 | "name": "ipython", 274 | "version": 3 275 | }, 276 | "file_extension": ".py", 277 | "mimetype": "text/x-python", 278 | "name": "python", 279 | "nbconvert_exporter": "python", 280 | "pygments_lexer": "ipython3", 281 | "version": "3.6.2" 282 | }, 283 | "varInspector": { 284 | "cols": { 285 | "lenName": 16, 286 | "lenType": 16, 287 | "lenVar": 40 288 | }, 289 | "kernels_config": { 290 | "python": { 291 | "delete_cmd_postfix": "", 292 | "delete_cmd_prefix": "del ", 293 | "library": "var_list.py", 294 | "varRefreshCmd": "print(var_dic_list())" 295 | }, 296 | "r": { 297 | "delete_cmd_postfix": ") ", 298 | "delete_cmd_prefix": "rm(", 299 | "library": "var_list.r", 300 | "varRefreshCmd": "cat(var_dic_list()) " 301 | } 302 | }, 303 | "types_to_exclude": [ 304 | "module", 305 | "function", 306 | "builtin_function_or_method", 307 | "instance", 308 | "_Feature" 309 | ], 310 | "window_display": false 311 | } 312 | }, 313 | "nbformat": 4, 314 | "nbformat_minor": 1 315 | } 316 | -------------------------------------------------------------------------------- /Chapter03/2. 三层简单神经网络的前向传播算法.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "#### 1. 三层简单神经网络" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 2, 13 | "metadata": { 14 | "collapsed": true 15 | }, 16 | "outputs": [], 17 | "source": [ 18 | "import tensorflow as tf" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": { 25 | "collapsed": true 26 | }, 27 | "outputs": [], 28 | "source": [] 29 | }, 30 | { 31 | "cell_type": "markdown", 32 | "metadata": {}, 33 | "source": [ 34 | "1.1 定义变量" 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": 5, 40 | "metadata": {}, 41 | "outputs": [ 42 | { 43 | "name": "stdout", 44 | "output_type": "stream", 45 | "text": [ 46 | "\n", 47 | "\n", 48 | "Tensor(\"Const_2:0\", shape=(1, 2), dtype=float32)\n" 49 | ] 50 | } 51 | ], 52 | "source": [ 53 | "w1= tf.Variable(tf.random_normal([2, 3], stddev=1, seed=1))\n", 54 | "w2= tf.Variable(tf.random_normal([3, 1], stddev=1, seed=1))\n", 55 | "x = tf.constant([[0.7, 0.9]])" 56 | ] 57 | }, 58 | { 59 | "cell_type": "markdown", 60 | "metadata": {}, 61 | "source": [ 62 | "1.2 定义前向传播的神经网络" 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": 7, 68 | "metadata": { 69 | "collapsed": true 70 | }, 71 | "outputs": [], 72 | "source": [ 73 | "a = tf.matmul(x, w1)\n", 74 | "y = tf.matmul(a, w2)" 75 | ] 76 | }, 77 | { 78 | "cell_type": "markdown", 79 | "metadata": {}, 80 | "source": [ 81 | "1.3 调用会话输出结果" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": 9, 87 | "metadata": {}, 88 | "outputs": [ 89 | { 90 | "name": "stdout", 91 | "output_type": "stream", 92 | "text": [ 93 | "[[ 3.95757794]]\n" 94 | ] 95 | } 96 | ], 97 | "source": [ 98 | "sess = tf.Session()\n", 99 | "sess.run(w1.initializer) \n", 100 | "sess.run(w2.initializer) \n", 101 | "print(sess.run(y)) \n", 102 | "sess.close()" 103 | ] 104 | }, 105 | { 106 | "cell_type": "markdown", 107 | "metadata": { 108 | "collapsed": true 109 | }, 110 | "source": [ 111 | "#### 2. 使用placeholder" 112 | ] 113 | }, 114 | { 115 | "cell_type": "code", 116 | "execution_count": 36, 117 | "metadata": {}, 118 | "outputs": [ 119 | { 120 | "name": "stdout", 121 | "output_type": "stream", 122 | "text": [ 123 | "None\n", 124 | "[[ 3.95757794]]\n", 125 | "[[ 3.95757794]]\n" 126 | ] 127 | } 128 | ], 129 | "source": [ 130 | "x = tf.placeholder(tf.float32, shape=(1, 2), name=\"input\")\n", 131 | "a = tf.matmul(x, w1)\n", 132 | "y = tf.matmul(a, w2)\n", 133 | "\n", 134 | "with tf.Session().as_default() as sess:\n", 135 | " init_op = tf.global_variables_initializer()\n", 136 | " print(sess.run(init_op))\n", 137 | " print(sess.run(y, feed_dict={x: [[0.7, 0.9]]}))\n", 138 | " print(y.eval(feed_dict={x: [[.7,.9]]}))" 139 | ] 140 | }, 141 | { 142 | "cell_type": "markdown", 143 | "metadata": {}, 144 | "source": [ 145 | "#### 3. 增加多个输入" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": 37, 151 | "metadata": {}, 152 | "outputs": [ 153 | { 154 | "name": "stdout", 155 | "output_type": "stream", 156 | "text": [ 157 | "[[ 3.95757794]\n", 158 | " [ 1.15376544]\n", 159 | " [ 3.16749239]]\n" 160 | ] 161 | } 162 | ], 163 | "source": [ 164 | "x = tf.placeholder(tf.float32, shape=(3, 2), name=\"input\")\n", 165 | "a = tf.matmul(x, w1)\n", 166 | "y = tf.matmul(a, w2)\n", 167 | "\n", 168 | "sess = tf.Session()\n", 169 | "#使用tf.global_variables_initializer()来初始化所有的变量\n", 170 | "init_op = tf.global_variables_initializer() \n", 171 | "sess.run(init_op)\n", 172 | "\n", 173 | "print(sess.run(y, feed_dict={x: [[0.7,0.9],[0.1,0.4],[0.5,0.8]]})) " 174 | ] 175 | } 176 | ], 177 | "metadata": { 178 | "kernelspec": { 179 | "display_name": "Python 3", 180 | "language": "python", 181 | "name": "python3" 182 | }, 183 | "language_info": { 184 | "codemirror_mode": { 185 | "name": "ipython", 186 | "version": 3 187 | }, 188 | "file_extension": ".py", 189 | "mimetype": "text/x-python", 190 | "name": "python", 191 | "nbconvert_exporter": "python", 192 | "pygments_lexer": "ipython3", 193 | "version": "3.6.2" 194 | }, 195 | "varInspector": { 196 | "cols": { 197 | "lenName": 16, 198 | "lenType": 16, 199 | "lenVar": 40 200 | }, 201 | "kernels_config": { 202 | "python": { 203 | "delete_cmd_postfix": "", 204 | "delete_cmd_prefix": "del ", 205 | "library": "var_list.py", 206 | "varRefreshCmd": "print(var_dic_list())" 207 | }, 208 | "r": { 209 | "delete_cmd_postfix": ") ", 210 | "delete_cmd_prefix": "rm(", 211 | "library": "var_list.r", 212 | "varRefreshCmd": "cat(var_dic_list()) " 213 | } 214 | }, 215 | "types_to_exclude": [ 216 | "module", 217 | "function", 218 | "builtin_function_or_method", 219 | "instance", 220 | "_Feature" 221 | ], 222 | "window_display": false 223 | } 224 | }, 225 | "nbformat": 4, 226 | "nbformat_minor": 1 227 | } 228 | -------------------------------------------------------------------------------- /Chapter03/3. 完整神经网络样例程序.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import tensorflow as tf\n", 12 | "from numpy.random import RandomState" 13 | ] 14 | }, 15 | { 16 | "cell_type": "markdown", 17 | "metadata": {}, 18 | "source": [ 19 | "#### 1. 定义神经网络的参数，输入和输出节点。" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 2, 25 | "metadata": { 26 | "collapsed": true 27 | }, 28 | "outputs": [], 29 | "source": [ 30 | "batch_size = 8\n", 31 | "dataset_size = 128\n", 32 | "w1= tf.Variable(tf.random_normal([2, 3], stddev=1, seed=1), trainable=True)\n", 33 | "w2= tf.Variable(tf.random_normal([3, 1], stddev=1, seed=1), trainable=True)\n", 34 | "x = tf.placeholder(tf.float32, shape=(None, 2), name=\"x-input\")\n", 35 | "y_= tf.placeholder(tf.float32, shape=(None, 1), name='y-input')" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": {}, 41 | "source": [ 42 | "#### 2. 定义前向传播过程，损失函数及反向传播算法。" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 3, 48 | "metadata": { 49 | "collapsed": true 50 | }, 51 | "outputs": [], 52 | "source": [ 53 | "a = tf.matmul(x, w1)\n", 54 | "y = tf.matmul(a, w2)\n", 55 | "cross_entropy = -tf.reduce_mean(y_ * tf.log(tf.clip_by_value(y, 1e-10, 1.0))) \n", 56 | "train_step = tf.train.AdamOptimizer(0.001).minimize(cross_entropy)" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "metadata": {}, 62 | "source": [ 63 | "#### 3. 生成模拟数据集。" 64 | ] 65 | }, 66 | { 67 | "cell_type": "code", 68 | "execution_count": 6, 69 | "metadata": {}, 70 | "outputs": [], 71 | "source": [ 72 | "rdm = RandomState(1)\n", 73 | "X = rdm.rand(dataset_size, 2)\n", 74 | "Y = [[int(x1+x2 < 1)] for (x1, x2) in X]" 75 | ] 76 | }, 77 | { 78 | "cell_type": "markdown", 79 | "metadata": {}, 80 | "source": [ 81 | "#### 4. 创建一个会话来运行TensorFlow程序。" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": 7, 87 | "metadata": {}, 88 | "outputs": [ 89 | { 90 | "name": "stdout", 91 | "output_type": "stream", 92 | "text": [ 93 | "w1: [[-0.81131822 1.48459876 0.06532937]\n", 94 | " [-2.4427042 0.0992484 0.59122431]]\n", 95 | "w2: [[-0.81131822]\n", 96 | " [ 1.48459876]\n", 97 | " [ 0.06532937]]\n", 98 | "\n", 99 | "\n", 100 | "After 0 training step(s), cross entropy on all data is 0.0674925\n", 101 | "w1: [[-0.81231821 1.4855988 0.06632921]\n", 102 | " [-2.44370413 0.1002484 0.59222424]]\n", 103 | "w2: [[-0.81231821]\n", 104 | " [ 1.4855988 ]\n", 105 | " [ 0.06632937]]\n", 106 | "After 1000 training step(s), cross entropy on all data is 0.0163385\n", 107 | "w1: [[-1.27549374 1.93239319 0.71818316]\n", 108 | " [-2.82764411 0.47066161 1.14189851]]\n", 109 | "w2: [[-1.21198606]\n", 110 | " [ 1.95956361]\n", 111 | " [ 0.55081969]]\n", 112 | "After 2000 training step(s), cross entropy on all data is 0.00907547\n", 113 | "w1: [[-1.51397622 2.1591146 1.07429051]\n", 114 | " [-3.01708913 0.64845419 1.46364999]]\n", 115 | "w2: [[-1.40506411]\n", 116 | " [ 2.20634365]\n", 117 | " [ 0.83948904]]\n", 118 | "After 3000 training step(s), cross entropy on all data is 0.00714436\n", 119 | "w1: [[-1.65394425 2.29218411 1.27439237]\n", 120 | " [-3.14156055 0.76467752 1.66820383]]\n", 121 | "w2: [[-1.52613485]\n", 122 | " [ 2.35394239]\n", 123 | " [ 1.01985705]]\n", 124 | "After 4000 training step(s), cross entropy on all data is 0.00578471\n", 125 | "w1: [[-1.79143536 2.42184758 1.46388769]\n", 126 | " [-3.28938985 0.90241849 1.88527477]]\n", 127 | "w2: [[-1.66073918]\n", 128 | " [ 2.50406837]\n", 129 | " [ 1.20711744]]\n", 130 | "After 5000 training step(s), cross entropy on all data is 0.00430222\n", 131 | "w1: [[-1.96215367 2.58266187 1.68243814]\n", 132 | " [-3.46851015 1.07014132 2.11830711]]\n", 133 | "w2: [[-1.82502723]\n", 134 | " [ 2.68580961]\n", 135 | " [ 1.41858089]]\n", 136 | "After 6000 training step(s), cross entropy on all data is 0.00280812\n", 137 | "w1: [[-2.15782928 2.76797581 1.91360271]\n", 138 | " [-3.66963673 1.26025426 2.35586643]]\n", 139 | "w2: [[-2.01201868]\n", 140 | " [ 2.88948369]\n", 141 | " [ 1.64122665]]\n", 142 | "After 7000 training step(s), cross entropy on all data is 0.00137464\n", 143 | "w1: [[-2.3685751 2.96918178 2.14771676]\n", 144 | " [-3.88335228 1.46428025 2.59262657]]\n", 145 | "w2: [[-2.21356416]\n", 146 | " [ 3.10541439]\n", 147 | " [ 1.86731029]]\n", 148 | "After 8000 training step(s), cross entropy on all data is 2.11566e-05\n", 149 | "w1: [[-2.58881521 3.18109441 2.38287973]\n", 150 | " [-4.10505152 1.67768049 2.82888079]]\n", 151 | "w2: [[-2.42511773]\n", 152 | " [ 3.32894397]\n", 153 | " [ 2.09544468]]\n", 154 | "After 9000 training step(s), cross entropy on all data is -0\n", 155 | "w1: [[-2.59392238 3.18602753 2.38825655]\n", 156 | " [-4.1101799 1.68263638 2.83427358]]\n", 157 | "w2: [[-2.4300375 ]\n", 158 | " [ 3.33411145]\n", 159 | " [ 2.10067439]]\n" 160 | ] 161 | } 162 | ], 163 | "source": [ 164 | "with tf.Session() as sess:\n", 165 | " init_op = tf.global_variables_initializer()\n", 166 | " sess.run(init_op)\n", 167 | " \n", 168 | " # 输出目前（未经训练）的参数取值。\n", 169 | " print(\"w1:\", sess.run(w1))\n", 170 | " print(\"w2:\", sess.run(w2))\n", 171 | " print(\"\\n\")\n", 172 | " \n", 173 | " # 训练模型。\n", 174 | " STEPS = 10000\n", 175 | " for i in range(STEPS):\n", 176 | " start = (i*batch_size) % dataset_size\n", 177 | " end = (i*batch_size) % dataset_size + batch_size\n", 178 | " sess.run(train_step, feed_dict={x: X[start:end], y_: Y[start:end]})\n", 179 | " if i % 1000 == 0:\n", 180 | " total_cross_entropy = sess.run(cross_entropy, feed_dict={x: X, y_: Y})\n", 181 | " print(\"After %d training step(s), cross entropy on all data is %g\" % (i, total_cross_entropy))\n", 182 | " print(\"w1:\", sess.run(w1))\n", 183 | " print(\"w2:\", sess.run(w2)) \n", 184 | " # 输出训练后的参数取值。\n", 185 | "# print(\"\\n\")\n" 186 | ] 187 | }, 188 | { 189 | "cell_type": "code", 190 | "execution_count": null, 191 | "metadata": { 192 | "collapsed": true 193 | }, 194 | "outputs": [], 195 | "source": [] 196 | } 197 | ], 198 | "metadata": { 199 | "kernelspec": { 200 | "display_name": "Python 3", 201 | "language": "python", 202 | "name": "python3" 203 | }, 204 | "language_info": { 205 | "codemirror_mode": { 206 | "name": "ipython", 207 | "version": 3 208 | }, 209 | "file_extension": ".py", 210 | "mimetype": "text/x-python", 211 | "name": "python", 212 | "nbconvert_exporter": "python", 213 | "pygments_lexer": "ipython3", 214 | "version": "3.6.2" 215 | }, 216 | "varInspector": { 217 | "cols": { 218 | "lenName": 16, 219 | "lenType": 16, 220 | "lenVar": 40 221 | }, 222 | "kernels_config": { 223 | "python": { 224 | "delete_cmd_postfix": "", 225 | "delete_cmd_prefix": "del ", 226 | "library": "var_list.py", 227 | "varRefreshCmd": "print(var_dic_list())" 228 | }, 229 | "r": { 230 | "delete_cmd_postfix": ") ", 231 | "delete_cmd_prefix": "rm(", 232 | "library": "var_list.r", 233 | "varRefreshCmd": "cat(var_dic_list()) " 234 | } 235 | }, 236 | "oldHeight": 305, 237 | "position": { 238 | "height": "333px", 239 | "left": "711px", 240 | "right": "20px", 241 | "top": "127px", 242 | "width": "521px" 243 | }, 244 | "types_to_exclude": [ 245 | "module", 246 | "function", 247 | "builtin_function_or_method", 248 | "instance", 249 | "_Feature" 250 | ], 251 | "varInspector_section_display": "block", 252 | "window_display": false 253 | } 254 | }, 255 | "nbformat": 4, 256 | "nbformat_minor": 1 257 | } 258 | -------------------------------------------------------------------------------- /Chapter04/2. 学习率的设置.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "#### 假设我们要最小化函数 $y=x^2$, 选择初始点 $x_0=5$" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "#### 1. 学习率为1的时候，x在5和-5之间震荡。" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": 1, 20 | "metadata": {}, 21 | "outputs": [ 22 | { 23 | "name": "stdout", 24 | "output_type": "stream", 25 | "text": [ 26 | "After 1 iteration(s): x1 is -5.000000.\n", 27 | "After 2 iteration(s): x2 is 5.000000.\n", 28 | "After 3 iteration(s): x3 is -5.000000.\n", 29 | "After 4 iteration(s): x4 is 5.000000.\n", 30 | "After 5 iteration(s): x5 is -5.000000.\n", 31 | "After 6 iteration(s): x6 is 5.000000.\n", 32 | "After 7 iteration(s): x7 is -5.000000.\n", 33 | "After 8 iteration(s): x8 is 5.000000.\n", 34 | "After 9 iteration(s): x9 is -5.000000.\n", 35 | "After 10 iteration(s): x10 is 5.000000.\n" 36 | ] 37 | } 38 | ], 39 | "source": [ 40 | "import tensorflow as tf\n", 41 | "TRAINING_STEPS = 10\n", 42 | "LEARNING_RATE = 1\n", 43 | "x = tf.Variable(tf.constant(5, dtype=tf.float32), name=\"x\")\n", 44 | "y = tf.square(x)\n", 45 | "\n", 46 | "train_op = tf.train.GradientDescentOptimizer(LEARNING_RATE).minimize(y)\n", 47 | "\n", 48 | "with tf.Session() as sess:\n", 49 | " sess.run(tf.global_variables_initializer())\n", 50 | " for i in range(TRAINING_STEPS):\n", 51 | " sess.run(train_op)\n", 52 | " x_value = sess.run(x)\n", 53 | " print(\"After %s iteration(s): x%s is %f.\"% (i+1, i+1, x_value))" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "#### 2. 学习率为0.001的时候，下降速度过慢，在901轮时才收敛到0.823355。" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": 4, 66 | "metadata": {}, 67 | "outputs": [ 68 | { 69 | "name": "stdout", 70 | "output_type": "stream", 71 | "text": [ 72 | "After 100 iteration(s): x100 is 4.092832.\n", 73 | "After 200 iteration(s): x200 is 3.350255.\n", 74 | "After 300 iteration(s): x300 is 2.742408.\n", 75 | "After 400 iteration(s): x400 is 2.244844.\n", 76 | "After 500 iteration(s): x500 is 1.837555.\n", 77 | "After 600 iteration(s): x600 is 1.504161.\n", 78 | "After 700 iteration(s): x700 is 1.231257.\n", 79 | "After 800 iteration(s): x800 is 1.007866.\n", 80 | "After 900 iteration(s): x900 is 0.825006.\n", 81 | "After 1000 iteration(s): x1000 is 0.675322.\n" 82 | ] 83 | } 84 | ], 85 | "source": [ 86 | "TRAINING_STEPS = 1000\n", 87 | "LEARNING_RATE = 0.001\n", 88 | "x = tf.Variable(tf.constant(5, dtype=tf.float32), name=\"x\")\n", 89 | "y = tf.square(x)\n", 90 | "\n", 91 | "train_op = tf.train.GradientDescentOptimizer(LEARNING_RATE).minimize(y)\n", 92 | "\n", 93 | "with tf.Session() as sess:\n", 94 | " sess.run(tf.global_variables_initializer())\n", 95 | " for i in range(TRAINING_STEPS):\n", 96 | " sess.run(train_op)\n", 97 | " if i % 100 == 99: \n", 98 | " x_value = sess.run(x)\n", 99 | " print(\"After %s iteration(s): x%s is %f.\"% (i+1, i+1, x_value))" 100 | ] 101 | }, 102 | { 103 | "cell_type": "markdown", 104 | "metadata": { 105 | "collapsed": true 106 | }, 107 | "source": [ 108 | "#### 3. 使用指数衰减的学习率，在迭代初期得到较高的下降速度，可以在较小的训练轮数下取得不错的收敛程度。" 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": 5, 114 | "metadata": {}, 115 | "outputs": [ 116 | { 117 | "name": "stdout", 118 | "output_type": "stream", 119 | "text": [ 120 | "After 10 iteration(s): x10 is 0.796464, learning rate is 0.066483.\n", 121 | "After 20 iteration(s): x20 is 0.244167, learning rate is 0.044200.\n", 122 | "After 30 iteration(s): x30 is 0.113049, learning rate is 0.029386.\n", 123 | "After 40 iteration(s): x40 is 0.068213, learning rate is 0.019537.\n", 124 | "After 50 iteration(s): x50 is 0.048895, learning rate is 0.012989.\n", 125 | "After 60 iteration(s): x60 is 0.039235, learning rate is 0.008635.\n", 126 | "After 70 iteration(s): x70 is 0.033913, learning rate is 0.005741.\n", 127 | "After 80 iteration(s): x80 is 0.030788, learning rate is 0.003817.\n", 128 | "After 90 iteration(s): x90 is 0.028874, learning rate is 0.002538.\n", 129 | "After 100 iteration(s): x100 is 0.027669, learning rate is 0.001687.\n" 130 | ] 131 | } 132 | ], 133 | "source": [ 134 | "TRAINING_STEPS = 100\n", 135 | "global_step = tf.Variable(0)\n", 136 | "LEARNING_RATE = tf.train.exponential_decay(0.1, global_step, 1, 0.96, staircase=True)\n", 137 | "\n", 138 | "x = tf.Variable(tf.constant(5, dtype=tf.float32), name=\"x\")\n", 139 | "y = tf.square(x)\n", 140 | "train_op = tf.train.GradientDescentOptimizer(LEARNING_RATE).minimize(y, global_step=global_step)\n", 141 | "\n", 142 | "with tf.Session() as sess:\n", 143 | " sess.run(tf.global_variables_initializer())\n", 144 | " for i in range(TRAINING_STEPS):\n", 145 | " sess.run(train_op)\n", 146 | " if i % 10 == 9:\n", 147 | " LEARNING_RATE_value = sess.run(LEARNING_RATE)\n", 148 | " x_value = sess.run(x)\n", 149 | " print(\"After %s iteration(s): x%s is %f, learning rate is %f.\"% (i+1, i+1, x_value, LEARNING_RATE_value))" 150 | ] 151 | }, 152 | { 153 | "cell_type": "code", 154 | "execution_count": null, 155 | "metadata": { 156 | "collapsed": true 157 | }, 158 | "outputs": [], 159 | "source": [] 160 | } 161 | ], 162 | "metadata": { 163 | "kernelspec": { 164 | "display_name": "Python 3", 165 | "language": "python", 166 | "name": "python3" 167 | }, 168 | "language_info": { 169 | "codemirror_mode": { 170 | "name": "ipython", 171 | "version": 3 172 | }, 173 | "file_extension": ".py", 174 | "mimetype": "text/x-python", 175 | "name": "python", 176 | "nbconvert_exporter": "python", 177 | "pygments_lexer": "ipython3", 178 | "version": "3.6.2" 179 | }, 180 | "varInspector": { 181 | "cols": { 182 | "lenName": 16, 183 | "lenType": 16, 184 | "lenVar": 40 185 | }, 186 | "kernels_config": { 187 | "python": { 188 | "delete_cmd_postfix": "", 189 | "delete_cmd_prefix": "del ", 190 | "library": "var_list.py", 191 | "varRefreshCmd": "print(var_dic_list())" 192 | }, 193 | "r": { 194 | "delete_cmd_postfix": ") ", 195 | "delete_cmd_prefix": "rm(", 196 | "library": "var_list.r", 197 | "varRefreshCmd": "cat(var_dic_list()) " 198 | } 199 | }, 200 | "types_to_exclude": [ 201 | "module", 202 | "function", 203 | "builtin_function_or_method", 204 | "instance", 205 | "_Feature" 206 | ], 207 | "window_display": false 208 | } 209 | }, 210 | "nbformat": 4, 211 | "nbformat_minor": 1 212 | } 213 | -------------------------------------------------------------------------------- /Chapter04/4. 滑动平均模型.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import tensorflow as tf" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "#### 1. 定义变量及滑动平均类" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 2, 24 | "metadata": { 25 | "collapsed": true 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "v1 = tf.Variable(0, dtype=tf.float32)\n", 30 | "step = tf.Variable(0, trainable=False)\n", 31 | "ema = tf.train.ExponentialMovingAverage(decay=0.99, num_updates=step)\n", 32 | "# 每一次操作的时候，列表变量[v1]都会被更新\n", 33 | "maintain_averages_op = ema.apply([v1]) " 34 | ] 35 | }, 36 | { 37 | "cell_type": "markdown", 38 | "metadata": {}, 39 | "source": [ 40 | "#### 2. 查看不同迭代中变量取值的变化。tf.assign用于变更变量的数值" 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": 12, 46 | "metadata": {}, 47 | "outputs": [ 48 | { 49 | "name": "stdout", 50 | "output_type": "stream", 51 | "text": [ 52 | "[0.0, 0.0]\n", 53 | "[5.0, 4.5]\n", 54 | "[10.0, 4.5549998]\n", 55 | "[10.0, 4.6094499]\n" 56 | ] 57 | } 58 | ], 59 | "source": [ 60 | "with tf.Session() as sess:\n", 61 | " \n", 62 | " # 初始化\n", 63 | " init_op = tf.global_variables_initializer()\n", 64 | " sess.run(init_op)\n", 65 | " print(sess.run([v1, ema.average(v1)]))\n", 66 | " \n", 67 | " # 更新变量v1的取值\n", 68 | " sess.run(tf.assign(v1, 5))\n", 69 | " sess.run(maintain_averages_op)\n", 70 | " print(sess.run([v1, ema.average(v1)]))\n", 71 | " \n", 72 | " # 更新step和v1的取值\n", 73 | " sess.run(tf.assign(step, 10000)) \n", 74 | " sess.run(tf.assign(v1, 10))\n", 75 | " sess.run(maintain_averages_op)\n", 76 | " print(sess.run([v1, ema.average(v1)]))\n", 77 | " \n", 78 | " # 更新一次v1的滑动平均值\n", 79 | " sess.run(maintain_averages_op)\n", 80 | " print(sess.run([v1, ema.average(v1)]))\n" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": null, 86 | "metadata": { 87 | "collapsed": true 88 | }, 89 | "outputs": [], 90 | "source": [] 91 | } 92 | ], 93 | "metadata": { 94 | "kernelspec": { 95 | "display_name": "Python 3", 96 | "language": "python", 97 | "name": "python3" 98 | }, 99 | "language_info": { 100 | "codemirror_mode": { 101 | "name": "ipython", 102 | "version": 3 103 | }, 104 | "file_extension": ".py", 105 | "mimetype": "text/x-python", 106 | "name": "python", 107 | "nbconvert_exporter": "python", 108 | "pygments_lexer": "ipython3", 109 | "version": "3.6.2" 110 | }, 111 | "varInspector": { 112 | "cols": { 113 | "lenName": 16, 114 | "lenType": 16, 115 | "lenVar": 40 116 | }, 117 | "kernels_config": { 118 | "python": { 119 | "delete_cmd_postfix": "", 120 | "delete_cmd_prefix": "del ", 121 | "library": "var_list.py", 122 | "varRefreshCmd": "print(var_dic_list())" 123 | }, 124 | "r": { 125 | "delete_cmd_postfix": ") ", 126 | "delete_cmd_prefix": "rm(", 127 | "library": "var_list.r", 128 | "varRefreshCmd": "cat(var_dic_list()) " 129 | } 130 | }, 131 | "oldHeight": 152, 132 | "position": { 133 | "height": "174px", 134 | "left": "159px", 135 | "right": "20px", 136 | "top": "13px", 137 | "width": "800px" 138 | }, 139 | "types_to_exclude": [ 140 | "module", 141 | "function", 142 | "builtin_function_or_method", 143 | "instance", 144 | "_Feature" 145 | ], 146 | "varInspector_section_display": "block", 147 | "window_display": false 148 | } 149 | }, 150 | "nbformat": 4, 151 | "nbformat_minor": 1 152 | } 153 | -------------------------------------------------------------------------------- /Chapter05/2. TensorFlow训练神经网络/2. 不使用正则化.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import tensorflow as tf\n", 10 | "from tensorflow.examples.tutorials.mnist import input_data" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": {}, 16 | "source": [ 17 | "#### 1.设置输入和输出节点的个数,配置神经网络的参数。" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 2, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "INPUT_NODE = 784 # 输入节点\n", 27 | "OUTPUT_NODE = 10 # 输出节点\n", 28 | "LAYER1_NODE = 500 # 隐藏层数 \n", 29 | " \n", 30 | "BATCH_SIZE = 100 # 每次batch打包的样本个数 \n", 31 | "\n", 32 | "# 模型相关的参数\n", 33 | "LEARNING_RATE_BASE = 0.8 \n", 34 | "LEARNING_RATE_DECAY = 0.99 \n", 35 | " \n", 36 | "TRAINING_STEPS = 5000 \n", 37 | "MOVING_AVERAGE_DECAY = 0.99 " 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "#### 2. 定义辅助函数来计算前向传播结果，使用ReLU做为激活函数。" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": 3, 50 | "metadata": {}, 51 | "outputs": [], 52 | "source": [ 53 | "def inference(input_tensor, avg_class, weights1, biases1, weights2, biases2):\n", 54 | " # 不使用滑动平均类\n", 55 | " if avg_class == None:\n", 56 | " layer1 = tf.nn.relu(tf.matmul(input_tensor, weights1) + biases1)\n", 57 | " return tf.matmul(layer1, weights2) + biases2\n", 58 | "\n", 59 | " else:\n", 60 | " # 使用滑动平均类\n", 61 | " layer1 = tf.nn.relu(tf.matmul(input_tensor, avg_class.average(weights1)) + avg_class.average(biases1))\n", 62 | " return tf.matmul(layer1, avg_class.average(weights2)) + avg_class.average(biases2) " 63 | ] 64 | }, 65 | { 66 | "cell_type": "markdown", 67 | "metadata": {}, 68 | "source": [ 69 | "#### 3. 定义训练过程。" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 4, 75 | "metadata": {}, 76 | "outputs": [], 77 | "source": [ 78 | "def train(mnist):\n", 79 | " x = tf.placeholder(tf.float32, [None, INPUT_NODE], name='x-input')\n", 80 | " y_ = tf.placeholder(tf.float32, [None, OUTPUT_NODE], name='y-input')\n", 81 | " # 生成隐藏层的参数。\n", 82 | " weights1 = tf.Variable(tf.truncated_normal([INPUT_NODE, LAYER1_NODE], stddev=0.1))\n", 83 | " biases1 = tf.Variable(tf.constant(0.1, shape=[LAYER1_NODE]))\n", 84 | " # 生成输出层的参数。\n", 85 | " weights2 = tf.Variable(tf.truncated_normal([LAYER1_NODE, OUTPUT_NODE], stddev=0.1))\n", 86 | " biases2 = tf.Variable(tf.constant(0.1, shape=[OUTPUT_NODE]))\n", 87 | "\n", 88 | " # 计算不含滑动平均类的前向传播结果\n", 89 | " y = inference(x, None, weights1, biases1, weights2, biases2)\n", 90 | " \n", 91 | " # 定义训练轮数及相关的滑动平均类 \n", 92 | " global_step = tf.Variable(0, trainable=False)\n", 93 | " variable_averages = tf.train.ExponentialMovingAverage(MOVING_AVERAGE_DECAY, global_step)\n", 94 | " variables_averages_op = variable_averages.apply(tf.trainable_variables())\n", 95 | " average_y = inference(x, variable_averages, weights1, biases1, weights2, biases2)\n", 96 | " \n", 97 | " # 计算交叉熵及其平均值\n", 98 | " cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y, labels=tf.argmax(y_, 1))\n", 99 | " cross_entropy_mean = tf.reduce_mean(cross_entropy)\n", 100 | " \n", 101 | " # 损失函数的计算\n", 102 | " loss = cross_entropy_mean\n", 103 | " \n", 104 | " # 设置指数衰减的学习率。\n", 105 | " learning_rate = tf.train.exponential_decay(\n", 106 | " LEARNING_RATE_BASE,\n", 107 | " global_step,\n", 108 | " mnist.train.num_examples / BATCH_SIZE,\n", 109 | " LEARNING_RATE_DECAY,\n", 110 | " staircase=True)\n", 111 | " \n", 112 | " # 优化损失函数\n", 113 | " train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=global_step)\n", 114 | " \n", 115 | " # 反向传播更新参数和更新每一个参数的滑动平均值\n", 116 | " with tf.control_dependencies([train_step, variables_averages_op]):\n", 117 | " train_op = tf.no_op(name='train')\n", 118 | "\n", 119 | " # 计算正确率\n", 120 | " correct_prediction = tf.equal(tf.argmax(average_y, 1), tf.argmax(y_, 1))\n", 121 | " accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n", 122 | " \n", 123 | " # 初始化会话，并开始训练过程。\n", 124 | " with tf.Session() as sess:\n", 125 | " tf.global_variables_initializer().run()\n", 126 | " validate_feed = {x: mnist.validation.images, y_: mnist.validation.labels}\n", 127 | " test_feed = {x: mnist.test.images, y_: mnist.test.labels} \n", 128 | " \n", 129 | " # 循环的训练神经网络。\n", 130 | " for i in range(TRAINING_STEPS):\n", 131 | " if i % 1000 == 0:\n", 132 | " validate_acc = sess.run(accuracy, feed_dict=validate_feed)\n", 133 | " print(\"After %d training step(s), validation accuracy using average model is %g \" % (i, validate_acc))\n", 134 | " \n", 135 | " xs,ys=mnist.train.next_batch(BATCH_SIZE)\n", 136 | " sess.run(train_op,feed_dict={x:xs,y_:ys})\n", 137 | "\n", 138 | " test_acc=sess.run(accuracy,feed_dict=test_feed)\n", 139 | " print((\"After %d training step(s), test accuracy using average model is %g\" %(TRAINING_STEPS, test_acc)))\n", 140 | " " 141 | ] 142 | }, 143 | { 144 | "cell_type": "markdown", 145 | "metadata": {}, 146 | "source": [ 147 | "#### 4. 主程序入口，这里设定模型训练次数为5000次。" 148 | ] 149 | }, 150 | { 151 | "cell_type": "code", 152 | "execution_count": 5, 153 | "metadata": {}, 154 | "outputs": [ 155 | { 156 | "name": "stdout", 157 | "output_type": "stream", 158 | "text": [ 159 | "Extracting ../../../datasets/MNIST_data/train-images-idx3-ubyte.gz\n", 160 | "Extracting ../../../datasets/MNIST_data/train-labels-idx1-ubyte.gz\n", 161 | "Extracting ../../../datasets/MNIST_data/t10k-images-idx3-ubyte.gz\n", 162 | "Extracting ../../../datasets/MNIST_data/t10k-labels-idx1-ubyte.gz\n", 163 | "After 0 training step(s), validation accuracy using average model is 0.1258 \n", 164 | "After 1000 training step(s), validation accuracy using average model is 0.9742 \n", 165 | "After 2000 training step(s), validation accuracy using average model is 0.9794 \n", 166 | "After 3000 training step(s), validation accuracy using average model is 0.9812 \n", 167 | "After 4000 training step(s), validation accuracy using average model is 0.9824 \n", 168 | "After 5000 training step(s), test accuracy using average model is 0.9827\n" 169 | ] 170 | } 171 | ], 172 | "source": [ 173 | "def main(argv=None):\n", 174 | " mnist = input_data.read_data_sets(\"../../datasets/MNIST_data\", one_hot=True)\n", 175 | " train(mnist)\n", 176 | "\n", 177 | "if __name__=='__main__':\n", 178 | " main()" 179 | ] 180 | } 181 | ], 182 | "metadata": { 183 | "kernelspec": { 184 | "display_name": "Python 3", 185 | "language": "python", 186 | "name": "python3" 187 | }, 188 | "language_info": { 189 | "codemirror_mode": { 190 | "name": "ipython", 191 | "version": 3 192 | }, 193 | "file_extension": ".py", 194 | "mimetype": "text/x-python", 195 | "name": "python", 196 | "nbconvert_exporter": "python", 197 | "pygments_lexer": "ipython3", 198 | "version": "3.6.2" 199 | }, 200 | "varInspector": { 201 | "cols": { 202 | "lenName": 16, 203 | "lenType": 16, 204 | "lenVar": 40 205 | }, 206 | "kernels_config": { 207 | "python": { 208 | "delete_cmd_postfix": "", 209 | "delete_cmd_prefix": "del ", 210 | "library": "var_list.py", 211 | "varRefreshCmd": "print(var_dic_list())" 212 | }, 213 | "r": { 214 | "delete_cmd_postfix": ") ", 215 | "delete_cmd_prefix": "rm(", 216 | "library": "var_list.r", 217 | "varRefreshCmd": "cat(var_dic_list()) " 218 | } 219 | }, 220 | "types_to_exclude": [ 221 | "module", 222 | "function", 223 | "builtin_function_or_method", 224 | "instance", 225 | "_Feature" 226 | ], 227 | "window_display": false 228 | } 229 | }, 230 | "nbformat": 4, 231 | "nbformat_minor": 1 232 | } 233 | -------------------------------------------------------------------------------- /Chapter05/2. TensorFlow训练神经网络/3. 不使用指数衰减的学习率.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import tensorflow as tf\n", 10 | "from tensorflow.examples.tutorials.mnist import input_data" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": {}, 16 | "source": [ 17 | "#### 1.设置输入和输出节点的个数,配置神经网络的参数。" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 2, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "INPUT_NODE = 784 # 输入节点\n", 27 | "OUTPUT_NODE = 10 # 输出节点\n", 28 | "LAYER1_NODE = 500 # 隐藏层数 \n", 29 | " \n", 30 | "BATCH_SIZE = 100 # 每次batch打包的样本个数 \n", 31 | "\n", 32 | "# 模型相关的参数\n", 33 | "LEARNING_RATE = 0.1 \n", 34 | "REGULARAZTION_RATE = 0.0001 \n", 35 | "TRAINING_STEPS = 5000 \n", 36 | "MOVING_AVERAGE_DECAY = 0.99 " 37 | ] 38 | }, 39 | { 40 | "cell_type": "markdown", 41 | "metadata": {}, 42 | "source": [ 43 | "#### 2. 定义辅助函数来计算前向传播结果，使用ReLU做为激活函数。" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": 3, 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "def inference(input_tensor, avg_class, weights1, biases1, weights2, biases2):\n", 53 | " # 不使用滑动平均类\n", 54 | " if avg_class == None:\n", 55 | " layer1 = tf.nn.relu(tf.matmul(input_tensor, weights1) + biases1)\n", 56 | " return tf.matmul(layer1, weights2) + biases2\n", 57 | "\n", 58 | " else:\n", 59 | " # 使用滑动平均类\n", 60 | " layer1 = tf.nn.relu(tf.matmul(input_tensor, avg_class.average(weights1)) + avg_class.average(biases1))\n", 61 | " return tf.matmul(layer1, avg_class.average(weights2)) + avg_class.average(biases2) " 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": {}, 67 | "source": [ 68 | "#### 3. 定义训练过程。" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": 4, 74 | "metadata": {}, 75 | "outputs": [], 76 | "source": [ 77 | "def train(mnist):\n", 78 | " x = tf.placeholder(tf.float32, [None, INPUT_NODE], name='x-input')\n", 79 | " y_ = tf.placeholder(tf.float32, [None, OUTPUT_NODE], name='y-input')\n", 80 | " # 生成隐藏层的参数。\n", 81 | " weights1 = tf.Variable(tf.truncated_normal([INPUT_NODE, LAYER1_NODE], stddev=0.1))\n", 82 | " biases1 = tf.Variable(tf.constant(0.1, shape=[LAYER1_NODE]))\n", 83 | " # 生成输出层的参数。\n", 84 | " weights2 = tf.Variable(tf.truncated_normal([LAYER1_NODE, OUTPUT_NODE], stddev=0.1))\n", 85 | " biases2 = tf.Variable(tf.constant(0.1, shape=[OUTPUT_NODE]))\n", 86 | "\n", 87 | " # 计算不含滑动平均类的前向传播结果\n", 88 | " y = inference(x, None, weights1, biases1, weights2, biases2)\n", 89 | " \n", 90 | " # 定义训练轮数及相关的滑动平均类 \n", 91 | " global_step = tf.Variable(0, trainable=False)\n", 92 | " variable_averages = tf.train.ExponentialMovingAverage(MOVING_AVERAGE_DECAY, global_step)\n", 93 | " variables_averages_op = variable_averages.apply(tf.trainable_variables())\n", 94 | " average_y = inference(x, variable_averages, weights1, biases1, weights2, biases2)\n", 95 | " \n", 96 | " # 计算交叉熵及其平均值\n", 97 | " cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y, labels=tf.argmax(y_, 1))\n", 98 | " cross_entropy_mean = tf.reduce_mean(cross_entropy)\n", 99 | " \n", 100 | " # 损失函数的计算\n", 101 | " regularizer = tf.contrib.layers.l2_regularizer(REGULARAZTION_RATE)\n", 102 | " regularaztion = regularizer(weights1) + regularizer(weights2)\n", 103 | " loss = cross_entropy_mean + regularaztion\n", 104 | " \n", 105 | " # 优化损失函数\n", 106 | " train_step = tf.train.GradientDescentOptimizer(LEARNING_RATE).minimize(loss, global_step=global_step)\n", 107 | " \n", 108 | " # 反向传播更新参数和更新每一个参数的滑动平均值\n", 109 | " with tf.control_dependencies([train_step, variables_averages_op]):\n", 110 | " train_op = tf.no_op(name='train')\n", 111 | "\n", 112 | " # 计算正确率\n", 113 | " correct_prediction = tf.equal(tf.argmax(average_y, 1), tf.argmax(y_, 1))\n", 114 | " accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n", 115 | " \n", 116 | " # 初始化会话，并开始训练过程。\n", 117 | " with tf.Session() as sess:\n", 118 | " tf.global_variables_initializer().run()\n", 119 | " validate_feed = {x: mnist.validation.images, y_: mnist.validation.labels}\n", 120 | " test_feed = {x: mnist.test.images, y_: mnist.test.labels} \n", 121 | " \n", 122 | " # 循环的训练神经网络。\n", 123 | " for i in range(TRAINING_STEPS):\n", 124 | " if i % 1000 == 0:\n", 125 | " validate_acc = sess.run(accuracy, feed_dict=validate_feed)\n", 126 | " print(\"After %d training step(s), validation accuracy using average model is %g \" % (i, validate_acc))\n", 127 | " \n", 128 | " xs,ys=mnist.train.next_batch(BATCH_SIZE)\n", 129 | " sess.run(train_op,feed_dict={x:xs,y_:ys})\n", 130 | "\n", 131 | " test_acc=sess.run(accuracy,feed_dict=test_feed)\n", 132 | " print((\"After %d training step(s), test accuracy using average model is %g\" %(TRAINING_STEPS, test_acc)))\n", 133 | " " 134 | ] 135 | }, 136 | { 137 | "cell_type": "markdown", 138 | "metadata": {}, 139 | "source": [ 140 | "#### 4. 主程序入口，这里设定模型训练次数为5000次。" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": 5, 146 | "metadata": {}, 147 | "outputs": [ 148 | { 149 | "name": "stdout", 150 | "output_type": "stream", 151 | "text": [ 152 | "Extracting ../../../datasets/MNIST_data/train-images-idx3-ubyte.gz\n", 153 | "Extracting ../../../datasets/MNIST_data/train-labels-idx1-ubyte.gz\n", 154 | "Extracting ../../../datasets/MNIST_data/t10k-images-idx3-ubyte.gz\n", 155 | "Extracting ../../../datasets/MNIST_data/t10k-labels-idx1-ubyte.gz\n", 156 | "After 0 training step(s), validation accuracy using average model is 0.0972 \n", 157 | "After 1000 training step(s), validation accuracy using average model is 0.9486 \n", 158 | "After 2000 training step(s), validation accuracy using average model is 0.964 \n", 159 | "After 3000 training step(s), validation accuracy using average model is 0.9704 \n", 160 | "After 4000 training step(s), validation accuracy using average model is 0.9726 \n", 161 | "After 5000 training step(s), test accuracy using average model is 0.9737\n" 162 | ] 163 | } 164 | ], 165 | "source": [ 166 | "def main(argv=None):\n", 167 | " mnist = input_data.read_data_sets(\"../../datasets/MNIST_data\", one_hot=True)\n", 168 | " train(mnist)\n", 169 | "\n", 170 | "if __name__=='__main__':\n", 171 | " main()" 172 | ] 173 | } 174 | ], 175 | "metadata": { 176 | "kernelspec": { 177 | "display_name": "Python 3", 178 | "language": "python", 179 | "name": "python3" 180 | }, 181 | "language_info": { 182 | "codemirror_mode": { 183 | "name": "ipython", 184 | "version": 3 185 | }, 186 | "file_extension": ".py", 187 | "mimetype": "text/x-python", 188 | "name": "python", 189 | "nbconvert_exporter": "python", 190 | "pygments_lexer": "ipython3", 191 | "version": "3.6.2" 192 | }, 193 | "varInspector": { 194 | "cols": { 195 | "lenName": 16, 196 | "lenType": 16, 197 | "lenVar": 40 198 | }, 199 | "kernels_config": { 200 | "python": { 201 | "delete_cmd_postfix": "", 202 | "delete_cmd_prefix": "del ", 203 | "library": "var_list.py", 204 | "varRefreshCmd": "print(var_dic_list())" 205 | }, 206 | "r": { 207 | "delete_cmd_postfix": ") ", 208 | "delete_cmd_prefix": "rm(", 209 | "library": "var_list.r", 210 | "varRefreshCmd": "cat(var_dic_list()) " 211 | } 212 | }, 213 | "types_to_exclude": [ 214 | "module", 215 | "function", 216 | "builtin_function_or_method", 217 | "instance", 218 | "_Feature" 219 | ], 220 | "window_display": false 221 | } 222 | }, 223 | "nbformat": 4, 224 | "nbformat_minor": 1 225 | } 226 | -------------------------------------------------------------------------------- /Chapter05/2. TensorFlow训练神经网络/4. 不使用激活函数.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import tensorflow as tf\n", 10 | "from tensorflow.examples.tutorials.mnist import input_data" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": {}, 16 | "source": [ 17 | "#### 1.设置输入和输出节点的个数,配置神经网络的参数。" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 2, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "INPUT_NODE = 784 # 输入节点\n", 27 | "OUTPUT_NODE = 10 # 输出节点\n", 28 | "LAYER1_NODE = 500 # 隐藏层数 \n", 29 | " \n", 30 | "BATCH_SIZE = 100 # 每次batch打包的样本个数 \n", 31 | "\n", 32 | "# 模型相关的参数\n", 33 | "LEARNING_RATE_BASE = 0.8 \n", 34 | "LEARNING_RATE_DECAY = 0.99 \n", 35 | "REGULARAZTION_RATE = 0.0001 \n", 36 | "TRAINING_STEPS = 5000 \n", 37 | "MOVING_AVERAGE_DECAY = 0.99 " 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "#### 2. 定义辅助函数来计算前向传播结果，使用ReLU做为激活函数。" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": 3, 50 | "metadata": {}, 51 | "outputs": [], 52 | "source": [ 53 | "def inference(input_tensor, avg_class, weights1, biases1, weights2, biases2):\n", 54 | " # 不使用滑动平均类\n", 55 | " if avg_class == None:\n", 56 | " layer1 = tf.matmul(input_tensor, weights1) + biases1\n", 57 | " return tf.matmul(layer1, weights2) + biases2\n", 58 | "\n", 59 | " else:\n", 60 | " # 使用滑动平均类\n", 61 | " layer1 = tf.matmul(input_tensor, avg_class.average(weights1)) + avg_class.average(biases1)\n", 62 | " return tf.matmul(layer1, avg_class.average(weights2)) + avg_class.average(biases2) " 63 | ] 64 | }, 65 | { 66 | "cell_type": "markdown", 67 | "metadata": {}, 68 | "source": [ 69 | "#### 3. 定义训练过程。" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 4, 75 | "metadata": {}, 76 | "outputs": [], 77 | "source": [ 78 | "def train(mnist):\n", 79 | " x = tf.placeholder(tf.float32, [None, INPUT_NODE], name='x-input')\n", 80 | " y_ = tf.placeholder(tf.float32, [None, OUTPUT_NODE], name='y-input')\n", 81 | " # 生成隐藏层的参数。\n", 82 | " weights1 = tf.Variable(tf.truncated_normal([INPUT_NODE, LAYER1_NODE], stddev=0.1))\n", 83 | " biases1 = tf.Variable(tf.constant(0.1, shape=[LAYER1_NODE]))\n", 84 | " # 生成输出层的参数。\n", 85 | " weights2 = tf.Variable(tf.truncated_normal([LAYER1_NODE, OUTPUT_NODE], stddev=0.1))\n", 86 | " biases2 = tf.Variable(tf.constant(0.1, shape=[OUTPUT_NODE]))\n", 87 | "\n", 88 | " # 计算不含滑动平均类的前向传播结果\n", 89 | " y = inference(x, None, weights1, biases1, weights2, biases2)\n", 90 | " \n", 91 | " # 定义训练轮数及相关的滑动平均类 \n", 92 | " global_step = tf.Variable(0, trainable=False)\n", 93 | " variable_averages = tf.train.ExponentialMovingAverage(MOVING_AVERAGE_DECAY, global_step)\n", 94 | " variables_averages_op = variable_averages.apply(tf.trainable_variables())\n", 95 | " average_y = inference(x, variable_averages, weights1, biases1, weights2, biases2)\n", 96 | " \n", 97 | " # 计算交叉熵及其平均值\n", 98 | " cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y, labels=tf.argmax(y_, 1))\n", 99 | " cross_entropy_mean = tf.reduce_mean(cross_entropy)\n", 100 | " \n", 101 | " # 损失函数的计算\n", 102 | " regularizer = tf.contrib.layers.l2_regularizer(REGULARAZTION_RATE)\n", 103 | " regularaztion = regularizer(weights1) + regularizer(weights2)\n", 104 | " loss = cross_entropy_mean + regularaztion\n", 105 | " \n", 106 | " # 设置指数衰减的学习率。\n", 107 | " learning_rate = tf.train.exponential_decay(\n", 108 | " LEARNING_RATE_BASE,\n", 109 | " global_step,\n", 110 | " mnist.train.num_examples / BATCH_SIZE,\n", 111 | " LEARNING_RATE_DECAY,\n", 112 | " staircase=True)\n", 113 | " \n", 114 | " # 优化损失函数\n", 115 | " train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=global_step)\n", 116 | " \n", 117 | " # 反向传播更新参数和更新每一个参数的滑动平均值\n", 118 | " with tf.control_dependencies([train_step, variables_averages_op]):\n", 119 | " train_op = tf.no_op(name='train')\n", 120 | "\n", 121 | " # 计算正确率\n", 122 | " correct_prediction = tf.equal(tf.argmax(average_y, 1), tf.argmax(y_, 1))\n", 123 | " accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n", 124 | " \n", 125 | " # 初始化会话，并开始训练过程。\n", 126 | " with tf.Session() as sess:\n", 127 | " tf.global_variables_initializer().run()\n", 128 | " validate_feed = {x: mnist.validation.images, y_: mnist.validation.labels}\n", 129 | " test_feed = {x: mnist.test.images, y_: mnist.test.labels} \n", 130 | " \n", 131 | " # 循环的训练神经网络。\n", 132 | " for i in range(TRAINING_STEPS):\n", 133 | " if i % 1000 == 0:\n", 134 | " validate_acc = sess.run(accuracy, feed_dict=validate_feed)\n", 135 | " print(\"After %d training step(s), validation accuracy using average model is %g \" % (i, validate_acc))\n", 136 | " \n", 137 | " xs,ys=mnist.train.next_batch(BATCH_SIZE)\n", 138 | " sess.run(train_op,feed_dict={x:xs,y_:ys})\n", 139 | "\n", 140 | " test_acc=sess.run(accuracy,feed_dict=test_feed)\n", 141 | " print((\"After %d training step(s), test accuracy using average model is %g\" %(TRAINING_STEPS, test_acc)))\n", 142 | " " 143 | ] 144 | }, 145 | { 146 | "cell_type": "markdown", 147 | "metadata": {}, 148 | "source": [ 149 | "#### 4. 主程序入口，这里设定模型训练次数为5000次。" 150 | ] 151 | }, 152 | { 153 | "cell_type": "code", 154 | "execution_count": 5, 155 | "metadata": {}, 156 | "outputs": [ 157 | { 158 | "name": "stdout", 159 | "output_type": "stream", 160 | "text": [ 161 | "Extracting ../../../datasets/MNIST_data/train-images-idx3-ubyte.gz\n", 162 | "Extracting ../../../datasets/MNIST_data/train-labels-idx1-ubyte.gz\n", 163 | "Extracting ../../../datasets/MNIST_data/t10k-images-idx3-ubyte.gz\n", 164 | "Extracting ../../../datasets/MNIST_data/t10k-labels-idx1-ubyte.gz\n", 165 | "After 0 training step(s), validation accuracy using average model is 0.097 \n", 166 | "After 1000 training step(s), validation accuracy using average model is 0.0958 \n", 167 | "After 2000 training step(s), validation accuracy using average model is 0.0958 \n", 168 | "After 3000 training step(s), validation accuracy using average model is 0.0958 \n", 169 | "After 4000 training step(s), validation accuracy using average model is 0.0958 \n", 170 | "After 5000 training step(s), test accuracy using average model is 0.098\n" 171 | ] 172 | } 173 | ], 174 | "source": [ 175 | "def main(argv=None):\n", 176 | " mnist = input_data.read_data_sets(\"../../datasets/MNIST_data\", one_hot=True)\n", 177 | " train(mnist)\n", 178 | "\n", 179 | "if __name__=='__main__':\n", 180 | " main()" 181 | ] 182 | } 183 | ], 184 | "metadata": { 185 | "kernelspec": { 186 | "display_name": "Python 3", 187 | "language": "python", 188 | "name": "python3" 189 | }, 190 | "language_info": { 191 | "codemirror_mode": { 192 | "name": "ipython", 193 | "version": 3 194 | }, 195 | "file_extension": ".py", 196 | "mimetype": "text/x-python", 197 | "name": "python", 198 | "nbconvert_exporter": "python", 199 | "pygments_lexer": "ipython3", 200 | "version": "3.6.2" 201 | }, 202 | "varInspector": { 203 | "cols": { 204 | "lenName": 16, 205 | "lenType": 16, 206 | "lenVar": 40 207 | }, 208 | "kernels_config": { 209 | "python": { 210 | "delete_cmd_postfix": "", 211 | "delete_cmd_prefix": "del ", 212 | "library": "var_list.py", 213 | "varRefreshCmd": "print(var_dic_list())" 214 | }, 215 | "r": { 216 | "delete_cmd_postfix": ") ", 217 | "delete_cmd_prefix": "rm(", 218 | "library": "var_list.r", 219 | "varRefreshCmd": "cat(var_dic_list()) " 220 | } 221 | }, 222 | "types_to_exclude": [ 223 | "module", 224 | "function", 225 | "builtin_function_or_method", 226 | "instance", 227 | "_Feature" 228 | ], 229 | "window_display": false 230 | } 231 | }, 232 | "nbformat": 4, 233 | "nbformat_minor": 1 234 | } 235 | -------------------------------------------------------------------------------- /Chapter05/2. TensorFlow训练神经网络/5. 不使用隐藏层.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import tensorflow as tf\n", 10 | "from tensorflow.examples.tutorials.mnist import input_data" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": {}, 16 | "source": [ 17 | "#### 1.设置输入和输出节点的个数,配置神经网络的参数。" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 2, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "INPUT_NODE = 784 # 输入节点\n", 27 | "OUTPUT_NODE = 10 # 输出节点\n", 28 | "\n", 29 | "BATCH_SIZE = 100 # 每次batch打包的样本个数 \n", 30 | "\n", 31 | "# 模型相关的参数\n", 32 | "LEARNING_RATE_BASE = 0.8 \n", 33 | "LEARNING_RATE_DECAY = 0.99 \n", 34 | "REGULARAZTION_RATE = 0.0001 \n", 35 | "TRAINING_STEPS = 5000 \n", 36 | "MOVING_AVERAGE_DECAY = 0.99 " 37 | ] 38 | }, 39 | { 40 | "cell_type": "markdown", 41 | "metadata": {}, 42 | "source": [ 43 | "#### 2. 定义辅助函数来计算前向传播结果，使用ReLU做为激活函数。" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": 3, 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "def inference(input_tensor, avg_class, weights1, biases1):\n", 53 | " # 不使用滑动平均类\n", 54 | " if avg_class == None:\n", 55 | " layer1 = tf.nn.relu(tf.matmul(input_tensor, weights1) + biases1)\n", 56 | " return layer1\n", 57 | "\n", 58 | " else:\n", 59 | " # 使用滑动平均类\n", 60 | " layer1 = tf.nn.relu(tf.matmul(input_tensor, avg_class.average(weights1)) + avg_class.average(biases1))\n", 61 | " return layer1" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": {}, 67 | "source": [ 68 | "#### 3. 定义训练过程。" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": 4, 74 | "metadata": {}, 75 | "outputs": [], 76 | "source": [ 77 | "def train(mnist):\n", 78 | " x = tf.placeholder(tf.float32, [None, INPUT_NODE], name='x-input')\n", 79 | " y_ = tf.placeholder(tf.float32, [None, OUTPUT_NODE], name='y-input')\n", 80 | " \n", 81 | " # 生成输出层的参数。\n", 82 | " weights1 = tf.Variable(tf.truncated_normal([INPUT_NODE, OUTPUT_NODE], stddev=0.1))\n", 83 | " biases1 = tf.Variable(tf.constant(0.1, shape=[OUTPUT_NODE]))\n", 84 | "\n", 85 | " # 计算不含滑动平均类的前向传播结果\n", 86 | " y = inference(x, None, weights1, biases1)\n", 87 | " \n", 88 | " # 定义训练轮数及相关的滑动平均类 \n", 89 | " global_step = tf.Variable(0, trainable=False)\n", 90 | " variable_averages = tf.train.ExponentialMovingAverage(MOVING_AVERAGE_DECAY, global_step)\n", 91 | " variables_averages_op = variable_averages.apply(tf.trainable_variables())\n", 92 | " average_y = inference(x, variable_averages, weights1, biases1)\n", 93 | " \n", 94 | " # 计算交叉熵及其平均值\n", 95 | " cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y, labels=tf.argmax(y_, 1))\n", 96 | " cross_entropy_mean = tf.reduce_mean(cross_entropy)\n", 97 | " \n", 98 | " # 损失函数的计算\n", 99 | " regularizer = tf.contrib.layers.l2_regularizer(REGULARAZTION_RATE)\n", 100 | " regularaztion = regularizer(weights1)\n", 101 | " loss = cross_entropy_mean + regularaztion\n", 102 | " \n", 103 | " # 设置指数衰减的学习率。\n", 104 | " learning_rate = tf.train.exponential_decay(\n", 105 | " LEARNING_RATE_BASE,\n", 106 | " global_step,\n", 107 | " mnist.train.num_examples / BATCH_SIZE,\n", 108 | " LEARNING_RATE_DECAY,\n", 109 | " staircase=True)\n", 110 | " \n", 111 | " # 优化损失函数\n", 112 | " train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=global_step)\n", 113 | " \n", 114 | " # 反向传播更新参数和更新每一个参数的滑动平均值\n", 115 | " with tf.control_dependencies([train_step, variables_averages_op]):\n", 116 | " train_op = tf.no_op(name='train')\n", 117 | "\n", 118 | " # 计算正确率\n", 119 | " correct_prediction = tf.equal(tf.argmax(average_y, 1), tf.argmax(y_, 1))\n", 120 | " accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n", 121 | " \n", 122 | " # 初始化会话，并开始训练过程。\n", 123 | " with tf.Session() as sess:\n", 124 | " tf.global_variables_initializer().run()\n", 125 | " validate_feed = {x: mnist.validation.images, y_: mnist.validation.labels}\n", 126 | " test_feed = {x: mnist.test.images, y_: mnist.test.labels} \n", 127 | " \n", 128 | " # 循环的训练神经网络。\n", 129 | " for i in range(TRAINING_STEPS):\n", 130 | " if i % 1000 == 0:\n", 131 | " validate_acc = sess.run(accuracy, feed_dict=validate_feed)\n", 132 | " print(\"After %d training step(s), validation accuracy using average model is %g \" % (i, validate_acc))\n", 133 | " \n", 134 | " xs,ys=mnist.train.next_batch(BATCH_SIZE)\n", 135 | " sess.run(train_op,feed_dict={x:xs,y_:ys})\n", 136 | "\n", 137 | " test_acc=sess.run(accuracy,feed_dict=test_feed)\n", 138 | " print((\"After %d training step(s), test accuracy using average model is %g\" %(TRAINING_STEPS, test_acc)))\n", 139 | " " 140 | ] 141 | }, 142 | { 143 | "cell_type": "markdown", 144 | "metadata": {}, 145 | "source": [ 146 | "#### 4. 主程序入口，这里设定模型训练次数为5000次。" 147 | ] 148 | }, 149 | { 150 | "cell_type": "code", 151 | "execution_count": 5, 152 | "metadata": {}, 153 | "outputs": [ 154 | { 155 | "name": "stdout", 156 | "output_type": "stream", 157 | "text": [ 158 | "Extracting ../../../datasets/MNIST_data/train-images-idx3-ubyte.gz\n", 159 | "Extracting ../../../datasets/MNIST_data/train-labels-idx1-ubyte.gz\n", 160 | "Extracting ../../../datasets/MNIST_data/t10k-images-idx3-ubyte.gz\n", 161 | "Extracting ../../../datasets/MNIST_data/t10k-labels-idx1-ubyte.gz\n", 162 | "After 0 training step(s), validation accuracy using average model is 0.0596 \n", 163 | "After 1000 training step(s), validation accuracy using average model is 0.6604 \n", 164 | "After 2000 training step(s), validation accuracy using average model is 0.665 \n", 165 | "After 3000 training step(s), validation accuracy using average model is 0.666 \n", 166 | "After 4000 training step(s), validation accuracy using average model is 0.6668 \n", 167 | "After 5000 training step(s), test accuracy using average model is 0.6648\n" 168 | ] 169 | } 170 | ], 171 | "source": [ 172 | "def main(argv=None):\n", 173 | " mnist = input_data.read_data_sets(\"../../datasets/MNIST_data\", one_hot=True)\n", 174 | " train(mnist)\n", 175 | "\n", 176 | "if __name__=='__main__':\n", 177 | " main()" 178 | ] 179 | } 180 | ], 181 | "metadata": { 182 | "kernelspec": { 183 | "display_name": "Python 3", 184 | "language": "python", 185 | "name": "python3" 186 | }, 187 | "language_info": { 188 | "codemirror_mode": { 189 | "name": "ipython", 190 | "version": 3 191 | }, 192 | "file_extension": ".py", 193 | "mimetype": "text/x-python", 194 | "name": "python", 195 | "nbconvert_exporter": "python", 196 | "pygments_lexer": "ipython3", 197 | "version": "3.6.2" 198 | }, 199 | "varInspector": { 200 | "cols": { 201 | "lenName": 16, 202 | "lenType": 16, 203 | "lenVar": 40 204 | }, 205 | "kernels_config": { 206 | "python": { 207 | "delete_cmd_postfix": "", 208 | "delete_cmd_prefix": "del ", 209 | "library": "var_list.py", 210 | "varRefreshCmd": "print(var_dic_list())" 211 | }, 212 | "r": { 213 | "delete_cmd_postfix": ") ", 214 | "delete_cmd_prefix": "rm(", 215 | "library": "var_list.r", 216 | "varRefreshCmd": "cat(var_dic_list()) " 217 | } 218 | }, 219 | "types_to_exclude": [ 220 | "module", 221 | "function", 222 | "builtin_function_or_method", 223 | "instance", 224 | "_Feature" 225 | ], 226 | "window_display": false 227 | } 228 | }, 229 | "nbformat": 4, 230 | "nbformat_minor": 1 231 | } 232 | -------------------------------------------------------------------------------- /Chapter05/2. TensorFlow训练神经网络/6. 不使用滑动平均.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import tensorflow as tf\n", 10 | "from tensorflow.examples.tutorials.mnist import input_data" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": {}, 16 | "source": [ 17 | "#### 1.设置输入和输出节点的个数,配置神经网络的参数。" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 2, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "INPUT_NODE = 784 # 输入节点\n", 27 | "OUTPUT_NODE = 10 # 输出节点\n", 28 | "LAYER1_NODE = 500 # 隐藏层数 \n", 29 | " \n", 30 | "BATCH_SIZE = 100 # 每次batch打包的样本个数 \n", 31 | "\n", 32 | "# 模型相关的参数\n", 33 | "LEARNING_RATE_BASE = 0.8 \n", 34 | "LEARNING_RATE_DECAY = 0.99 \n", 35 | "REGULARAZTION_RATE = 0.0001 \n", 36 | "TRAINING_STEPS = 5000 " 37 | ] 38 | }, 39 | { 40 | "cell_type": "markdown", 41 | "metadata": {}, 42 | "source": [ 43 | "#### 2. 定义辅助函数来计算前向传播结果，使用ReLU做为激活函数。" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": 3, 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "def inference(input_tensor, avg_class, weights1, biases1, weights2, biases2):\n", 53 | " # 不使用滑动平均类\n", 54 | " if avg_class == None:\n", 55 | " layer1 = tf.nn.relu(tf.matmul(input_tensor, weights1) + biases1)\n", 56 | " return tf.matmul(layer1, weights2) + biases2\n", 57 | "\n", 58 | " else:\n", 59 | " # 使用滑动平均类\n", 60 | " layer1 = tf.nn.relu(tf.matmul(input_tensor, avg_class.average(weights1)) + avg_class.average(biases1))\n", 61 | " return tf.matmul(layer1, avg_class.average(weights2)) + avg_class.average(biases2) " 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": {}, 67 | "source": [ 68 | "#### 3. 定义训练过程。" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": 4, 74 | "metadata": {}, 75 | "outputs": [], 76 | "source": [ 77 | "def train(mnist):\n", 78 | " x = tf.placeholder(tf.float32, [None, INPUT_NODE], name='x-input')\n", 79 | " y_ = tf.placeholder(tf.float32, [None, OUTPUT_NODE], name='y-input')\n", 80 | " # 生成隐藏层的参数。\n", 81 | " weights1 = tf.Variable(tf.truncated_normal([INPUT_NODE, LAYER1_NODE], stddev=0.1))\n", 82 | " biases1 = tf.Variable(tf.constant(0.1, shape=[LAYER1_NODE]))\n", 83 | " # 生成输出层的参数。\n", 84 | " weights2 = tf.Variable(tf.truncated_normal([LAYER1_NODE, OUTPUT_NODE], stddev=0.1))\n", 85 | " biases2 = tf.Variable(tf.constant(0.1, shape=[OUTPUT_NODE]))\n", 86 | "\n", 87 | " # 计算不含滑动平均类的前向传播结果\n", 88 | " y = inference(x, None, weights1, biases1, weights2, biases2)\n", 89 | " \n", 90 | " # 定义训练轮数及相关的滑动平均类 \n", 91 | " global_step = tf.Variable(0, trainable=False)\n", 92 | " \n", 93 | " # 计算交叉熵及其平均值\n", 94 | " cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y, labels=tf.argmax(y_, 1))\n", 95 | " cross_entropy_mean = tf.reduce_mean(cross_entropy)\n", 96 | " \n", 97 | " # 损失函数的计算\n", 98 | " regularizer = tf.contrib.layers.l2_regularizer(REGULARAZTION_RATE)\n", 99 | " regularaztion = regularizer(weights1) + regularizer(weights2)\n", 100 | " loss = cross_entropy_mean + regularaztion\n", 101 | " \n", 102 | " # 设置指数衰减的学习率。\n", 103 | " learning_rate = tf.train.exponential_decay(\n", 104 | " LEARNING_RATE_BASE,\n", 105 | " global_step,\n", 106 | " mnist.train.num_examples / BATCH_SIZE,\n", 107 | " LEARNING_RATE_DECAY,\n", 108 | " staircase=True)\n", 109 | " \n", 110 | " # 优化损失函数\n", 111 | " train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=global_step)\n", 112 | " \n", 113 | " # 反向传播更新参数\n", 114 | " with tf.control_dependencies([train_step]):\n", 115 | " train_op = tf.no_op(name='train')\n", 116 | "\n", 117 | " # 计算正确率\n", 118 | " correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))\n", 119 | " accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n", 120 | " \n", 121 | " # 初始化会话，并开始训练过程。\n", 122 | " with tf.Session() as sess:\n", 123 | " tf.global_variables_initializer().run()\n", 124 | " validate_feed = {x: mnist.validation.images, y_: mnist.validation.labels}\n", 125 | " test_feed = {x: mnist.test.images, y_: mnist.test.labels} \n", 126 | " \n", 127 | " # 循环的训练神经网络。\n", 128 | " for i in range(TRAINING_STEPS):\n", 129 | " if i % 1000 == 0:\n", 130 | " validate_acc = sess.run(accuracy, feed_dict=validate_feed)\n", 131 | " print(\"After %d training step(s), validation accuracy using average model is %g \" % (i, validate_acc))\n", 132 | " \n", 133 | " xs,ys=mnist.train.next_batch(BATCH_SIZE)\n", 134 | " sess.run(train_op,feed_dict={x:xs,y_:ys})\n", 135 | "\n", 136 | " test_acc=sess.run(accuracy,feed_dict=test_feed)\n", 137 | " print((\"After %d training step(s), test accuracy using average model is %g\" %(TRAINING_STEPS, test_acc)))" 138 | ] 139 | }, 140 | { 141 | "cell_type": "markdown", 142 | "metadata": {}, 143 | "source": [ 144 | "#### 4. 主程序入口，这里设定模型训练次数为5000次。" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": 5, 150 | "metadata": {}, 151 | "outputs": [ 152 | { 153 | "name": "stdout", 154 | "output_type": "stream", 155 | "text": [ 156 | "Extracting ../../datasets/MNIST_data/train-images-idx3-ubyte.gz\n", 157 | "Extracting ../../datasets/MNIST_data/train-labels-idx1-ubyte.gz\n", 158 | "Extracting ../../datasets/MNIST_data/t10k-images-idx3-ubyte.gz\n", 159 | "Extracting ../../datasets/MNIST_data/t10k-labels-idx1-ubyte.gz\n", 160 | "After 0 training step(s), validation accuracy using average model is 0.0572 \n", 161 | "After 1000 training step(s), validation accuracy using average model is 0.9714 \n", 162 | "After 2000 training step(s), validation accuracy using average model is 0.9762 \n", 163 | "After 3000 training step(s), validation accuracy using average model is 0.979 \n", 164 | "After 4000 training step(s), validation accuracy using average model is 0.98 \n", 165 | "After 5000 training step(s), test accuracy using average model is 0.982\n" 166 | ] 167 | } 168 | ], 169 | "source": [ 170 | "def main(argv=None):\n", 171 | " mnist = input_data.read_data_sets(\"../../datasets/MNIST_data\", one_hot=True)\n", 172 | " train(mnist)\n", 173 | "\n", 174 | "if __name__=='__main__':\n", 175 | " main()" 176 | ] 177 | }, 178 | { 179 | "cell_type": "code", 180 | "execution_count": null, 181 | "metadata": { 182 | "collapsed": true 183 | }, 184 | "outputs": [], 185 | "source": [] 186 | } 187 | ], 188 | "metadata": { 189 | "kernelspec": { 190 | "display_name": "Python 3", 191 | "language": "python", 192 | "name": "python3" 193 | }, 194 | "language_info": { 195 | "codemirror_mode": { 196 | "name": "ipython", 197 | "version": 3 198 | }, 199 | "file_extension": ".py", 200 | "mimetype": "text/x-python", 201 | "name": "python", 202 | "nbconvert_exporter": "python", 203 | "pygments_lexer": "ipython3", 204 | "version": "3.6.2" 205 | }, 206 | "varInspector": { 207 | "cols": { 208 | "lenName": 16, 209 | "lenType": 16, 210 | "lenVar": 40 211 | }, 212 | "kernels_config": { 213 | "python": { 214 | "delete_cmd_postfix": "", 215 | "delete_cmd_prefix": "del ", 216 | "library": "var_list.py", 217 | "varRefreshCmd": "print(var_dic_list())" 218 | }, 219 | "r": { 220 | "delete_cmd_postfix": ") ", 221 | "delete_cmd_prefix": "rm(", 222 | "library": "var_list.r", 223 | "varRefreshCmd": "cat(var_dic_list()) " 224 | } 225 | }, 226 | "types_to_exclude": [ 227 | "module", 228 | "function", 229 | "builtin_function_or_method", 230 | "instance", 231 | "_Feature" 232 | ], 233 | "window_display": false 234 | } 235 | }, 236 | "nbformat": 4, 237 | "nbformat_minor": 1 238 | } 239 | -------------------------------------------------------------------------------- /Chapter05/3. 变量管理.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "name": "stdout", 10 | "output_type": "stream", 11 | "text": [ 12 | "var1.name: var1:0\n", 13 | "var2.name: var2:0\n", 14 | "var3.name: var:0\n", 15 | "var4.name: var_1:0\n", 16 | "before update var1: 1.0\n", 17 | "after update var1: 2.5\n", 18 | "before update var2: [ 1.]\n", 19 | "after update var2: [ 3.5]\n", 20 | "var5: [[-0.31131822 1.98459876 0.56532937]\n", 21 | " [-1.9427042 0.59924841 1.09122431]]\n", 22 | "var6: [[ 1. 1.]\n", 23 | " [ 1. 1.]]\n", 24 | "var7: [[ 0.]\n", 25 | " [ 0.]]\n" 26 | ] 27 | } 28 | ], 29 | "source": [ 30 | "import tensorflow as tf\n", 31 | "\n", 32 | "# var1是一个数值\n", 33 | "var1 = tf.get_variable(\"var1\", shape=None, initializer=tf.constant(1.0))\n", 34 | "# var2是一个一维数组\n", 35 | "var2 = tf.Variable(tf.constant(1.0, shape=[1], dtype=tf.float32), name=\"var2\")\n", 36 | "\n", 37 | "#即使设置相同的name，实际上也是不同的变量\n", 38 | "var3 = tf.Variable(tf.constant(1.0, shape=[1], dtype=tf.float32), name=\"var\")\n", 39 | "var4 = tf.Variable(1.0, name=\"var\")\n", 40 | "print(\"var1.name:\", var1.name)\n", 41 | "print(\"var2.name:\", var2.name)\n", 42 | "print(\"var3.name:\", var3.name)\n", 43 | "print(\"var4.name:\", var4.name)\n", 44 | "\n", 45 | "#变量初始化函数\n", 46 | "var5 = tf.Variable(tf.random_normal(shape=[2, 3], mean=0.5, stddev=1.0, seed=1.0), name='var5')\n", 47 | "var6 = tf.Variable(tf.ones(shape=[2, 2]), name='var6')\n", 48 | "var7 = tf.Variable(tf.zeros(shape=[2, 1]), name='var7')\n", 49 | "\n", 50 | "with tf.Session() as sess:\n", 51 | " #变量赋值函数，会自动创建一个operation，必须执行operation，否则不会进行赋值\n", 52 | " # var1是一个数值\n", 53 | " assign_op1 = tf.assign(var1, 2.5)\n", 54 | " # var2是一个一维数组\n", 55 | " assign_op2 = var2.assign([3.5])\n", 56 | "\n", 57 | " sess.run(tf.global_variables_initializer())\n", 58 | " print(\"before update var1:\", var1.eval())\n", 59 | " sess.run(assign_op1)\n", 60 | " print(\"after update var1:\", var1.eval())\n", 61 | " print(\"before update var2:\", var2.eval())\n", 62 | " sess.run(assign_op2)\n", 63 | " print(\"after update var2:\", var2.eval())\n", 64 | " print(\"var5:\", var5.eval())\n", 65 | " print(\"var6:\", var6.eval())\n", 66 | " print(\"var7:\", var7.eval())\n" 67 | ] 68 | }, 69 | { 70 | "cell_type": "markdown", 71 | "metadata": {}, 72 | "source": [ 73 | "#### 1. 在上下文管理器“foo”中创建变量“v”。" 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": 2, 79 | "metadata": { 80 | "code_folding": [] 81 | }, 82 | "outputs": [ 83 | { 84 | "name": "stdout", 85 | "output_type": "stream", 86 | "text": [ 87 | "True\n", 88 | "v.name foo/v:0\n", 89 | "v1.name foo/v:0\n", 90 | "v.name bar/v2:0\n" 91 | ] 92 | } 93 | ], 94 | "source": [ 95 | "with tf.variable_scope(\"foo\"):\n", 96 | " v = tf.get_variable(\"v\", shape=[1], initializer=tf.constant_initializer(1.0))\n", 97 | " \n", 98 | "#with tf.variable_scope(\"foo\"):\n", 99 | " # v = tf.get_variable(\"v\", [1])\n", 100 | " \n", 101 | "with tf.variable_scope(\"foo\", reuse=True):\n", 102 | " v1 = tf.get_variable(\"v\", shape=[1])\n", 103 | "print(v == v1)\n", 104 | "print(\"v.name\", v.name)\n", 105 | "print(\"v1.name\", v1.name)\n", 106 | "\n", 107 | "# 如果reuse设置为True会产生错误\n", 108 | "with tf.variable_scope(\"bar\", reuse=False):\n", 109 | " v = tf.get_variable(\"v2\", shape=[1])\n", 110 | "\n", 111 | "print(\"v.name\", v.name)\n" 112 | ] 113 | }, 114 | { 115 | "cell_type": "markdown", 116 | "metadata": {}, 117 | "source": [ 118 | "#### 2. 嵌套上下文管理器中的reuse参数的使用。" 119 | ] 120 | }, 121 | { 122 | "cell_type": "code", 123 | "execution_count": 3, 124 | "metadata": {}, 125 | "outputs": [ 126 | { 127 | "name": "stdout", 128 | "output_type": "stream", 129 | "text": [ 130 | "False\n", 131 | "True\n", 132 | "True\n", 133 | "False\n" 134 | ] 135 | } 136 | ], 137 | "source": [ 138 | "with tf.variable_scope(\"root\"):\n", 139 | " print(tf.get_variable_scope().reuse)\n", 140 | " \n", 141 | " with tf.variable_scope(\"foo\", reuse=True):\n", 142 | " print(tf.get_variable_scope().reuse)\n", 143 | " \n", 144 | " with tf.variable_scope(\"bar\"):\n", 145 | " print(tf.get_variable_scope().reuse)\n", 146 | " \n", 147 | " print(tf.get_variable_scope().reuse)\n" 148 | ] 149 | }, 150 | { 151 | "cell_type": "markdown", 152 | "metadata": {}, 153 | "source": [ 154 | "#### 3. 通过variable_scope来管理变量。" 155 | ] 156 | }, 157 | { 158 | "cell_type": "code", 159 | "execution_count": 4, 160 | "metadata": {}, 161 | "outputs": [ 162 | { 163 | "name": "stdout", 164 | "output_type": "stream", 165 | "text": [ 166 | "v1.name: v:0\n", 167 | "v2.name: foo/v:0\n", 168 | "v3.name: foo/bar/v:0\n", 169 | "v4.name: v1:0\n" 170 | ] 171 | } 172 | ], 173 | "source": [ 174 | "v1 = tf.get_variable(\"v\", [1])\n", 175 | "print(\"v1.name:\", v1.name)\n", 176 | "\n", 177 | "with tf.variable_scope(\"foo\",reuse=True):\n", 178 | " v2 = tf.get_variable(\"v\", [1])\n", 179 | "print(\"v2.name:\", v2.name)\n", 180 | "\n", 181 | "# 变量范围嵌套\n", 182 | "with tf.variable_scope(\"foo\"):\n", 183 | " with tf.variable_scope(\"bar\"):\n", 184 | " v3 = tf.get_variable(\"v\", [1])\n", 185 | " print(\"v3.name:\", v3.name)\n", 186 | " \n", 187 | "v4 = tf.get_variable(\"v1\", [1])\n", 188 | "print(\"v4.name:\", v4.name)" 189 | ] 190 | }, 191 | { 192 | "cell_type": "markdown", 193 | "metadata": {}, 194 | "source": [ 195 | "#### 4. 我们可以通过变量的名称来获取变量。" 196 | ] 197 | }, 198 | { 199 | "cell_type": "code", 200 | "execution_count": 5, 201 | "metadata": {}, 202 | "outputs": [ 203 | { 204 | "name": "stdout", 205 | "output_type": "stream", 206 | "text": [ 207 | "v5 == v3: True\n", 208 | "v6 == v4: True\n" 209 | ] 210 | } 211 | ], 212 | "source": [ 213 | "with tf.variable_scope(\"\",reuse=True):\n", 214 | " v5 = tf.get_variable(\"foo/bar/v\", [1])\n", 215 | " print(\"v5 == v3:\", v5 == v3)\n", 216 | " v6 = tf.get_variable(\"v1\", [1]) \n", 217 | " print(\"v6 == v4:\", v6 == v4)" 218 | ] 219 | }, 220 | { 221 | "cell_type": "code", 222 | "execution_count": null, 223 | "metadata": { 224 | "collapsed": true 225 | }, 226 | "outputs": [], 227 | "source": [] 228 | } 229 | ], 230 | "metadata": { 231 | "kernelspec": { 232 | "display_name": "Python 3", 233 | "language": "python", 234 | "name": "python3" 235 | }, 236 | "language_info": { 237 | "codemirror_mode": { 238 | "name": "ipython", 239 | "version": 3 240 | }, 241 | "file_extension": ".py", 242 | "mimetype": "text/x-python", 243 | "name": "python", 244 | "nbconvert_exporter": "python", 245 | "pygments_lexer": "ipython3", 246 | "version": "3.6.2" 247 | }, 248 | "varInspector": { 249 | "cols": { 250 | "lenName": 16, 251 | "lenType": 16, 252 | "lenVar": 40 253 | }, 254 | "kernels_config": { 255 | "python": { 256 | "delete_cmd_postfix": "", 257 | "delete_cmd_prefix": "del ", 258 | "library": "var_list.py", 259 | "varRefreshCmd": "print(var_dic_list())" 260 | }, 261 | "r": { 262 | "delete_cmd_postfix": ") ", 263 | "delete_cmd_prefix": "rm(", 264 | "library": "var_list.r", 265 | "varRefreshCmd": "cat(var_dic_list()) " 266 | } 267 | }, 268 | "types_to_exclude": [ 269 | "module", 270 | "function", 271 | "builtin_function_or_method", 272 | "instance", 273 | "_Feature" 274 | ], 275 | "window_display": false 276 | } 277 | }, 278 | "nbformat": 4, 279 | "nbformat_minor": 1 280 | } 281 | -------------------------------------------------------------------------------- /Chapter05/4.1. ckpt文件保存方法.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import tensorflow as tf" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "#### 1. 保存计算两个变量和的模型。" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 2, 24 | "metadata": { 25 | "collapsed": true 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "v1 = tf.Variable(tf.constant(1.0, shape=[1]), name = \"v1\")\n", 30 | "v2 = tf.Variable(tf.constant(2.0, shape=[1]), name = \"v2\")\n", 31 | "result = v1 + v2\n", 32 | "\n", 33 | "init_op = tf.global_variables_initializer()\n", 34 | "# 可以指定saver保存什么变量，如果只保存v1，那么下边要恢复result就会出错，因为缺少v2\n", 35 | "saver = tf.train.Saver()\n", 36 | "\n", 37 | "with tf.Session() as sess:\n", 38 | " sess.run(init_op)\n", 39 | " saver.save(sess, \"Saved_model/model.ckpt\")\n", 40 | " writer = tf.summary.FileWriter('Saved_Graph',graph=tf.get_default_graph())\n", 41 | " writer.close()" 42 | ] 43 | }, 44 | { 45 | "cell_type": "code", 46 | "execution_count": null, 47 | "metadata": {}, 48 | "outputs": [ 49 | { 50 | "name": "stdout", 51 | "output_type": "stream", 52 | "text": [ 53 | "TensorBoard 0.1.4 at http://cookeemMac.local:6006 (Press CTRL+C to quit) " 54 | ] 55 | } 56 | ], 57 | "source": [ 58 | "!tensorboard --logdir=Saved_Graph" 59 | ] 60 | }, 61 | { 62 | "cell_type": "markdown", 63 | "metadata": {}, 64 | "source": [ 65 | "#### 2. 加载保存了两个变量和的模型。" 66 | ] 67 | }, 68 | { 69 | "cell_type": "code", 70 | "execution_count": 3, 71 | "metadata": {}, 72 | "outputs": [ 73 | { 74 | "name": "stdout", 75 | "output_type": "stream", 76 | "text": [ 77 | "INFO:tensorflow:Restoring parameters from Saved_model/model.ckpt\n", 78 | "[ 3.]\n" 79 | ] 80 | } 81 | ], 82 | "source": [ 83 | "with tf.Session() as sess:\n", 84 | " saver.restore(sess, \"Saved_model/model.ckpt\")\n", 85 | " print(result.eval())" 86 | ] 87 | }, 88 | { 89 | "cell_type": "markdown", 90 | "metadata": {}, 91 | "source": [ 92 | "#### 3. 如果不希望重复定义的模型，可以直接加载持久化的图：tf.train.import_meta_graph。" 93 | ] 94 | }, 95 | { 96 | "cell_type": "code", 97 | "execution_count": 4, 98 | "metadata": {}, 99 | "outputs": [ 100 | { 101 | "name": "stdout", 102 | "output_type": "stream", 103 | "text": [ 104 | "INFO:tensorflow:Restoring parameters from Saved_model/model.ckpt\n", 105 | "[ 3.]\n" 106 | ] 107 | } 108 | ], 109 | "source": [ 110 | "saver = tf.train.import_meta_graph(\"Saved_model/model.ckpt.meta\")\n", 111 | "with tf.Session() as sess:\n", 112 | " saver.restore(sess, \"Saved_model/model.ckpt\")\n", 113 | " print(sess.run(tf.get_default_graph().get_tensor_by_name(\"add:0\")))\n", 114 | " " 115 | ] 116 | }, 117 | { 118 | "cell_type": "markdown", 119 | "metadata": {}, 120 | "source": [ 121 | "#### 4. 变量重命名。" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": 5, 127 | "metadata": { 128 | "collapsed": true 129 | }, 130 | "outputs": [], 131 | "source": [ 132 | "v1 = tf.Variable(tf.constant(1.0, shape=[1]), name = \"other-v1\")\n", 133 | "v2 = tf.Variable(tf.constant(2.0, shape=[1]), name = \"other-v2\")\n", 134 | "saver = tf.train.Saver({\"v1\": v1, \"v2\": v2})" 135 | ] 136 | } 137 | ], 138 | "metadata": { 139 | "kernelspec": { 140 | "display_name": "Python 3", 141 | "language": "python", 142 | "name": "python3" 143 | }, 144 | "language_info": { 145 | "codemirror_mode": { 146 | "name": "ipython", 147 | "version": 3 148 | }, 149 | "file_extension": ".py", 150 | "mimetype": "text/x-python", 151 | "name": "python", 152 | "nbconvert_exporter": "python", 153 | "pygments_lexer": "ipython3", 154 | "version": "3.6.2" 155 | }, 156 | "varInspector": { 157 | "cols": { 158 | "lenName": 16, 159 | "lenType": 16, 160 | "lenVar": 40 161 | }, 162 | "kernels_config": { 163 | "python": { 164 | "delete_cmd_postfix": "", 165 | "delete_cmd_prefix": "del ", 166 | "library": "var_list.py", 167 | "varRefreshCmd": "print(var_dic_list())" 168 | }, 169 | "r": { 170 | "delete_cmd_postfix": ") ", 171 | "delete_cmd_prefix": "rm(", 172 | "library": "var_list.r", 173 | "varRefreshCmd": "cat(var_dic_list()) " 174 | } 175 | }, 176 | "types_to_exclude": [ 177 | "module", 178 | "function", 179 | "builtin_function_or_method", 180 | "instance", 181 | "_Feature" 182 | ], 183 | "window_display": false 184 | } 185 | }, 186 | "nbformat": 4, 187 | "nbformat_minor": 1 188 | } 189 | -------------------------------------------------------------------------------- /Chapter05/4.2.1 滑动平均类的保存.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import tensorflow as tf" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "#### 1. 使用滑动平均。" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 2, 24 | "metadata": {}, 25 | "outputs": [ 26 | { 27 | "name": "stdout", 28 | "output_type": "stream", 29 | "text": [ 30 | "variables.name: v:0\n", 31 | "variables.name: v:0\n", 32 | "variables.name: v/ExponentialMovingAverage:0\n" 33 | ] 34 | } 35 | ], 36 | "source": [ 37 | "v = tf.Variable(0, dtype=tf.float32, name=\"v\")\n", 38 | "for variables in tf.global_variables(): \n", 39 | " print(\"variables.name:\", variables.name)\n", 40 | " \n", 41 | "ema = tf.train.ExponentialMovingAverage(0.99)\n", 42 | "maintain_averages_op = ema.apply(tf.global_variables())\n", 43 | "for variables in tf.global_variables(): \n", 44 | " print(\"variables.name:\", variables.name)" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": {}, 50 | "source": [ 51 | "#### 2. 保存滑动平均模型。" 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": 3, 57 | "metadata": {}, 58 | "outputs": [ 59 | { 60 | "name": "stdout", 61 | "output_type": "stream", 62 | "text": [ 63 | "[10.0, 0.099999905]\n" 64 | ] 65 | } 66 | ], 67 | "source": [ 68 | "saver = tf.train.Saver()\n", 69 | "with tf.Session() as sess:\n", 70 | " init_op = tf.global_variables_initializer()\n", 71 | " sess.run(init_op)\n", 72 | " \n", 73 | " sess.run(tf.assign(v, 10))\n", 74 | " sess.run(maintain_averages_op)\n", 75 | " # 保存的时候会将v:0  v/ExponentialMovingAverage:0这两个变量都存下来。\n", 76 | " saver.save(sess, \"Saved_model/model2.ckpt\")\n", 77 | " print(sess.run([v, ema.average(v)]))" 78 | ] 79 | }, 80 | { 81 | "cell_type": "markdown", 82 | "metadata": {}, 83 | "source": [ 84 | "#### 3. 加载滑动平均模型。" 85 | ] 86 | }, 87 | { 88 | "cell_type": "code", 89 | "execution_count": 4, 90 | "metadata": {}, 91 | "outputs": [ 92 | { 93 | "name": "stdout", 94 | "output_type": "stream", 95 | "text": [ 96 | "INFO:tensorflow:Restoring parameters from Saved_model/model2.ckpt\n", 97 | "0.0999999\n" 98 | ] 99 | } 100 | ], 101 | "source": [ 102 | "v = tf.Variable(0, dtype=tf.float32, name=\"v\")\n", 103 | "\n", 104 | "# 通过变量重命名将原来变量v的滑动平均值直接赋值给v。\n", 105 | "saver = tf.train.Saver({\"v/ExponentialMovingAverage\": v})\n", 106 | "with tf.Session() as sess:\n", 107 | " saver.restore(sess, \"Saved_model/model2.ckpt\")\n", 108 | " print(sess.run(v))" 109 | ] 110 | } 111 | ], 112 | "metadata": { 113 | "kernelspec": { 114 | "display_name": "Python 3", 115 | "language": "python", 116 | "name": "python3" 117 | }, 118 | "language_info": { 119 | "codemirror_mode": { 120 | "name": "ipython", 121 | "version": 3 122 | }, 123 | "file_extension": ".py", 124 | "mimetype": "text/x-python", 125 | "name": "python", 126 | "nbconvert_exporter": "python", 127 | "pygments_lexer": "ipython3", 128 | "version": "3.6.2" 129 | }, 130 | "varInspector": { 131 | "cols": { 132 | "lenName": 16, 133 | "lenType": 16, 134 | "lenVar": 40 135 | }, 136 | "kernels_config": { 137 | "python": { 138 | "delete_cmd_postfix": "", 139 | "delete_cmd_prefix": "del ", 140 | "library": "var_list.py", 141 | "varRefreshCmd": "print(var_dic_list())" 142 | }, 143 | "r": { 144 | "delete_cmd_postfix": ") ", 145 | "delete_cmd_prefix": "rm(", 146 | "library": "var_list.r", 147 | "varRefreshCmd": "cat(var_dic_list()) " 148 | } 149 | }, 150 | "types_to_exclude": [ 151 | "module", 152 | "function", 153 | "builtin_function_or_method", 154 | "instance", 155 | "_Feature" 156 | ], 157 | "window_display": false 158 | } 159 | }, 160 | "nbformat": 4, 161 | "nbformat_minor": 1 162 | } 163 | -------------------------------------------------------------------------------- /Chapter05/4.3. pb文件保存方法.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "#### 1. pb文件的保存方法。" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": {}, 14 | "outputs": [ 15 | { 16 | "name": "stdout", 17 | "output_type": "stream", 18 | "text": [ 19 | "INFO:tensorflow:Froze 2 variables.\n", 20 | "Converted 2 variables to const ops.\n" 21 | ] 22 | } 23 | ], 24 | "source": [ 25 | "import tensorflow as tf\n", 26 | "from tensorflow.python.framework import graph_util\n", 27 | "\n", 28 | "v1 = tf.Variable(tf.constant(1.0, shape=[1]), name = \"v1\")\n", 29 | "v2 = tf.Variable(tf.constant(2.0, shape=[1]), name = \"v2\")\n", 30 | "result = v1 + v2\n", 31 | "\n", 32 | "init_op = tf.global_variables_initializer()\n", 33 | "with tf.Session() as sess:\n", 34 | " sess.run(init_op)\n", 35 | " graph_def = tf.get_default_graph().as_graph_def()\n", 36 | " output_graph_def = graph_util.convert_variables_to_constants(sess, graph_def, ['add'])\n", 37 | " with tf.gfile.GFile(\"Saved_model/combined_model.pb\", \"wb\") as f:\n", 38 | " f.write(output_graph_def.SerializeToString())" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": {}, 44 | "source": [ 45 | "#### 2. 加载pb文件。" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 3, 51 | "metadata": {}, 52 | "outputs": [ 53 | { 54 | "name": "stdout", 55 | "output_type": "stream", 56 | "text": [ 57 | "[array([ 3.], dtype=float32)]\n" 58 | ] 59 | } 60 | ], 61 | "source": [ 62 | "from tensorflow.python.platform import gfile\n", 63 | "with tf.Session() as sess:\n", 64 | " model_filename = \"Saved_model/combined_model.pb\"\n", 65 | " \n", 66 | " with gfile.FastGFile(model_filename, 'rb') as f:\n", 67 | " graph_def = tf.GraphDef()\n", 68 | " graph_def.ParseFromString(f.read())\n", 69 | "\n", 70 | " result = tf.import_graph_def(graph_def, return_elements=[\"add:0\"])\n", 71 | " print(sess.run(result))" 72 | ] 73 | } 74 | ], 75 | "metadata": { 76 | "kernelspec": { 77 | "display_name": "Python 3", 78 | "language": "python", 79 | "name": "python3" 80 | }, 81 | "language_info": { 82 | "codemirror_mode": { 83 | "name": "ipython", 84 | "version": 3 85 | }, 86 | "file_extension": ".py", 87 | "mimetype": "text/x-python", 88 | "name": "python", 89 | "nbconvert_exporter": "python", 90 | "pygments_lexer": "ipython3", 91 | "version": "3.6.2" 92 | }, 93 | "varInspector": { 94 | "cols": { 95 | "lenName": 16, 96 | "lenType": 16, 97 | "lenVar": 40 98 | }, 99 | "kernels_config": { 100 | "python": { 101 | "delete_cmd_postfix": "", 102 | "delete_cmd_prefix": "del ", 103 | "library": "var_list.py", 104 | "varRefreshCmd": "print(var_dic_list())" 105 | }, 106 | "r": { 107 | "delete_cmd_postfix": ") ", 108 | "delete_cmd_prefix": "rm(", 109 | "library": "var_list.r", 110 | "varRefreshCmd": "cat(var_dic_list()) " 111 | } 112 | }, 113 | "types_to_exclude": [ 114 | "module", 115 | "function", 116 | "builtin_function_or_method", 117 | "instance", 118 | "_Feature" 119 | ], 120 | "window_display": false 121 | } 122 | }, 123 | "nbformat": 4, 124 | "nbformat_minor": 1 125 | } 126 | -------------------------------------------------------------------------------- /Chapter05/4.4. 恢复变量以及模型.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "#### 1. 通过tf.train.import_meta_graph获取模型" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": {}, 14 | "outputs": [ 15 | { 16 | "name": "stdout", 17 | "output_type": "stream", 18 | "text": [ 19 | "INFO:tensorflow:Restoring parameters from Saved_model/model.ckpt\n", 20 | "v1:0 : [ 1.]\n", 21 | "v2:0 : [ 2.]\n", 22 | "v1 VariableV2 :\n", 23 | "[ 1.]\n", 24 | "##########\n", 25 | "v2 VariableV2 :\n", 26 | "[ 2.]\n", 27 | "##########\n", 28 | "add Add :\n", 29 | "[ 3.]\n", 30 | "##########\n", 31 | "v1: [ 1.]\n", 32 | "v2: [ 2.]\n", 33 | "add: [ 3.]\n" 34 | ] 35 | } 36 | ], 37 | "source": [ 38 | "import tensorflow as tf\n", 39 | "\n", 40 | "saver = tf.train.import_meta_graph(\"Saved_model/model.ckpt.meta\")\n", 41 | "with tf.Session() as sess:\n", 42 | " saver.restore(sess, \"Saved_model/model.ckpt\")\n", 43 | " for v in tf.global_variables():\n", 44 | " print(v.name, \":\", v.eval())\n", 45 | " for op in tf.get_default_graph().get_operations():\n", 46 | " if op.type == \"VariableV2\" or op.type == \"Add\":\n", 47 | " print(op.name, op.type, \":\")\n", 48 | " for o in op.outputs:\n", 49 | " print(o.eval())\n", 50 | " print(\"##########\") \n", 51 | " print(\"v1:\", sess.run(tf.get_default_graph().get_tensor_by_name(\"v1:0\")))\n", 52 | " print(\"v2:\", sess.run(tf.get_default_graph().get_tensor_by_name(\"v2:0\")))\n", 53 | " print(\"add:\", sess.run(tf.get_default_graph().get_tensor_by_name(\"add:0\")))\n" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "#### 2. 通过NewCheckpointReader获取变量" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": 2, 66 | "metadata": {}, 67 | "outputs": [ 68 | { 69 | "name": "stdout", 70 | "output_type": "stream", 71 | "text": [ 72 | "varname: v2 ,varshape: [1] ,value: [ 2.]\n", 73 | "varname: v1 ,varshape: [1] ,value: [ 1.]\n" 74 | ] 75 | } 76 | ], 77 | "source": [ 78 | "import tensorflow as tf\n", 79 | "\n", 80 | "reader = tf.train.NewCheckpointReader(\"Saved_model/model.ckpt\")\n", 81 | "\n", 82 | "# 获取所有变量列表\n", 83 | "all_vars = reader.get_variable_to_shape_map()\n", 84 | "for var_name in all_vars:\n", 85 | " print(\"varname:\", var_name, \",varshape:\", all_vars[var_name], \",value:\", reader.get_tensor(var_name))\n", 86 | " " 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": null, 92 | "metadata": { 93 | "collapsed": true 94 | }, 95 | "outputs": [], 96 | "source": [] 97 | } 98 | ], 99 | "metadata": { 100 | "kernelspec": { 101 | "display_name": "Python 3", 102 | "language": "python", 103 | "name": "python3" 104 | }, 105 | "language_info": { 106 | "codemirror_mode": { 107 | "name": "ipython", 108 | "version": 3 109 | }, 110 | "file_extension": ".py", 111 | "mimetype": "text/x-python", 112 | "name": "python", 113 | "nbconvert_exporter": "python", 114 | "pygments_lexer": "ipython3", 115 | "version": "3.6.2" 116 | }, 117 | "varInspector": { 118 | "cols": { 119 | "lenName": 16, 120 | "lenType": 16, 121 | "lenVar": 40 122 | }, 123 | "kernels_config": { 124 | "python": { 125 | "delete_cmd_postfix": "", 126 | "delete_cmd_prefix": "del ", 127 | "library": "var_list.py", 128 | "varRefreshCmd": "print(var_dic_list())" 129 | }, 130 | "r": { 131 | "delete_cmd_postfix": ") ", 132 | "delete_cmd_prefix": "rm(", 133 | "library": "var_list.r", 134 | "varRefreshCmd": "cat(var_dic_list()) " 135 | } 136 | }, 137 | "types_to_exclude": [ 138 | "module", 139 | "function", 140 | "builtin_function_or_method", 141 | "instance", 142 | "_Feature" 143 | ], 144 | "window_display": false 145 | } 146 | }, 147 | "nbformat": 4, 148 | "nbformat_minor": 2 149 | } 150 | -------------------------------------------------------------------------------- /Chapter05/5. MNIST最佳实践/mnist_inference.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import tensorflow as tf" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "#### 1. 定义神经网络结构相关的参数。" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 2, 24 | "metadata": { 25 | "collapsed": false 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "INPUT_NODE = 784\n", 30 | "OUTPUT_NODE = 10\n", 31 | "LAYER1_NODE = 500" 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": {}, 37 | "source": [ 38 | "#### 2. 通过tf.get_variable函数来获取变量。" 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": 3, 44 | "metadata": { 45 | "collapsed": true 46 | }, 47 | "outputs": [], 48 | "source": [ 49 | "def get_weight_variable(shape, regularizer):\n", 50 | " weights = tf.get_variable(\"weights\", shape, initializer=tf.truncated_normal_initializer(stddev=0.1))\n", 51 | " if regularizer != None: tf.add_to_collection('losses', regularizer(weights))\n", 52 | " return weights" 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "metadata": {}, 58 | "source": [ 59 | "#### 3. 定义神经网络的前向传播过程。" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 4, 65 | "metadata": { 66 | "collapsed": true 67 | }, 68 | "outputs": [], 69 | "source": [ 70 | "def inference(input_tensor, regularizer):\n", 71 | " with tf.variable_scope('layer1'):\n", 72 | "\n", 73 | " weights = get_weight_variable([INPUT_NODE, LAYER1_NODE], regularizer)\n", 74 | " biases = tf.get_variable(\"biases\", [LAYER1_NODE], initializer=tf.constant_initializer(0.0))\n", 75 | " layer1 = tf.nn.relu(tf.matmul(input_tensor, weights) + biases)\n", 76 | "\n", 77 | "\n", 78 | " with tf.variable_scope('layer2'):\n", 79 | " weights = get_weight_variable([LAYER1_NODE, OUTPUT_NODE], regularizer)\n", 80 | " biases = tf.get_variable(\"biases\", [OUTPUT_NODE], initializer=tf.constant_initializer(0.0))\n", 81 | " layer2 = tf.matmul(layer1, weights) + biases\n", 82 | "\n", 83 | " return layer2" 84 | ] 85 | } 86 | ], 87 | "metadata": { 88 | "kernelspec": { 89 | "display_name": "Python 2", 90 | "language": "python", 91 | "name": "python2" 92 | }, 93 | "language_info": { 94 | "codemirror_mode": { 95 | "name": "ipython", 96 | "version": 2 97 | }, 98 | "file_extension": ".py", 99 | "mimetype": "text/x-python", 100 | "name": "python", 101 | "nbconvert_exporter": "python", 102 | "pygments_lexer": "ipython2", 103 | "version": "2.7.10" 104 | } 105 | }, 106 | "nbformat": 4, 107 | "nbformat_minor": 1 108 | } 109 | -------------------------------------------------------------------------------- /Chapter05/5. MNIST最佳实践/mnist_inference.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | 3 | INPUT_NODE = 784 4 | OUTPUT_NODE = 10 5 | LAYER1_NODE = 500 6 | 7 | def get_weight_variable(shape, regularizer): 8 | weights = tf.get_variable("weights", shape, initializer=tf.truncated_normal_initializer(stddev=0.1)) 9 | if regularizer != None: tf.add_to_collection('losses', regularizer(weights)) 10 | return weights 11 | 12 | 13 | def inference(input_tensor, regularizer): 14 | with tf.variable_scope('layer1'): 15 | 16 | weights = get_weight_variable([INPUT_NODE, LAYER1_NODE], regularizer) 17 | biases = tf.get_variable("biases", [LAYER1_NODE], initializer=tf.constant_initializer(0.0)) 18 | layer1 = tf.nn.relu(tf.matmul(input_tensor, weights) + biases) 19 | 20 | with tf.variable_scope('layer2'): 21 | weights = get_weight_variable([LAYER1_NODE, OUTPUT_NODE], regularizer) 22 | biases = tf.get_variable("biases", [OUTPUT_NODE], initializer=tf.constant_initializer(0.0)) 23 | layer2 = tf.matmul(layer1, weights) + biases 24 | 25 | return layer2 -------------------------------------------------------------------------------- /Chapter05/5. MNIST最佳实践/mnist_train.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | from tensorflow.examples.tutorials.mnist import input_data 3 | import mnist_inference 4 | import os 5 | 6 | BATCH_SIZE = 100 7 | LEARNING_RATE_BASE = 0.8 8 | LEARNING_RATE_DECAY = 0.99 9 | REGULARIZATION_RATE = 0.0001 10 | TRAINING_STEPS = 30000 11 | MOVING_AVERAGE_DECAY = 0.99 12 | MODEL_SAVE_PATH="MNIST_model/" 13 | MODEL_NAME="mnist_model" 14 | 15 | 16 | def train(mnist): 17 | 18 | x = tf.placeholder(tf.float32, [None, mnist_inference.INPUT_NODE], name='x-input') 19 | y_ = tf.placeholder(tf.float32, [None, mnist_inference.OUTPUT_NODE], name='y-input') 20 | 21 | regularizer = tf.contrib.layers.l2_regularizer(REGULARIZATION_RATE) 22 | y = mnist_inference.inference(x, regularizer) 23 | global_step = tf.Variable(0, trainable=False) 24 | 25 | 26 | variable_averages = tf.train.ExponentialMovingAverage(MOVING_AVERAGE_DECAY, global_step) 27 | variables_averages_op = variable_averages.apply(tf.trainable_variables()) 28 | cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y, labels=tf.argmax(y_, 1)) 29 | cross_entropy_mean = tf.reduce_mean(cross_entropy) 30 | loss = cross_entropy_mean + tf.add_n(tf.get_collection('losses')) 31 | learning_rate = tf.train.exponential_decay( 32 | LEARNING_RATE_BASE, 33 | global_step, 34 | mnist.train.num_examples / BATCH_SIZE, LEARNING_RATE_DECAY, 35 | staircase=True) 36 | train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=global_step) 37 | with tf.control_dependencies([train_step, variables_averages_op]): 38 | train_op = tf.no_op(name='train') 39 | 40 | 41 | saver = tf.train.Saver() 42 | with tf.Session() as sess: 43 | tf.global_variables_initializer().run() 44 | 45 | for i in range(TRAINING_STEPS): 46 | xs, ys = mnist.train.next_batch(BATCH_SIZE) 47 | _, loss_value, step = sess.run([train_op, loss, global_step], feed_dict={x: xs, y_: ys}) 48 | if i % 1000 == 0: 49 | print("After %d training step(s), loss on training batch is %g." % (step, loss_value)) 50 | saver.save(sess, os.path.join(MODEL_SAVE_PATH, MODEL_NAME), global_step=global_step) 51 | 52 | 53 | def main(argv=None): 54 | mnist = input_data.read_data_sets("../../datasets/MNIST_data", one_hot=True) 55 | train(mnist) 56 | 57 | if __name__ == '__main__': 58 | tf.app.run() 59 | 60 | 61 | -------------------------------------------------------------------------------- /Chapter05/Saved_Graph/events.out.tfevents.1505460689.cookeemMac.local: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cookeem/TensorFlow_learning_notes/187a93bba93b74a44641c9d3f0990adb5236fdbc/Chapter05/Saved_Graph/events.out.tfevents.1505460689.cookeemMac.local -------------------------------------------------------------------------------- /Chapter06/1. 卷积层、池化层样例.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "#### 1. 假设我们输入矩阵\n", 8 | "\n", 9 | "$\n", 10 | "M=\\left(\\begin{array}{c}\n", 11 | "1&-1&0\\\\\n", 12 | "-1&2&1\\\\\n", 13 | "0&2&-2\n", 14 | "\\end{array}\\right)\n", 15 | "$" 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": 2, 21 | "metadata": {}, 22 | "outputs": [ 23 | { 24 | "name": "stdout", 25 | "output_type": "stream", 26 | "text": [ 27 | "Matrix shape is: (3, 3, 1)\n" 28 | ] 29 | } 30 | ], 31 | "source": [ 32 | "import tensorflow as tf\n", 33 | "import numpy as np\n", 34 | "\n", 35 | "M = np.array([\n", 36 | " [[1],[-1],[0]],\n", 37 | " [[-1],[2],[1]],\n", 38 | " [[0],[2],[-2]]\n", 39 | " ])\n", 40 | "\n", 41 | "print(\"Matrix shape is: \",M.shape)" 42 | ] 43 | }, 44 | { 45 | "cell_type": "markdown", 46 | "metadata": {}, 47 | "source": [ 48 | "#### 2. 定义卷积过滤器, 深度为1。\n", 49 | "$\n", 50 | "W=\\left(\\begin{array}{c}\n", 51 | "1&-1\\\\\n", 52 | "0&2\n", 53 | "\\end{array}\\right)\n", 54 | "$" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": 6, 60 | "metadata": { 61 | "collapsed": true 62 | }, 63 | "outputs": [], 64 | "source": [ 65 | "filter_weight = tf.get_variable('weights', shape=[2, 2, 1, 1], initializer = tf.constant_initializer([\n", 66 | " [1, -1],\n", 67 | " [0, 2]]))\n", 68 | "biases = tf.get_variable('biases', shape=[1], initializer = tf.constant_initializer(1))" 69 | ] 70 | }, 71 | { 72 | "cell_type": "markdown", 73 | "metadata": {}, 74 | "source": [ 75 | "#### 3. 调整输入的格式符合TensorFlow的要求。" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": 8, 81 | "metadata": {}, 82 | "outputs": [ 83 | { 84 | "data": { 85 | "text/plain": [ 86 | "array([[[[ 1.],\n", 87 | " [-1.],\n", 88 | " [ 0.]],\n", 89 | "\n", 90 | " [[-1.],\n", 91 | " [ 2.],\n", 92 | " [ 1.]],\n", 93 | "\n", 94 | " [[ 0.],\n", 95 | " [ 2.],\n", 96 | " [-2.]]]], dtype=float32)" 97 | ] 98 | }, 99 | "execution_count": 8, 100 | "metadata": {}, 101 | "output_type": "execute_result" 102 | } 103 | ], 104 | "source": [ 105 | "M = np.asarray(M, dtype='float32')\n", 106 | "M = M.reshape(1, 3, 3, 1)\n", 107 | "M" 108 | ] 109 | }, 110 | { 111 | "cell_type": "markdown", 112 | "metadata": {}, 113 | "source": [ 114 | "#### 4. 计算矩阵通过卷积层过滤器和池化层过滤器计算后的结果。" 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": 9, 120 | "metadata": {}, 121 | "outputs": [ 122 | { 123 | "name": "stdout", 124 | "output_type": "stream", 125 | "text": [ 126 | "convoluted_M: \n", 127 | " [[[[ 7.]\n", 128 | " [ 1.]]\n", 129 | "\n", 130 | " [[-1.]\n", 131 | " [-1.]]]]\n", 132 | "pooled_M: \n", 133 | " [[[[ 0.25]\n", 134 | " [ 0.5 ]]\n", 135 | "\n", 136 | " [[ 1. ]\n", 137 | " [-2. ]]]]\n" 138 | ] 139 | } 140 | ], 141 | "source": [ 142 | "x = tf.placeholder('float32', [1, None, None, 1])\n", 143 | "conv = tf.nn.conv2d(x, filter_weight, strides = [1, 2, 2, 1], padding = 'SAME')\n", 144 | "bias = tf.nn.bias_add(conv, biases)\n", 145 | "pool = tf.nn.avg_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')\n", 146 | "with tf.Session() as sess:\n", 147 | " tf.global_variables_initializer().run()\n", 148 | " convoluted_M = sess.run(bias,feed_dict={x:M})\n", 149 | " pooled_M = sess.run(pool,feed_dict={x:M})\n", 150 | " \n", 151 | " print(\"convoluted_M: \\n\", convoluted_M)\n", 152 | " print(\"pooled_M: \\n\", pooled_M)" 153 | ] 154 | } 155 | ], 156 | "metadata": { 157 | "kernelspec": { 158 | "display_name": "Python 3", 159 | "language": "python", 160 | "name": "python3" 161 | }, 162 | "language_info": { 163 | "codemirror_mode": { 164 | "name": "ipython", 165 | "version": 3 166 | }, 167 | "file_extension": ".py", 168 | "mimetype": "text/x-python", 169 | "name": "python", 170 | "nbconvert_exporter": "python", 171 | "pygments_lexer": "ipython3", 172 | "version": "3.6.2" 173 | }, 174 | "varInspector": { 175 | "cols": { 176 | "lenName": 16, 177 | "lenType": 16, 178 | "lenVar": 40 179 | }, 180 | "kernels_config": { 181 | "python": { 182 | "delete_cmd_postfix": "", 183 | "delete_cmd_prefix": "del ", 184 | "library": "var_list.py", 185 | "varRefreshCmd": "print(var_dic_list())" 186 | }, 187 | "r": { 188 | "delete_cmd_postfix": ") ", 189 | "delete_cmd_prefix": "rm(", 190 | "library": "var_list.r", 191 | "varRefreshCmd": "cat(var_dic_list()) " 192 | } 193 | }, 194 | "types_to_exclude": [ 195 | "module", 196 | "function", 197 | "builtin_function_or_method", 198 | "instance", 199 | "_Feature" 200 | ], 201 | "window_display": false 202 | } 203 | }, 204 | "nbformat": 4, 205 | "nbformat_minor": 1 206 | } 207 | -------------------------------------------------------------------------------- /Chapter06/1.2. MNIST卷积和池化.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "from tensorflow.examples.tutorials.mnist import input_data\n", 12 | "import tensorflow as tf" 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": 2, 18 | "metadata": { 19 | "collapsed": true 20 | }, 21 | "outputs": [], 22 | "source": [ 23 | "def deepnn(x):\n", 24 | " with tf.name_scope('reshape'):\n", 25 | " x_image = tf.reshape(x, [-1, 28, 28, 1])\n", 26 | "\n", 27 | " # First convolutional layer - maps one grayscale image to 32 feature maps.\n", 28 | " with tf.name_scope('conv1'):\n", 29 | " W_conv1 = weight_variable([5, 5, 1, 32])\n", 30 | " b_conv1 = bias_variable([32])\n", 31 | " h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)\n", 32 | "\n", 33 | " # Pooling layer - downsamples by 2X.\n", 34 | " with tf.name_scope('pool1'):\n", 35 | " h_pool1 = max_pool_2x2(h_conv1)\n", 36 | "\n", 37 | " # Second convolutional layer -- maps 32 feature maps to 64.\n", 38 | " with tf.name_scope('conv2'):\n", 39 | " W_conv2 = weight_variable([5, 5, 32, 64])\n", 40 | " b_conv2 = bias_variable([64])\n", 41 | " h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)\n", 42 | "\n", 43 | " # Second pooling layer.\n", 44 | " with tf.name_scope('pool2'):\n", 45 | " h_pool2 = max_pool_2x2(h_conv2)\n", 46 | "\n", 47 | " # Fully connected layer 1 -- after 2 round of downsampling, our 28x28 image\n", 48 | " # is down to 7x7x64 feature maps -- maps this to 1024 features.\n", 49 | " with tf.name_scope('fc1'):\n", 50 | " W_fc1 = weight_variable([7 * 7 * 64, 1024])\n", 51 | " b_fc1 = bias_variable([1024])\n", 52 | "\n", 53 | " h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])\n", 54 | " h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)\n", 55 | "\n", 56 | " # Dropout - controls the complexity of the model, prevents co-adaptation of\n", 57 | " # features.\n", 58 | " with tf.name_scope('dropout'):\n", 59 | " keep_prob = tf.placeholder(tf.float32)\n", 60 | " h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)\n", 61 | "\n", 62 | " # Map the 1024 features to 10 classes, one for each digit\n", 63 | " with tf.name_scope('fc2'):\n", 64 | " W_fc2 = weight_variable([1024, 10])\n", 65 | " b_fc2 = bias_variable([10])\n", 66 | "\n", 67 | " y_conv = tf.matmul(h_fc1_drop, W_fc2) + b_fc2\n", 68 | " return y_conv, keep_prob\n", 69 | "\n", 70 | "\n", 71 | "def conv2d(x, W):\n", 72 | " \"\"\"conv2d returns a 2d convolution layer with full stride.\"\"\"\n", 73 | " return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')\n", 74 | "\n", 75 | "\n", 76 | "def max_pool_2x2(x):\n", 77 | " \"\"\"max_pool_2x2 downsamples a feature map by 2X.\"\"\"\n", 78 | " return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],\n", 79 | " strides=[1, 2, 2, 1], padding='SAME')\n", 80 | "\n", 81 | "\n", 82 | "def weight_variable(shape):\n", 83 | " \"\"\"weight_variable generates a weight variable of a given shape.\"\"\"\n", 84 | " initial = tf.truncated_normal(shape, stddev=0.1)\n", 85 | " return tf.Variable(initial)\n", 86 | "\n", 87 | "\n", 88 | "def bias_variable(shape):\n", 89 | " \"\"\"bias_variable generates a bias variable of a given shape.\"\"\"\n", 90 | " initial = tf.constant(0.1, shape=shape)\n", 91 | " return tf.Variable(initial)" 92 | ] 93 | }, 94 | { 95 | "cell_type": "code", 96 | "execution_count": 3, 97 | "metadata": { 98 | "collapsed": true 99 | }, 100 | "outputs": [], 101 | "source": [ 102 | "def main():\n", 103 | " # Import data\n", 104 | " mnist = input_data.read_data_sets(\"../datasets/MNIST_data\", one_hot=True)\n", 105 | " # Create the model\n", 106 | " x = tf.placeholder(tf.float32, [None, 784])\n", 107 | "\n", 108 | " # Define loss and optimizer\n", 109 | " y_ = tf.placeholder(tf.float32, [None, 10])\n", 110 | "\n", 111 | " # Build the graph for the deep net\n", 112 | " y_conv, keep_prob = deepnn(x)\n", 113 | "\n", 114 | " with tf.name_scope('loss'):\n", 115 | " cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=y_,\n", 116 | " logits=y_conv)\n", 117 | " cross_entropy = tf.reduce_mean(cross_entropy)\n", 118 | "\n", 119 | " with tf.name_scope('adam_optimizer'):\n", 120 | " train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)\n", 121 | "\n", 122 | " with tf.name_scope('accuracy'):\n", 123 | " correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))\n", 124 | " correct_prediction = tf.cast(correct_prediction, tf.float32)\n", 125 | " accuracy = tf.reduce_mean(correct_prediction)\n", 126 | "\n", 127 | " train_writer = tf.summary.FileWriter(\"model\")\n", 128 | " train_writer.add_graph(tf.get_default_graph())\n", 129 | "\n", 130 | " with tf.Session() as sess:\n", 131 | " sess.run(tf.global_variables_initializer())\n", 132 | " for i in range(1000):\n", 133 | " batch = mnist.train.next_batch(50)\n", 134 | " if i % 100 == 0:\n", 135 | " train_accuracy = accuracy.eval(feed_dict={\n", 136 | " x: batch[0], y_: batch[1], keep_prob: 1.0})\n", 137 | " print('step %d, training accuracy %g' % (i, train_accuracy))\n", 138 | " train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})\n", 139 | "\n", 140 | " print('test accuracy %g' % accuracy.eval(feed_dict={\n", 141 | " x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))" 142 | ] 143 | }, 144 | { 145 | "cell_type": "code", 146 | "execution_count": 4, 147 | "metadata": {}, 148 | "outputs": [ 149 | { 150 | "name": "stdout", 151 | "output_type": "stream", 152 | "text": [ 153 | "Extracting ../datasets/MNIST_data/train-images-idx3-ubyte.gz\n", 154 | "Extracting ../datasets/MNIST_data/train-labels-idx1-ubyte.gz\n", 155 | "Extracting ../datasets/MNIST_data/t10k-images-idx3-ubyte.gz\n", 156 | "Extracting ../datasets/MNIST_data/t10k-labels-idx1-ubyte.gz\n", 157 | "step 0, training accuracy 0.18\n", 158 | "step 100, training accuracy 0.76\n", 159 | "step 200, training accuracy 0.92\n", 160 | "step 300, training accuracy 1\n", 161 | "step 400, training accuracy 0.94\n", 162 | "step 500, training accuracy 0.94\n", 163 | "step 600, training accuracy 0.96\n", 164 | "step 700, training accuracy 0.94\n", 165 | "step 800, training accuracy 0.96\n", 166 | "step 900, training accuracy 0.96\n", 167 | "test accuracy 0.962\n" 168 | ] 169 | } 170 | ], 171 | "source": [ 172 | "if __name__ == '__main__':\n", 173 | " main()" 174 | ] 175 | } 176 | ], 177 | "metadata": { 178 | "kernelspec": { 179 | "display_name": "Python 3", 180 | "language": "python", 181 | "name": "python3" 182 | }, 183 | "varInspector": { 184 | "cols": { 185 | "lenName": 16, 186 | "lenType": 16, 187 | "lenVar": 40 188 | }, 189 | "kernels_config": { 190 | "python": { 191 | "delete_cmd_postfix": "", 192 | "delete_cmd_prefix": "del ", 193 | "library": "var_list.py", 194 | "varRefreshCmd": "print(var_dic_list())" 195 | }, 196 | "r": { 197 | "delete_cmd_postfix": ") ", 198 | "delete_cmd_prefix": "rm(", 199 | "library": "var_list.r", 200 | "varRefreshCmd": "cat(var_dic_list()) " 201 | } 202 | }, 203 | "types_to_exclude": [ 204 | "module", 205 | "function", 206 | "builtin_function_or_method", 207 | "instance", 208 | "_Feature" 209 | ], 210 | "window_display": false 211 | } 212 | }, 213 | "nbformat": 4, 214 | "nbformat_minor": 2 215 | } 216 | -------------------------------------------------------------------------------- /Chapter06/LeNet-5/LeNet5_infernece.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import tensorflow as tf" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "#### 1. 设定神经网络的参数" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 2, 24 | "metadata": { 25 | "collapsed": true 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "# 输入为28*28的图像[28, 28]\n", 30 | "INPUT_NODE = 784\n", 31 | "# 输出为1~10的可能性[10]\n", 32 | "OUTPUT_NODE = 10\n", 33 | "\n", 34 | "# 图像尺寸\n", 35 | "IMAGE_SIZE = 28\n", 36 | "# 图像的颜色通道数，这里只有黑白一种通道\n", 37 | "NUM_CHANNELS = 1\n", 38 | "# 标签的数量\n", 39 | "NUM_LABELS = 10\n", 40 | "\n", 41 | "# 第一层卷积的深度\n", 42 | "CONV1_DEEP = 32\n", 43 | "# 第一层卷积的过滤器尺寸\n", 44 | "CONV1_SIZE = 5\n", 45 | "\n", 46 | "# 第二层卷积的深度\n", 47 | "CONV2_DEEP = 64\n", 48 | "# 第二层卷积的过滤器尺寸\n", 49 | "CONV2_SIZE = 5\n", 50 | "\n", 51 | "# 全连接层的节点个数\n", 52 | "FC_SIZE = 512\n", 53 | "\n", 54 | "\n", 55 | "# 常见的卷积模型\n", 56 | "# 本例子卷积模型 输入 -> 卷积层 -> 池化层 -> 卷积层 -> 池化层 -> 全连接层 -> 全连接层\n", 57 | "# 输入 -> (卷积层+ -> 池化层?)+ -> 全连接层+" 58 | ] 59 | }, 60 | { 61 | "cell_type": "markdown", 62 | "metadata": {}, 63 | "source": [ 64 | "#### 2. 定义前向传播的过程" 65 | ] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": 3, 70 | "metadata": { 71 | "collapsed": true 72 | }, 73 | "outputs": [], 74 | "source": [ 75 | "def inference(input_tensor, train, regularizer):\n", 76 | " # 第一层卷积1\n", 77 | " # 输入为[x-size=28, y-size=28, channel=1]的图像\n", 78 | " # 过滤器尺寸[x-size=5, y-size=5, channel=1, deep=32]\n", 79 | " # 过滤器步长=1\n", 80 | " # 输出为[x-size=28, y-size=28, deep=32]的矩阵\n", 81 | " with tf.variable_scope('layer1-conv1'):\n", 82 | " conv1_weights = tf.get_variable(\n", 83 | " name=\"weight\", \n", 84 | " shape=[CONV1_SIZE, CONV1_SIZE, NUM_CHANNELS, CONV1_DEEP],\n", 85 | " initializer=tf.truncated_normal_initializer(stddev=0.1)\n", 86 | " )\n", 87 | " conv1_biases = tf.get_variable(\n", 88 | " name=\"bias\", \n", 89 | " shape=[CONV1_DEEP], \n", 90 | " initializer=tf.constant_initializer(0.0)\n", 91 | " )\n", 92 | " conv1 = tf.nn.conv2d(input_tensor, conv1_weights, strides=[1, 1, 1, 1], padding='SAME')\n", 93 | " relu1 = tf.nn.relu(tf.nn.bias_add(conv1, conv1_biases))\n", 94 | "\n", 95 | " # 第二层池化1\n", 96 | " # 输入为[x-size=28, y-size=28, deep=32]的矩阵\n", 97 | " # 过滤器尺寸[x-size=2, y-size=2]\n", 98 | " # 过滤器步长=2\n", 99 | " # 输出为[x-size=14, y-size=14, deep=32]的矩阵\n", 100 | " with tf.name_scope(\"layer2-pool1\"):\n", 101 | " pool1 = tf.nn.max_pool(relu1, ksize = [1,2,2,1],strides=[1,2,2,1],padding=\"SAME\")\n", 102 | "\n", 103 | " # 第三层卷积2\n", 104 | " # 输入为[x-size=14, y-size=14, deep=32]的矩阵\n", 105 | " # 过滤器尺寸[x-size=5, y-size=5, channel=1, deep=64]\n", 106 | " # 过滤器步长=1\n", 107 | " # 输出为[x-size=14, y-size=14, deep=64]的矩阵\n", 108 | " with tf.variable_scope(\"layer3-conv2\"):\n", 109 | " conv2_weights = tf.get_variable(\n", 110 | " name=\"weight\", \n", 111 | " shape=[CONV2_SIZE, CONV2_SIZE, CONV1_DEEP, CONV2_DEEP],\n", 112 | " initializer=tf.truncated_normal_initializer(stddev=0.1)\n", 113 | " )\n", 114 | " conv2_biases = tf.get_variable(\n", 115 | " name=\"bias\", \n", 116 | " shape=[CONV2_DEEP], \n", 117 | " initializer=tf.constant_initializer(0.0)\n", 118 | " )\n", 119 | " conv2 = tf.nn.conv2d(pool1, conv2_weights, strides=[1, 1, 1, 1], padding='SAME')\n", 120 | " relu2 = tf.nn.relu(tf.nn.bias_add(conv2, conv2_biases))\n", 121 | "\n", 122 | " # 第四层池化2\n", 123 | " # 输入为[x-size=14, y-size=14, deep=64]的矩阵\n", 124 | " # 过滤器尺寸[x-size=2, y-size=2]\n", 125 | " # 过滤器步长=2\n", 126 | " # 输出为[x-size=7, y-size=7, deep=64]的矩阵\n", 127 | " with tf.name_scope(\"layer4-pool2\"):\n", 128 | " pool2 = tf.nn.max_pool(relu2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')\n", 129 | " # 把[batch, x-size, y-size, deep]4维矩阵转化为[batch, vector]2维矩阵，长*宽*深度转换为1维向量\n", 130 | " pool_shape = pool2.get_shape().as_list()\n", 131 | " nodes = pool_shape[1] * pool_shape[2] * pool_shape[3]\n", 132 | " reshaped = tf.reshape(pool2, [pool_shape[0], nodes])\n", 133 | "\n", 134 | " # 全连接层 \n", 135 | " with tf.variable_scope('layer5-fc1'):\n", 136 | " fc1_weights = tf.get_variable(\n", 137 | " name=\"weight\", \n", 138 | " shape=[nodes, FC_SIZE],\n", 139 | " initializer=tf.truncated_normal_initializer(stddev=0.1)\n", 140 | " )\n", 141 | " # 只有全连接的权重需要加入正则化\n", 142 | " if regularizer != None: tf.add_to_collection('losses', regularizer(fc1_weights))\n", 143 | " fc1_biases = tf.get_variable(\"bias\", [FC_SIZE], initializer=tf.constant_initializer(0.1))\n", 144 | "\n", 145 | " fc1 = tf.nn.relu(tf.matmul(reshaped, fc1_weights) + fc1_biases)\n", 146 | " # dropout在训练数据的时候，会随机把部分输出改为0\n", 147 | " # dropout可以避免过度拟合，dropout一般只在全连接层，而不是在卷积层或者池化层使用\n", 148 | " if train: fc1 = tf.nn.dropout(fc1, 0.5)\n", 149 | "\n", 150 | " # 全连接层\n", 151 | " # 输入为[512]的向量\n", 152 | " # 输出为[10]的向量\n", 153 | " with tf.variable_scope('layer6-fc2'):\n", 154 | " fc2_weights = tf.get_variable(\n", 155 | " name=\"weight\", \n", 156 | " shape=[FC_SIZE, NUM_LABELS],\n", 157 | " initializer=tf.truncated_normal_initializer(stddev=0.1)\n", 158 | " )\n", 159 | " if regularizer != None: tf.add_to_collection('losses', regularizer(fc2_weights))\n", 160 | " fc2_biases = tf.get_variable(\"bias\", [NUM_LABELS], initializer=tf.constant_initializer(0.1))\n", 161 | " logit = tf.matmul(fc1, fc2_weights) + fc2_biases\n", 162 | "\n", 163 | " return logit" 164 | ] 165 | } 166 | ], 167 | "metadata": { 168 | "kernelspec": { 169 | "display_name": "Python 3", 170 | "language": "python", 171 | "name": "python3" 172 | }, 173 | "language_info": { 174 | "codemirror_mode": { 175 | "name": "ipython", 176 | "version": 3 177 | }, 178 | "file_extension": ".py", 179 | "mimetype": "text/x-python", 180 | "name": "python", 181 | "nbconvert_exporter": "python", 182 | "pygments_lexer": "ipython3", 183 | "version": "3.6.2" 184 | }, 185 | "varInspector": { 186 | "cols": { 187 | "lenName": 16, 188 | "lenType": 16, 189 | "lenVar": 40 190 | }, 191 | "kernels_config": { 192 | "python": { 193 | "delete_cmd_postfix": "", 194 | "delete_cmd_prefix": "del ", 195 | "library": "var_list.py", 196 | "varRefreshCmd": "print(var_dic_list())" 197 | }, 198 | "r": { 199 | "delete_cmd_postfix": ") ", 200 | "delete_cmd_prefix": "rm(", 201 | "library": "var_list.r", 202 | "varRefreshCmd": "cat(var_dic_list()) " 203 | } 204 | }, 205 | "types_to_exclude": [ 206 | "module", 207 | "function", 208 | "builtin_function_or_method", 209 | "instance", 210 | "_Feature" 211 | ], 212 | "window_display": false 213 | } 214 | }, 215 | "nbformat": 4, 216 | "nbformat_minor": 1 217 | } 218 | -------------------------------------------------------------------------------- /Chapter06/LeNet-5/LeNet5_infernece.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | 3 | INPUT_NODE = 784 4 | OUTPUT_NODE = 10 5 | 6 | IMAGE_SIZE = 28 7 | NUM_CHANNELS = 1 8 | NUM_LABELS = 10 9 | 10 | CONV1_DEEP = 32 11 | CONV1_SIZE = 5 12 | 13 | CONV2_DEEP = 64 14 | CONV2_SIZE = 5 15 | 16 | FC_SIZE = 512 17 | 18 | def inference(input_tensor, train, regularizer): 19 | with tf.variable_scope('layer1-conv1'): 20 | conv1_weights = tf.get_variable( 21 | "weight", [CONV1_SIZE, CONV1_SIZE, NUM_CHANNELS, CONV1_DEEP], 22 | initializer=tf.truncated_normal_initializer(stddev=0.1)) 23 | conv1_biases = tf.get_variable("bias", [CONV1_DEEP], initializer=tf.constant_initializer(0.0)) 24 | conv1 = tf.nn.conv2d(input_tensor, conv1_weights, strides=[1, 1, 1, 1], padding='SAME') 25 | relu1 = tf.nn.relu(tf.nn.bias_add(conv1, conv1_biases)) 26 | 27 | with tf.name_scope("layer2-pool1"): 28 | pool1 = tf.nn.max_pool(relu1, ksize = [1,2,2,1],strides=[1,2,2,1],padding="SAME") 29 | 30 | with tf.variable_scope("layer3-conv2"): 31 | conv2_weights = tf.get_variable( 32 | "weight", [CONV2_SIZE, CONV2_SIZE, CONV1_DEEP, CONV2_DEEP], 33 | initializer=tf.truncated_normal_initializer(stddev=0.1)) 34 | conv2_biases = tf.get_variable("bias", [CONV2_DEEP], initializer=tf.constant_initializer(0.0)) 35 | conv2 = tf.nn.conv2d(pool1, conv2_weights, strides=[1, 1, 1, 1], padding='SAME') 36 | relu2 = tf.nn.relu(tf.nn.bias_add(conv2, conv2_biases)) 37 | 38 | with tf.name_scope("layer4-pool2"): 39 | pool2 = tf.nn.max_pool(relu2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME') 40 | pool_shape = pool2.get_shape().as_list() 41 | nodes = pool_shape[1] * pool_shape[2] * pool_shape[3] 42 | reshaped = tf.reshape(pool2, [pool_shape[0], nodes]) 43 | 44 | with tf.variable_scope('layer5-fc1'): 45 | fc1_weights = tf.get_variable("weight", [nodes, FC_SIZE], 46 | initializer=tf.truncated_normal_initializer(stddev=0.1)) 47 | if regularizer != None: tf.add_to_collection('losses', regularizer(fc1_weights)) 48 | fc1_biases = tf.get_variable("bias", [FC_SIZE], initializer=tf.constant_initializer(0.1)) 49 | 50 | fc1 = tf.nn.relu(tf.matmul(reshaped, fc1_weights) + fc1_biases) 51 | if train: fc1 = tf.nn.dropout(fc1, 0.5) 52 | 53 | with tf.variable_scope('layer6-fc2'): 54 | fc2_weights = tf.get_variable("weight", [FC_SIZE, NUM_LABELS], 55 | initializer=tf.truncated_normal_initializer(stddev=0.1)) 56 | if regularizer != None: tf.add_to_collection('losses', regularizer(fc2_weights)) 57 | fc2_biases = tf.get_variable("bias", [NUM_LABELS], initializer=tf.constant_initializer(0.1)) 58 | logit = tf.matmul(fc1, fc2_weights) + fc2_biases 59 | 60 | return logit -------------------------------------------------------------------------------- /Chapter06/LeNet-5/LeNet5_train.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import tensorflow as tf\n", 12 | "from tensorflow.examples.tutorials.mnist import input_data\n", 13 | "import LeNet5_infernece\n", 14 | "import os\n", 15 | "import numpy as np" 16 | ] 17 | }, 18 | { 19 | "cell_type": "markdown", 20 | "metadata": { 21 | "collapsed": true 22 | }, 23 | "source": [ 24 | "#### 1. 定义神经网络相关的参数" 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": 2, 30 | "metadata": { 31 | "collapsed": true 32 | }, 33 | "outputs": [], 34 | "source": [ 35 | "BATCH_SIZE = 100\n", 36 | "LEARNING_RATE_BASE = 0.01\n", 37 | "LEARNING_RATE_DECAY = 0.99\n", 38 | "REGULARIZATION_RATE = 0.0001\n", 39 | "TRAINING_STEPS = 6000\n", 40 | "MOVING_AVERAGE_DECAY = 0.99" 41 | ] 42 | }, 43 | { 44 | "cell_type": "markdown", 45 | "metadata": {}, 46 | "source": [ 47 | "#### 2. 定义训练过程" 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": null, 53 | "metadata": { 54 | "collapsed": true 55 | }, 56 | "outputs": [], 57 | "source": [ 58 | "def train(mnist):\n", 59 | " # 定义输出为4维矩阵的placeholder\n", 60 | " x = tf.placeholder(tf.float32, [\n", 61 | " BATCH_SIZE,\n", 62 | " LeNet5_infernece.IMAGE_SIZE,\n", 63 | " LeNet5_infernece.IMAGE_SIZE,\n", 64 | " LeNet5_infernece.NUM_CHANNELS],\n", 65 | " name='x-input')\n", 66 | " y_ = tf.placeholder(tf.float32, [None, LeNet5_infernece.OUTPUT_NODE], name='y-input')\n", 67 | " \n", 68 | " regularizer = tf.contrib.layers.l2_regularizer(REGULARIZATION_RATE)\n", 69 | " y = LeNet5_infernece.inference(x,False,regularizer)\n", 70 | " global_step = tf.Variable(0, trainable=False)\n", 71 | "\n", 72 | " # 定义损失函数、学习率、滑动平均操作以及训练过程。\n", 73 | " variable_averages = tf.train.ExponentialMovingAverage(MOVING_AVERAGE_DECAY, global_step)\n", 74 | " variables_averages_op = variable_averages.apply(tf.trainable_variables())\n", 75 | " cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y, labels=tf.argmax(y_, 1))\n", 76 | " cross_entropy_mean = tf.reduce_mean(cross_entropy)\n", 77 | " loss = cross_entropy_mean + tf.add_n(tf.get_collection('losses'))\n", 78 | " learning_rate = tf.train.exponential_decay(\n", 79 | " LEARNING_RATE_BASE,\n", 80 | " global_step,\n", 81 | " mnist.train.num_examples / BATCH_SIZE, LEARNING_RATE_DECAY,\n", 82 | " staircase=True)\n", 83 | "\n", 84 | " train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=global_step)\n", 85 | " with tf.control_dependencies([train_step, variables_averages_op]):\n", 86 | " train_op = tf.no_op(name='train')\n", 87 | " \n", 88 | " # 初始化TensorFlow持久化类。\n", 89 | " saver = tf.train.Saver()\n", 90 | " with tf.Session() as sess:\n", 91 | " tf.global_variables_initializer().run()\n", 92 | " for i in range(TRAINING_STEPS):\n", 93 | " xs, ys = mnist.train.next_batch(BATCH_SIZE)\n", 94 | "\n", 95 | " reshaped_xs = np.reshape(xs, (\n", 96 | " BATCH_SIZE,\n", 97 | " LeNet5_infernece.IMAGE_SIZE,\n", 98 | " LeNet5_infernece.IMAGE_SIZE,\n", 99 | " LeNet5_infernece.NUM_CHANNELS))\n", 100 | " _, loss_value, step = sess.run([train_op, loss, global_step], feed_dict={x: reshaped_xs, y_: ys})\n", 101 | "\n", 102 | " if i % 50 == 0:\n", 103 | " print(\"After %d training step(s), loss on training batch is %g.\" % (step, loss_value))" 104 | ] 105 | }, 106 | { 107 | "cell_type": "markdown", 108 | "metadata": {}, 109 | "source": [ 110 | "#### 3. 主程序入口" 111 | ] 112 | }, 113 | { 114 | "cell_type": "code", 115 | "execution_count": null, 116 | "metadata": {}, 117 | "outputs": [ 118 | { 119 | "name": "stdout", 120 | "output_type": "stream", 121 | "text": [ 122 | "Extracting ../../datasets/MNIST_data/train-images-idx3-ubyte.gz\n", 123 | "Extracting ../../datasets/MNIST_data/train-labels-idx1-ubyte.gz\n", 124 | "Extracting ../../datasets/MNIST_data/t10k-images-idx3-ubyte.gz\n", 125 | "Extracting ../../datasets/MNIST_data/t10k-labels-idx1-ubyte.gz\n", 126 | "After 1 training step(s), loss on training batch is 4.26947.\n", 127 | "After 101 training step(s), loss on training batch is 1.01421.\n", 128 | "After 201 training step(s), loss on training batch is 0.928992.\n", 129 | "After 301 training step(s), loss on training batch is 0.810302.\n" 130 | ] 131 | } 132 | ], 133 | "source": [ 134 | "def main(argv=None):\n", 135 | " mnist = input_data.read_data_sets(\"../../datasets/MNIST_data\", one_hot=True)\n", 136 | " train(mnist)\n", 137 | "\n", 138 | "if __name__ == '__main__':\n", 139 | " main()\n" 140 | ] 141 | } 142 | ], 143 | "metadata": { 144 | "kernelspec": { 145 | "display_name": "Python 3", 146 | "language": "python", 147 | "name": "python3" 148 | }, 149 | "language_info": { 150 | "codemirror_mode": { 151 | "name": "ipython", 152 | "version": 3 153 | }, 154 | "file_extension": ".py", 155 | "mimetype": "text/x-python", 156 | "name": "python", 157 | "nbconvert_exporter": "python", 158 | "pygments_lexer": "ipython3", 159 | "version": "3.6.2" 160 | }, 161 | "varInspector": { 162 | "cols": { 163 | "lenName": 16, 164 | "lenType": 16, 165 | "lenVar": 40 166 | }, 167 | "kernels_config": { 168 | "python": { 169 | "delete_cmd_postfix": "", 170 | "delete_cmd_prefix": "del ", 171 | "library": "var_list.py", 172 | "varRefreshCmd": "print(var_dic_list())" 173 | }, 174 | "r": { 175 | "delete_cmd_postfix": ") ", 176 | "delete_cmd_prefix": "rm(", 177 | "library": "var_list.r", 178 | "varRefreshCmd": "cat(var_dic_list()) " 179 | } 180 | }, 181 | "types_to_exclude": [ 182 | "module", 183 | "function", 184 | "builtin_function_or_method", 185 | "instance", 186 | "_Feature" 187 | ], 188 | "window_display": false 189 | } 190 | }, 191 | "nbformat": 4, 192 | "nbformat_minor": 1 193 | } 194 | -------------------------------------------------------------------------------- /Chapter07/1. TFRecord样例程序.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import tensorflow as tf\n", 12 | "from tensorflow.examples.tutorials.mnist import input_data\n", 13 | "import numpy as np" 14 | ] 15 | }, 16 | { 17 | "cell_type": "markdown", 18 | "metadata": {}, 19 | "source": [ 20 | "#### 1. 将输入转化成TFRecord格式并保存。" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": 2, 26 | "metadata": {}, 27 | "outputs": [ 28 | { 29 | "name": "stdout", 30 | "output_type": "stream", 31 | "text": [ 32 | "Extracting ../datasets/MNIST_data/train-images-idx3-ubyte.gz\n", 33 | "Extracting ../datasets/MNIST_data/train-labels-idx1-ubyte.gz\n", 34 | "Extracting ../datasets/MNIST_data/t10k-images-idx3-ubyte.gz\n", 35 | "Extracting ../datasets/MNIST_data/t10k-labels-idx1-ubyte.gz\n", 36 | "TFRecord文件已保存。\n" 37 | ] 38 | } 39 | ], 40 | "source": [ 41 | "# 定义函数转化变量类型。\n", 42 | "def _int64_feature(value):\n", 43 | " return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))\n", 44 | "\n", 45 | "def _bytes_feature(value):\n", 46 | " return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))\n", 47 | "\n", 48 | "# 读取mnist数据。\n", 49 | "mnist = input_data.read_data_sets(\"../datasets/MNIST_data\",dtype=tf.uint8, one_hot=True)\n", 50 | "images = mnist.train.images\n", 51 | "labels = mnist.train.labels\n", 52 | "pixels = images.shape[1]\n", 53 | "num_examples = mnist.train.num_examples\n", 54 | "\n", 55 | "# 输出TFRecord文件的地址。\n", 56 | "filename = \"Records/output.tfrecords\"\n", 57 | "writer = tf.python_io.TFRecordWriter(filename)\n", 58 | "for index in range(num_examples):\n", 59 | " image_raw = images[index].tostring()\n", 60 | "\n", 61 | " example = tf.train.Example(features=tf.train.Features(feature={\n", 62 | " 'pixels': _int64_feature(pixels),\n", 63 | " 'label': _int64_feature(np.argmax(labels[index])),\n", 64 | " 'image_raw': _bytes_feature(image_raw)\n", 65 | " }))\n", 66 | " writer.write(example.SerializeToString())\n", 67 | "writer.close()\n", 68 | "print(\"TFRecord文件已保存。\")" 69 | ] 70 | }, 71 | { 72 | "cell_type": "markdown", 73 | "metadata": {}, 74 | "source": [ 75 | "#### 2. 读取TFRecord文件" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": 3, 81 | "metadata": {}, 82 | "outputs": [ 83 | { 84 | "name": "stdout", 85 | "output_type": "stream", 86 | "text": [ 87 | "7 784\n", 88 | "3 784\n", 89 | "4 784\n", 90 | "6 784\n", 91 | "1 784\n", 92 | "8 784\n", 93 | "1 784\n", 94 | "0 784\n", 95 | "9 784\n", 96 | "8 784\n" 97 | ] 98 | } 99 | ], 100 | "source": [ 101 | "# 读取文件。\n", 102 | "reader = tf.TFRecordReader()\n", 103 | "filename_queue = tf.train.string_input_producer([\"Records/output.tfrecords\"])\n", 104 | "_,serialized_example = reader.read(filename_queue)\n", 105 | "\n", 106 | "# 解析读取的样例。\n", 107 | "features = tf.parse_single_example(\n", 108 | " serialized_example,\n", 109 | " features={\n", 110 | " 'image_raw':tf.FixedLenFeature([],tf.string),\n", 111 | " 'pixels':tf.FixedLenFeature([],tf.int64),\n", 112 | " 'label':tf.FixedLenFeature([],tf.int64)\n", 113 | " })\n", 114 | "\n", 115 | "images = tf.decode_raw(features['image_raw'],tf.uint8)\n", 116 | "labels = tf.cast(features['label'],tf.int32)\n", 117 | "pixels = tf.cast(features['pixels'],tf.int32)\n", 118 | "\n", 119 | "sess = tf.Session()\n", 120 | "\n", 121 | "# 启动多线程处理输入数据。\n", 122 | "coord = tf.train.Coordinator()\n", 123 | "threads = tf.train.start_queue_runners(sess=sess,coord=coord)\n", 124 | "\n", 125 | "for i in range(10):\n", 126 | " image, label, pixel = sess.run([images, labels, pixels])\n", 127 | " print(label, pixel)" 128 | ] 129 | } 130 | ], 131 | "metadata": { 132 | "kernelspec": { 133 | "display_name": "Python 3", 134 | "language": "python", 135 | "name": "python3" 136 | }, 137 | "language_info": { 138 | "codemirror_mode": { 139 | "name": "ipython", 140 | "version": 3 141 | }, 142 | "file_extension": ".py", 143 | "mimetype": "text/x-python", 144 | "name": "python", 145 | "nbconvert_exporter": "python", 146 | "pygments_lexer": "ipython3", 147 | "version": "3.6.2" 148 | }, 149 | "varInspector": { 150 | "cols": { 151 | "lenName": 16, 152 | "lenType": 16, 153 | "lenVar": 40 154 | }, 155 | "kernels_config": { 156 | "python": { 157 | "delete_cmd_postfix": "", 158 | "delete_cmd_prefix": "del ", 159 | "library": "var_list.py", 160 | "varRefreshCmd": "print(var_dic_list())" 161 | }, 162 | "r": { 163 | "delete_cmd_postfix": ") ", 164 | "delete_cmd_prefix": "rm(", 165 | "library": "var_list.r", 166 | "varRefreshCmd": "cat(var_dic_list()) " 167 | } 168 | }, 169 | "types_to_exclude": [ 170 | "module", 171 | "function", 172 | "builtin_function_or_method", 173 | "instance", 174 | "_Feature" 175 | ], 176 | "window_display": false 177 | } 178 | }, 179 | "nbformat": 4, 180 | "nbformat_minor": 1 181 | } 182 | -------------------------------------------------------------------------------- /Chapter07/3. 队列操作.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import tensorflow as tf" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "#### 1. 创建队列，并操作里面的元素。" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 3, 24 | "metadata": {}, 25 | "outputs": [ 26 | { 27 | "name": "stdout", 28 | "output_type": "stream", 29 | "text": [ 30 | "0\n", 31 | "10\n", 32 | "1\n", 33 | "11\n", 34 | "2\n" 35 | ] 36 | } 37 | ], 38 | "source": [ 39 | "# 创建一个FIFO队列，队列中最多2个元素，类型为int32\n", 40 | "q = tf.FIFOQueue(2, \"int32\")\n", 41 | "# enqueue_many初始化队列中的元素\n", 42 | "init = q.enqueue_many(([0, 10], ))\n", 43 | "# dequeue把第一个元素0，出队列并保存到变量x\n", 44 | "x = q.dequeue()\n", 45 | "y = x + 1\n", 46 | "# enqueue把1插入到队列\n", 47 | "q_inc = q.enqueue([y])\n", 48 | "with tf.Session() as sess:\n", 49 | " # 运行初始化队列\n", 50 | " init.run()\n", 51 | " for _ in range(5):\n", 52 | " v, _ = sess.run([x, q_inc])\n", 53 | " print(v)\n", 54 | " \n", 55 | "# TensorFlow有两种队列：\n", 56 | "# tf.FIFOQueue：先进先出队列\n", 57 | "# tf.RandomShuffleQueue：随机调度队列" 58 | ] 59 | }, 60 | { 61 | "cell_type": "code", 62 | "execution_count": 5, 63 | "metadata": { 64 | "collapsed": true 65 | }, 66 | "outputs": [], 67 | "source": [ 68 | "import numpy as np\n", 69 | "import threading\n", 70 | "import time" 71 | ] 72 | }, 73 | { 74 | "cell_type": "markdown", 75 | "metadata": {}, 76 | "source": [ 77 | "#### 2. 这个程序每隔1秒判断是否需要停止并打印自己的ID。" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": 6, 83 | "metadata": { 84 | "collapsed": true 85 | }, 86 | "outputs": [], 87 | "source": [ 88 | "def MyLoop(coord, worker_id):\n", 89 | " while not coord.should_stop():\n", 90 | " if np.random.rand()<0.1:\n", 91 | " print(\"Stoping from id: %d\\n\" % worker_id)\n", 92 | " coord.request_stop()\n", 93 | " else:\n", 94 | " print(\"Working on id: %d\\n\" % worker_id)\n", 95 | " time.sleep(1)" 96 | ] 97 | }, 98 | { 99 | "cell_type": "markdown", 100 | "metadata": {}, 101 | "source": [ 102 | "#### 3. 创建、启动并退出线程。" 103 | ] 104 | }, 105 | { 106 | "cell_type": "code", 107 | "execution_count": 9, 108 | "metadata": {}, 109 | "outputs": [ 110 | { 111 | "name": "stdout", 112 | "output_type": "stream", 113 | "text": [ 114 | "Working on id: 0\n", 115 | "\n", 116 | "Working on id: 1\n", 117 | "Working on id: 2\n", 118 | "\n", 119 | "\n", 120 | "Working on id: 3\n", 121 | "\n", 122 | "Working on id: 4\n", 123 | "\n", 124 | "Working on id: 1\n", 125 | "Working on id: 2\n", 126 | "\n", 127 | "Working on id: 0\n", 128 | "Stoping from id: 4\n", 129 | "Working on id: 3\n", 130 | "\n", 131 | "\n", 132 | "\n", 133 | "\n" 134 | ] 135 | } 136 | ], 137 | "source": [ 138 | "# 声明一个Coordinator类来协同多个线程\n", 139 | "coord = tf.train.Coordinator()\n", 140 | "# 声明创建5个线程\n", 141 | "threads = [threading.Thread(target=MyLoop, args=(coord, i, )) for i in range(5)]\n", 142 | "# 启动所有线程\n", 143 | "for t in threads:t.start()\n", 144 | "# 等待所有线程退出\n", 145 | "coord.join(threads)" 146 | ] 147 | } 148 | ], 149 | "metadata": { 150 | "kernelspec": { 151 | "display_name": "Python 3", 152 | "language": "python", 153 | "name": "python3" 154 | }, 155 | "language_info": { 156 | "codemirror_mode": { 157 | "name": "ipython", 158 | "version": 3 159 | }, 160 | "file_extension": ".py", 161 | "mimetype": "text/x-python", 162 | "name": "python", 163 | "nbconvert_exporter": "python", 164 | "pygments_lexer": "ipython3", 165 | "version": "3.6.2" 166 | }, 167 | "varInspector": { 168 | "cols": { 169 | "lenName": 16, 170 | "lenType": 16, 171 | "lenVar": 40 172 | }, 173 | "kernels_config": { 174 | "python": { 175 | "delete_cmd_postfix": "", 176 | "delete_cmd_prefix": "del ", 177 | "library": "var_list.py", 178 | "varRefreshCmd": "print(var_dic_list())" 179 | }, 180 | "r": { 181 | "delete_cmd_postfix": ") ", 182 | "delete_cmd_prefix": "rm(", 183 | "library": "var_list.r", 184 | "varRefreshCmd": "cat(var_dic_list()) " 185 | } 186 | }, 187 | "types_to_exclude": [ 188 | "module", 189 | "function", 190 | "builtin_function_or_method", 191 | "instance", 192 | "_Feature" 193 | ], 194 | "window_display": false 195 | } 196 | }, 197 | "nbformat": 4, 198 | "nbformat_minor": 1 199 | } 200 | -------------------------------------------------------------------------------- /Chapter07/4. 多线程队列操作.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import tensorflow as tf" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "#### 1. 定义队列及其操作。" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 2, 24 | "metadata": { 25 | "collapsed": true 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "queue = tf.FIFOQueue(100,\"float\")\n", 30 | "# 把随机数插入到队列中\n", 31 | "enqueue_op = queue.enqueue([tf.random_normal([1])])\n", 32 | "# 创建一个5个线程运行队列的入队操作，表示同时启动五个线程把随机数插入队列\n", 33 | "qr = tf.train.QueueRunner(queue, [enqueue_op] * 5)\n", 34 | "# 把QueueRunner放到计算图的tf.GraphKeys.QUENE_RUNNERS中\n", 35 | "tf.train.add_queue_runner(qr)\n", 36 | "# 定义出队操作\n", 37 | "out_tensor = queue.dequeue()" 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "#### 2. 启动线程。" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": 20, 50 | "metadata": {}, 51 | "outputs": [ 52 | { 53 | "name": "stdout", 54 | "output_type": "stream", 55 | "text": [ 56 | "[ 0.75037265]\n", 57 | "[-0.09710068]\n", 58 | "[ 1.1645664]\n", 59 | "[ 0.09584331]\n", 60 | "[ 0.86558545]\n" 61 | ] 62 | } 63 | ], 64 | "source": [ 65 | "with tf.Session() as sess:\n", 66 | " # 协同启动线程\n", 67 | " coord = tf.train.Coordinator()\n", 68 | " # tf.train.start_queue_runners会启动tf.GraphKeys.QUENE_RUNNERS\n", 69 | " # 集合中的所有QueueRunner\n", 70 | " threads = tf.train.start_queue_runners(sess=sess, coord=coord)\n", 71 | " for _ in range(5): \n", 72 | " print(sess.run(out_tensor))\n", 73 | " # 停止所有线程\n", 74 | " coord.request_stop()\n", 75 | " coord.join(threads)" 76 | ] 77 | } 78 | ], 79 | "metadata": { 80 | "kernelspec": { 81 | "display_name": "Python 3", 82 | "language": "python", 83 | "name": "python3" 84 | }, 85 | "language_info": { 86 | "codemirror_mode": { 87 | "name": "ipython", 88 | "version": 3 89 | }, 90 | "file_extension": ".py", 91 | "mimetype": "text/x-python", 92 | "name": "python", 93 | "nbconvert_exporter": "python", 94 | "pygments_lexer": "ipython3", 95 | "version": "3.6.2" 96 | }, 97 | "varInspector": { 98 | "cols": { 99 | "lenName": 16, 100 | "lenType": 16, 101 | "lenVar": 40 102 | }, 103 | "kernels_config": { 104 | "python": { 105 | "delete_cmd_postfix": "", 106 | "delete_cmd_prefix": "del ", 107 | "library": "var_list.py", 108 | "varRefreshCmd": "print(var_dic_list())" 109 | }, 110 | "r": { 111 | "delete_cmd_postfix": ") ", 112 | "delete_cmd_prefix": "rm(", 113 | "library": "var_list.r", 114 | "varRefreshCmd": "cat(var_dic_list()) " 115 | } 116 | }, 117 | "types_to_exclude": [ 118 | "module", 119 | "function", 120 | "builtin_function_or_method", 121 | "instance", 122 | "_Feature" 123 | ], 124 | "window_display": false 125 | } 126 | }, 127 | "nbformat": 4, 128 | "nbformat_minor": 1 129 | } 130 | -------------------------------------------------------------------------------- /Chapter07/5. 输入文件队列.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import tensorflow as tf" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "#### 1. 生成文件存储样例数据。" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 2, 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "def _int64_feature(value):\n", 28 | " return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))\n", 29 | "num_shards = 2\n", 30 | "instances_per_shard = 2\n", 31 | "for i in range(num_shards):\n", 32 | " filename = ('Records/data.tfrecords-%.5d-of-%.5d' % (i, num_shards)) \n", 33 | " # 将Example结构写入TFRecord文件。\n", 34 | " writer = tf.python_io.TFRecordWriter(filename)\n", 35 | " for j in range(instances_per_shard):\n", 36 | " # Example结构仅包含当前样例属于第几个文件以及是当前文件的第几个样本。\n", 37 | " example = tf.train.Example(features=tf.train.Features(feature={\n", 38 | " 'i': _int64_feature(i),\n", 39 | " 'j': _int64_feature(j)}))\n", 40 | " writer.write(example.SerializeToString())\n", 41 | " writer.close() " 42 | ] 43 | }, 44 | { 45 | "cell_type": "markdown", 46 | "metadata": {}, 47 | "source": [ 48 | "#### 2. 读取文件。" 49 | ] 50 | }, 51 | { 52 | "cell_type": "code", 53 | "execution_count": 5, 54 | "metadata": {}, 55 | "outputs": [ 56 | { 57 | "data": { 58 | "text/plain": [ 59 | "[None, None]" 60 | ] 61 | }, 62 | "execution_count": 5, 63 | "metadata": {}, 64 | "output_type": "execute_result" 65 | }, 66 | { 67 | "name": "stdout", 68 | "output_type": "stream", 69 | "text": [ 70 | "[b'Records/data.tfrecords-00000-of-00002'\n", 71 | " b'Records/data.tfrecords-00001-of-00002']\n", 72 | "[0, 0]\n", 73 | "[0, 1]\n", 74 | "[1, 0]\n", 75 | "[1, 1]\n", 76 | "[0, 0]\n", 77 | "[0, 1]\n" 78 | ] 79 | } 80 | ], 81 | "source": [ 82 | "files = tf.train.match_filenames_once(\"Records/data.tfrecords-*\")\n", 83 | "filename_queue = tf.train.string_input_producer(files, shuffle=False) \n", 84 | "reader = tf.TFRecordReader()\n", 85 | "_, serialized_example = reader.read(filename_queue)\n", 86 | "features = tf.parse_single_example(\n", 87 | " serialized_example,\n", 88 | " features={\n", 89 | " 'i': tf.FixedLenFeature([], tf.int64),\n", 90 | " 'j': tf.FixedLenFeature([], tf.int64),\n", 91 | " })\n", 92 | "with tf.Session() as sess:\n", 93 | " sess.run([tf.global_variables_initializer(), tf.local_variables_initializer()])\n", 94 | " print(sess.run(files))\n", 95 | " coord = tf.train.Coordinator()\n", 96 | " threads = tf.train.start_queue_runners(sess=sess, coord=coord)\n", 97 | " for i in range(6):\n", 98 | " print(sess.run([features['i'], features['j']]))\n", 99 | " coord.request_stop()\n", 100 | " coord.join(threads)" 101 | ] 102 | }, 103 | { 104 | "cell_type": "markdown", 105 | "metadata": {}, 106 | "source": [ 107 | "#### 3. 组合训练数据（Batching）" 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": 8, 113 | "metadata": {}, 114 | "outputs": [ 115 | { 116 | "name": "stdout", 117 | "output_type": "stream", 118 | "text": [ 119 | "[0 1] [1 1]\n", 120 | "[0 1] [1 1]\n", 121 | "[0 0] [0 1]\n" 122 | ] 123 | } 124 | ], 125 | "source": [ 126 | "example, label = features['i'], features['j']\n", 127 | "batch_size = 2\n", 128 | "capacity = 1000 + 3 * batch_size\n", 129 | "example_batch, label_batch = tf.train.batch([example, label], batch_size=batch_size, capacity=capacity)\n", 130 | "\n", 131 | "with tf.Session() as sess:\n", 132 | " tf.global_variables_initializer().run()\n", 133 | " tf.local_variables_initializer().run()\n", 134 | " coord = tf.train.Coordinator()\n", 135 | " threads = tf.train.start_queue_runners(sess=sess, coord=coord)\n", 136 | " for i in range(3):\n", 137 | " cur_example_batch, cur_label_batch = sess.run([example_batch, label_batch])\n", 138 | " print(cur_example_batch, cur_label_batch)\n", 139 | " coord.request_stop()\n", 140 | " coord.join(threads)\n" 141 | ] 142 | } 143 | ], 144 | "metadata": { 145 | "kernelspec": { 146 | "display_name": "Python 3", 147 | "language": "python", 148 | "name": "python3" 149 | }, 150 | "language_info": { 151 | "codemirror_mode": { 152 | "name": "ipython", 153 | "version": 3 154 | }, 155 | "file_extension": ".py", 156 | "mimetype": "text/x-python", 157 | "name": "python", 158 | "nbconvert_exporter": "python", 159 | "pygments_lexer": "ipython3", 160 | "version": "3.6.2" 161 | }, 162 | "varInspector": { 163 | "cols": { 164 | "lenName": 16, 165 | "lenType": 16, 166 | "lenVar": 40 167 | }, 168 | "kernels_config": { 169 | "python": { 170 | "delete_cmd_postfix": "", 171 | "delete_cmd_prefix": "del ", 172 | "library": "var_list.py", 173 | "varRefreshCmd": "print(var_dic_list())" 174 | }, 175 | "r": { 176 | "delete_cmd_postfix": ") ", 177 | "delete_cmd_prefix": "rm(", 178 | "library": "var_list.r", 179 | "varRefreshCmd": "cat(var_dic_list()) " 180 | } 181 | }, 182 | "types_to_exclude": [ 183 | "module", 184 | "function", 185 | "builtin_function_or_method", 186 | "instance", 187 | "_Feature" 188 | ], 189 | "window_display": false 190 | } 191 | }, 192 | "nbformat": 4, 193 | "nbformat_minor": 1 194 | } 195 | -------------------------------------------------------------------------------- /Chapter07/6. 输入数据处理框架.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import tensorflow as tf" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "#### 1. 创建文件列表，通过文件列表创建输入文件队列，读取文件为本章第一节创建的文件。" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 2, 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "files = tf.train.match_filenames_once(\"Records/output.tfrecords\")\n", 28 | "filename_queue = tf.train.string_input_producer(files, shuffle=False) " 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": {}, 34 | "source": [ 35 | "#### 3. 解析TFRecord文件里的数据。" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": 3, 41 | "metadata": {}, 42 | "outputs": [], 43 | "source": [ 44 | "# 读取文件。\n", 45 | "\n", 46 | "reader = tf.TFRecordReader()\n", 47 | "_,serialized_example = reader.read(filename_queue)\n", 48 | "\n", 49 | "# 解析读取的样例。\n", 50 | "features = tf.parse_single_example(\n", 51 | " serialized_example,\n", 52 | " features={\n", 53 | " 'image_raw':tf.FixedLenFeature([],tf.string),\n", 54 | " 'pixels':tf.FixedLenFeature([],tf.int64),\n", 55 | " 'label':tf.FixedLenFeature([],tf.int64)\n", 56 | " })\n", 57 | "\n", 58 | "decoded_images = tf.decode_raw(features['image_raw'],tf.uint8)\n", 59 | "retyped_images = tf.cast(decoded_images, tf.float32)\n", 60 | "labels = tf.cast(features['label'],tf.int32)\n", 61 | "#pixels = tf.cast(features['pixels'],tf.int32)\n", 62 | "images = tf.reshape(retyped_images, [784])" 63 | ] 64 | }, 65 | { 66 | "cell_type": "markdown", 67 | "metadata": {}, 68 | "source": [ 69 | "#### 4. 将文件以100个为一组打包。" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 4, 75 | "metadata": {}, 76 | "outputs": [], 77 | "source": [ 78 | "min_after_dequeue = 10000\n", 79 | "batch_size = 100\n", 80 | "capacity = min_after_dequeue + 3 * batch_size\n", 81 | "\n", 82 | "image_batch, label_batch = tf.train.shuffle_batch([images, labels], \n", 83 | " batch_size=batch_size, \n", 84 | " capacity=capacity, \n", 85 | " min_after_dequeue=min_after_dequeue)" 86 | ] 87 | }, 88 | { 89 | "cell_type": "markdown", 90 | "metadata": {}, 91 | "source": [ 92 | "#### 5. 训练模型。" 93 | ] 94 | }, 95 | { 96 | "cell_type": "code", 97 | "execution_count": 5, 98 | "metadata": { 99 | "collapsed": true 100 | }, 101 | "outputs": [], 102 | "source": [ 103 | "def inference(input_tensor, weights1, biases1, weights2, biases2):\n", 104 | " layer1 = tf.nn.relu(tf.matmul(input_tensor, weights1) + biases1)\n", 105 | " return tf.matmul(layer1, weights2) + biases2" 106 | ] 107 | }, 108 | { 109 | "cell_type": "code", 110 | "execution_count": 7, 111 | "metadata": {}, 112 | "outputs": [ 113 | { 114 | "name": "stdout", 115 | "output_type": "stream", 116 | "text": [ 117 | "After 0 training step(s), loss is 497.954 \n", 118 | "After 1000 training step(s), loss is 8.30417 \n", 119 | "After 2000 training step(s), loss is 8.35147 \n", 120 | "After 3000 training step(s), loss is 8.12348 \n", 121 | "After 4000 training step(s), loss is 7.99348 \n" 122 | ] 123 | } 124 | ], 125 | "source": [ 126 | "# 模型相关的参数\n", 127 | "INPUT_NODE = 784\n", 128 | "OUTPUT_NODE = 10\n", 129 | "LAYER1_NODE = 500\n", 130 | "REGULARAZTION_RATE = 0.0001 \n", 131 | "TRAINING_STEPS = 5000 \n", 132 | "\n", 133 | "weights1 = tf.Variable(tf.truncated_normal([INPUT_NODE, LAYER1_NODE], stddev=0.1))\n", 134 | "biases1 = tf.Variable(tf.constant(0.1, shape=[LAYER1_NODE]))\n", 135 | "\n", 136 | "weights2 = tf.Variable(tf.truncated_normal([LAYER1_NODE, OUTPUT_NODE], stddev=0.1))\n", 137 | "biases2 = tf.Variable(tf.constant(0.1, shape=[OUTPUT_NODE]))\n", 138 | "\n", 139 | "y = inference(image_batch, weights1, biases1, weights2, biases2)\n", 140 | " \n", 141 | "# 计算交叉熵及其平均值\n", 142 | "cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y, labels=label_batch)\n", 143 | "cross_entropy_mean = tf.reduce_mean(cross_entropy)\n", 144 | " \n", 145 | "# 损失函数的计算\n", 146 | "regularizer = tf.contrib.layers.l2_regularizer(REGULARAZTION_RATE)\n", 147 | "regularaztion = regularizer(weights1) + regularizer(weights2)\n", 148 | "loss = cross_entropy_mean + regularaztion\n", 149 | "\n", 150 | "# 优化损失函数\n", 151 | "train_step = tf.train.GradientDescentOptimizer(0.01).minimize(loss)\n", 152 | " \n", 153 | "# 初始化会话，并开始训练过程。\n", 154 | "with tf.Session() as sess:\n", 155 | " # 由于使用了Coordinator，必须对local和global变量进行初始化\n", 156 | " sess.run(tf.local_variables_initializer())\n", 157 | " sess.run(tf.global_variables_initializer())\n", 158 | " coord = tf.train.Coordinator()\n", 159 | " threads = tf.train.start_queue_runners(sess=sess, coord=coord)\n", 160 | " # 循环的训练神经网络。\n", 161 | " for i in range(TRAINING_STEPS):\n", 162 | " if i % 1000 == 0:\n", 163 | " print(\"After %d training step(s), loss is %g \" % (i, sess.run(loss)))\n", 164 | " \n", 165 | " sess.run(train_step) \n", 166 | " coord.request_stop()\n", 167 | " coord.join(threads) " 168 | ] 169 | } 170 | ], 171 | "metadata": { 172 | "kernelspec": { 173 | "display_name": "Python 3", 174 | "language": "python", 175 | "name": "python3" 176 | }, 177 | "language_info": { 178 | "codemirror_mode": { 179 | "name": "ipython", 180 | "version": 3 181 | }, 182 | "file_extension": ".py", 183 | "mimetype": "text/x-python", 184 | "name": "python", 185 | "nbconvert_exporter": "python", 186 | "pygments_lexer": "ipython3", 187 | "version": "3.6.2" 188 | }, 189 | "varInspector": { 190 | "cols": { 191 | "lenName": 16, 192 | "lenType": 16, 193 | "lenVar": 40 194 | }, 195 | "kernels_config": { 196 | "python": { 197 | "delete_cmd_postfix": "", 198 | "delete_cmd_prefix": "del ", 199 | "library": "var_list.py", 200 | "varRefreshCmd": "print(var_dic_list())" 201 | }, 202 | "r": { 203 | "delete_cmd_postfix": ") ", 204 | "delete_cmd_prefix": "rm(", 205 | "library": "var_list.r", 206 | "varRefreshCmd": "cat(var_dic_list()) " 207 | } 208 | }, 209 | "types_to_exclude": [ 210 | "module", 211 | "function", 212 | "builtin_function_or_method", 213 | "instance", 214 | "_Feature" 215 | ], 216 | "window_display": false 217 | } 218 | }, 219 | "nbformat": 4, 220 | "nbformat_minor": 1 221 | } 222 | -------------------------------------------------------------------------------- /Chapter07/Records/README: -------------------------------------------------------------------------------- 1 | This folder saves records -------------------------------------------------------------------------------- /Chapter08/1. 循环神经网络前向传播.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import numpy as np" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "#### 1. 定义RNN的参数。" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 2, 24 | "metadata": { 25 | "collapsed": true 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "X = [1,2]\n", 30 | "state = [0.0, 0.0]\n", 31 | "w_cell_state = np.asarray([[0.1, 0.2], [0.3, 0.4]])\n", 32 | "w_cell_input = np.asarray([0.5, 0.6])\n", 33 | "b_cell = np.asarray([0.1, -0.1])\n", 34 | "w_output = np.asarray([[1.0], [2.0]])\n", 35 | "b_output = 0.1" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": {}, 41 | "source": [ 42 | "#### 2. 执行前向传播过程。" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 3, 48 | "metadata": { 49 | "collapsed": false 50 | }, 51 | "outputs": [ 52 | { 53 | "name": "stdout", 54 | "output_type": "stream", 55 | "text": [ 56 | "before activation: [ 0.6 0.5]\n", 57 | "state: [ 0.53704957 0.46211716]\n", 58 | "output: [ 1.56128388]\n", 59 | "before activation: [ 1.2923401 1.39225678]\n", 60 | "state: [ 0.85973818 0.88366641]\n", 61 | "output: [ 2.72707101]\n" 62 | ] 63 | } 64 | ], 65 | "source": [ 66 | "for i in range(len(X)):\n", 67 | " before_activation = np.dot(state, w_cell_state) + X[i] * w_cell_input + b_cell\n", 68 | " state = np.tanh(before_activation)\n", 69 | " final_output = np.dot(state, w_output) + b_output\n", 70 | " print \"before activation: \", before_activation\n", 71 | " print \"state: \", state\n", 72 | " print \"output: \", final_output" 73 | ] 74 | } 75 | ], 76 | "metadata": { 77 | "kernelspec": { 78 | "display_name": "Python 2", 79 | "language": "python", 80 | "name": "python2" 81 | }, 82 | "language_info": { 83 | "codemirror_mode": { 84 | "name": "ipython", 85 | "version": 2 86 | }, 87 | "file_extension": ".py", 88 | "mimetype": "text/x-python", 89 | "name": "python", 90 | "nbconvert_exporter": "python", 91 | "pygments_lexer": "ipython2", 92 | "version": "2.7.10" 93 | } 94 | }, 95 | "nbformat": 4, 96 | "nbformat_minor": 1 97 | } 98 | -------------------------------------------------------------------------------- /Chapter08/2. PTB数据集介绍.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import tensorflow as tf\n", 12 | "import reader" 13 | ] 14 | }, 15 | { 16 | "cell_type": "markdown", 17 | "metadata": {}, 18 | "source": [ 19 | "#### 1. 读取数据并打印长度及前100位数据。" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 2, 25 | "metadata": { 26 | "collapsed": false 27 | }, 28 | "outputs": [ 29 | { 30 | "name": "stdout", 31 | "output_type": "stream", 32 | "text": [ 33 | "929589\n", 34 | "[9970, 9971, 9972, 9974, 9975, 9976, 9980, 9981, 9982, 9983, 9984, 9986, 9987, 9988, 9989, 9991, 9992, 9993, 9994, 9995, 9996, 9997, 9998, 9999, 2, 9256, 1, 3, 72, 393, 33, 2133, 0, 146, 19, 6, 9207, 276, 407, 3, 2, 23, 1, 13, 141, 4, 1, 5465, 0, 3081, 1596, 96, 2, 7682, 1, 3, 72, 393, 8, 337, 141, 4, 2477, 657, 2170, 955, 24, 521, 6, 9207, 276, 4, 39, 303, 438, 3684, 2, 6, 942, 4, 3150, 496, 263, 5, 138, 6092, 4241, 6036, 30, 988, 6, 241, 760, 4, 1015, 2786, 211, 6, 96, 4]\n" 35 | ] 36 | } 37 | ], 38 | "source": [ 39 | "DATA_PATH = \"../../datasets/PTB_data\"\n", 40 | "train_data, valid_data, test_data, _ = reader.ptb_raw_data(DATA_PATH)\n", 41 | "print len(train_data)\n", 42 | "print train_data[:100]" 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": {}, 48 | "source": [ 49 | "#### 2. 将训练数据组织成batch大小为4、截断长度为5的数据组。并使用队列读取前3个batch。" 50 | ] 51 | }, 52 | { 53 | "cell_type": "code", 54 | "execution_count": 3, 55 | "metadata": { 56 | "collapsed": false 57 | }, 58 | "outputs": [ 59 | { 60 | "name": "stdout", 61 | "output_type": "stream", 62 | "text": [ 63 | "X0: [[9970 9971 9972 9974 9975]\n", 64 | " [ 332 7147 328 1452 8595]\n", 65 | " [1969 0 98 89 2254]\n", 66 | " [ 3 3 2 14 24]]\n", 67 | "Y0: [[9971 9972 9974 9975 9976]\n", 68 | " [7147 328 1452 8595 59]\n", 69 | " [ 0 98 89 2254 0]\n", 70 | " [ 3 2 14 24 198]]\n", 71 | "X1: [[9976 9980 9981 9982 9983]\n", 72 | " [ 59 1569 105 2231 1]\n", 73 | " [ 0 312 1641 4 1063]\n", 74 | " [ 198 150 2262 10 0]]\n", 75 | "Y1: [[9980 9981 9982 9983 9984]\n", 76 | " [1569 105 2231 1 895]\n", 77 | " [ 312 1641 4 1063 8]\n", 78 | " [ 150 2262 10 0 507]]\n", 79 | "X2: [[9984 9986 9987 9988 9989]\n", 80 | " [ 895 1 5574 4 618]\n", 81 | " [ 8 713 0 264 820]\n", 82 | " [ 507 74 2619 0 1]]\n", 83 | "Y2: [[9986 9987 9988 9989 9991]\n", 84 | " [ 1 5574 4 618 2]\n", 85 | " [ 713 0 264 820 2]\n", 86 | " [ 74 2619 0 1 8]]\n" 87 | ] 88 | } 89 | ], 90 | "source": [ 91 | "# ptb_producer返回的为一个二维的tuple数据。\n", 92 | "result = reader.ptb_producer(train_data, 4, 5)\n", 93 | "\n", 94 | "# 通过队列依次读取batch。\n", 95 | "with tf.Session() as sess:\n", 96 | " coord = tf.train.Coordinator()\n", 97 | " threads = tf.train.start_queue_runners(sess=sess, coord=coord)\n", 98 | " for i in range(3):\n", 99 | " x, y = sess.run(result)\n", 100 | " print \"X%d: \"%i, x\n", 101 | " print \"Y%d: \"%i, y\n", 102 | " coord.request_stop()\n", 103 | " coord.join(threads)" 104 | ] 105 | } 106 | ], 107 | "metadata": { 108 | "kernelspec": { 109 | "display_name": "Python 2", 110 | "language": "python", 111 | "name": "python2" 112 | }, 113 | "language_info": { 114 | "codemirror_mode": { 115 | "name": "ipython", 116 | "version": 2 117 | }, 118 | "file_extension": ".py", 119 | "mimetype": "text/x-python", 120 | "name": "python", 121 | "nbconvert_exporter": "python", 122 | "pygments_lexer": "ipython2", 123 | "version": "2.7.10" 124 | } 125 | }, 126 | "nbformat": 4, 127 | "nbformat_minor": 1 128 | } 129 | -------------------------------------------------------------------------------- /Chapter08/4. SKlearn封装例子.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "from sklearn import model_selection\n", 12 | "from sklearn import datasets\n", 13 | "from sklearn import metrics\n", 14 | "import tensorflow as tf\n", 15 | "import numpy as np\n", 16 | "from tensorflow.contrib.learn.python.learn.estimators.estimator import SKCompat\n", 17 | "learn = tf.contrib.learn" 18 | ] 19 | }, 20 | { 21 | "cell_type": "markdown", 22 | "metadata": {}, 23 | "source": [ 24 | "#### 1. 自定义softmax回归模型。" 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": 2, 30 | "metadata": { 31 | "collapsed": false 32 | }, 33 | "outputs": [], 34 | "source": [ 35 | "def my_model(features, target):\n", 36 | " target = tf.one_hot(target, 3, 1, 0)\n", 37 | " \n", 38 | " # 计算预测值及损失函数。\n", 39 | " logits = tf.contrib.layers.fully_connected(features, 3, tf.nn.softmax)\n", 40 | " loss = tf.losses.softmax_cross_entropy(target, logits)\n", 41 | " \n", 42 | " # 创建优化步骤。\n", 43 | " train_op = tf.contrib.layers.optimize_loss(\n", 44 | " loss,\n", 45 | " tf.contrib.framework.get_global_step(),\n", 46 | " optimizer='Adam',\n", 47 | " learning_rate=0.01)\n", 48 | " return tf.arg_max(logits, 1), loss, train_op" 49 | ] 50 | }, 51 | { 52 | "cell_type": "markdown", 53 | "metadata": {}, 54 | "source": [ 55 | "#### 2. 读取数据并将数据转化成TensorFlow要求的float32格式。" 56 | ] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "execution_count": 3, 61 | "metadata": { 62 | "collapsed": false 63 | }, 64 | "outputs": [], 65 | "source": [ 66 | "iris = datasets.load_iris()\n", 67 | "x_train, x_test, y_train, y_test = model_selection.train_test_split(\n", 68 | " iris.data, iris.target, test_size=0.2, random_state=0)\n", 69 | "\n", 70 | "x_train, x_test = map(np.float32, [x_train, x_test])" 71 | ] 72 | }, 73 | { 74 | "cell_type": "markdown", 75 | "metadata": {}, 76 | "source": [ 77 | "#### 3. 封装和训练模型，输出准确率。" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": 4, 83 | "metadata": { 84 | "collapsed": false 85 | }, 86 | "outputs": [ 87 | { 88 | "name": "stdout", 89 | "output_type": "stream", 90 | "text": [ 91 | "INFO:tensorflow:Using default config.\n", 92 | "INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_num_ps_replicas': 0, '_keep_checkpoint_max': 5, '_tf_random_seed': None, '_task_type': None, '_environment': 'local', '_is_chief': True, '_cluster_spec': , '_tf_config': gpu_options {\n", 93 | " per_process_gpu_memory_fraction: 1\n", 94 | "}\n", 95 | ", '_task_id': 0, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_evaluation_master': '', '_keep_checkpoint_every_n_hours': 10000, '_master': ''}\n", 96 | "INFO:tensorflow:Create CheckpointSaverHook.\n", 97 | "INFO:tensorflow:Saving checkpoints for 1 into Models/model_1/model.ckpt.\n", 98 | "INFO:tensorflow:loss = 1.09918, step = 1\n", 99 | "INFO:tensorflow:global_step/sec: 677.497\n", 100 | "INFO:tensorflow:loss = 0.783171, step = 101\n", 101 | "INFO:tensorflow:global_step/sec: 706.629\n", 102 | "INFO:tensorflow:loss = 0.696257, step = 201\n", 103 | "INFO:tensorflow:global_step/sec: 701.641\n", 104 | "INFO:tensorflow:loss = 0.655656, step = 301\n", 105 | "INFO:tensorflow:global_step/sec: 562.503\n", 106 | "INFO:tensorflow:loss = 0.634671, step = 401\n", 107 | "INFO:tensorflow:global_step/sec: 582.211\n", 108 | "INFO:tensorflow:loss = 0.622115, step = 501\n", 109 | "INFO:tensorflow:global_step/sec: 522.269\n", 110 | "INFO:tensorflow:loss = 0.613794, step = 601\n", 111 | "INFO:tensorflow:global_step/sec: 583.315\n", 112 | "INFO:tensorflow:loss = 0.607876, step = 701\n", 113 | "INFO:tensorflow:Saving checkpoints for 800 into Models/model_1/model.ckpt.\n", 114 | "INFO:tensorflow:Loss for final step: 0.603483.\n", 115 | "Accuracy: 100.00%\n" 116 | ] 117 | } 118 | ], 119 | "source": [ 120 | "classifier = SKCompat(learn.Estimator(model_fn=my_model, model_dir=\"Models/model_1\"))\n", 121 | "classifier.fit(x_train, y_train, steps=800)\n", 122 | "\n", 123 | "y_predicted = [i for i in classifier.predict(x_test)]\n", 124 | "score = metrics.accuracy_score(y_test, y_predicted)\n", 125 | "print('Accuracy: %.2f%%' % (score * 100))" 126 | ] 127 | } 128 | ], 129 | "metadata": { 130 | "kernelspec": { 131 | "display_name": "Python 2", 132 | "language": "python", 133 | "name": "python2" 134 | }, 135 | "language_info": { 136 | "codemirror_mode": { 137 | "name": "ipython", 138 | "version": 2 139 | }, 140 | "file_extension": ".py", 141 | "mimetype": "text/x-python", 142 | "name": "python", 143 | "nbconvert_exporter": "python", 144 | "pygments_lexer": "ipython2", 145 | "version": "2.7.10" 146 | } 147 | }, 148 | "nbformat": 4, 149 | "nbformat_minor": 1 150 | } 151 | -------------------------------------------------------------------------------- /Chapter08/Models/README: -------------------------------------------------------------------------------- 1 | This folder saves model files -------------------------------------------------------------------------------- /Chapter08/reader.py: -------------------------------------------------------------------------------- 1 | # Copyright 2015 The TensorFlow Authors. All Rights Reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | # ============================================================================== 15 | 16 | 17 | """Utilities for parsing PTB text files.""" 18 | from __future__ import absolute_import 19 | from __future__ import division 20 | from __future__ import print_function 21 | 22 | import collections 23 | import os 24 | 25 | import tensorflow as tf 26 | 27 | 28 | def _read_words(filename): 29 | with tf.gfile.GFile(filename, "r") as f: 30 | return f.read().decode("utf-8").replace("\n", "").split() 31 | 32 | 33 | def _build_vocab(filename): 34 | data = _read_words(filename) 35 | 36 | counter = collections.Counter(data) 37 | count_pairs = sorted(counter.items(), key=lambda x: (-x[1], x[0])) 38 | 39 | words, _ = list(zip(*count_pairs)) 40 | word_to_id = dict(zip(words, range(len(words)))) 41 | 42 | return word_to_id 43 | 44 | 45 | def _file_to_word_ids(filename, word_to_id): 46 | data = _read_words(filename) 47 | return [word_to_id[word] for word in data if word in word_to_id] 48 | 49 | 50 | def ptb_raw_data(data_path=None): 51 | """Load PTB raw data from data directory "data_path". 52 | 53 | Reads PTB text files, converts strings to integer ids, 54 | and performs mini-batching of the inputs. 55 | 56 | The PTB dataset comes from Tomas Mikolov's webpage: 57 | 58 | http://www.fit.vutbr.cz/~imikolov/rnnlm/simple-examples.tgz 59 | 60 | Args: 61 | data_path: string path to the directory where simple-examples.tgz has 62 | been extracted. 63 | 64 | Returns: 65 | tuple (train_data, valid_data, test_data, vocabulary) 66 | where each of the data objects can be passed to PTBIterator. 67 | """ 68 | 69 | train_path = os.path.join(data_path, "ptb.train.txt") 70 | valid_path = os.path.join(data_path, "ptb.valid.txt") 71 | test_path = os.path.join(data_path, "ptb.test.txt") 72 | 73 | word_to_id = _build_vocab(train_path) 74 | train_data = _file_to_word_ids(train_path, word_to_id) 75 | valid_data = _file_to_word_ids(valid_path, word_to_id) 76 | test_data = _file_to_word_ids(test_path, word_to_id) 77 | vocabulary = len(word_to_id) 78 | return train_data, valid_data, test_data, vocabulary 79 | 80 | 81 | def ptb_producer(raw_data, batch_size, num_steps, name=None): 82 | """Iterate on the raw PTB data. 83 | 84 | This chunks up raw_data into batches of examples and returns Tensors that 85 | are drawn from these batches. 86 | 87 | Args: 88 | raw_data: one of the raw data outputs from ptb_raw_data. 89 | batch_size: int, the batch size. 90 | num_steps: int, the number of unrolls. 91 | name: the name of this operation (optional). 92 | 93 | Returns: 94 | A pair of Tensors, each shaped [batch_size, num_steps]. The second element 95 | of the tuple is the same data time-shifted to the right by one. 96 | 97 | Raises: 98 | tf.errors.InvalidArgumentError: if batch_size or num_steps are too high. 99 | """ 100 | with tf.name_scope(name, "PTBProducer", [raw_data, batch_size, num_steps]): 101 | raw_data = tf.convert_to_tensor(raw_data, name="raw_data", dtype=tf.int32) 102 | 103 | data_len = tf.size(raw_data) 104 | batch_len = data_len // batch_size 105 | data = tf.reshape(raw_data[0 : batch_size * batch_len], 106 | [batch_size, batch_len]) 107 | 108 | epoch_size = (batch_len - 1) // num_steps 109 | assertion = tf.assert_positive( 110 | epoch_size, 111 | message="epoch_size == 0, decrease batch_size or num_steps") 112 | with tf.control_dependencies([assertion]): 113 | epoch_size = tf.identity(epoch_size, name="epoch_size") 114 | 115 | i = tf.train.range_input_producer(epoch_size, shuffle=False).dequeue() 116 | x = tf.strided_slice(data, [0, i * num_steps], 117 | [batch_size, (i + 1) * num_steps]) 118 | x.set_shape([batch_size, num_steps]) 119 | y = tf.strided_slice(data, [0, i * num_steps + 1], 120 | [batch_size, (i + 1) * num_steps + 1]) 121 | y.set_shape([batch_size, num_steps]) 122 | return x, y 123 | -------------------------------------------------------------------------------- /Chapter09/1. 命名空间.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "#### 1. 不同的命名空间。" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": {}, 14 | "outputs": [ 15 | { 16 | "name": "stdout", 17 | "output_type": "stream", 18 | "text": [ 19 | "foo/bar:0\n", 20 | "bar/bar:0\n" 21 | ] 22 | } 23 | ], 24 | "source": [ 25 | "import tensorflow as tf\n", 26 | "with tf.variable_scope(\"foo\"):\n", 27 | " a = tf.get_variable(\"bar\", [1])\n", 28 | " print(a.name)\n", 29 | "\n", 30 | "with tf.variable_scope(\"bar\"):\n", 31 | " b = tf.get_variable(\"bar\", [1])\n", 32 | " print(b.name)" 33 | ] 34 | }, 35 | { 36 | "cell_type": "markdown", 37 | "metadata": {}, 38 | "source": [ 39 | "#### 2. tf.Variable和tf.get_variable的区别。" 40 | ] 41 | }, 42 | { 43 | "cell_type": "code", 44 | "execution_count": 2, 45 | "metadata": {}, 46 | "outputs": [ 47 | { 48 | "name": "stdout", 49 | "output_type": "stream", 50 | "text": [ 51 | "a/Variable:0\n", 52 | "b:0\n" 53 | ] 54 | } 55 | ], 56 | "source": [ 57 | "with tf.name_scope(\"a\"):\n", 58 | " a = tf.Variable([1])\n", 59 | " print(a.name)\n", 60 | " \n", 61 | " a = tf.get_variable(\"b\", [1])\n", 62 | " print(a.name)" 63 | ] 64 | }, 65 | { 66 | "cell_type": "markdown", 67 | "metadata": {}, 68 | "source": [ 69 | "#### 3. TensorBoard可以根据命名空间来整理可视化效果图上的节点。" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 3, 75 | "metadata": { 76 | "collapsed": true 77 | }, 78 | "outputs": [], 79 | "source": [ 80 | "with tf.name_scope(\"input1\"):\n", 81 | " input1 = tf.constant([1.0, 2.0, 3.0], name=\"input2\")\n", 82 | "with tf.name_scope(\"input2\"):\n", 83 | " input2 = tf.Variable(tf.random_uniform([3]), name=\"input2\")\n", 84 | "output = tf.add_n([input1, input2], name=\"add\")\n", 85 | "\n", 86 | "writer = tf.summary.FileWriter(\"log/simple_example.log\", tf.get_default_graph())\n", 87 | "writer.close()" 88 | ] 89 | } 90 | ], 91 | "metadata": { 92 | "kernelspec": { 93 | "display_name": "Python 3", 94 | "language": "python", 95 | "name": "python3" 96 | }, 97 | "language_info": { 98 | "codemirror_mode": { 99 | "name": "ipython", 100 | "version": 3 101 | }, 102 | "file_extension": ".py", 103 | "mimetype": "text/x-python", 104 | "name": "python", 105 | "nbconvert_exporter": "python", 106 | "pygments_lexer": "ipython3", 107 | "version": "3.6.2" 108 | }, 109 | "varInspector": { 110 | "cols": { 111 | "lenName": 16, 112 | "lenType": 16, 113 | "lenVar": 40 114 | }, 115 | "kernels_config": { 116 | "python": { 117 | "delete_cmd_postfix": "", 118 | "delete_cmd_prefix": "del ", 119 | "library": "var_list.py", 120 | "varRefreshCmd": "print(var_dic_list())" 121 | }, 122 | "r": { 123 | "delete_cmd_postfix": ") ", 124 | "delete_cmd_prefix": "rm(", 125 | "library": "var_list.r", 126 | "varRefreshCmd": "cat(var_dic_list()) " 127 | } 128 | }, 129 | "types_to_exclude": [ 130 | "module", 131 | "function", 132 | "builtin_function_or_method", 133 | "instance", 134 | "_Feature" 135 | ], 136 | "window_display": false 137 | } 138 | }, 139 | "nbformat": 4, 140 | "nbformat_minor": 1 141 | } 142 | -------------------------------------------------------------------------------- /Chapter09/2. 改造后的mnist_train.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import tensorflow as tf\n", 12 | "from tensorflow.examples.tutorials.mnist import input_data\n", 13 | "import mnist_inference" 14 | ] 15 | }, 16 | { 17 | "cell_type": "markdown", 18 | "metadata": {}, 19 | "source": [ 20 | "#### 1. 定义神经网络的参数。" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": 2, 26 | "metadata": { 27 | "collapsed": true 28 | }, 29 | "outputs": [], 30 | "source": [ 31 | "BATCH_SIZE = 100\n", 32 | "LEARNING_RATE_BASE = 0.8\n", 33 | "LEARNING_RATE_DECAY = 0.99\n", 34 | "REGULARIZATION_RATE = 0.0001\n", 35 | "TRAINING_STEPS = 3000\n", 36 | "MOVING_AVERAGE_DECAY = 0.99" 37 | ] 38 | }, 39 | { 40 | "cell_type": "markdown", 41 | "metadata": {}, 42 | "source": [ 43 | "#### 2. 定义训练的过程并保存TensorBoard的log文件。" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": 3, 49 | "metadata": { 50 | "collapsed": true 51 | }, 52 | "outputs": [], 53 | "source": [ 54 | "def train(mnist):\n", 55 | " # 输入数据的命名空间。\n", 56 | " with tf.name_scope('input'):\n", 57 | " x = tf.placeholder(tf.float32, [None, mnist_inference.INPUT_NODE], name='x-input')\n", 58 | " y_ = tf.placeholder(tf.float32, [None, mnist_inference.OUTPUT_NODE], name='y-input')\n", 59 | " regularizer = tf.contrib.layers.l2_regularizer(REGULARIZATION_RATE)\n", 60 | " y = mnist_inference.inference(x, regularizer)\n", 61 | " global_step = tf.Variable(0, trainable=False)\n", 62 | " \n", 63 | " # 处理滑动平均的命名空间。\n", 64 | " with tf.name_scope(\"moving_average\"):\n", 65 | " variable_averages = tf.train.ExponentialMovingAverage(MOVING_AVERAGE_DECAY, global_step)\n", 66 | " variables_averages_op = variable_averages.apply(tf.trainable_variables())\n", 67 | " \n", 68 | " # 计算损失函数的命名空间。\n", 69 | " with tf.name_scope(\"loss_function\"):\n", 70 | " cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y, labels=tf.argmax(y_, 1))\n", 71 | " cross_entropy_mean = tf.reduce_mean(cross_entropy)\n", 72 | " loss = cross_entropy_mean + tf.add_n(tf.get_collection('losses'))\n", 73 | " \n", 74 | " # 定义学习率、优化方法及每一轮执行训练的操作的命名空间。\n", 75 | " with tf.name_scope(\"train_step\"):\n", 76 | " learning_rate = tf.train.exponential_decay(\n", 77 | " LEARNING_RATE_BASE,\n", 78 | " global_step,\n", 79 | " mnist.train.num_examples / BATCH_SIZE, LEARNING_RATE_DECAY,\n", 80 | " staircase=True)\n", 81 | "\n", 82 | " train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=global_step)\n", 83 | "\n", 84 | " with tf.control_dependencies([train_step, variables_averages_op]):\n", 85 | " train_op = tf.no_op(name='train')\n", 86 | " \n", 87 | " writer = tf.summary.FileWriter(\"log/modified_mnist_train.log\", tf.get_default_graph())\n", 88 | " \n", 89 | " # 训练模型。\n", 90 | " with tf.Session() as sess:\n", 91 | " tf.global_variables_initializer().run()\n", 92 | " for i in range(TRAINING_STEPS):\n", 93 | " xs, ys = mnist.train.next_batch(BATCH_SIZE)\n", 94 | "\n", 95 | " if i % 1000 == 0:\n", 96 | " # 配置运行时需要记录的信息。\n", 97 | " run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)\n", 98 | " # 运行时记录运行信息的proto。\n", 99 | " run_metadata = tf.RunMetadata()\n", 100 | " _, loss_value, step = sess.run(\n", 101 | " [train_op, loss, global_step], feed_dict={x: xs, y_: ys},\n", 102 | " options=run_options, run_metadata=run_metadata)\n", 103 | " writer.add_run_metadata(run_metadata=run_metadata, tag=(\"tag%d\" % i), global_step=i)\n", 104 | " print(\"After %d training step(s), loss on training batch is %g.\" % (step, loss_value))\n", 105 | " else:\n", 106 | " _, loss_value, step = sess.run([train_op, loss, global_step], feed_dict={x: xs, y_: ys})\n", 107 | " \n", 108 | " writer.close()" 109 | ] 110 | }, 111 | { 112 | "cell_type": "markdown", 113 | "metadata": {}, 114 | "source": [ 115 | "#### 3. 主函数。" 116 | ] 117 | }, 118 | { 119 | "cell_type": "code", 120 | "execution_count": 4, 121 | "metadata": {}, 122 | "outputs": [ 123 | { 124 | "name": "stdout", 125 | "output_type": "stream", 126 | "text": [ 127 | "Extracting ../datasets/MNIST_data/train-images-idx3-ubyte.gz\n", 128 | "Extracting ../datasets/MNIST_data/train-labels-idx1-ubyte.gz\n", 129 | "Extracting ../datasets/MNIST_data/t10k-images-idx3-ubyte.gz\n", 130 | "Extracting ../datasets/MNIST_data/t10k-labels-idx1-ubyte.gz\n", 131 | "After 1 training step(s), loss on training batch is 3.15454.\n", 132 | "After 1001 training step(s), loss on training batch is 0.203072.\n", 133 | "After 2001 training step(s), loss on training batch is 0.166225.\n" 134 | ] 135 | } 136 | ], 137 | "source": [ 138 | "def main(argv=None): \n", 139 | " mnist = input_data.read_data_sets(\"../datasets/MNIST_data\", one_hot=True)\n", 140 | " train(mnist)\n", 141 | "\n", 142 | "if __name__ == '__main__':\n", 143 | " main()" 144 | ] 145 | } 146 | ], 147 | "metadata": { 148 | "kernelspec": { 149 | "display_name": "Python 3", 150 | "language": "python", 151 | "name": "python3" 152 | }, 153 | "language_info": { 154 | "codemirror_mode": { 155 | "name": "ipython", 156 | "version": 3 157 | }, 158 | "file_extension": ".py", 159 | "mimetype": "text/x-python", 160 | "name": "python", 161 | "nbconvert_exporter": "python", 162 | "pygments_lexer": "ipython3", 163 | "version": "3.6.2" 164 | }, 165 | "varInspector": { 166 | "cols": { 167 | "lenName": 16, 168 | "lenType": 16, 169 | "lenVar": 40 170 | }, 171 | "kernels_config": { 172 | "python": { 173 | "delete_cmd_postfix": "", 174 | "delete_cmd_prefix": "del ", 175 | "library": "var_list.py", 176 | "varRefreshCmd": "print(var_dic_list())" 177 | }, 178 | "r": { 179 | "delete_cmd_postfix": ") ", 180 | "delete_cmd_prefix": "rm(", 181 | "library": "var_list.r", 182 | "varRefreshCmd": "cat(var_dic_list()) " 183 | } 184 | }, 185 | "types_to_exclude": [ 186 | "module", 187 | "function", 188 | "builtin_function_or_method", 189 | "instance", 190 | "_Feature" 191 | ], 192 | "window_display": false 193 | } 194 | }, 195 | "nbformat": 4, 196 | "nbformat_minor": 1 197 | } 198 | -------------------------------------------------------------------------------- /Chapter09/3. 监控指标可视化.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import tensorflow as tf\n", 12 | "from tensorflow.examples.tutorials.mnist import input_data" 13 | ] 14 | }, 15 | { 16 | "cell_type": "markdown", 17 | "metadata": {}, 18 | "source": [ 19 | "#### 1. 生成变量监控信息并定义生成监控信息日志的操作。" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 2, 25 | "metadata": { 26 | "collapsed": true 27 | }, 28 | "outputs": [], 29 | "source": [ 30 | "SUMMARY_DIR = \"log/supervisor.log\"\n", 31 | "BATCH_SIZE = 100\n", 32 | "TRAIN_STEPS = 3000\n", 33 | "\n", 34 | "def variable_summaries(var, name):\n", 35 | " with tf.name_scope('summaries'):\n", 36 | " tf.summary.histogram(name, var)\n", 37 | " mean = tf.reduce_mean(var)\n", 38 | " tf.summary.scalar('mean/' + name, mean)\n", 39 | " stddev = tf.sqrt(tf.reduce_mean(tf.square(var - mean)))\n", 40 | " tf.summary.scalar('stddev/' + name, stddev) " 41 | ] 42 | }, 43 | { 44 | "cell_type": "markdown", 45 | "metadata": {}, 46 | "source": [ 47 | "#### 2. 生成一层全链接的神经网络。 " 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": 3, 53 | "metadata": { 54 | "collapsed": true 55 | }, 56 | "outputs": [], 57 | "source": [ 58 | "def nn_layer(input_tensor, input_dim, output_dim, layer_name, act=tf.nn.relu):\n", 59 | " with tf.name_scope(layer_name):\n", 60 | " with tf.name_scope('weights'):\n", 61 | " weights = tf.Variable(tf.truncated_normal([input_dim, output_dim], stddev=0.1))\n", 62 | " variable_summaries(weights, layer_name + '/weights')\n", 63 | " with tf.name_scope('biases'):\n", 64 | " biases = tf.Variable(tf.constant(0.0, shape=[output_dim]))\n", 65 | " variable_summaries(biases, layer_name + '/biases')\n", 66 | " with tf.name_scope('Wx_plus_b'):\n", 67 | " preactivate = tf.matmul(input_tensor, weights) + biases\n", 68 | " tf.summary.histogram(layer_name + '/pre_activations', preactivate)\n", 69 | " activations = act(preactivate, name='activation') \n", 70 | " \n", 71 | " # 记录神经网络节点输出在经过激活函数之后的分布。\n", 72 | " tf.summary.histogram(layer_name + '/activations', activations)\n", 73 | " return activations" 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": 4, 79 | "metadata": { 80 | "collapsed": true 81 | }, 82 | "outputs": [], 83 | "source": [ 84 | "def main():\n", 85 | " mnist = input_data.read_data_sets(\"../datasets/MNIST_data\", one_hot=True)\n", 86 | "\n", 87 | " with tf.name_scope('input'):\n", 88 | " x = tf.placeholder(tf.float32, [None, 784], name='x-input')\n", 89 | " y_ = tf.placeholder(tf.float32, [None, 10], name='y-input')\n", 90 | "\n", 91 | " with tf.name_scope('input_reshape'):\n", 92 | " image_shaped_input = tf.reshape(x, [-1, 28, 28, 1])\n", 93 | " tf.summary.image('input', image_shaped_input, 10)\n", 94 | "\n", 95 | " hidden1 = nn_layer(x, 784, 500, 'layer1')\n", 96 | " y = nn_layer(hidden1, 500, 10, 'layer2', act=tf.identity)\n", 97 | "\n", 98 | " with tf.name_scope('cross_entropy'):\n", 99 | " cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y, labels=y_))\n", 100 | " tf.summary.scalar('cross_entropy', cross_entropy)\n", 101 | "\n", 102 | " with tf.name_scope('train'):\n", 103 | " train_step = tf.train.AdamOptimizer(0.001).minimize(cross_entropy)\n", 104 | "\n", 105 | " with tf.name_scope('accuracy'):\n", 106 | " with tf.name_scope('correct_prediction'):\n", 107 | " correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))\n", 108 | " with tf.name_scope('accuracy'):\n", 109 | " accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n", 110 | " tf.summary.scalar('accuracy', accuracy)\n", 111 | "\n", 112 | " merged = tf.summary.merge_all()\n", 113 | "\n", 114 | " with tf.Session() as sess:\n", 115 | " \n", 116 | " summary_writer = tf.summary.FileWriter(SUMMARY_DIR, sess.graph)\n", 117 | " tf.global_variables_initializer().run()\n", 118 | "\n", 119 | " for i in range(TRAIN_STEPS):\n", 120 | " xs, ys = mnist.train.next_batch(BATCH_SIZE)\n", 121 | " # 运行训练步骤以及所有的日志生成操作，得到这次运行的日志。\n", 122 | " summary, _ = sess.run([merged, train_step], feed_dict={x: xs, y_: ys})\n", 123 | " # 将得到的所有日志写入日志文件，这样TensorBoard程序就可以拿到这次运行所对应的\n", 124 | " # 运行信息。\n", 125 | " summary_writer.add_summary(summary, i)\n", 126 | "\n", 127 | " summary_writer.close()" 128 | ] 129 | }, 130 | { 131 | "cell_type": "code", 132 | "execution_count": 5, 133 | "metadata": {}, 134 | "outputs": [ 135 | { 136 | "name": "stdout", 137 | "output_type": "stream", 138 | "text": [ 139 | "Extracting ../datasets/MNIST_data/train-images-idx3-ubyte.gz\n", 140 | "Extracting ../datasets/MNIST_data/train-labels-idx1-ubyte.gz\n", 141 | "Extracting ../datasets/MNIST_data/t10k-images-idx3-ubyte.gz\n", 142 | "Extracting ../datasets/MNIST_data/t10k-labels-idx1-ubyte.gz\n" 143 | ] 144 | } 145 | ], 146 | "source": [ 147 | "if __name__ == '__main__':\n", 148 | " main()" 149 | ] 150 | } 151 | ], 152 | "metadata": { 153 | "kernelspec": { 154 | "display_name": "Python 3", 155 | "language": "python", 156 | "name": "python3" 157 | }, 158 | "language_info": { 159 | "codemirror_mode": { 160 | "name": "ipython", 161 | "version": 3 162 | }, 163 | "file_extension": ".py", 164 | "mimetype": "text/x-python", 165 | "name": "python", 166 | "nbconvert_exporter": "python", 167 | "pygments_lexer": "ipython3", 168 | "version": "3.6.2" 169 | }, 170 | "varInspector": { 171 | "cols": { 172 | "lenName": 16, 173 | "lenType": 16, 174 | "lenVar": 40 175 | }, 176 | "kernels_config": { 177 | "python": { 178 | "delete_cmd_postfix": "", 179 | "delete_cmd_prefix": "del ", 180 | "library": "var_list.py", 181 | "varRefreshCmd": "print(var_dic_list())" 182 | }, 183 | "r": { 184 | "delete_cmd_postfix": ") ", 185 | "delete_cmd_prefix": "rm(", 186 | "library": "var_list.r", 187 | "varRefreshCmd": "cat(var_dic_list()) " 188 | } 189 | }, 190 | "types_to_exclude": [ 191 | "module", 192 | "function", 193 | "builtin_function_or_method", 194 | "instance", 195 | "_Feature" 196 | ], 197 | "window_display": false 198 | } 199 | }, 200 | "nbformat": 4, 201 | "nbformat_minor": 1 202 | } 203 | -------------------------------------------------------------------------------- /Chapter09/mnist_inference.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | 3 | INPUT_NODE = 784 4 | OUTPUT_NODE = 10 5 | LAYER1_NODE = 500 6 | 7 | def get_weight_variable(shape, regularizer): 8 | weights = tf.get_variable("weights", shape, initializer=tf.truncated_normal_initializer(stddev=0.1)) 9 | if regularizer != None: tf.add_to_collection('losses', regularizer(weights)) 10 | return weights 11 | 12 | 13 | def inference(input_tensor, regularizer): 14 | with tf.variable_scope('layer1'): 15 | 16 | weights = get_weight_variable([INPUT_NODE, LAYER1_NODE], regularizer) 17 | biases = tf.get_variable("biases", [LAYER1_NODE], initializer=tf.constant_initializer(0.0)) 18 | layer1 = tf.nn.relu(tf.matmul(input_tensor, weights) + biases) 19 | 20 | with tf.variable_scope('layer2'): 21 | weights = get_weight_variable([LAYER1_NODE, OUTPUT_NODE], regularizer) 22 | biases = tf.get_variable("biases", [OUTPUT_NODE], initializer=tf.constant_initializer(0.0)) 23 | layer2 = tf.matmul(layer1, weights) + biases 24 | 25 | return layer2 -------------------------------------------------------------------------------- /Chapter10/1. GPU基本操作.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | import tensorflow as tf 3 | 4 | a = tf.constant([1.0, 2.0, 3.0], shape=[3], name='a') 5 | b = tf.constant([1.0, 2.0, 3.0], shape=[3], name='b') 6 | c = a + b 7 | 8 | # 通过log_device_placement参数来记录运行每一个运算的设备。 9 | sess = tf.Session(config=tf.ConfigProto(log_device_placement=True)) 10 | print sess.run(c) 11 | 12 | 13 | # 通过tf.device将运算指定到特定的设备上。 14 | with tf.device('/cpu:0'): 15 | a = tf.constant([1.0, 2.0, 3.0], shape=[3], name='a') 16 | b = tf.constant([1.0, 2.0, 3.0], shape=[3], name='b') 17 | with tf.device('/gpu:1'): 18 | c = a + b 19 | 20 | sess = tf.Session(config=tf.ConfigProto(log_device_placement=True)) 21 | print sess.run(c) 22 | 23 | 24 | a_cpu = tf.Variable(0, name="a_cpu") 25 | with tf.device('/gpu:0'): 26 | a_gpu = tf.Variable(0, name="a_gpu") 27 | # 通过allow_soft_placement参数自动将无法放在GPU上的操作放回CPU上。 28 | sess = tf.Session(config=tf.ConfigProto(allow_soft_placement=True, log_device_placement=True)) 29 | sess.run(tf.global_variables_initializer()) 30 | 31 | 32 | -------------------------------------------------------------------------------- /Chapter10/2. 多GPU并行.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | from datetime import datetime 3 | import os 4 | import time 5 | 6 | import tensorflow as tf 7 | import mnist_inference 8 | 9 | # 定义训练神经网络时需要用到的参数。 10 | BATCH_SIZE = 100 11 | LEARNING_RATE_BASE = 0.001 12 | LEARNING_RATE_DECAY = 0.99 13 | REGULARAZTION_RATE = 0.0001 14 | TRAINING_STEPS = 1000 15 | MOVING_AVERAGE_DECAY = 0.99 16 | N_GPU = 4 17 | 18 | # 定义日志和模型输出的路径。 19 | MODEL_SAVE_PATH = "logs_and_models/" 20 | MODEL_NAME = "model.ckpt" 21 | DATA_PATH = "output.tfrecords" 22 | 23 | # 定义输入队列得到训练数据，具体细节可以参考第七章。 24 | def get_input(): 25 | filename_queue = tf.train.string_input_producer([DATA_PATH]) 26 | reader = tf.TFRecordReader() 27 | _, serialized_example = reader.read(filename_queue) 28 | 29 | # 定义数据解析格式。 30 | features = tf.parse_single_example( 31 | serialized_example, 32 | features={ 33 | 'image_raw': tf.FixedLenFeature([], tf.string), 34 | 'pixels': tf.FixedLenFeature([], tf.int64), 35 | 'label': tf.FixedLenFeature([], tf.int64), 36 | }) 37 | 38 | # 解析图片和标签信息。 39 | decoded_image = tf.decode_raw(features['image_raw'], tf.uint8) 40 | reshaped_image = tf.reshape(decoded_image, [784]) 41 | retyped_image = tf.cast(reshaped_image, tf.float32) 42 | label = tf.cast(features['label'], tf.int32) 43 | 44 | # 定义输入队列并返回。 45 | min_after_dequeue = 10000 46 | capacity = min_after_dequeue + 3 * BATCH_SIZE 47 | return tf.train.shuffle_batch( 48 | [retyped_image, label], 49 | batch_size=BATCH_SIZE, 50 | capacity=capacity, 51 | min_after_dequeue=min_after_dequeue) 52 | 53 | # 定义损失函数。 54 | def get_loss(x, y_, regularizer, scope, reuse_variables=None): 55 | with tf.variable_scope(tf.get_variable_scope(), reuse=reuse_variables): 56 | y = mnist_inference.inference(x, regularizer) 57 | cross_entropy = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(y, y_)) 58 | regularization_loss = tf.add_n(tf.get_collection('losses', scope)) 59 | loss = cross_entropy + regularization_loss 60 | return loss 61 | 62 | # 计算每一个变量梯度的平均值。 63 | def average_gradients(tower_grads): 64 | average_grads = [] 65 | 66 | # 枚举所有的变量和变量在不同GPU上计算得出的梯度。 67 | for grad_and_vars in zip(*tower_grads): 68 | # 计算所有GPU上的梯度平均值。 69 | grads = [] 70 | for g, _ in grad_and_vars: 71 | expanded_g = tf.expand_dims(g, 0) 72 | grads.append(expanded_g) 73 | grad = tf.concat(grads, 0) 74 | grad = tf.reduce_mean(grad, 0) 75 | 76 | v = grad_and_vars[0][1] 77 | grad_and_var = (grad, v) 78 | # 将变量和它的平均梯度对应起来。 79 | average_grads.append(grad_and_var) 80 | # 返回所有变量的平均梯度，这个将被用于变量的更新。 81 | return average_grads 82 | 83 | # 主训练过程。 84 | def main(argv=None): 85 | # 将简单的运算放在CPU上，只有神经网络的训练过程放在GPU上。 86 | with tf.Graph().as_default(), tf.device('/cpu:0'): 87 | 88 | # 定义基本的训练过程 89 | x, y_ = get_input() 90 | regularizer = tf.contrib.layers.l2_regularizer(REGULARAZTION_RATE) 91 | 92 | global_step = tf.get_variable('global_step', [], initializer=tf.constant_initializer(0), trainable=False) 93 | learning_rate = tf.train.exponential_decay( 94 | LEARNING_RATE_BASE, global_step, 60000 / BATCH_SIZE, LEARNING_RATE_DECAY) 95 | 96 | opt = tf.train.GradientDescentOptimizer(learning_rate) 97 | 98 | tower_grads = [] 99 | reuse_variables = False 100 | # 将神经网络的优化过程跑在不同的GPU上。 101 | for i in range(N_GPU): 102 | # 将优化过程指定在一个GPU上。 103 | with tf.device('/gpu:%d' % i): 104 | with tf.name_scope('GPU_%d' % i) as scope: 105 | cur_loss = get_loss(x, y_, regularizer, scope, reuse_variables) 106 | reuse_variables = True 107 | grads = opt.compute_gradients(cur_loss) 108 | tower_grads.append(grads) 109 | 110 | # 计算变量的平均梯度。 111 | grads = average_gradients(tower_grads) 112 | for grad, var in grads: 113 | if grad is not None: 114 | tf.histogram_summary('gradients_on_average/%s' % var.op.name, grad) 115 | 116 | # 使用平均梯度更新参数。 117 | apply_gradient_op = opt.apply_gradients(grads, global_step=global_step) 118 | for var in tf.trainable_variables(): 119 | tf.histogram_summary(var.op.name, var) 120 | 121 | # 计算变量的滑动平均值。 122 | variable_averages = tf.train.ExponentialMovingAverage(MOVING_AVERAGE_DECAY, global_step) 123 | variables_to_average = (tf.trainable_variables() +tf.moving_average_variables()) 124 | variables_averages_op = variable_averages.apply(variables_to_average) 125 | # 每一轮迭代需要更新变量的取值并更新变量的滑动平均值。 126 | train_op = tf.group(apply_gradient_op, variables_averages_op) 127 | 128 | saver = tf.train.Saver(tf.all_variables()) 129 | summary_op = tf.merge_all_summaries() 130 | init = tf.initialize_all_variables() 131 | with tf.Session(config=tf.ConfigProto( 132 | allow_soft_placement=True, log_device_placement=True)) as sess: 133 | # 初始化所有变量并启动队列。 134 | init.run() 135 | coord = tf.train.Coordinator() 136 | threads = tf.train.start_queue_runners(sess=sess, coord=coord) 137 | summary_writer = tf.train.SummaryWriter(MODEL_SAVE_PATH, sess.graph) 138 | 139 | for step in range(TRAINING_STEPS): 140 | # 执行神经网络训练操作，并记录训练操作的运行时间。 141 | start_time = time.time() 142 | _, loss_value = sess.run([train_op, cur_loss]) 143 | duration = time.time() - start_time 144 | 145 | # 每隔一段时间数据当前的训练进度，并统计训练速度。 146 | if step != 0 and step % 10 == 0: 147 | # 计算使用过的训练数据个数。 148 | num_examples_per_step = BATCH_SIZE * N_GPU 149 | examples_per_sec = num_examples_per_step / duration 150 | sec_per_batch = duration / N_GPU 151 | 152 | # 输出训练信息。 153 | format_str = ('%s: step %d, loss = %.2f (%.1f examples/sec; %.3f sec/batch)') 154 | print (format_str % (datetime.now(), step, loss_value, examples_per_sec, sec_per_batch)) 155 | 156 | # 通过TensorBoard可视化训练过程。 157 | summary = sess.run(summary_op) 158 | summary_writer.add_summary(summary, step) 159 | 160 | # 每隔一段时间保存当前的模型。 161 | if step % 1000 == 0 or (step + 1) == TRAINING_STEPS: 162 | checkpoint_path = os.path.join(MODEL_SAVE_PATH, MODEL_NAME) 163 | saver.save(sess, checkpoint_path, global_step=step) 164 | 165 | coord.request_stop() 166 | coord.join(threads) 167 | 168 | if __name__ == '__main__': 169 | tf.app.run() 170 | 171 | 172 | 173 | -------------------------------------------------------------------------------- /Chapter10/3. 分布式TensorFlow.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | 3 | # 创建一个本地集群。 4 | c = tf.constant("Hello, distributed TensorFlow!") 5 | server = tf.train.Server.create_local_server() 6 | sess = tf.Session(server.target) 7 | print sess.run(c) 8 | 9 | 10 | # 创建两个集群 11 | c = tf.constant("Hello from server1!") 12 | cluster = tf.train.ClusterSpec({"local": ["localhost:2222", "localhost:2223"]}) 13 | server = tf.train.Server(cluster, job_name="local", task_index=0) 14 | sess = tf.Session(server.target, config=tf.ConfigProto(log_device_placement=True)) 15 | print sess.run(c) 16 | server.join() 17 | 18 | 19 | import tensorflow as tf 20 | c = tf.constant("Hello from server2!") 21 | cluster = tf.train.ClusterSpec({"local": ["localhost:2222", "localhost:2223"]}) 22 | server = tf.train.Server(cluster, job_name="local", task_index=1) 23 | sess = tf.Session(server.target, config=tf.ConfigProto(log_device_placement=True)) 24 | print sess.run(c) 25 | server.join() 26 | 27 | 28 | -------------------------------------------------------------------------------- /Chapter10/4. 异步更新模式样例程序.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | import time 3 | import tensorflow as tf 4 | from tensorflow.examples.tutorials.mnist import input_data 5 | 6 | import mnist_inference 7 | 8 | # 配置神经网络的参数。 9 | BATCH_SIZE = 100 10 | LEARNING_RATE_BASE = 0.01 11 | LEARNING_RATE_DECAY = 0.99 12 | REGULARAZTION_RATE = 0.0001 13 | TRAINING_STEPS = 1000 14 | MOVING_AVERAGE_DECAY = 0.99 15 | 16 | # 模型保存的路径和文件名。 17 | MODEL_SAVE_PATH = "logs/log_async" 18 | DATA_PATH = "../../datasets/MNIST_data" 19 | 20 | FLAGS = tf.app.flags.FLAGS 21 | 22 | # 指定当前程序是参数服务器还是计算服务器。 23 | tf.app.flags.DEFINE_string('job_name', 'worker', ' "ps" or "worker" ') 24 | # 指定集群中的参数服务器地址。 25 | tf.app.flags.DEFINE_string( 26 | 'ps_hosts', ' tf-ps0:2222,tf-ps1:1111', 27 | 'Comma-separated list of hostname:port for the parameter server jobs. e.g. "tf-ps0:2222,tf-ps1:1111" ') 28 | # 指定集群中的计算服务器地址。 29 | tf.app.flags.DEFINE_string( 30 | 'worker_hosts', ' tf-worker0:2222,tf-worker1:1111', 31 | 'Comma-separated list of hostname:port for the worker jobs. e.g. "tf-worker0:2222,tf-worker1:1111" ') 32 | # 指定当前程序的任务ID。 33 | tf.app.flags.DEFINE_integer('task_id', 0, 'Task ID of the worker/replica running the training.') 34 | 35 | # 定义TensorFlow的计算图，并返回每一轮迭代时需要运行的操作。 36 | def build_model(x, y_, is_chief): 37 | regularizer = tf.contrib.layers.l2_regularizer(REGULARAZTION_RATE) 38 | # 通过和5.5节给出的mnist_inference.py代码计算神经网络前向传播的结果。 39 | y = mnist_inference.inference(x, regularizer) 40 | global_step = tf.Variable(0, trainable=False) 41 | 42 | # 计算损失函数并定义反向传播过程。 43 | cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y, labels=tf.argmax(y_, 1)) 44 | cross_entropy_mean = tf.reduce_mean(cross_entropy) 45 | loss = cross_entropy_mean + tf.add_n(tf.get_collection('losses')) 46 | learning_rate = tf.train.exponential_decay( 47 | LEARNING_RATE_BASE, global_step, 60000 / BATCH_SIZE, LEARNING_RATE_DECAY) 48 | train_op = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=global_step) 49 | 50 | # 定义每一轮迭代需要运行的操作。 51 | if is_chief: 52 | # 计算变量的滑动平均值。 53 | variable_averages = tf.train.ExponentialMovingAverage(MOVING_AVERAGE_DECAY, global_step) 54 | variables_averages_op = variable_averages.apply(tf.trainable_variables()) 55 | with tf.control_dependencies([variables_averages_op, train_op]): 56 | train_op = tf.no_op() 57 | return global_step, loss, train_op 58 | 59 | 60 | def main(argv=None): 61 | # 解析flags并通过tf.train.ClusterSpec配置TensorFlow集群。 62 | ps_hosts = FLAGS.ps_hosts.split(',') 63 | worker_hosts = FLAGS.worker_hosts.split(',') 64 | cluster = tf.train.ClusterSpec({"ps": ps_hosts, "worker": worker_hosts}) 65 | # 通过tf.train.ClusterSpec以及当前任务创建tf.train.Server。 66 | server = tf.train.Server(cluster, job_name = FLAGS.job_name, task_index=FLAGS.task_id) 67 | 68 | # 参数服务器只需要管理TensorFlow中的变量，不需要执行训练的过程。server.join()会 69 | # 一致停在这条语句上。 70 | if FLAGS.job_name == 'ps': 71 | with tf.device("/cpu:0"): 72 | server.join() 73 | 74 | # 定义计算服务器需要运行的操作。 75 | is_chief = (FLAGS.task_id == 0) 76 | mnist = input_data.read_data_sets(DATA_PATH, one_hot=True) 77 | 78 | device_setter = tf.train.replica_device_setter(worker_device="/job:worker/task:%d" % FLAGS.task_id, cluster=cluster) 79 | with tf.device(device_setter): 80 | 81 | # 定义输入并得到每一轮迭代需要运行的操作。 82 | x = tf.placeholder(tf.float32, [None, mnist_inference.INPUT_NODE], name='x-input') 83 | y_ = tf.placeholder(tf.float32, [None, mnist_inference.OUTPUT_NODE], name='y-input') 84 | global_step, loss, train_op = build_model(x, y_, is_chief) 85 | 86 | # 定义用于保存模型的saver。 87 | saver = tf.train.Saver() 88 | # 定义日志输出操作。 89 | summary_op = tf.summary.merge_all() 90 | # 定义变量初始化操作。 91 | init_op = tf.global_variables_initializer() 92 | # 通过tf.train.Supervisor管理训练深度学习模型时的通用功能。 93 | sv = tf.train.Supervisor( 94 | is_chief=is_chief, 95 | logdir=MODEL_SAVE_PATH, 96 | init_op=init_op, 97 | summary_op=summary_op, 98 | saver=saver, 99 | global_step=global_step, 100 | save_model_secs=60, 101 | save_summaries_secs=60) 102 | 103 | sess_config = tf.ConfigProto(allow_soft_placement=True, log_device_placement=False) 104 | # 通过tf.train.Supervisor生成会话。 105 | sess = sv.prepare_or_wait_for_session(server.target, config=sess_config) 106 | 107 | step = 0 108 | start_time = time.time() 109 | 110 | # 执行迭代过程。 111 | while not sv.should_stop(): 112 | xs, ys = mnist.train.next_batch(BATCH_SIZE) 113 | _, loss_value, global_step_value = sess.run([train_op, loss, global_step], feed_dict={x: xs, y_: ys}) 114 | if global_step_value >= TRAINING_STEPS: break 115 | 116 | # 每隔一段时间输出训练信息。 117 | if step > 0 and step % 100 == 0: 118 | duration = time.time() - start_time 119 | sec_per_batch = duration / global_step_value 120 | format_str = "After %d training steps (%d global steps), loss on training batch is %g. (%.3f sec/batch)" 121 | print format_str % (step, global_step_value, loss_value, sec_per_batch) 122 | step += 1 123 | sv.stop() 124 | 125 | if __name__ == "__main__": 126 | tf.app.run() 127 | 128 | 129 | 130 | 131 | -------------------------------------------------------------------------------- /Chapter10/5. 同步更新模式样例程序.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | import time 3 | import tensorflow as tf 4 | from tensorflow.examples.tutorials.mnist import input_data 5 | 6 | import mnist_inference 7 | 8 | # 配置神经网络的参数。 9 | BATCH_SIZE = 100 10 | LEARNING_RATE_BASE = 0.8 11 | LEARNING_RATE_DECAY = 0.99 12 | REGULARAZTION_RATE = 0.0001 13 | TRAINING_STEPS = 10000 14 | MOVING_AVERAGE_DECAY = 0.99 15 | MODEL_SAVE_PATH = "logs/log_sync" 16 | DATA_PATH = "../../datasets/MNIST_data" 17 | 18 | 19 | # 和异步模式类似的设置flags。 20 | FLAGS = tf.app.flags.FLAGS 21 | 22 | tf.app.flags.DEFINE_string('job_name', 'worker', ' "ps" or "worker" ') 23 | tf.app.flags.DEFINE_string( 24 | 'ps_hosts', ' tf-ps0:2222,tf-ps1:1111', 25 | 'Comma-separated list of hostname:port for the parameter server jobs. e.g. "tf-ps0:2222,tf-ps1:1111" ') 26 | tf.app.flags.DEFINE_string( 27 | 'worker_hosts', ' tf-worker0:2222,tf-worker1:1111', 28 | 'Comma-separated list of hostname:port for the worker jobs. e.g. "tf-worker0:2222,tf-worker1:1111" ') 29 | tf.app.flags.DEFINE_integer('task_id', 0, 'Task ID of the worker/replica running the training.') 30 | 31 | # 和异步模式类似的定义TensorFlow的计算图。唯一的区别在于使用 32 | # tf.train.SyncReplicasOptimizer函数处理同步更新。 33 | def build_model(x, y_, n_workers, is_chief): 34 | regularizer = tf.contrib.layers.l2_regularizer(REGULARAZTION_RATE) 35 | y = mnist_inference.inference(x, regularizer) 36 | global_step = tf.Variable(0, trainable=False) 37 | 38 | variable_averages = tf.train.ExponentialMovingAverage(MOVING_AVERAGE_DECAY, global_step) 39 | variables_averages_op = variable_averages.apply(tf.trainable_variables()) 40 | 41 | cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y, labels=tf.argmax(y_, 1)) 42 | cross_entropy_mean = tf.reduce_mean(cross_entropy) 43 | loss = cross_entropy_mean + tf.add_n(tf.get_collection('losses')) 44 | learning_rate = tf.train.exponential_decay( 45 | LEARNING_RATE_BASE, global_step, 60000 / BATCH_SIZE, LEARNING_RATE_DECAY) 46 | 47 | # 通过tf.train.SyncReplicasOptimizer函数实现同步更新。 48 | opt = tf.train.SyncReplicasOptimizer( 49 | tf.train.GradientDescentOptimizer(learning_rate), 50 | replicas_to_aggregate=n_workers, 51 | total_num_replicas=n_workers) 52 | 53 | train_op = opt.minimize(loss, global_step=global_step) 54 | if is_chief: 55 | variable_averages = tf.train.ExponentialMovingAverage(MOVING_AVERAGE_DECAY, global_step) 56 | variables_averages_op = variable_averages.apply(tf.trainable_variables()) 57 | with tf.control_dependencies([variables_averages_op, train_op]): 58 | train_op = tf.no_op() 59 | 60 | return global_step, loss, train_op, opt 61 | 62 | def main(argv=None): 63 | # 和异步模式类似的创建TensorFlow集群。 64 | ps_hosts = FLAGS.ps_hosts.split(',') 65 | worker_hosts = FLAGS.worker_hosts.split(',') 66 | print ('PS hosts are: %s' % ps_hosts) 67 | print ('Worker hosts are: %s' % worker_hosts) 68 | n_workers = len(worker_hosts) 69 | 70 | cluster = tf.train.ClusterSpec({"ps": ps_hosts, "worker": worker_hosts}) 71 | server = tf.train.Server( 72 | cluster, job_name = FLAGS.job_name, task_index=FLAGS.task_id) 73 | 74 | if FLAGS.job_name == 'ps': 75 | with tf.device("/cpu:0"): 76 | server.join() 77 | 78 | is_chief = (FLAGS.task_id == 0) 79 | mnist = input_data.read_data_sets(DATA_PATH, one_hot=True) 80 | 81 | with tf.device(tf.train.replica_device_setter( 82 | worker_device="/job:worker/task:%d" % FLAGS.task_id, cluster=cluster)): 83 | x = tf.placeholder(tf.float32, [None, mnist_inference.INPUT_NODE], name='x-input') 84 | y_ = tf.placeholder(tf.float32, [None, mnist_inference.OUTPUT_NODE], name='y-input') 85 | global_step, loss, train_op, opt = build_model(x, y_, n_workers, is_chief) 86 | # 和异步模式类似的声明一些辅助函数。 87 | saver = tf.train.Saver() 88 | summary_op = tf.summary.merge_all() 89 | init_op = tf.global_variables_initializer() 90 | 91 | # 在同步模式下，主计算服务器需要协调不同计算服务器计算得到的参数梯度并最终更新参数。 92 | # 这需要主计算服务器完成一些额外的初始化工作。 93 | if is_chief: 94 | # 获取协调不同计算服务器的队列。在更新参数之前，主计算服务器需要先启动这些队列。 95 | chief_queue_runner = opt.get_chief_queue_runner() 96 | # 初始化同步更新队列的操作。 97 | init_tokens_op = opt.get_init_tokens_op(0) 98 | 99 | # 和异步模式类似的声明tf.train.Supervisor。 100 | sv = tf.train.Supervisor(is_chief=is_chief, 101 | logdir=MODEL_SAVE_PATH, 102 | init_op=init_op, 103 | summary_op=summary_op, 104 | saver = saver, 105 | global_step=global_step, 106 | save_model_secs=60, 107 | save_summaries_secs=60) 108 | sess_config = tf.ConfigProto(allow_soft_placement=True, log_device_placement=False) 109 | sess = sv.prepare_or_wait_for_session(server.target, config=sess_config) 110 | 111 | # 在主计算服务器上启动协调同步更新的队列并执行初始化操作。 112 | if is_chief: 113 | sv.start_queue_runners(sess, [chief_queue_runner]) 114 | sess.run(init_tokens_op) 115 | 116 | # 和异步模式类似的运行迭代的训练过程。 117 | step = 0 118 | start_time = time.time() 119 | while not sv.should_stop(): 120 | xs, ys = mnist.train.next_batch(BATCH_SIZE) 121 | _, loss_value, global_step_value = sess.run([train_op, loss, global_step], feed_dict={x: xs, y_: ys}) 122 | if global_step_value >= TRAINING_STEPS: break 123 | 124 | if step > 0 and step % 100 == 0: 125 | duration = time.time() - start_time 126 | sec_per_batch = duration / (global_step_value * n_workers) 127 | format_str = "After %d training steps (%d global steps), loss on training batch is %g. (%.3f sec/batch)" 128 | print format_str % (step, global_step_value, loss_value, sec_per_batch) 129 | step += 1 130 | sv.stop() 131 | 132 | if __name__ == "__main__": 133 | tf.app.run() 134 | -------------------------------------------------------------------------------- /Chapter10/mnist_inference.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | 3 | INPUT_NODE = 784 4 | OUTPUT_NODE = 10 5 | LAYER1_NODE = 500 6 | 7 | def get_weight_variable(shape, regularizer): 8 | weights = tf.get_variable("weights", shape, initializer=tf.truncated_normal_initializer(stddev=0.1)) 9 | if regularizer != None: tf.add_to_collection('losses', regularizer(weights)) 10 | return weights 11 | 12 | 13 | def inference(input_tensor, regularizer): 14 | with tf.variable_scope('layer1'): 15 | 16 | weights = get_weight_variable([INPUT_NODE, LAYER1_NODE], regularizer) 17 | biases = tf.get_variable("biases", [LAYER1_NODE], initializer=tf.constant_initializer(0.0)) 18 | layer1 = tf.nn.relu(tf.matmul(input_tensor, weights) + biases) 19 | 20 | with tf.variable_scope('layer2'): 21 | weights = get_weight_variable([LAYER1_NODE, OUTPUT_NODE], regularizer) 22 | biases = tf.get_variable("biases", [OUTPUT_NODE], initializer=tf.constant_initializer(0.0)) 23 | layer2 = tf.matmul(layer1, weights) + biases 24 | 25 | return layer2 -------------------------------------------------------------------------------- /Python技巧（04.迭代器与生成器）.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 13, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "name": "stdout", 10 | "output_type": "stream", 11 | "text": [ 12 | "#\n", 13 | "# Mac OS X Notice\n", 14 | "#\n", 15 | "# This file is not used by the host name and address resolution\n", 16 | "# or the DNS query routing mechanisms used by most processes on\n", 17 | "# this Mac OS X system.\n", 18 | "#\n", 19 | "# This file is automatically generated.\n", 20 | "#\n", 21 | "nameserver 114.114.114.114\n", 22 | "@@@@@@\n", 23 | "#\n", 24 | "# Mac OS X Notice\n", 25 | "#\n", 26 | "# This file is not used by the host name and address resolution\n", 27 | "# or the DNS query routing mechanisms used by most processes on\n", 28 | "# this Mac OS X system.\n", 29 | "#\n", 30 | "# This file is automatically generated.\n", 31 | "#\n", 32 | "nameserver 114.114.114.114\n" 33 | ] 34 | } 35 | ], 36 | "source": [ 37 | "with open('/etc/resolv.conf') as f:\n", 38 | " while True:\n", 39 | " line = next(f, None)\n", 40 | " if line is None:\n", 41 | " break\n", 42 | " print(line, end='')\n", 43 | "\n", 44 | "print('@@@@@@')\n", 45 | "\n", 46 | "with open('/etc/resolv.conf') as f:\n", 47 | " for l in f:\n", 48 | " print(l, end='')" 49 | ] 50 | }, 51 | { 52 | "cell_type": "code", 53 | "execution_count": 16, 54 | "metadata": {}, 55 | "outputs": [ 56 | { 57 | "name": "stdout", 58 | "output_type": "stream", 59 | "text": [ 60 | "1\n", 61 | "2\n", 62 | "3\n" 63 | ] 64 | } 65 | ], 66 | "source": [ 67 | "items = [1, 2, 3]\n", 68 | "it = iter(items)\n", 69 | "for i in it:\n", 70 | " print(i)\n", 71 | " \n" 72 | ] 73 | } 74 | ], 75 | "metadata": { 76 | "kernelspec": { 77 | "display_name": "Python 3", 78 | "language": "python", 79 | "name": "python3" 80 | }, 81 | "language_info": { 82 | "codemirror_mode": { 83 | "name": "ipython", 84 | "version": 3 85 | }, 86 | "file_extension": ".py", 87 | "mimetype": "text/x-python", 88 | "name": "python", 89 | "nbconvert_exporter": "python", 90 | "pygments_lexer": "ipython3", 91 | "version": "3.6.2" 92 | }, 93 | "varInspector": { 94 | "cols": { 95 | "lenName": 16, 96 | "lenType": 16, 97 | "lenVar": 40 98 | }, 99 | "kernels_config": { 100 | "python": { 101 | "delete_cmd_postfix": "", 102 | "delete_cmd_prefix": "del ", 103 | "library": "var_list.py", 104 | "varRefreshCmd": "print(var_dic_list())" 105 | }, 106 | "r": { 107 | "delete_cmd_postfix": ") ", 108 | "delete_cmd_prefix": "rm(", 109 | "library": "var_list.r", 110 | "varRefreshCmd": "cat(var_dic_list()) " 111 | } 112 | }, 113 | "types_to_exclude": [ 114 | "module", 115 | "function", 116 | "builtin_function_or_method", 117 | "instance", 118 | "_Feature" 119 | ], 120 | "window_display": false 121 | } 122 | }, 123 | "nbformat": 4, 124 | "nbformat_minor": 2 125 | } 126 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # tensorflow学习笔记 2 | 3 | - 学习内容来源于：[caicloud / tensorflow-tutorial](https://github.com/caicloud/tensorflow-tutorial) 4 | 5 | - 电子书：《Tensorflow实战Google深度学习框架》 6 | 7 | ![Tensorflow实战Google深度学习框架](images/cover.JPEG) 8 | 9 | 对样例中python2.x代码以及注释进行补充和修正，修正为python3.x代码。代码使用[jupyter/jupyter](https://github.com/jupyter/jupyter)进行交互演示，运行以下命令演示： 10 | 11 | ```sh 12 | git clone https://github.com/cookeem/TensorFlow_learning_notes 13 | cd TensorFlow_learning_notes 14 | jupyter notebook 15 | ``` 16 | 17 | ## 目录： 18 | 1. [第3章 TensorFlow入门](Chapter03) 19 | 1. [第4章 深层神经网络](Chapter04) 20 | 1. [第5章 MNIST数字识别问题](Chapter05) 21 | 1. [第6章 图像识别与卷积神经网络](Chapter06) 22 | 1. [第7章 图像数据处理](Chapter07) 23 | 1. [第8章 循环神经网络](Chapter08) 24 | 1. [第9章 TensorBoard可视化](Chapter09) 25 | 1. [第10章 TensorFlow计算加速](Chapter10) 26 | 27 | ## 学习笔记： 28 | 29 | ### 1、多层：使用多层权重，例如多层全连接方式 30 | > 以下定义了三个隐藏层的全连接方式的神经网络 31 | > 样例代码： 32 | 33 | ```python 34 | import tensorflow as tf 35 | 36 | l1 = tf.matmul(x, w1) 37 | l2 = tf.matmul(l1, w2) 38 | y = tf.matmul(l2,w3) 39 | ``` 40 | 41 | ### 2、激活层：引入激活函数，让每一层去线性化 42 | > 激活函数有多种，例如常用的： 43 | > tf.nn.relu 44 | > tf.nn.tanh 45 | > tf.nn.sigmoid 46 | > tf.nn.elu 47 | > 样例代码： 48 | 49 | ```python 50 | import tensorflow as tf 51 | 52 | a = tf.nn.relu(tf.matmul(x, w1) + biase1) 53 | y = tf.nn.relu(tf.matmul(a, w2) + biase2) 54 | ``` 55 | 56 | ### 3、损失函数： 57 | > 经典损失函数，交叉熵（cross entropy） 58 | > 用于计算预测结果矩阵Y和实际结果矩阵Y_之间的距离 59 | > 样例代码： 60 | 61 | ```python 62 | import tensorflow as tf 63 | 64 | cross_entropy = -tf.reduce_mean(y_ * tf.log(tf.clip_by_value(y, 1e-10, 1.0))) 65 | ``` 66 | 67 | ```python 68 | import tensorflow as tf 69 | 70 | v = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) 71 | tf.reduce_mean(tf.clip_by_value(v, 0.0, 10.0)) 72 | ``` 73 | 74 | > 对于分类问题，通常把交叉熵与softmax回归一起使用 75 | 76 | ```python 77 | import tensorflow as tf 78 | 79 | cross_entropy = tf.nn.softmax_cross_entropy_with_logits(y, y_) 80 | ``` 81 | 82 | > 对于回归问题，通常使用mse（均方误差函数）计算损失函数 83 | 84 | ```python 85 | import tensorflow as tf 86 | 87 | mse_loss = tf.reduce_mean(tf.square(y_ - y)) 88 | 89 | # 与以下函数计算结果完全一致 90 | dataset_size = 1000 91 | mse_loss = tf.reduce_sum(tf.pow(y_ - y, 2)) / dataset_size 92 | ``` 93 | 94 | > 自定义条件化的损失函数 95 | 96 | ```python 97 | import tensorflow as tf 98 | 99 | loss_less = 10 100 | loss_more = 1 101 | loss = tf.reduce_sum(tf.where(tf.greater(y, y_), (y - y_) * loss_more, (y_ - y) * loss_less)) 102 | train_step = tf.train.AdamOptimizer(0.001).minimize(loss) 103 | ``` 104 | 105 | ### 4、神经网络优化算法，训练优化器 106 | > 一般优化器的目标是优化权重W和偏差biases，最小化损失函数的结果 107 | > 以下优化器会不断优化W和biases 108 | 109 | ```python 110 | import tensorflow as tf 111 | 112 | LEARNING_RATE = 0.001 113 | mse_loss = tf.reduce_mean(tf.square(y_ - y)) 114 | train_op = tf.train.AdamOptimizer(LEARNING_RATE).minimize(mse_loss) 115 | ``` 116 | 117 | ### 5、优化学习率LEARNING_RATE 118 | > 学习率设置过大可能导致无法收敛，学习率设置过小可能导致收敛过慢 119 | 120 | ```python 121 | import tensorflow as tf 122 | 123 | global_step = tf.Variable(0) 124 | learning_rate = tf.train.exponential_decay( 125 | learning_rate=0.1, 126 | global_step=global_step, 127 | decay_steps=100, 128 | decay_rate=0.96, 129 | staircase=True, 130 | name=None 131 | ) 132 | train_op = tf.train.AdamOptimizer(learning_rate).minimize(loss, global_step=global_step) 133 | ``` 134 | 135 | ### 6、过拟合问题（正则化） 136 | > 避免训练出来的模型过分复杂，即模型记住了所有数据（包括噪声引起的误差） 137 | > 因此需要引入正则化函数叠加的方式，避免模型出现过拟合 138 | 139 | ```python 140 | import tensorflow as tf 141 | 142 | v_lambda = 0.001 143 | w = tf.Variable(tf.random_normal([2, 1], stddev=1, seed=1)) 144 | y = tf.matmul(x, w) 145 | mse_loss = tf.reduce_mean(tf.square(y_ - y) + tf.contrib.layers.l2_regularizer(v_lambda)(w)) 146 | ``` 147 | 148 | ### 7、滑动平均模型 149 | > 用于控制模型的变化速度，可以控制权重W以及偏差biases 150 | > 例如：avg_class.average(w) avg_class.average(biases) 151 | 152 | ```python 153 | import tensorflow as tf 154 | 155 | v1 = tf.Variable(0, dtype=tf.float32) 156 | step = tf.Variable(0, trainable=False) 157 | ema = tf.train.ExponentialMovingAverage(decay=0.99, num_updates=step) 158 | # 每一次操作的时候，列表变量[v1]都会被更新 159 | maintain_averages_op = ema.apply([v1]) 160 | 161 | with tf.Session() as sess: 162 | 163 | # 初始化 164 | init_op = tf.global_variables_initializer() 165 | sess.run(init_op) 166 | print(sess.run([v1, ema.average(v1)])) 167 | 168 | # 更新step和v1的取值 169 | sess.run(tf.assign(step, 10000)) 170 | sess.run(tf.assign(v1, 10)) 171 | sess.run(maintain_averages_op) 172 | print(sess.run([v1, ema.average(v1)])) 173 | ``` 174 | -------------------------------------------------------------------------------- /images/cover.JPEG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cookeem/TensorFlow_learning_notes/187a93bba93b74a44641c9d3f0990adb5236fdbc/images/cover.JPEG --------------------------------------------------------------------------------