├── .gitignore ├── PyTorch-Handbook-QR.png ├── README.md ├── changelog-v1.0.md ├── chapter1 ├── 1.1-pytorch-introduction.md ├── 1.2-pytorch-installation.md ├── 1.3-deep-learning-with-pytorch-60-minute-blitz.md ├── 1.4-pytorch-resource.md ├── 1_tensor_tutorial.ipynb ├── 2_autograd_tutorial.ipynb ├── 3_neural_networks_tutorial.ipynb ├── 4_cifar10_tutorial.ipynb ├── 5_data_parallel_tutorial.ipynb ├── autograd_tutorial.ipynb ├── cifar10_tutorial.ipynb ├── data_parallel_tutorial.ipynb ├── neural_networks_tutorial.ipynb ├── pic1.png ├── readme.md └── tensor_tutorial.ipynb ├── chapter2 ├── 1.png ├── 10.png ├── 11.png ├── 12.png ├── 2.1.1.pytorch-basics-tensor.ipynb ├── 2.1.2-pytorch-basics-autograd.ipynb ├── 2.1.3-pytorch-basics-nerual-network.ipynb ├── 2.1.4-pytorch-basics-data-lorder.ipynb ├── 2.2-deep-learning-basic-mathematics.ipynb ├── 2.3-deep-learning-neural-network-introduction.ipynb ├── 2.4-cnn.ipynb ├── 2.5-rnn.ipynb ├── 2.png ├── 3.png ├── 4.png ├── 5.png ├── 6.png ├── 7.png ├── 8.png ├── 9.gif ├── alexnet.png ├── cnn.png ├── googlenet.png ├── gru.gif ├── lenet5.jpg ├── lstm.gif ├── median_benchmark.csv ├── readme.md ├── resnet.png ├── resnet18.jpg └── vgg16.png ├── chapter3 ├── 3.1-logistic-regression.ipynb ├── 3.2-mnist.ipynb ├── 3.3-rnn.ipynb ├── german.data-numeric └── readme.md └── chapter4 ├── 1.PNG ├── 1280px-Felis_silvestris_catus_lying_on_rice_straw.jpg ├── 2.PNG ├── 3.PNG ├── 4.1-fine-tuning.ipynb ├── 4.2.1-visdom.ipynb ├── 4.2.2-tensorboardx.ipynb ├── readme.md └── static.zip /.gitignore: -------------------------------------------------------------------------------- 1 | .ipynb_checkpoints 2 | -------------------------------------------------------------------------------- /PyTorch-Handbook-QR.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/PyTorch-Handbook-QR.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # PyTorch 中文手册（pytorch handbook） 2 | ![pytorch](https://raw.githubusercontent.com/pytorch/pytorch/master/docs/source/_static/img/pytorch-logo-dark.png) 3 | 4 | ## 书籍介绍 5 | 这是一本开源的书籍，目标是帮助那些希望和使用PyTorch进行深度学习开发和研究的朋友快速入门。 6 | 7 | 由于本人水平有限，在写此教程的时候参考了一些网上的资料，在这里对他们表示敬意，我会在每个引用中附上原文地址，方便大家参考。 8 | 9 | 深度学习的技术在飞速的发展，同时PyTorch也在不断更新，且本人会逐步完善相关内容。 10 | 11 | ## 版本说明 12 | 由于PyTorch版本更迭，教程的版本会与PyTorch版本，保持一致。 13 | 14 | 12月8日PyTorch已经发布1.0的稳定版。 15 | API的改动不是很大，本教程已经通过测试，保证能够在1.0中正常运行。 16 | 不过目前看影响不大，因为毕竟内容还不多。 v0.4.1已经新建了分支作为存档，并且该不会再进行更新了。 17 | 18 | [官方1.0说明](https://github.com/pytorch/pytorch/releases/tag/v1.0.0) 19 | [主要改动中文说明](changelog-v1.0.md) 20 | 21 | ## QQ群 22 | 群号：985896536 23 | 24 | 扫描二维码 25 | 26 | ![QR](PyTorch-Handbook-QR.png) 27 | 28 | 29 | 或 30 | 31 | [点击链接加入群聊 『PyTorch Handbook 交流群』](https://jq.qq.com/?_wv=1027&k=5L28MSZ) 32 | 33 | ## 问题 34 | 35 | 修改错别字请直接提issue或者fork后直接 pull request 36 | 37 | 有问题也请直接提issue 38 | 39 | 感谢 40 | 41 | ## 目录 42 | 43 | ### 第一章：PyTorch 入门 44 | 45 | 1. [PyTorch 简介](chapter1/1.1-pytorch-introduction.md) 46 | 2. [PyTorch 环境搭建](chapter1/1.2-pytorch-installation.md) 47 | 3. [PyTorch 深度学习：60分钟快速入门（官方）](chapter1/1.3-deep-learning-with-pytorch-60-minute-blitz.md) 48 | - [张量](chapter1/1_tensor_tutorial.ipynb) 49 | - [Autograd：自动求导](chapter1/2_autograd_tutorial.ipynb) 50 | - [神经网络](chapter1/3_neural_networks_tutorial.ipynb) 51 | - [训练一个分类器](chapter1/4_cifar10_tutorial.ipynb) 52 | - [选读：数据并行处理（多GPU）](chapter1/5_data_parallel_tutorial.ipynb) 53 | 4. [相关资源介绍](chapter1/1.4-pytorch-resource.md) 54 | 55 | ### 第二章 基础 56 | #### 第一节 PyTorch 基础 57 | 1. [张量](chapter2/2.1.1.pytorch-basics-tensor.ipynb) 58 | 2. [自动求导](chapter2/2.1.2-pytorch-basics-autograd.ipynb) 59 | 3. [神经网络包nn和优化器optm](chapter2/2.1.3-pytorch-basics-nerual-network.ipynb) 60 | 4. [数据的加载和预处理](chapter2/2.1.4-pytorch-basics-data-lorder.ipynb) 61 | #### 第二节 深度学习基础及数学原理 62 | 63 | [深度学习基础及数学原理](chapter2/2.2-deep-learning-basic-mathematics.ipynb) 64 | 65 | #### 第三节 神经网络简介 66 | 67 | [神经网络简介](chapter2/2.3-deep-learning-neural-network-introduction.ipynb) 68 | 69 | #### 第四节 卷积神经网络 70 | 71 | [卷积神经网络](chapter2/2.4-cnn.ipynb) 72 | 73 | #### 第五节 循环神经网络 74 | 75 | [循环神经网络](chapter2/2.5-rnn.ipynb) 76 | 77 | ### 第三章 实践 78 | #### 第一节 logistic回归二元分类 79 | 80 | [logistic回归二元分类](chapter3/3.1-logistic-regression.ipynb) 81 | 82 | 83 | #### 第二节 CNN:MNIST数据集手写数字识别 84 | 85 | [CNN:MNIST数据集手写数字识别](chapter3/3.2-mnist.ipynb) 86 | 87 | #### 第三节 RNN实例：通过Sin预测Cos 88 | 89 | [RNN实例：通过Sin预测Cos](chapter3/3.3-rnn.ipynb) 90 | 91 | ### 第四章 提高 92 | #### 第一节 Fine-tuning 93 | 94 | [Fine-tuning](chapter4/4.1-fine-tuning.ipynb) 95 | 96 | #### 第二节 可视化 97 | 98 | [visdom](chapter4/4.2.1-visdom.ipynb) 99 | 100 | [tensorboardx](chapter4/4.2.2-tensorboardx.ipynb) 101 | 102 | #### 第三节 fastai 103 | #### 第四节 训练的一些技巧 104 | #### 第五节 并行计算 105 | 106 | ### 第五章 应用 107 | #### 第一节 Kaggle介绍 108 | #### 第二节 结构化数据 109 | #### 第三节 计算机视觉 110 | #### 第四节 自然语言处理 111 | #### 第五节 协同过滤 112 | 113 | ### 第六章 资源 114 | 115 | 116 | ### 第七章 附录 117 | transforms的常用操作总结 118 | 119 | pytorch的损失函数总结 120 | 121 | pytorch的优化器总结 122 | 123 | ## License 124 | 125 | ![](https://i.creativecommons.org/l/by-nc-sa/3.0/88x31.png) 126 | 127 | [本作品采用知识共享署名-非商业性使用-相同方式共享 3.0 中国大陆许可协议进行许可](http://creativecommons.org/licenses/by-nc-sa/3.0/cn) 128 | -------------------------------------------------------------------------------- /changelog-v1.0.md: -------------------------------------------------------------------------------- 1 | # PyTorch 1.0主要改动说明 2 | 本改动说明只针对主要部分进行阐述，详情还请以官方英文为准 3 | 4 | ## JIT 5 | JIT是一组编译工具，用来弥补研究和产品部署之间的差距。 6 | 7 | 新增的torch.jit包，包含了一组编译器工具，用于搭建PyTorch的研究与工业生产之间的桥梁。 8 | 它包括一种名为Torch Script的语言(单从语法上来看，这是Python语言的一个子集，所以我们不用再学一门新语言了)，以及两种使现有代码与JIT兼容的方法。 9 | Torch Script的代码是可以进行积极的优化（类似TensorFlow代码），可以序列化，以供以后在C++ API中使用，它完全不依赖于Python。 10 | 11 | `@torch.jit.script`注解，官方给出的注释是Write in Python, run anywhere! 12 | 13 | ## 改进的分布式库torch.distributed 14 | - 新的torch.distributed是性能驱动的，并且对所有后端（Gloo，NCCL和MPI）完全异步操作 15 | - 显著的分布式数据并行性能改进，尤其适用于网络较慢的主机，如基于以太网的主机 16 | - 为torch.distributed包中的所有分布式集合操作添加异步支持 17 | - 在Gloo后端添加以下CPU操作：send，recv，reduce，all_gather，gather，scatter 18 | - 在NCCL后端添加障碍操作 19 | - 为NCCL后端添加new_group支持 20 | 21 | 其实主要含义就是分布式的性能得到了改进，具体改进多少后面还要进行测试了 22 | 23 | ## C++的前端（C++ 版 Torch) 24 | C++前端是到PyTorch后端的纯C++接口，遵循已建立的Python前端的API和体系结构。它旨在支持在高性能、低延迟和硬核C++应用程序中的研究。它相当于torch.nn, torch.data等PyTorch的Python前端组件。 25 | 26 | 这个对我来说意义不大，不详细说了 27 | 28 | ## Torch Hub 29 | 这个一听就知道，google刚推出TensorFlow Hub不久，FB就跟进了，受益最大的就是“没有技术含量的公司可以再吹一波牛逼了” 30 | 31 | ## 官方提供免费课程 32 | 与Udacity合作，免费提供成人AI课程： 33 | 34 | [PyTorch深度学习简介](https://cn.udacity.com/course/deep-learning-pytorch--ud188) 35 | 36 | 看样子如果出中文版的话可能要有一阵子了，不过 I love study，study makes me happy. 37 | 38 | 39 | ## 比较重要的改动 40 | 41 | - torch.distributed的 TCP 后端被移除了，官方建议：CPU用Gloo或者MPI, GPU用英伟达的NCCL 42 | - 使用tensor的0下标返回数据的方法（如：loss[0]），彻底移除, 你需要使用loss.item() 这个可能比较重要，因为标量（0阶张量）的操作一定要用到这个 43 | - 移除了直接在CUDA上调用numpy函数的隐式类型转换，现在你需要手动把数据从CUDA先移到CPU上 这个对我们来说不重要，我们一直都是先移动到cpu再转化成numpy函数的 -------------------------------------------------------------------------------- /chapter1/1.1-pytorch-introduction.md: -------------------------------------------------------------------------------- 1 | # 1.1 Pytorch 简介 2 | 3 | ## 1.1.1 PyTorch的由来 4 | 很多人都会拿PyTorch和Google的Tensorflow进行比较，这个肯定是没有问题的，因为他们是最火的两个深度学习框架了。但是说到PyTorch，其实应该先说[Torch](http://torch.ch)。 5 | 6 | ## 1.1.2 Torch是什么？ 7 | 8 | **Torch英译中：火炬** 9 | 10 | A Tensor library like Numpy, unlike Numpy it has strong GPU support. Lua is a wrapper for Torch (Yes! you need to have a good understanding of Lua), and for that you will need LuaRocks package manager. 11 | [1](https://stackoverflow.com/questions/44371560/what-is-the-relationship-between-pytorch-and-torch) 12 | 13 | Torch是一个与Numpy类似的张量（Tensor）操作库，与Numpy不同的是Torch对GPU支持的很好，Lua是Torch的上层包装。 14 | 15 | Torch is not going anywhere. PyTorch and Torch use the same C libraries that contain all the performance: TH, THC, THNN, THCUNN and they will continue to be shared. 16 | We still and will have continued engineering on Torch itself, and we have no immediate plan to remove that. 17 | [2](https://discuss.pytorch.org/t/roadmap-for-torch-and-pytorch/38) 18 | 19 | PyTorch和Torch使用包含所有相同性能的C库：TH, THC, THNN, THCUNN，并且它们将继续共享这些库。 20 | 21 | 这样的回答就很明确了，其实PyTorch和Torch都使用的是相同的底层，只是使用了不同的上层包装语言。 22 | 23 | 注：LUA虽然快，但是太小众了，所以才会有PyTorch的出现。 24 | 25 | ## 1.1.3 重新介绍 PyTorch 26 | PyTorch is an open source machine learning library for Python, based on Torch, used for applications such as natural language processing. It is primarily developed by Facebook's artificial-intelligence research group, and Uber's "Pyro" software for probabilistic programming is built on it. 27 | [3](https://en.wikipedia.org/wiki/PyTorch) 28 | 29 | PyTorch是一个基于Torch的Python开源机器学习库，用于自然语言处理等应用程序。 它主要由Facebook的人工智能研究小组开发。Uber的"Pyro"也是使用的这个库。 30 | 31 | PyTorch is a Python package that provides two high-level features: 32 | 33 | Tensor computation (like NumPy) with strong GPU acceleration 34 | 35 | Deep neural networks built on a tape-based autograd system 36 | You can reuse your favorite Python packages such as NumPy, SciPy and Cython to extend PyTorch when needed. 37 | [4](https://github.com/pytorch/pytorch) 38 | 39 | PyTorch是一个Python包，提供两个高级功能： 40 | * 具有强大的GPU加速的张量计算（如NumPy） 41 | * 包含自动求导系统的的深度神经网络 42 | 43 | ## 1.1.4 对比PyTorch和Tensorflow 44 | 没有好的框架，只有合适的框架， 这里有个简单的对比，所以这里就不详细再说了 45 | https://zhuanlan.zhihu.com/p/28636490 46 | 并且技术是发展的，这里的对比也不是绝对的，比如Tensorflow在1.5版的时候就引入了Eager Execution机制实现了动态图，PyTorch的可视化,windows支持，沿维翻转张量等问题都已经不是问题了。 47 | 48 | ## 1.1.5 再次总结 49 | 50 | - PyTorch算是相当简洁优雅且高效快速的框架 51 | - 设计追求最少的封装，尽量避免重复造轮子 52 | - 算是所有的框架中面向对象设计的最优雅的一个，设计最符合人们的思维，它让用户尽可能地专注于实现自己的想法 53 | - 大佬支持,与google的Tensorflow类似，FAIR的支持足以确保PyTorch获得持续的开发更新 54 | - 不错的的文档（相比FB的其他项目，PyTorch的文档简直算是完善了，参考Thrift），PyTorch作者亲自维护的论坛 供用户交流和求教问题 55 | - 入门简单 56 | 57 | 所以如果以上信息有吸引你的内容，那么请一定要读完这本书:) -------------------------------------------------------------------------------- /chapter1/1.2-pytorch-installation.md: -------------------------------------------------------------------------------- 1 | # 1.2 Pytorch环境搭建 2 | PyTorch的安装十分简单，根据[PyTorch官网](https://pytorch.org/)，对系统选择和安装方式等灵活选择即可。 3 | 这里以[anaconda](https://www.anaconda.com/)为例，简单的说一下步骤和要点。 4 | 国内安装anaconda建议使用[清华](https://mirrors.tuna.tsinghua.edu.cn/help/anaconda/)或者中科大[http://mirrors.ustc.edu.cn/help/anaconda.html]镜像，快的不是一点半点。 5 | 6 | # 1.2.1 安装Pytorch 7 | anaconda安装完成后可以开始创建环境，这里以win10 系统为例。打开Anaconda Prompt 8 | ```bash 9 | #pytorch为环境名，这里创建python3.6版。 10 | conda create -n pytorch python=3.6 11 | # 切换到pytorch环境 12 | activate pytorch 13 | #安装GPU版本，根据cuda版本选择cuda80，cuda92，如果cuda是9.0版，则不需要 14 | #直接conda install pytorch -c pytorch 即可 15 | # win下查看cuda版本命令nvcc -V 16 | conda install pytorch cuda92 -c pytorch 17 | # cpu版本使用 18 | # conda install pytorch-cpu -c pytorch 19 | 20 | # torchvision 是torch提供的计算机视觉工具包，后面介绍 21 | pip install torchvision 22 | ``` 23 | 需要说明的一点是如果使用清华源，可以直接添加pytorch源镜像去掉，并且去掉`-c pytorch` 这样才能使用镜像源。 24 | 25 | 验证输入python 进入 26 | ```python 27 | import torch 28 | torch.__version__ 29 | # 得到结果'0.4.1' 30 | ``` 31 | 32 | ## 1.2.2 配置 Jupyter Notebook 33 | 新建的环境是没有安装安装ipykernel的所以无法注册到Jupyter Notebook中，所以先要准备下环境 34 | ```bash 35 | #安装ipykernel 36 | conda install ipykernel 37 | #写入环境 38 | python -m ipykernel install --name pytorch --display-name "Pytorch for Deeplearning" 39 | ``` 40 | 下一步就是定制 Jupyter Notebook 41 | ```bash 42 | #切换回基础环境 43 | activate base 44 | #创建jupyter notebook配置文件 45 | jupyter notebook --generate-config 46 | ## 这里会显示创建jupyter_notebook_config.py的具体位置 47 | ``` 48 | 打开文件，修改 49 | ``` 50 | c.NotebookApp.notebook_dir = '' 默认目录位置 51 | c.NotebookApp.iopub_data_rate_limit = 100000000 这个改大一些否则有可能报错 52 | ``` 53 | 54 | ## 1.2.3 测试 55 | 至此 Pytorch 的开发环境安装完成，可以在开始菜单中打开Jupyter Notebook 在New 菜单中创建文件时选择`Pytorch for Deeplearning` 创建PyTorch的相关开发环境了 56 | 57 | ## 1.2.4 问题解决 58 | 59 | ### 问题1：启动python提示编码错误 60 | 61 | 删除 .python_history [来源](http://tantai.org/posts/install-keras-pytorch-jupyter-notebook-Anaconda-window-10-cpu/) 62 | ### 问题2 默认目录设置不起效 63 | 打开快捷方式，看看快捷方式是否跟这个截图一样，如果是则删除 `%USERPROFILE%` 改参数会覆盖掉notebook_dir设置，导致配置不起效 64 | 65 | ![Alt text](pic1.png) 66 | 67 | 如果你还发现其他问题，请直接留言 68 | 69 | 70 | -------------------------------------------------------------------------------- /chapter1/1.3-deep-learning-with-pytorch-60-minute-blitz.md: -------------------------------------------------------------------------------- 1 | # PyTorch 深度学习:60分钟快速入门 （官方） 2 | 3 | 本章为官方网站的 4 | [Deep Learning with PyTorch: A 60 Minute Blitz] 5 | (https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html) 6 | 的中文翻译，目前在网上看到所有中文翻译版本都已经过时了，所以才又从新翻译了一遍，确保与官方同步 7 | 8 | ## 目录 9 | 10 | 1. [张量](1_tensor_tutorial.ipynb) 11 | 2. [Autograd: 自动求导](2_autograd_tutorial.ipynb) **本章是冲突的重灾区，建议阅读** 12 | 3. [神经网络](3_neural_networks_tutorial.ipynb) 13 | 4. [训练一个分类器](4_cifar10_tutorial.ipynb) 14 | 5. [选读：数据并行处理(多GPU)](5_data_parallel_tutorial.ipynb) 15 | 16 | ## 说明 17 | 本章中的所有图片均来自于PyTorch官网，版权归PyTorch所有. -------------------------------------------------------------------------------- /chapter1/1.4-pytorch-resource.md: -------------------------------------------------------------------------------- 1 | # 相关资源列表 -------------------------------------------------------------------------------- /chapter1/1_tensor_tutorial.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%matplotlib inline" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "\n", 17 | "PyTorch是什么?\n", 18 | "================\n", 19 | "\n", 20 | "基于Python的科学计算包，服务于以下两种场景:\n", 21 | "\n", 22 | "- 作为NumPy的替代品，可以使用GPU的强大计算能力\n", 23 | "- 提供最大的灵活性和高速的深度学习研究平台\n", 24 | " \n", 25 | "\n", 26 | "开始\n", 27 | "---------------\n", 28 | "\n", 29 | "Tensors（张量）\n", 30 | "\n", 31 | "^^^^^^^\n", 32 | "\n", 33 | "Tensors与Numpy中的 ndarrays类似，但是在PyTorch中\n", 34 | "Tensors 可以使用GPU进行计算.\n", 35 | "\n" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": 2, 41 | "metadata": {}, 42 | "outputs": [], 43 | "source": [ 44 | "from __future__ import print_function\n", 45 | "import torch" 46 | ] 47 | }, 48 | { 49 | "cell_type": "markdown", 50 | "metadata": {}, 51 | "source": [ 52 | "创建一个 5x3 矩阵, 但是未初始化:\n", 53 | "\n" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": 3, 59 | "metadata": {}, 60 | "outputs": [ 61 | { 62 | "name": "stdout", 63 | "output_type": "stream", 64 | "text": [ 65 | "tensor([[0.0000, 0.0000, 0.0000],\n", 66 | " [0.0000, 0.0000, 0.0000],\n", 67 | " [0.0000, 0.0000, 0.0000],\n", 68 | " [0.0000, 0.0000, 0.0000],\n", 69 | " [0.0000, 0.0000, 0.0000]])\n" 70 | ] 71 | } 72 | ], 73 | "source": [ 74 | "x = torch.empty(5, 3)\n", 75 | "print(x)" 76 | ] 77 | }, 78 | { 79 | "cell_type": "markdown", 80 | "metadata": {}, 81 | "source": [ 82 | "创建一个随机初始化的矩阵:\n", 83 | "\n" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": 4, 89 | "metadata": {}, 90 | "outputs": [ 91 | { 92 | "name": "stdout", 93 | "output_type": "stream", 94 | "text": [ 95 | "tensor([[0.6972, 0.0231, 0.3087],\n", 96 | " [0.2083, 0.6141, 0.6896],\n", 97 | " [0.7228, 0.9715, 0.5304],\n", 98 | " [0.7727, 0.1621, 0.9777],\n", 99 | " [0.6526, 0.6170, 0.2605]])\n" 100 | ] 101 | } 102 | ], 103 | "source": [ 104 | "x = torch.rand(5, 3)\n", 105 | "print(x)" 106 | ] 107 | }, 108 | { 109 | "cell_type": "markdown", 110 | "metadata": {}, 111 | "source": [ 112 | "创建一个0填充的矩阵，数据类型为long:\n", 113 | "\n" 114 | ] 115 | }, 116 | { 117 | "cell_type": "code", 118 | "execution_count": 5, 119 | "metadata": {}, 120 | "outputs": [ 121 | { 122 | "name": "stdout", 123 | "output_type": "stream", 124 | "text": [ 125 | "tensor([[0, 0, 0],\n", 126 | " [0, 0, 0],\n", 127 | " [0, 0, 0],\n", 128 | " [0, 0, 0],\n", 129 | " [0, 0, 0]])\n" 130 | ] 131 | } 132 | ], 133 | "source": [ 134 | "x = torch.zeros(5, 3, dtype=torch.long)\n", 135 | "print(x)" 136 | ] 137 | }, 138 | { 139 | "cell_type": "markdown", 140 | "metadata": {}, 141 | "source": [ 142 | "创建tensor并使用现有数据初始化:\n", 143 | "\n" 144 | ] 145 | }, 146 | { 147 | "cell_type": "code", 148 | "execution_count": 6, 149 | "metadata": {}, 150 | "outputs": [ 151 | { 152 | "name": "stdout", 153 | "output_type": "stream", 154 | "text": [ 155 | "tensor([5.5000, 3.0000])\n" 156 | ] 157 | } 158 | ], 159 | "source": [ 160 | "x = torch.tensor([5.5, 3])\n", 161 | "print(x)" 162 | ] 163 | }, 164 | { 165 | "cell_type": "markdown", 166 | "metadata": {}, 167 | "source": [ 168 | "根据现有的张量创建张量。 这些方法将重用输入张量的属性，例如， dtype，除非设置新的值进行覆盖" 169 | ] 170 | }, 171 | { 172 | "cell_type": "code", 173 | "execution_count": 7, 174 | "metadata": {}, 175 | "outputs": [ 176 | { 177 | "name": "stdout", 178 | "output_type": "stream", 179 | "text": [ 180 | "tensor([[1., 1., 1.],\n", 181 | " [1., 1., 1.],\n", 182 | " [1., 1., 1.],\n", 183 | " [1., 1., 1.],\n", 184 | " [1., 1., 1.]], dtype=torch.float64)\n", 185 | "tensor([[ 0.5691, -2.0126, -0.4064],\n", 186 | " [-0.0863, 0.4692, -1.1209],\n", 187 | " [-1.1177, -0.5764, -0.5363],\n", 188 | " [-0.4390, 0.6688, 0.0889],\n", 189 | " [ 1.3334, -1.1600, 1.8457]])\n" 190 | ] 191 | } 192 | ], 193 | "source": [ 194 | "x = x.new_ones(5, 3, dtype=torch.double) # new_* 方法来创建对象\n", 195 | "print(x)\n", 196 | "\n", 197 | "x = torch.randn_like(x, dtype=torch.float) # 覆盖 dtype!\n", 198 | "print(x) # 对象的size 是相同的，只是值和类型发生了变化" 199 | ] 200 | }, 201 | { 202 | "cell_type": "markdown", 203 | "metadata": {}, 204 | "source": [ 205 | "获取 size\n", 206 | "\n", 207 | "***译者注：使用size方法与Numpy的shape属性返回的相同，张量也支持shape属性，后面会详细介绍***\n" 208 | ] 209 | }, 210 | { 211 | "cell_type": "code", 212 | "execution_count": 8, 213 | "metadata": {}, 214 | "outputs": [ 215 | { 216 | "name": "stdout", 217 | "output_type": "stream", 218 | "text": [ 219 | "torch.Size([5, 3])\n" 220 | ] 221 | } 222 | ], 223 | "source": [ 224 | "print(x.size())" 225 | ] 226 | }, 227 | { 228 | "cell_type": "markdown", 229 | "metadata": {}, 230 | "source": [ 231 | "

Note

``torch.Size`` 返回值是 tuple类型, 所以它支持tuple类型的所有操作.

\n", 232 | "\n", 233 | "操作\n", 234 | "\n", 235 | "^^^^^^^^^^\n", 236 | "\n", 237 | "操作有多种语法。 \n", 238 | "\n", 239 | "我们将看一下加法运算。\n", 240 | "\n", 241 | "加法1:" 242 | ] 243 | }, 244 | { 245 | "cell_type": "code", 246 | "execution_count": 9, 247 | "metadata": {}, 248 | "outputs": [ 249 | { 250 | "name": "stdout", 251 | "output_type": "stream", 252 | "text": [ 253 | "tensor([[ 0.7808, -1.4388, 0.3151],\n", 254 | " [-0.0076, 1.0716, -0.8465],\n", 255 | " [-0.8175, 0.3625, -0.2005],\n", 256 | " [ 0.2435, 0.8512, 0.7142],\n", 257 | " [ 1.4737, -0.8545, 2.4833]])\n" 258 | ] 259 | } 260 | ], 261 | "source": [ 262 | "y = torch.rand(5, 3)\n", 263 | "print(x + y)" 264 | ] 265 | }, 266 | { 267 | "cell_type": "markdown", 268 | "metadata": {}, 269 | "source": [ 270 | "加法2\n" 271 | ] 272 | }, 273 | { 274 | "cell_type": "code", 275 | "execution_count": 10, 276 | "metadata": {}, 277 | "outputs": [ 278 | { 279 | "name": "stdout", 280 | "output_type": "stream", 281 | "text": [ 282 | "tensor([[ 0.7808, -1.4388, 0.3151],\n", 283 | " [-0.0076, 1.0716, -0.8465],\n", 284 | " [-0.8175, 0.3625, -0.2005],\n", 285 | " [ 0.2435, 0.8512, 0.7142],\n", 286 | " [ 1.4737, -0.8545, 2.4833]])\n" 287 | ] 288 | } 289 | ], 290 | "source": [ 291 | "print(torch.add(x, y))" 292 | ] 293 | }, 294 | { 295 | "cell_type": "markdown", 296 | "metadata": {}, 297 | "source": [ 298 | "提供输出tensor作为参数\n", 299 | "\n" 300 | ] 301 | }, 302 | { 303 | "cell_type": "code", 304 | "execution_count": 11, 305 | "metadata": {}, 306 | "outputs": [ 307 | { 308 | "name": "stdout", 309 | "output_type": "stream", 310 | "text": [ 311 | "tensor([[ 0.7808, -1.4388, 0.3151],\n", 312 | " [-0.0076, 1.0716, -0.8465],\n", 313 | " [-0.8175, 0.3625, -0.2005],\n", 314 | " [ 0.2435, 0.8512, 0.7142],\n", 315 | " [ 1.4737, -0.8545, 2.4833]])\n" 316 | ] 317 | } 318 | ], 319 | "source": [ 320 | "result = torch.empty(5, 3)\n", 321 | "torch.add(x, y, out=result)\n", 322 | "print(result)" 323 | ] 324 | }, 325 | { 326 | "cell_type": "markdown", 327 | "metadata": {}, 328 | "source": [ 329 | "替换\n", 330 | "\n" 331 | ] 332 | }, 333 | { 334 | "cell_type": "code", 335 | "execution_count": 12, 336 | "metadata": {}, 337 | "outputs": [ 338 | { 339 | "name": "stdout", 340 | "output_type": "stream", 341 | "text": [ 342 | "tensor([[ 0.7808, -1.4388, 0.3151],\n", 343 | " [-0.0076, 1.0716, -0.8465],\n", 344 | " [-0.8175, 0.3625, -0.2005],\n", 345 | " [ 0.2435, 0.8512, 0.7142],\n", 346 | " [ 1.4737, -0.8545, 2.4833]])\n" 347 | ] 348 | } 349 | ], 350 | "source": [ 351 | "# adds x to y\n", 352 | "y.add_(x)\n", 353 | "print(y)" 354 | ] 355 | }, 356 | { 357 | "cell_type": "markdown", 358 | "metadata": {}, 359 | "source": [ 360 | "

Note

任何 以``_`` 结尾的操作都会用结果替换原变量.\n", 361 | " 例如: ``x.copy_(y)``, ``x.t_()``, 都会改变 ``x``.

\n", 362 | "\n", 363 | "你可以使用与NumPy索引方式相同的操作来进行对张量的操作\n", 364 | "\n" 365 | ] 366 | }, 367 | { 368 | "cell_type": "code", 369 | "execution_count": 13, 370 | "metadata": {}, 371 | "outputs": [ 372 | { 373 | "name": "stdout", 374 | "output_type": "stream", 375 | "text": [ 376 | "tensor([-2.0126, 0.4692, -0.5764, 0.6688, -1.1600])\n" 377 | ] 378 | } 379 | ], 380 | "source": [ 381 | "print(x[:, 1])" 382 | ] 383 | }, 384 | { 385 | "cell_type": "markdown", 386 | "metadata": {}, 387 | "source": [ 388 | "``torch.view``: 可以改变张量的维度和大小\n", 389 | "\n", 390 | "***译者注：torch.view 与Numpy的reshape类似***\n", 391 | "\n" 392 | ] 393 | }, 394 | { 395 | "cell_type": "code", 396 | "execution_count": 14, 397 | "metadata": {}, 398 | "outputs": [ 399 | { 400 | "name": "stdout", 401 | "output_type": "stream", 402 | "text": [ 403 | "torch.Size([4, 4]) torch.Size([16]) torch.Size([2, 8])\n" 404 | ] 405 | } 406 | ], 407 | "source": [ 408 | "x = torch.randn(4, 4)\n", 409 | "y = x.view(16)\n", 410 | "z = x.view(-1, 8) # size -1 从其他维度推断\n", 411 | "print(x.size(), y.size(), z.size())" 412 | ] 413 | }, 414 | { 415 | "cell_type": "markdown", 416 | "metadata": {}, 417 | "source": [ 418 | "如果你有只有一个元素的张量，使用``.item()``来得到Python数据类型的数值\n" 419 | ] 420 | }, 421 | { 422 | "cell_type": "code", 423 | "execution_count": 15, 424 | "metadata": {}, 425 | "outputs": [ 426 | { 427 | "name": "stdout", 428 | "output_type": "stream", 429 | "text": [ 430 | "tensor([-0.2368])\n", 431 | "-0.23680149018764496\n" 432 | ] 433 | } 434 | ], 435 | "source": [ 436 | "x = torch.randn(1)\n", 437 | "print(x)\n", 438 | "print(x.item())" 439 | ] 440 | }, 441 | { 442 | "cell_type": "markdown", 443 | "metadata": {}, 444 | "source": [ 445 | "**Read later:**\n", 446 | "\n", 447 | "\n", 448 | " 100+ Tensor operations, including transposing, indexing, slicing,\n", 449 | " mathematical operations, linear algebra, random numbers, etc.,\n", 450 | " are described\n", 451 | " `here `_.\n", 452 | "\n", 453 | "NumPy 转换\n", 454 | "------------\n", 455 | "\n", 456 | "Converting a Torch Tensor to a NumPy array and vice versa is a breeze.\n", 457 | "\n", 458 | "The Torch Tensor and NumPy array will share their underlying memory\n", 459 | "locations, and changing one will change the other.\n", 460 | "\n", 461 | "Converting a Torch Tensor to a NumPy Array\n", 462 | "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n", 463 | "\n" 464 | ] 465 | }, 466 | { 467 | "cell_type": "code", 468 | "execution_count": 16, 469 | "metadata": {}, 470 | "outputs": [ 471 | { 472 | "name": "stdout", 473 | "output_type": "stream", 474 | "text": [ 475 | "tensor([1., 1., 1., 1., 1.])\n" 476 | ] 477 | } 478 | ], 479 | "source": [ 480 | "a = torch.ones(5)\n", 481 | "print(a)" 482 | ] 483 | }, 484 | { 485 | "cell_type": "code", 486 | "execution_count": 17, 487 | "metadata": {}, 488 | "outputs": [ 489 | { 490 | "name": "stdout", 491 | "output_type": "stream", 492 | "text": [ 493 | "[1. 1. 1. 1. 1.]\n" 494 | ] 495 | } 496 | ], 497 | "source": [ 498 | "b = a.numpy()\n", 499 | "print(b)" 500 | ] 501 | }, 502 | { 503 | "cell_type": "markdown", 504 | "metadata": {}, 505 | "source": [ 506 | "See how the numpy array changed in value.\n", 507 | "\n" 508 | ] 509 | }, 510 | { 511 | "cell_type": "code", 512 | "execution_count": 18, 513 | "metadata": {}, 514 | "outputs": [ 515 | { 516 | "name": "stdout", 517 | "output_type": "stream", 518 | "text": [ 519 | "tensor([2., 2., 2., 2., 2.])\n", 520 | "[2. 2. 2. 2. 2.]\n" 521 | ] 522 | } 523 | ], 524 | "source": [ 525 | "a.add_(1)\n", 526 | "print(a)\n", 527 | "print(b)" 528 | ] 529 | }, 530 | { 531 | "cell_type": "markdown", 532 | "metadata": {}, 533 | "source": [ 534 | " NumPy Array 转化成 Torch Tensor\n", 535 | "\n", 536 | "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n", 537 | "\n", 538 | "使用from_numpy自动转化\n", 539 | "\n" 540 | ] 541 | }, 542 | { 543 | "cell_type": "code", 544 | "execution_count": 19, 545 | "metadata": {}, 546 | "outputs": [ 547 | { 548 | "name": "stdout", 549 | "output_type": "stream", 550 | "text": [ 551 | "[2. 2. 2. 2. 2.]\n", 552 | "tensor([2., 2., 2., 2., 2.], dtype=torch.float64)\n" 553 | ] 554 | } 555 | ], 556 | "source": [ 557 | "import numpy as np\n", 558 | "a = np.ones(5)\n", 559 | "b = torch.from_numpy(a)\n", 560 | "np.add(a, 1, out=a)\n", 561 | "print(a)\n", 562 | "print(b)" 563 | ] 564 | }, 565 | { 566 | "cell_type": "markdown", 567 | "metadata": {}, 568 | "source": [ 569 | "\n", 570 | "所有的 Tensor 类型默认都是基于CPU， CharTensor 类型不支持到\n", 571 | "NumPy 的转换.\n", 572 | "CUDA 张量\n", 573 | "------------\n", 574 | "\n", 575 | "使用``.to`` 方法 可以将Tensor移动到任何设备中\n", 576 | "\n" 577 | ] 578 | }, 579 | { 580 | "cell_type": "code", 581 | "execution_count": 20, 582 | "metadata": { 583 | "scrolled": true 584 | }, 585 | "outputs": [ 586 | { 587 | "name": "stdout", 588 | "output_type": "stream", 589 | "text": [ 590 | "tensor([0.7632], device='cuda:0')\n", 591 | "tensor([0.7632], dtype=torch.float64)\n" 592 | ] 593 | } 594 | ], 595 | "source": [ 596 | "# is_available 函数判断是否有cuda可以使用\n", 597 | "# ``torch.device``将张量移动到指定的设备中\n", 598 | "if torch.cuda.is_available():\n", 599 | " device = torch.device(\"cuda\") # a CUDA 设备对象\n", 600 | " y = torch.ones_like(x, device=device) # 直接从GPU创建张量\n", 601 | " x = x.to(device) # 或者直接使用``.to(\"cuda\")``将张量移动到cuda中\n", 602 | " z = x + y\n", 603 | " print(z)\n", 604 | " print(z.to(\"cpu\", torch.double)) # ``.to`` 也会对变量的类型做更改" 605 | ] 606 | } 607 | ], 608 | "metadata": { 609 | "kernelspec": { 610 | "display_name": "Pytorch for Deeplearning", 611 | "language": "python", 612 | "name": "pytorch" 613 | }, 614 | "language_info": { 615 | "codemirror_mode": { 616 | "name": "ipython", 617 | "version": 3 618 | }, 619 | "file_extension": ".py", 620 | "mimetype": "text/x-python", 621 | "name": "python", 622 | "nbconvert_exporter": "python", 623 | "pygments_lexer": "ipython3", 624 | "version": "3.6.7" 625 | } 626 | }, 627 | "nbformat": 4, 628 | "nbformat_minor": 1 629 | } 630 | -------------------------------------------------------------------------------- /chapter1/2_autograd_tutorial.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%matplotlib inline" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "\n", 17 | "Autograd: 自动求导机制\n", 18 | "===================================\n", 19 | "\n", 20 | "PyTorch 中所有神经网络的核心是 ``autograd`` 包。\n", 21 | "我们先简单介绍一下这个包，然后训练第一个简单的神经网络。\n", 22 | "\n", 23 | "``autograd``包为张量上的所有操作提供了自动求导。\n", 24 | "它是一个在运行时定义的框架，这意味着反向传播是根据你的代码来确定如何运行，并且每次迭代可以是不同的。\n", 25 | "\n", 26 | "\n", 27 | "示例\n", 28 | "\n", 29 | "张量（Tensor）\n", 30 | "--------\n", 31 | "\n", 32 | "``torch.Tensor``是这个包的核心类。如果设置\n", 33 | "``.requires_grad`` 为 ``True``，那么将会追踪所有对于该张量的操作。 \n", 34 | "当完成计算后通过调用 ``.backward()``，自动计算所有的梯度，\n", 35 | "这个张量的所有梯度将会自动积累到 ``.grad`` 属性。\n", 36 | "\n", 37 | "要阻止张量跟踪历史记录，可以调用``.detach()``方法将其与计算历史记录分离，并禁止跟踪它将来的计算记录。\n", 38 | "\n", 39 | "为了防止跟踪历史记录（和使用内存），可以将代码块包装在``with torch.no_grad()：``中。\n", 40 | "在评估模型时特别有用，因为模型可能具有`requires_grad = True`的可训练参数，但是我们不需要梯度计算。\n", 41 | "\n", 42 | "在自动梯度计算中还有另外一个重要的类``Function``.\n", 43 | "\n", 44 | "``Tensor`` and ``Function`` are interconnected and build up an acyclic\n", 45 | "graph, that encodes a complete history of computation. Each tensor has\n", 46 | "a ``.grad_fn`` attribute that references a ``Function`` that has created\n", 47 | "the ``Tensor`` (except for Tensors created by the user - their\n", 48 | "``grad_fn is None``).\n", 49 | "\n", 50 | "``Tensor`` 和 ``Function``互相连接并生成一个非循环图，它表示和存储了完整的计算历史。\n", 51 | "每个张量都有一个``.grad_fn``属性，这个属性引用了一个创建了``Tensor``的``Function``（除非这个张量是用户手动创建的，即，这个张量的\n", 52 | "``grad_fn`` 是 ``None``）。\n", 53 | "\n", 54 | "如果需要计算导数，你可以在``Tensor``上调用``.backward()``。 \n", 55 | "如果``Tensor``是一个标量（即它包含一个元素数据）则不需要为``backward()``指定任何参数，\n", 56 | "但是如果它有更多的元素，你需要指定一个``gradient`` 参数来匹配张量的形状。\n" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "metadata": {}, 62 | "source": [ 63 | "***译者注：在其他的文章中你可能会看到说将Tensor包裹到Variable中提供自动梯度计算，Variable 这个在0.41版中已经被标注为过期了，现在可以直接使用Tensor，官方文档在这里：***\n", 64 | "(https://pytorch.org/docs/stable/autograd.html#variable-deprecated) \n", 65 | "\n", 66 | "具体的后面会有详细说明" 67 | ] 68 | }, 69 | { 70 | "cell_type": "code", 71 | "execution_count": 2, 72 | "metadata": {}, 73 | "outputs": [], 74 | "source": [ 75 | "import torch" 76 | ] 77 | }, 78 | { 79 | "cell_type": "markdown", 80 | "metadata": {}, 81 | "source": [ 82 | "创建一个张量并设置 requires_grad=True 用来追踪他的计算历史\n" 83 | ] 84 | }, 85 | { 86 | "cell_type": "code", 87 | "execution_count": 3, 88 | "metadata": {}, 89 | "outputs": [ 90 | { 91 | "name": "stdout", 92 | "output_type": "stream", 93 | "text": [ 94 | "tensor([[1., 1.],\n", 95 | " [1., 1.]], requires_grad=True)\n" 96 | ] 97 | } 98 | ], 99 | "source": [ 100 | "x = torch.ones(2, 2, requires_grad=True)\n", 101 | "print(x)" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "对张量进行操作:\n", 109 | "\n" 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": 4, 115 | "metadata": {}, 116 | "outputs": [ 117 | { 118 | "name": "stdout", 119 | "output_type": "stream", 120 | "text": [ 121 | "tensor([[3., 3.],\n", 122 | " [3., 3.]], grad_fn=)\n" 123 | ] 124 | } 125 | ], 126 | "source": [ 127 | "y = x + 2\n", 128 | "print(y)" 129 | ] 130 | }, 131 | { 132 | "cell_type": "markdown", 133 | "metadata": {}, 134 | "source": [ 135 | "结果``y``已经被计算出来了，所以，``grad_fn``已经被自动生成了。\n", 136 | "\n" 137 | ] 138 | }, 139 | { 140 | "cell_type": "code", 141 | "execution_count": 5, 142 | "metadata": {}, 143 | "outputs": [ 144 | { 145 | "name": "stdout", 146 | "output_type": "stream", 147 | "text": [ 148 | "\n" 149 | ] 150 | } 151 | ], 152 | "source": [ 153 | "print(y.grad_fn)" 154 | ] 155 | }, 156 | { 157 | "cell_type": "markdown", 158 | "metadata": {}, 159 | "source": [ 160 | "对y进行一个操作\n", 161 | "\n" 162 | ] 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": 6, 167 | "metadata": {}, 168 | "outputs": [ 169 | { 170 | "name": "stdout", 171 | "output_type": "stream", 172 | "text": [ 173 | "tensor([[27., 27.],\n", 174 | " [27., 27.]], grad_fn=) tensor(27., grad_fn=)\n" 175 | ] 176 | } 177 | ], 178 | "source": [ 179 | "z = y * y * 3\n", 180 | "out = z.mean()\n", 181 | "\n", 182 | "print(z, out)" 183 | ] 184 | }, 185 | { 186 | "cell_type": "markdown", 187 | "metadata": {}, 188 | "source": [ 189 | "``.requires_grad_( ... )`` 可以改变现有张量的 ``requires_grad``属性。\n", 190 | "如果没有指定的话，默认输入的flag是 ``False``。\n", 191 | "\n" 192 | ] 193 | }, 194 | { 195 | "cell_type": "code", 196 | "execution_count": 7, 197 | "metadata": {}, 198 | "outputs": [ 199 | { 200 | "name": "stdout", 201 | "output_type": "stream", 202 | "text": [ 203 | "False\n", 204 | "True\n", 205 | "\n" 206 | ] 207 | } 208 | ], 209 | "source": [ 210 | "a = torch.randn(2, 2)\n", 211 | "a = ((a * 3) / (a - 1))\n", 212 | "print(a.requires_grad)\n", 213 | "a.requires_grad_(True)\n", 214 | "print(a.requires_grad)\n", 215 | "b = (a * a).sum()\n", 216 | "print(b.grad_fn)" 217 | ] 218 | }, 219 | { 220 | "cell_type": "markdown", 221 | "metadata": {}, 222 | "source": [ 223 | "梯度\n", 224 | "---------\n", 225 | "反向传播\n", 226 | "因为 ``out``是一个纯量（scalar），``out.backward()`` 等于``out.backward(torch.tensor(1))``。\n", 227 | "\n" 228 | ] 229 | }, 230 | { 231 | "cell_type": "code", 232 | "execution_count": 8, 233 | "metadata": {}, 234 | "outputs": [], 235 | "source": [ 236 | "out.backward()" 237 | ] 238 | }, 239 | { 240 | "cell_type": "markdown", 241 | "metadata": {}, 242 | "source": [ 243 | "print gradients d(out)/dx\n", 244 | "\n", 245 | "\n" 246 | ] 247 | }, 248 | { 249 | "cell_type": "code", 250 | "execution_count": 9, 251 | "metadata": {}, 252 | "outputs": [ 253 | { 254 | "name": "stdout", 255 | "output_type": "stream", 256 | "text": [ 257 | "tensor([[4.5000, 4.5000],\n", 258 | " [4.5000, 4.5000]])\n" 259 | ] 260 | } 261 | ], 262 | "source": [ 263 | "print(x.grad)" 264 | ] 265 | }, 266 | { 267 | "cell_type": "markdown", 268 | "metadata": {}, 269 | "source": [ 270 | "得到矩阵 ``4.5``.调用 ``out``\n", 271 | "*Tensor* “$o$”.\n", 272 | "\n", 273 | "得到 $o = \\frac{1}{4}\\sum_i z_i$,\n", 274 | "$z_i = 3(x_i+2)^2$ and $z_i\\bigr\\rvert_{x_i=1} = 27$.\n", 275 | "\n", 276 | "因此,\n", 277 | "$\\frac{\\partial o}{\\partial x_i} = \\frac{3}{2}(x_i+2)$, hence\n", 278 | "$\\frac{\\partial o}{\\partial x_i}\\bigr\\rvert_{x_i=1} = \\frac{9}{2} = 4.5$.\n", 279 | "\n" 280 | ] 281 | }, 282 | { 283 | "cell_type": "markdown", 284 | "metadata": {}, 285 | "source": [ 286 | "可以使用 autograd 做更多的操作\n", 287 | "\n" 288 | ] 289 | }, 290 | { 291 | "cell_type": "code", 292 | "execution_count": 10, 293 | "metadata": {}, 294 | "outputs": [ 295 | { 296 | "name": "stdout", 297 | "output_type": "stream", 298 | "text": [ 299 | "tensor([-920.6895, -115.7301, -867.6995], grad_fn=)\n" 300 | ] 301 | } 302 | ], 303 | "source": [ 304 | "x = torch.randn(3, requires_grad=True)\n", 305 | "\n", 306 | "y = x * 2\n", 307 | "while y.data.norm() < 1000:\n", 308 | " y = y * 2\n", 309 | "\n", 310 | "print(y)" 311 | ] 312 | }, 313 | { 314 | "cell_type": "code", 315 | "execution_count": 11, 316 | "metadata": {}, 317 | "outputs": [ 318 | { 319 | "name": "stdout", 320 | "output_type": "stream", 321 | "text": [ 322 | "tensor([ 51.2000, 512.0000, 0.0512])\n" 323 | ] 324 | } 325 | ], 326 | "source": [ 327 | "gradients = torch.tensor([0.1, 1.0, 0.0001], dtype=torch.float)\n", 328 | "y.backward(gradients)\n", 329 | "\n", 330 | "print(x.grad)" 331 | ] 332 | }, 333 | { 334 | "cell_type": "markdown", 335 | "metadata": {}, 336 | "source": [ 337 | "如果``.requires_grad=True``但是你又不希望进行autograd的计算，\n", 338 | "那么可以将变量包裹在 ``with torch.no_grad()``中:\n" 339 | ] 340 | }, 341 | { 342 | "cell_type": "code", 343 | "execution_count": 12, 344 | "metadata": {}, 345 | "outputs": [ 346 | { 347 | "name": "stdout", 348 | "output_type": "stream", 349 | "text": [ 350 | "True\n", 351 | "True\n", 352 | "False\n" 353 | ] 354 | } 355 | ], 356 | "source": [ 357 | "print(x.requires_grad)\n", 358 | "print((x ** 2).requires_grad)\n", 359 | "\n", 360 | "with torch.no_grad():\n", 361 | "\tprint((x ** 2).requires_grad)" 362 | ] 363 | }, 364 | { 365 | "cell_type": "markdown", 366 | "metadata": {}, 367 | "source": [ 368 | "**稍后阅读:**\n", 369 | "\n", 370 | " ``autograd`` 和 ``Function`` 的官方文档 https://pytorch.org/docs/autograd\n", 371 | "\n" 372 | ] 373 | }, 374 | { 375 | "cell_type": "code", 376 | "execution_count": null, 377 | "metadata": {}, 378 | "outputs": [], 379 | "source": [] 380 | } 381 | ], 382 | "metadata": { 383 | "kernelspec": { 384 | "display_name": "Pytorch for Deeplearning", 385 | "language": "python", 386 | "name": "pytorch" 387 | }, 388 | "language_info": { 389 | "codemirror_mode": { 390 | "name": "ipython", 391 | "version": 3 392 | }, 393 | "file_extension": ".py", 394 | "mimetype": "text/x-python", 395 | "name": "python", 396 | "nbconvert_exporter": "python", 397 | "pygments_lexer": "ipython3", 398 | "version": "3.6.7" 399 | } 400 | }, 401 | "nbformat": 4, 402 | "nbformat_minor": 1 403 | } 404 | -------------------------------------------------------------------------------- /chapter1/3_neural_networks_tutorial.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%matplotlib inline" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "\n", 17 | "Neural Networks\n", 18 | "===============\n", 19 | "\n", 20 | "使用torch.nn包来构建神经网络。\n", 21 | "\n", 22 | "上一讲已经讲过了``autograd``，``nn``包依赖``autograd``包来定义模型并求导。\n", 23 | "一个``nn.Module``包含各个层和一个``forward(input)``方法，该方法返回``output``。\n", 24 | "\n", 25 | "\n", 26 | "\n", 27 | "例如：\n", 28 | "\n", 29 | "![](https://pytorch.org/tutorials/_images/mnist.png)\n", 30 | "\n", 31 | "它是一个简单的前馈神经网络，它接受一个输入，然后一层接着一层地传递，最后输出计算的结果。\n", 32 | "\n", 33 | "神经网络的典型训练过程如下：\n", 34 | "\n", 35 | "1. 定义包含一些可学习的参数(或者叫权重)神经网络模型； \n", 36 | "2. 在数据集上迭代； \n", 37 | "3. 通过神经网络处理输入； \n", 38 | "4. 计算损失(输出结果和正确值的差值大小)；\n", 39 | "5. 将梯度反向传播会网络的参数； \n", 40 | "6. 更新网络的参数，主要使用如下简单的更新原则： \n", 41 | "``weight = weight - learning_rate * gradient``\n", 42 | "\n", 43 | " \n", 44 | "\n", 45 | "定义网络\n", 46 | "------------------\n", 47 | "\n", 48 | "开始定义一个网络：\n", 49 | "\n" 50 | ] 51 | }, 52 | { 53 | "cell_type": "code", 54 | "execution_count": 2, 55 | "metadata": {}, 56 | "outputs": [ 57 | { 58 | "name": "stdout", 59 | "output_type": "stream", 60 | "text": [ 61 | "Net(\n", 62 | " (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))\n", 63 | " (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n", 64 | " (fc1): Linear(in_features=400, out_features=120, bias=True)\n", 65 | " (fc2): Linear(in_features=120, out_features=84, bias=True)\n", 66 | " (fc3): Linear(in_features=84, out_features=10, bias=True)\n", 67 | ")\n" 68 | ] 69 | } 70 | ], 71 | "source": [ 72 | "import torch\n", 73 | "import torch.nn as nn\n", 74 | "import torch.nn.functional as F\n", 75 | "\n", 76 | "\n", 77 | "class Net(nn.Module):\n", 78 | "\n", 79 | " def __init__(self):\n", 80 | " super(Net, self).__init__()\n", 81 | " # 1 input image channel, 6 output channels, 5x5 square convolution\n", 82 | " # kernel\n", 83 | " self.conv1 = nn.Conv2d(1, 6, 5)\n", 84 | " self.conv2 = nn.Conv2d(6, 16, 5)\n", 85 | " # an affine operation: y = Wx + b\n", 86 | " self.fc1 = nn.Linear(16 * 5 * 5, 120)\n", 87 | " self.fc2 = nn.Linear(120, 84)\n", 88 | " self.fc3 = nn.Linear(84, 10)\n", 89 | "\n", 90 | " def forward(self, x):\n", 91 | " # Max pooling over a (2, 2) window\n", 92 | " x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))\n", 93 | " # If the size is a square you can only specify a single number\n", 94 | " x = F.max_pool2d(F.relu(self.conv2(x)), 2)\n", 95 | " x = x.view(-1, self.num_flat_features(x))\n", 96 | " x = F.relu(self.fc1(x))\n", 97 | " x = F.relu(self.fc2(x))\n", 98 | " x = self.fc3(x)\n", 99 | " return x\n", 100 | "\n", 101 | " def num_flat_features(self, x):\n", 102 | " size = x.size()[1:] # all dimensions except the batch dimension\n", 103 | " num_features = 1\n", 104 | " for s in size:\n", 105 | " num_features *= s\n", 106 | " return num_features\n", 107 | "\n", 108 | "\n", 109 | "net = Net()\n", 110 | "print(net)" 111 | ] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "metadata": {}, 116 | "source": [ 117 | "在模型中必须要定义 ``forward`` 函数，``backward``\n", 118 | "函数（用来计算梯度）会被``autograd``自动创建。\n", 119 | "可以在 ``forward`` 函数中使用任何针对 Tensor 的操作。\n", 120 | "\n", 121 | " ``net.parameters()``返回可被学习的参数（权重）列表和值\n", 122 | "\n" 123 | ] 124 | }, 125 | { 126 | "cell_type": "code", 127 | "execution_count": 4, 128 | "metadata": {}, 129 | "outputs": [ 130 | { 131 | "name": "stdout", 132 | "output_type": "stream", 133 | "text": [ 134 | "10\n", 135 | "torch.Size([6, 1, 5, 5])\n" 136 | ] 137 | } 138 | ], 139 | "source": [ 140 | "params = list(net.parameters())\n", 141 | "print(len(params))\n", 142 | "print(params[0].size()) # conv1's .weight" 143 | ] 144 | }, 145 | { 146 | "cell_type": "markdown", 147 | "metadata": {}, 148 | "source": [ 149 | "测试随机输入32×32。\n", 150 | "注：这个网络（LeNet）期望的输入大小是32×32，如果使用MNIST数据集来训练这个网络，请把图片大小重新调整到32×32。\n", 151 | "\n" 152 | ] 153 | }, 154 | { 155 | "cell_type": "code", 156 | "execution_count": 5, 157 | "metadata": {}, 158 | "outputs": [ 159 | { 160 | "name": "stdout", 161 | "output_type": "stream", 162 | "text": [ 163 | "tensor([[-0.0204, -0.0268, -0.0829, 0.1420, -0.0192, 0.1848, 0.0723, -0.0393,\n", 164 | " -0.0275, 0.0867]], grad_fn=)\n" 165 | ] 166 | } 167 | ], 168 | "source": [ 169 | "input = torch.randn(1, 1, 32, 32)\n", 170 | "out = net(input)\n", 171 | "print(out)" 172 | ] 173 | }, 174 | { 175 | "cell_type": "markdown", 176 | "metadata": {}, 177 | "source": [ 178 | "将所有参数的梯度缓存清零，然后进行随机梯度的的反向传播：\n", 179 | "\n" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": 6, 185 | "metadata": {}, 186 | "outputs": [], 187 | "source": [ 188 | "net.zero_grad()\n", 189 | "out.backward(torch.randn(1, 10))" 190 | ] 191 | }, 192 | { 193 | "cell_type": "markdown", 194 | "metadata": {}, 195 | "source": [ 196 | "

Note

``torch.nn`` 只支持小批量输入。整个 ``torch.nn``\n", 197 | "包都只支持小批量样本，而不支持单个样本。\n", 198 | "\n", 199 | " 例如，``nn.Conv2d`` 接受一个4维的张量，\n", 200 | " ``每一维分别是sSamples * nChannels * Height * Width（样本数*通道数*高*宽）``。\n", 201 | "\n", 202 | " 如果你有单个样本，只需使用 ``input.unsqueeze(0)`` 来添加其它的维数

\n", 203 | "\n", 204 | "在继续之前，我们回顾一下到目前为止用到的类。\n", 205 | "\n", 206 | "**回顾:**\n", 207 | " - ``torch.Tensor``：一个用过自动调用 ``backward()``实现支持自动梯度计算的 *多维数组* ，\n", 208 | " 并且保存关于这个向量的*梯度* w.r.t.\n", 209 | " - ``nn.Module``：神经网络模块。封装参数、移动到GPU上运行、导出、加载等。\n", 210 | " - ``nn.Parameter``：一种变量，当把它赋值给一个``Module``时，被 *自动* 地注册为一个参数。\n", 211 | " - ``autograd.Function``：实现一个自动求导操作的前向和反向定义，每个变量操作至少创建一个函数节点，每一个``Tensor``的操作都回创建一个接到创建``Tensor``和 *编码其历史* 的函数的``Function``节点。\n", 212 | "\n", 213 | "**重点如下：**\n", 214 | " - 定义一个网络\n", 215 | " - 处理输入，调用backword\n", 216 | "\n", 217 | "**还剩：**\n", 218 | " - 计算损失\n", 219 | " - 更新网络权重\n", 220 | "\n", 221 | "损失函数\n", 222 | "-------------\n", 223 | "一个损失函数接受一对 (output, target) 作为输入，计算一个值来估计网络的输出和目标值相差多少。\n", 224 | "\n", 225 | "***译者注：output为网络的输出，target为实际值***\n", 226 | "\n", 227 | "nn包中有很多不同的[损失函数](https://pytorch.org/docs/nn.html#loss-functions)。\n", 228 | "``nn.MSELoss``是一个比较简单的损失函数，它计算输出和目标间的**均方误差**，\n", 229 | "例如：\n", 230 | "\n" 231 | ] 232 | }, 233 | { 234 | "cell_type": "code", 235 | "execution_count": 7, 236 | "metadata": {}, 237 | "outputs": [ 238 | { 239 | "name": "stdout", 240 | "output_type": "stream", 241 | "text": [ 242 | "tensor(1.3172, grad_fn=)\n" 243 | ] 244 | } 245 | ], 246 | "source": [ 247 | "output = net(input)\n", 248 | "target = torch.randn(10) # 随机值作为样例\n", 249 | "target = target.view(1, -1) # 使target和output的shape相同\n", 250 | "criterion = nn.MSELoss()\n", 251 | "\n", 252 | "loss = criterion(output, target)\n", 253 | "print(loss)" 254 | ] 255 | }, 256 | { 257 | "cell_type": "markdown", 258 | "metadata": {}, 259 | "source": [ 260 | "Now, if you follow ``loss`` in the backward direction, using its\n", 261 | "``.grad_fn`` attribute, you will see a graph of computations that looks\n", 262 | "like this:\n", 263 | "\n", 264 | "::\n", 265 | "\n", 266 | " input -> conv2d -> relu -> maxpool2d -> conv2d -> relu -> maxpool2d\n", 267 | " -> view -> linear -> relu -> linear -> relu -> linear\n", 268 | " -> MSELoss\n", 269 | " -> loss\n", 270 | "\n", 271 | "So, when we call ``loss.backward()``, the whole graph is differentiated\n", 272 | "w.r.t. the loss, and all Tensors in the graph that has ``requires_grad=True``\n", 273 | "will have their ``.grad`` Tensor accumulated with the gradient.\n", 274 | "\n", 275 | "For illustration, let us follow a few steps backward:\n", 276 | "\n" 277 | ] 278 | }, 279 | { 280 | "cell_type": "code", 281 | "execution_count": null, 282 | "metadata": {}, 283 | "outputs": [], 284 | "source": [ 285 | "print(loss.grad_fn) # MSELoss\n", 286 | "print(loss.grad_fn.next_functions[0][0]) # Linear\n", 287 | "print(loss.grad_fn.next_functions[0][0].next_functions[0][0]) # ReLU" 288 | ] 289 | }, 290 | { 291 | "cell_type": "markdown", 292 | "metadata": {}, 293 | "source": [ 294 | "反向传播\n", 295 | "--------\n", 296 | "调用loss.backward()获得反向传播的误差。\n", 297 | "\n", 298 | "但是在调用前需要清除已存在的梯度，否则梯度将被累加到已存在的梯度。\n", 299 | "\n", 300 | "现在，我们将调用loss.backward()，并查看conv1层的偏差（bias）项在反向传播前后的梯度。\n", 301 | "\n", 302 | "\n" 303 | ] 304 | }, 305 | { 306 | "cell_type": "code", 307 | "execution_count": 8, 308 | "metadata": {}, 309 | "outputs": [ 310 | { 311 | "name": "stdout", 312 | "output_type": "stream", 313 | "text": [ 314 | "conv1.bias.grad before backward\n", 315 | "tensor([0., 0., 0., 0., 0., 0.])\n", 316 | "conv1.bias.grad after backward\n", 317 | "tensor([ 0.0074, -0.0249, -0.0107, 0.0326, -0.0017, -0.0059])\n" 318 | ] 319 | } 320 | ], 321 | "source": [ 322 | "net.zero_grad() # 清除梯度\n", 323 | "\n", 324 | "print('conv1.bias.grad before backward')\n", 325 | "print(net.conv1.bias.grad)\n", 326 | "\n", 327 | "loss.backward()\n", 328 | "\n", 329 | "print('conv1.bias.grad after backward')\n", 330 | "print(net.conv1.bias.grad)" 331 | ] 332 | }, 333 | { 334 | "cell_type": "markdown", 335 | "metadata": {}, 336 | "source": [ 337 | "如何使用损失函数\n", 338 | "\n", 339 | "**稍后阅读：**\n", 340 | "\n", 341 | " `nn`包，包含了各种用来构成深度神经网络构建块的模块和损失函数，完整的文档请查看[here](https://pytorch.org/docs/nn)。\n", 342 | "\n", 343 | "**剩下的最后一件事:**\n", 344 | "\n", 345 | " - 新网络的权重\n", 346 | "\n", 347 | "更新权重\n", 348 | "------------------\n", 349 | "在实践中最简单的权重更新规则是随机梯度下降（SGD）：\n", 350 | "\n", 351 | " ``weight = weight - learning_rate * gradient``\n", 352 | "\n", 353 | "我们可以使用简单的Python代码实现这个规则：\n", 354 | "\n", 355 | "```python\n", 356 | "\n", 357 | " learning_rate = 0.01\n", 358 | " for f in net.parameters():\n", 359 | " f.data.sub_(f.grad.data * learning_rate)\n", 360 | "```\n", 361 | "但是当使用神经网络是想要使用各种不同的更新规则时，比如SGD、Nesterov-SGD、Adam、RMSPROP等，PyTorch中构建了一个包``torch.optim``实现了所有的这些规则。\n", 362 | "使用它们非常简单：\n" 363 | ] 364 | }, 365 | { 366 | "cell_type": "code", 367 | "execution_count": 9, 368 | "metadata": {}, 369 | "outputs": [], 370 | "source": [ 371 | "import torch.optim as optim\n", 372 | "\n", 373 | "# create your optimizer\n", 374 | "optimizer = optim.SGD(net.parameters(), lr=0.01)\n", 375 | "\n", 376 | "# in your training loop:\n", 377 | "optimizer.zero_grad() # zero the gradient buffers\n", 378 | "output = net(input)\n", 379 | "loss = criterion(output, target)\n", 380 | "loss.backward()\n", 381 | "optimizer.step() # Does the update" 382 | ] 383 | }, 384 | { 385 | "cell_type": "markdown", 386 | "metadata": {}, 387 | "source": [ 388 | ".. Note::\n", 389 | " \n", 390 | " Observe how gradient buffers had to be manually set to zero using\n", 391 | " ``optimizer.zero_grad()``. This is because gradients are accumulated\n", 392 | " as explained in `Backprop`_ section.\n", 393 | "\n" 394 | ] 395 | }, 396 | { 397 | "cell_type": "code", 398 | "execution_count": null, 399 | "metadata": {}, 400 | "outputs": [], 401 | "source": [] 402 | } 403 | ], 404 | "metadata": { 405 | "kernelspec": { 406 | "display_name": "Pytorch for Deeplearning", 407 | "language": "python", 408 | "name": "pytorch" 409 | }, 410 | "language_info": { 411 | "codemirror_mode": { 412 | "name": "ipython", 413 | "version": 3 414 | }, 415 | "file_extension": ".py", 416 | "mimetype": "text/x-python", 417 | "name": "python", 418 | "nbconvert_exporter": "python", 419 | "pygments_lexer": "ipython3", 420 | "version": "3.6.7" 421 | } 422 | }, 423 | "nbformat": 4, 424 | "nbformat_minor": 1 425 | } 426 | -------------------------------------------------------------------------------- /chapter1/5_data_parallel_tutorial.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%matplotlib inline" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "\n", 17 | "数据并行（选读）\n", 18 | "==========================\n", 19 | "**Authors**: [Sung Kim](https://github.com/hunkim) and [Jenny Kang](https://github.com/jennykang)\n", 20 | "\n", 21 | "在这个教程里，我们将学习如何使用 ``DataParallel`` 来使用多GPU。 \n", 22 | "\n", 23 | "PyTorch非常容易就可以使用多GPU，用如下方式把一个模型放到GPU上：\n", 24 | "\n", 25 | "```python\n", 26 | "\n", 27 | " device = torch.device(\"cuda:0\")\n", 28 | " model.to(device)\n", 29 | "```\n", 30 | " GPU:\n", 31 | "然后复制所有的张量到GPU上：\n", 32 | "```python\n", 33 | "\n", 34 | " mytensor = my_tensor.to(device)\n", 35 | "```\n", 36 | "请注意，只调用``my_tensor.to(device)``并没有复制张量到GPU上，而是返回了一个copy。所以你需要把它赋值给一个新的张量并在GPU上使用这个张量。\n", 37 | "\n", 38 | "在多GPU上执行前向和反向传播是自然而然的事。\n", 39 | "但是PyTorch默认将只使用一个GPU。\n", 40 | "\n", 41 | "使用``DataParallel``可以轻易的让模型并行运行在多个GPU上。\n", 42 | "\n", 43 | "\n", 44 | "```python\n", 45 | "\n", 46 | " model = nn.DataParallel(model)\n", 47 | "```\n", 48 | "这才是这篇教程的核心，接下来我们将更详细的介绍它。\n", 49 | "\n" 50 | ] 51 | }, 52 | { 53 | "cell_type": "markdown", 54 | "metadata": {}, 55 | "source": [ 56 | "导入和参数\n", 57 | "----------------------\n", 58 | "\n", 59 | "导入PyTorch模块和定义参数。\n", 60 | "\n", 61 | "\n" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 2, 67 | "metadata": {}, 68 | "outputs": [], 69 | "source": [ 70 | "import torch\n", 71 | "import torch.nn as nn\n", 72 | "from torch.utils.data import Dataset, DataLoader\n", 73 | "\n", 74 | "# Parameters and DataLoaders\n", 75 | "input_size = 5\n", 76 | "output_size = 2\n", 77 | "\n", 78 | "batch_size = 30\n", 79 | "data_size = 100" 80 | ] 81 | }, 82 | { 83 | "cell_type": "markdown", 84 | "metadata": {}, 85 | "source": [ 86 | "Device\n", 87 | "\n", 88 | "\n" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": 3, 94 | "metadata": {}, 95 | "outputs": [], 96 | "source": [ 97 | "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")" 98 | ] 99 | }, 100 | { 101 | "cell_type": "markdown", 102 | "metadata": {}, 103 | "source": [ 104 | "虚拟数据集\n", 105 | "-------------\n", 106 | "\n", 107 | "制作一个虚拟（随机）数据集，\n", 108 | "你只需实现 `__getitem__`\n", 109 | "\n", 110 | "\n" 111 | ] 112 | }, 113 | { 114 | "cell_type": "code", 115 | "execution_count": 4, 116 | "metadata": {}, 117 | "outputs": [], 118 | "source": [ 119 | "class RandomDataset(Dataset):\n", 120 | "\n", 121 | " def __init__(self, size, length):\n", 122 | " self.len = length\n", 123 | " self.data = torch.randn(length, size)\n", 124 | "\n", 125 | " def __getitem__(self, index):\n", 126 | " return self.data[index]\n", 127 | "\n", 128 | " def __len__(self):\n", 129 | " return self.len\n", 130 | "\n", 131 | "rand_loader = DataLoader(dataset=RandomDataset(input_size, data_size),\n", 132 | " batch_size=batch_size, shuffle=True)" 133 | ] 134 | }, 135 | { 136 | "cell_type": "markdown", 137 | "metadata": {}, 138 | "source": [ 139 | "简单模型\n", 140 | "------------\n", 141 | "作为演示，我们的模型只接受一个输入，执行一个线性操作，然后得到结果。\n", 142 | "说明：``DataParallel``能在任何模型（CNN，RNN，Capsule Net等）上使用。\n", 143 | "\n", 144 | "\n", 145 | "我们在模型内部放置了一条打印语句来打印输入和输出向量的大小。\n", 146 | "\n", 147 | "请注意批次的秩为0时打印的内容。\n", 148 | "\n", 149 | "\n" 150 | ] 151 | }, 152 | { 153 | "cell_type": "code", 154 | "execution_count": 5, 155 | "metadata": {}, 156 | "outputs": [], 157 | "source": [ 158 | "class Model(nn.Module):\n", 159 | " # Our model\n", 160 | "\n", 161 | " def __init__(self, input_size, output_size):\n", 162 | " super(Model, self).__init__()\n", 163 | " self.fc = nn.Linear(input_size, output_size)\n", 164 | "\n", 165 | " def forward(self, input):\n", 166 | " output = self.fc(input)\n", 167 | " print(\"\\tIn Model: input size\", input.size(),\n", 168 | " \"output size\", output.size())\n", 169 | "\n", 170 | " return output" 171 | ] 172 | }, 173 | { 174 | "cell_type": "markdown", 175 | "metadata": {}, 176 | "source": [ 177 | "创建一个模型和数据并行\n", 178 | "-----------------------------\n", 179 | "\n", 180 | "这是本教程的核心部分。\n", 181 | "\n", 182 | "首先，我们需要创建一个模型实例和检测我们是否有多个GPU。\n", 183 | "如果有多个GPU，使用``nn.DataParallel``来包装我们的模型。\n", 184 | "然后通过m``model.to(device)``把模型放到GPU上。\n", 185 | "\n" 186 | ] 187 | }, 188 | { 189 | "cell_type": "code", 190 | "execution_count": 6, 191 | "metadata": {}, 192 | "outputs": [ 193 | { 194 | "data": { 195 | "text/plain": [ 196 | "Model(\n", 197 | " (fc): Linear(in_features=5, out_features=2, bias=True)\n", 198 | ")" 199 | ] 200 | }, 201 | "execution_count": 6, 202 | "metadata": {}, 203 | "output_type": "execute_result" 204 | } 205 | ], 206 | "source": [ 207 | "model = Model(input_size, output_size)\n", 208 | "if torch.cuda.device_count() > 1:\n", 209 | " print(\"Let's use\", torch.cuda.device_count(), \"GPUs!\")\n", 210 | " # dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs\n", 211 | " model = nn.DataParallel(model)\n", 212 | "\n", 213 | "model.to(device)" 214 | ] 215 | }, 216 | { 217 | "cell_type": "markdown", 218 | "metadata": {}, 219 | "source": [ 220 | "运行模型\n", 221 | "-------------\n", 222 | "\n", 223 | "现在可以看到输入和输出张量的大小。\n", 224 | "\n", 225 | "\n" 226 | ] 227 | }, 228 | { 229 | "cell_type": "code", 230 | "execution_count": 7, 231 | "metadata": {}, 232 | "outputs": [ 233 | { 234 | "name": "stdout", 235 | "output_type": "stream", 236 | "text": [ 237 | "\tIn Model: input size torch.Size([30, 5]) output size torch.Size([30, 2])\n", 238 | "Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n", 239 | "\tIn Model: input size torch.Size([30, 5]) output size torch.Size([30, 2])\n", 240 | "Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n", 241 | "\tIn Model: input size torch.Size([30, 5]) output size torch.Size([30, 2])\n", 242 | "Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n", 243 | "\tIn Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n", 244 | "Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])\n" 245 | ] 246 | } 247 | ], 248 | "source": [ 249 | "for data in rand_loader:\n", 250 | " input = data.to(device)\n", 251 | " output = model(input)\n", 252 | " print(\"Outside: input size\", input.size(),\n", 253 | " \"output_size\", output.size())" 254 | ] 255 | }, 256 | { 257 | "cell_type": "markdown", 258 | "metadata": {}, 259 | "source": [ 260 | "结果\n", 261 | "-------\n", 262 | "\n", 263 | "当没有或者只有一个GPU时，对30个输入和输出进行批处理，得到了期望的一样得到30个输入和输出，但是如果你有多个GPU，你得到如下的结果。\n", 264 | "\n", 265 | "\n", 266 | "2 GPUs\n", 267 | "~\n", 268 | "\n", 269 | "If you have 2, you will see:\n", 270 | "\n", 271 | ".. code:: bash\n", 272 | "\n", 273 | " # on 2 GPUs\n", 274 | " Let's use 2 GPUs!\n", 275 | " In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])\n", 276 | " In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])\n", 277 | " Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n", 278 | " In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])\n", 279 | " In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])\n", 280 | " Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n", 281 | " In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])\n", 282 | " In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])\n", 283 | " Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n", 284 | " In Model: input size torch.Size([5, 5]) output size torch.Size([5, 2])\n", 285 | " In Model: input size torch.Size([5, 5]) output size torch.Size([5, 2])\n", 286 | " Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])\n", 287 | "\n", 288 | "3 GPUs\n", 289 | "~\n", 290 | "\n", 291 | "If you have 3 GPUs, you will see:\n", 292 | "\n", 293 | ".. code:: bash\n", 294 | "\n", 295 | " Let's use 3 GPUs!\n", 296 | " In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n", 297 | " In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n", 298 | " In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n", 299 | " Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n", 300 | " In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n", 301 | " In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n", 302 | " In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n", 303 | " Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n", 304 | " In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n", 305 | " In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n", 306 | " In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n", 307 | " Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n", 308 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 309 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 310 | " In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n", 311 | " Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])\n", 312 | "\n", 313 | "8 GPUs\n", 314 | "~~\n", 315 | "\n", 316 | "If you have 8, you will see:\n", 317 | "\n", 318 | ".. code:: bash\n", 319 | "\n", 320 | " Let's use 8 GPUs!\n", 321 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 322 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 323 | " In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n", 324 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 325 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 326 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 327 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 328 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 329 | " Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n", 330 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 331 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 332 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 333 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 334 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 335 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 336 | " In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n", 337 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 338 | " Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n", 339 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 340 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 341 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 342 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 343 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 344 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 345 | " In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n", 346 | " In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n", 347 | " Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n", 348 | " In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n", 349 | " In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n", 350 | " In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n", 351 | " In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n", 352 | " In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n", 353 | " Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])\n", 354 | "\n", 355 | "\n" 356 | ] 357 | }, 358 | { 359 | "cell_type": "markdown", 360 | "metadata": {}, 361 | "source": [ 362 | "总结\n", 363 | "-------\n", 364 | "\n", 365 | "DataParallel会自动的划分数据，并将作业发送到多个GPU上的多个模型。\n", 366 | "并在每个模型完成作业后，收集合并结果并返回。\n", 367 | "\n", 368 | "更多信息请看这里：\n", 369 | "https://pytorch.org/tutorials/beginner/former\\_torchies/parallelism\\_tutorial.html.\n", 370 | "\n", 371 | "\n" 372 | ] 373 | }, 374 | { 375 | "cell_type": "code", 376 | "execution_count": null, 377 | "metadata": {}, 378 | "outputs": [], 379 | "source": [] 380 | } 381 | ], 382 | "metadata": { 383 | "kernelspec": { 384 | "display_name": "Pytorch for Deeplearning", 385 | "language": "python", 386 | "name": "pytorch" 387 | }, 388 | "language_info": { 389 | "codemirror_mode": { 390 | "name": "ipython", 391 | "version": 3 392 | }, 393 | "file_extension": ".py", 394 | "mimetype": "text/x-python", 395 | "name": "python", 396 | "nbconvert_exporter": "python", 397 | "pygments_lexer": "ipython3", 398 | "version": "3.6.7" 399 | } 400 | }, 401 | "nbformat": 4, 402 | "nbformat_minor": 1 403 | } 404 | -------------------------------------------------------------------------------- /chapter1/autograd_tutorial.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%matplotlib inline" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "\nAutograd: Automatic Differentiation\n===================================\n\nCentral to all neural networks in PyTorch is the ``autograd`` package.\nLet\u2019s first briefly visit this, and we will then go to training our\nfirst neural network.\n\n\nThe ``autograd`` package provides automatic differentiation for all operations\non Tensors. It is a define-by-run framework, which means that your backprop is\ndefined by how your code is run, and that every single iteration can be\ndifferent.\n\nLet us see this in more simple terms with some examples.\n\nTensor\n--------\n\n``torch.Tensor`` is the central class of the package. If you set its attribute\n``.requires_grad`` as ``True``, it starts to track all operations on it. When\nyou finish your computation you can call ``.backward()`` and have all the\ngradients computed automatically. The gradient for this tensor will be\naccumulated into ``.grad`` attribute.\n\nTo stop a tensor from tracking history, you can call ``.detach()`` to detach\nit from the computation history, and to prevent future computation from being\ntracked.\n\nTo prevent tracking history (and using memory), you can also wrap the code block\nin ``with torch.no_grad():``. This can be particularly helpful when evaluating a\nmodel because the model may have trainable parameters with `requires_grad=True`,\nbut for which we don't need the gradients.\n\nThere\u2019s one more class which is very important for autograd\nimplementation - a ``Function``.\n\n``Tensor`` and ``Function`` are interconnected and build up an acyclic\ngraph, that encodes a complete history of computation. Each tensor has\na ``.grad_fn`` attribute that references a ``Function`` that has created\nthe ``Tensor`` (except for Tensors created by the user - their\n``grad_fn is None``).\n\nIf you want to compute the derivatives, you can call ``.backward()`` on\na ``Tensor``. If ``Tensor`` is a scalar (i.e. it holds a one element\ndata), you don\u2019t need to specify any arguments to ``backward()``,\nhowever if it has more elements, you need to specify a ``gradient``\nargument that is a tensor of matching shape.\n\n" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": { 25 | "collapsed": false 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "import torch" 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "metadata": {}, 35 | "source": [ 36 | "Create a tensor and set requires_grad=True to track computation with it\n\n" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": null, 42 | "metadata": { 43 | "collapsed": false 44 | }, 45 | "outputs": [], 46 | "source": [ 47 | "x = torch.ones(2, 2, requires_grad=True)\nprint(x)" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": {}, 53 | "source": [ 54 | "Do an operation of tensor:\n\n" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": { 61 | "collapsed": false 62 | }, 63 | "outputs": [], 64 | "source": [ 65 | "y = x + 2\nprint(y)" 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "metadata": {}, 71 | "source": [ 72 | "``y`` was created as a result of an operation, so it has a ``grad_fn``.\n\n" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": null, 78 | "metadata": { 79 | "collapsed": false 80 | }, 81 | "outputs": [], 82 | "source": [ 83 | "print(y.grad_fn)" 84 | ] 85 | }, 86 | { 87 | "cell_type": "markdown", 88 | "metadata": {}, 89 | "source": [ 90 | "Do more operations on y\n\n" 91 | ] 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": null, 96 | "metadata": { 97 | "collapsed": false 98 | }, 99 | "outputs": [], 100 | "source": [ 101 | "z = y * y * 3\nout = z.mean()\n\nprint(z, out)" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "``.requires_grad_( ... )`` changes an existing Tensor's ``requires_grad``\nflag in-place. The input flag defaults to ``False`` if not given.\n\n" 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": null, 114 | "metadata": { 115 | "collapsed": false 116 | }, 117 | "outputs": [], 118 | "source": [ 119 | "a = torch.randn(2, 2)\na = ((a * 3) / (a - 1))\nprint(a.requires_grad)\na.requires_grad_(True)\nprint(a.requires_grad)\nb = (a * a).sum()\nprint(b.grad_fn)" 120 | ] 121 | }, 122 | { 123 | "cell_type": "markdown", 124 | "metadata": {}, 125 | "source": [ 126 | "Gradients\n---------\nLet's backprop now\nBecause ``out`` contains a single scalar, ``out.backward()`` is\nequivalent to ``out.backward(torch.tensor(1))``.\n\n" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": null, 132 | "metadata": { 133 | "collapsed": false 134 | }, 135 | "outputs": [], 136 | "source": [ 137 | "out.backward()" 138 | ] 139 | }, 140 | { 141 | "cell_type": "markdown", 142 | "metadata": {}, 143 | "source": [ 144 | "print gradients d(out)/dx\n\n\n" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": null, 150 | "metadata": { 151 | "collapsed": false 152 | }, 153 | "outputs": [], 154 | "source": [ 155 | "print(x.grad)" 156 | ] 157 | }, 158 | { 159 | "cell_type": "markdown", 160 | "metadata": {}, 161 | "source": [ 162 | "You should have got a matrix of ``4.5``. Let\u2019s call the ``out``\n*Tensor* \u201c$o$\u201d.\nWe have that $o = \\frac{1}{4}\\sum_i z_i$,\n$z_i = 3(x_i+2)^2$ and $z_i\\bigr\\rvert_{x_i=1} = 27$.\nTherefore,\n$\\frac{\\partial o}{\\partial x_i} = \\frac{3}{2}(x_i+2)$, hence\n$\\frac{\\partial o}{\\partial x_i}\\bigr\\rvert_{x_i=1} = \\frac{9}{2} = 4.5$.\n\n" 163 | ] 164 | }, 165 | { 166 | "cell_type": "markdown", 167 | "metadata": {}, 168 | "source": [ 169 | "You can do many crazy things with autograd!\n\n" 170 | ] 171 | }, 172 | { 173 | "cell_type": "code", 174 | "execution_count": null, 175 | "metadata": { 176 | "collapsed": false 177 | }, 178 | "outputs": [], 179 | "source": [ 180 | "x = torch.randn(3, requires_grad=True)\n\ny = x * 2\nwhile y.data.norm() < 1000:\n y = y * 2\n\nprint(y)" 181 | ] 182 | }, 183 | { 184 | "cell_type": "code", 185 | "execution_count": null, 186 | "metadata": { 187 | "collapsed": false 188 | }, 189 | "outputs": [], 190 | "source": [ 191 | "gradients = torch.tensor([0.1, 1.0, 0.0001], dtype=torch.float)\ny.backward(gradients)\n\nprint(x.grad)" 192 | ] 193 | }, 194 | { 195 | "cell_type": "markdown", 196 | "metadata": {}, 197 | "source": [ 198 | "You can also stop autograd from tracking history on Tensors\nwith ``.requires_grad=True`` by wrapping the code block in\n``with torch.no_grad()``:\n\n" 199 | ] 200 | }, 201 | { 202 | "cell_type": "code", 203 | "execution_count": null, 204 | "metadata": { 205 | "collapsed": false 206 | }, 207 | "outputs": [], 208 | "source": [ 209 | "print(x.requires_grad)\nprint((x ** 2).requires_grad)\n\nwith torch.no_grad():\n\tprint((x ** 2).requires_grad)" 210 | ] 211 | }, 212 | { 213 | "cell_type": "markdown", 214 | "metadata": {}, 215 | "source": [ 216 | "**Read Later:**\n\nDocumentation of ``autograd`` and ``Function`` is at\nhttps://pytorch.org/docs/autograd\n\n" 217 | ] 218 | } 219 | ], 220 | "metadata": { 221 | "kernelspec": { 222 | "display_name": "Python 3", 223 | "language": "python", 224 | "name": "python3" 225 | }, 226 | "language_info": { 227 | "codemirror_mode": { 228 | "name": "ipython", 229 | "version": 3 230 | }, 231 | "file_extension": ".py", 232 | "mimetype": "text/x-python", 233 | "name": "python", 234 | "nbconvert_exporter": "python", 235 | "pygments_lexer": "ipython3", 236 | "version": "3.6.7" 237 | } 238 | }, 239 | "nbformat": 4, 240 | "nbformat_minor": 0 241 | } -------------------------------------------------------------------------------- /chapter1/cifar10_tutorial.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%matplotlib inline" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "\nTraining a Classifier\n=====================\n\nThis is it. You have seen how to define neural networks, compute loss and make\nupdates to the weights of the network.\n\nNow you might be thinking,\n\nWhat about data?\n----------------\n\nGenerally, when you have to deal with image, text, audio or video data,\nyou can use standard python packages that load data into a numpy array.\nThen you can convert this array into a ``torch.*Tensor``.\n\n- For images, packages such as Pillow, OpenCV are useful\n- For audio, packages such as scipy and librosa\n- For text, either raw Python or Cython based loading, or NLTK and\n SpaCy are useful\n\nSpecifically for vision, we have created a package called\n``torchvision``, that has data loaders for common datasets such as\nImagenet, CIFAR10, MNIST, etc. and data transformers for images, viz.,\n``torchvision.datasets`` and ``torch.utils.data.DataLoader``.\n\nThis provides a huge convenience and avoids writing boilerplate code.\n\nFor this tutorial, we will use the CIFAR10 dataset.\nIt has the classes: \u2018airplane\u2019, \u2018automobile\u2019, \u2018bird\u2019, \u2018cat\u2019, \u2018deer\u2019,\n\u2018dog\u2019, \u2018frog\u2019, \u2018horse\u2019, \u2018ship\u2019, \u2018truck\u2019. The images in CIFAR-10 are of\nsize 3x32x32, i.e. 3-channel color images of 32x32 pixels in size.\n\n.. figure:: /_static/img/cifar10.png\n :alt: cifar10\n\n cifar10\n\n\nTraining an image classifier\n----------------------------\n\nWe will do the following steps in order:\n\n1. Load and normalizing the CIFAR10 training and test datasets using\n ``torchvision``\n2. Define a Convolution Neural Network\n3. Define a loss function\n4. Train the network on the training data\n5. Test the network on the test data\n\n1. Loading and normalizing CIFAR10\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nUsing ``torchvision``, it\u2019s extremely easy to load CIFAR10.\n\n" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": { 25 | "collapsed": false 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "import torch\nimport torchvision\nimport torchvision.transforms as transforms" 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "metadata": {}, 35 | "source": [ 36 | "The output of torchvision datasets are PILImage images of range [0, 1].\nWe transform them to Tensors of normalized range [-1, 1].\n\n" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": null, 42 | "metadata": { 43 | "collapsed": false 44 | }, 45 | "outputs": [], 46 | "source": [ 47 | "transform = transforms.Compose(\n [transforms.ToTensor(),\n transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])\n\ntrainset = torchvision.datasets.CIFAR10(root='./data', train=True,\n download=True, transform=transform)\ntrainloader = torch.utils.data.DataLoader(trainset, batch_size=4,\n shuffle=True, num_workers=2)\n\ntestset = torchvision.datasets.CIFAR10(root='./data', train=False,\n download=True, transform=transform)\ntestloader = torch.utils.data.DataLoader(testset, batch_size=4,\n shuffle=False, num_workers=2)\n\nclasses = ('plane', 'car', 'bird', 'cat',\n 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": {}, 53 | "source": [ 54 | "Let us show some of the training images, for fun.\n\n" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": { 61 | "collapsed": false 62 | }, 63 | "outputs": [], 64 | "source": [ 65 | "import matplotlib.pyplot as plt\nimport numpy as np\n\n# functions to show an image\n\n\ndef imshow(img):\n img = img / 2 + 0.5 # unnormalize\n npimg = img.numpy()\n plt.imshow(np.transpose(npimg, (1, 2, 0)))\n\n\n# get some random training images\ndataiter = iter(trainloader)\nimages, labels = dataiter.next()\n\n# show images\nimshow(torchvision.utils.make_grid(images))\n# print labels\nprint(' '.join('%5s' % classes[labels[j]] for j in range(4)))" 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "metadata": {}, 71 | "source": [ 72 | "2. Define a Convolution Neural Network\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nCopy the neural network from the Neural Networks section before and modify it to\ntake 3-channel images (instead of 1-channel images as it was defined).\n\n" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": null, 78 | "metadata": { 79 | "collapsed": false 80 | }, 81 | "outputs": [], 82 | "source": [ 83 | "import torch.nn as nn\nimport torch.nn.functional as F\n\n\nclass Net(nn.Module):\n def __init__(self):\n super(Net, self).__init__()\n self.conv1 = nn.Conv2d(3, 6, 5)\n self.pool = nn.MaxPool2d(2, 2)\n self.conv2 = nn.Conv2d(6, 16, 5)\n self.fc1 = nn.Linear(16 * 5 * 5, 120)\n self.fc2 = nn.Linear(120, 84)\n self.fc3 = nn.Linear(84, 10)\n\n def forward(self, x):\n x = self.pool(F.relu(self.conv1(x)))\n x = self.pool(F.relu(self.conv2(x)))\n x = x.view(-1, 16 * 5 * 5)\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = self.fc3(x)\n return x\n\n\nnet = Net()" 84 | ] 85 | }, 86 | { 87 | "cell_type": "markdown", 88 | "metadata": {}, 89 | "source": [ 90 | "3. Define a Loss function and optimizer\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nLet's use a Classification Cross-Entropy loss and SGD with momentum.\n\n" 91 | ] 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": null, 96 | "metadata": { 97 | "collapsed": false 98 | }, 99 | "outputs": [], 100 | "source": [ 101 | "import torch.optim as optim\n\ncriterion = nn.CrossEntropyLoss()\noptimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "4. Train the network\n^^^^^^^^^^^^^^^^^^^^\n\nThis is when things start to get interesting.\nWe simply have to loop over our data iterator, and feed the inputs to the\nnetwork and optimize.\n\n" 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": null, 114 | "metadata": { 115 | "collapsed": false 116 | }, 117 | "outputs": [], 118 | "source": [ 119 | "for epoch in range(2): # loop over the dataset multiple times\n\n running_loss = 0.0\n for i, data in enumerate(trainloader, 0):\n # get the inputs\n inputs, labels = data\n\n # zero the parameter gradients\n optimizer.zero_grad()\n\n # forward + backward + optimize\n outputs = net(inputs)\n loss = criterion(outputs, labels)\n loss.backward()\n optimizer.step()\n\n # print statistics\n running_loss += loss.item()\n if i % 2000 == 1999: # print every 2000 mini-batches\n print('[%d, %5d] loss: %.3f' %\n (epoch + 1, i + 1, running_loss / 2000))\n running_loss = 0.0\n\nprint('Finished Training')" 120 | ] 121 | }, 122 | { 123 | "cell_type": "markdown", 124 | "metadata": {}, 125 | "source": [ 126 | "5. Test the network on the test data\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWe have trained the network for 2 passes over the training dataset.\nBut we need to check if the network has learnt anything at all.\n\nWe will check this by predicting the class label that the neural network\noutputs, and checking it against the ground-truth. If the prediction is\ncorrect, we add the sample to the list of correct predictions.\n\nOkay, first step. Let us display an image from the test set to get familiar.\n\n" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": null, 132 | "metadata": { 133 | "collapsed": false 134 | }, 135 | "outputs": [], 136 | "source": [ 137 | "dataiter = iter(testloader)\nimages, labels = dataiter.next()\n\n# print images\nimshow(torchvision.utils.make_grid(images))\nprint('GroundTruth: ', ' '.join('%5s' % classes[labels[j]] for j in range(4)))" 138 | ] 139 | }, 140 | { 141 | "cell_type": "markdown", 142 | "metadata": {}, 143 | "source": [ 144 | "Okay, now let us see what the neural network thinks these examples above are:\n\n" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": null, 150 | "metadata": { 151 | "collapsed": false 152 | }, 153 | "outputs": [], 154 | "source": [ 155 | "outputs = net(images)" 156 | ] 157 | }, 158 | { 159 | "cell_type": "markdown", 160 | "metadata": {}, 161 | "source": [ 162 | "The outputs are energies for the 10 classes.\nHigher the energy for a class, the more the network\nthinks that the image is of the particular class.\nSo, let's get the index of the highest energy:\n\n" 163 | ] 164 | }, 165 | { 166 | "cell_type": "code", 167 | "execution_count": null, 168 | "metadata": { 169 | "collapsed": false 170 | }, 171 | "outputs": [], 172 | "source": [ 173 | "_, predicted = torch.max(outputs, 1)\n\nprint('Predicted: ', ' '.join('%5s' % classes[predicted[j]]\n for j in range(4)))" 174 | ] 175 | }, 176 | { 177 | "cell_type": "markdown", 178 | "metadata": {}, 179 | "source": [ 180 | "The results seem pretty good.\n\nLet us look at how the network performs on the whole dataset.\n\n" 181 | ] 182 | }, 183 | { 184 | "cell_type": "code", 185 | "execution_count": null, 186 | "metadata": { 187 | "collapsed": false 188 | }, 189 | "outputs": [], 190 | "source": [ 191 | "correct = 0\ntotal = 0\nwith torch.no_grad():\n for data in testloader:\n images, labels = data\n outputs = net(images)\n _, predicted = torch.max(outputs.data, 1)\n total += labels.size(0)\n correct += (predicted == labels).sum().item()\n\nprint('Accuracy of the network on the 10000 test images: %d %%' % (\n 100 * correct / total))" 192 | ] 193 | }, 194 | { 195 | "cell_type": "markdown", 196 | "metadata": {}, 197 | "source": [ 198 | "That looks waaay better than chance, which is 10% accuracy (randomly picking\na class out of 10 classes).\nSeems like the network learnt something.\n\nHmmm, what are the classes that performed well, and the classes that did\nnot perform well:\n\n" 199 | ] 200 | }, 201 | { 202 | "cell_type": "code", 203 | "execution_count": null, 204 | "metadata": { 205 | "collapsed": false 206 | }, 207 | "outputs": [], 208 | "source": [ 209 | "class_correct = list(0. for i in range(10))\nclass_total = list(0. for i in range(10))\nwith torch.no_grad():\n for data in testloader:\n images, labels = data\n outputs = net(images)\n _, predicted = torch.max(outputs, 1)\n c = (predicted == labels).squeeze()\n for i in range(4):\n label = labels[i]\n class_correct[label] += c[i].item()\n class_total[label] += 1\n\n\nfor i in range(10):\n print('Accuracy of %5s : %2d %%' % (\n classes[i], 100 * class_correct[i] / class_total[i]))" 210 | ] 211 | }, 212 | { 213 | "cell_type": "markdown", 214 | "metadata": {}, 215 | "source": [ 216 | "Okay, so what next?\n\nHow do we run these neural networks on the GPU?\n\nTraining on GPU\n----------------\nJust like how you transfer a Tensor on to the GPU, you transfer the neural\nnet onto the GPU.\n\nLet's first define our device as the first visible cuda device if we have\nCUDA available:\n\n" 217 | ] 218 | }, 219 | { 220 | "cell_type": "code", 221 | "execution_count": null, 222 | "metadata": { 223 | "collapsed": false 224 | }, 225 | "outputs": [], 226 | "source": [ 227 | "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n\n# Assume that we are on a CUDA machine, then this should print a CUDA device:\n\nprint(device)" 228 | ] 229 | }, 230 | { 231 | "cell_type": "markdown", 232 | "metadata": {}, 233 | "source": [ 234 | "The rest of this section assumes that `device` is a CUDA device.\n\nThen these methods will recursively go over all modules and convert their\nparameters and buffers to CUDA tensors:\n\n.. code:: python\n\n net.to(device)\n\n\nRemember that you will have to send the inputs and targets at every step\nto the GPU too:\n\n.. code:: python\n\n inputs, labels = inputs.to(device), labels.to(device)\n\nWhy dont I notice MASSIVE speedup compared to CPU? Because your network\nis realllly small.\n\n**Exercise:** Try increasing the width of your network (argument 2 of\nthe first ``nn.Conv2d``, and argument 1 of the second ``nn.Conv2d`` \u2013\nthey need to be the same number), see what kind of speedup you get.\n\n**Goals achieved**:\n\n- Understanding PyTorch's Tensor library and neural networks at a high level.\n- Train a small neural network to classify images\n\nTraining on multiple GPUs\n-------------------------\nIf you want to see even more MASSIVE speedup using all of your GPUs,\nplease check out :doc:`data_parallel_tutorial`.\n\nWhere do I go next?\n-------------------\n\n- :doc:`Train neural nets to play video games `\n- `Train a state-of-the-art ResNet network on imagenet`_\n- `Train a face generator using Generative Adversarial Networks`_\n- `Train a word-level language model using Recurrent LSTM networks`_\n- `More examples`_\n- `More tutorials`_\n- `Discuss PyTorch on the Forums`_\n- `Chat with other users on Slack`_\n\n\n" 235 | ] 236 | } 237 | ], 238 | "metadata": { 239 | "kernelspec": { 240 | "display_name": "Python 3", 241 | "language": "python", 242 | "name": "python3" 243 | }, 244 | "language_info": { 245 | "codemirror_mode": { 246 | "name": "ipython", 247 | "version": 3 248 | }, 249 | "file_extension": ".py", 250 | "mimetype": "text/x-python", 251 | "name": "python", 252 | "nbconvert_exporter": "python", 253 | "pygments_lexer": "ipython3", 254 | "version": "3.6.7" 255 | } 256 | }, 257 | "nbformat": 4, 258 | "nbformat_minor": 0 259 | } -------------------------------------------------------------------------------- /chapter1/data_parallel_tutorial.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%matplotlib inline" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "\nOptional: Data Parallelism\n==========================\n**Authors**: `Sung Kim `_ and `Jenny Kang `_\n\nIn this tutorial, we will learn how to use multiple GPUs using ``DataParallel``.\n\nIt's very easy to use GPUs with PyTorch. You can put the model on a GPU:\n\n.. code:: python\n\n device = torch.device(\"cuda:0\")\n model.to(device)\n\nThen, you can copy all your tensors to the GPU:\n\n.. code:: python\n\n mytensor = my_tensor.to(device)\n\nPlease note that just calling ``my_tensor.to(device)`` returns a new copy of\n``my_tensor`` on GPU instead of rewriting ``my_tensor``. You need to assign it to\na new tensor and use that tensor on the GPU.\n\nIt's natural to execute your forward, backward propagations on multiple GPUs.\nHowever, Pytorch will only use one GPU by default. You can easily run your\noperations on multiple GPUs by making your model run parallelly using\n``DataParallel``:\n\n.. code:: python\n\n model = nn.DataParallel(model)\n\nThat's the core behind this tutorial. We will explore it in more detail below.\n\n" 19 | ] 20 | }, 21 | { 22 | "cell_type": "markdown", 23 | "metadata": {}, 24 | "source": [ 25 | "Imports and parameters\n----------------------\n\nImport PyTorch modules and define parameters.\n\n\n" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": null, 31 | "metadata": { 32 | "collapsed": false 33 | }, 34 | "outputs": [], 35 | "source": [ 36 | "import torch\nimport torch.nn as nn\nfrom torch.utils.data import Dataset, DataLoader\n\n# Parameters and DataLoaders\ninput_size = 5\noutput_size = 2\n\nbatch_size = 30\ndata_size = 100" 37 | ] 38 | }, 39 | { 40 | "cell_type": "markdown", 41 | "metadata": {}, 42 | "source": [ 43 | "Device\n\n\n" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": null, 49 | "metadata": { 50 | "collapsed": false 51 | }, 52 | "outputs": [], 53 | "source": [ 54 | "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")" 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "Dummy DataSet\n-------------\n\nMake a dummy (random) dataset. You just need to implement the\ngetitem\n\n\n" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": null, 67 | "metadata": { 68 | "collapsed": false 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "class RandomDataset(Dataset):\n\n def __init__(self, size, length):\n self.len = length\n self.data = torch.randn(length, size)\n\n def __getitem__(self, index):\n return self.data[index]\n\n def __len__(self):\n return self.len\n\nrand_loader = DataLoader(dataset=RandomDataset(input_size, data_size),\n batch_size=batch_size, shuffle=True)" 73 | ] 74 | }, 75 | { 76 | "cell_type": "markdown", 77 | "metadata": {}, 78 | "source": [ 79 | "Simple Model\n------------\n\nFor the demo, our model just gets an input, performs a linear operation, and\ngives an output. However, you can use ``DataParallel`` on any model (CNN, RNN,\nCapsule Net etc.)\n\nWe've placed a print statement inside the model to monitor the size of input\nand output tensors.\nPlease pay attention to what is printed at batch rank 0.\n\n\n" 80 | ] 81 | }, 82 | { 83 | "cell_type": "code", 84 | "execution_count": null, 85 | "metadata": { 86 | "collapsed": false 87 | }, 88 | "outputs": [], 89 | "source": [ 90 | "class Model(nn.Module):\n # Our model\n\n def __init__(self, input_size, output_size):\n super(Model, self).__init__()\n self.fc = nn.Linear(input_size, output_size)\n\n def forward(self, input):\n output = self.fc(input)\n print(\"\\tIn Model: input size\", input.size(),\n \"output size\", output.size())\n\n return output" 91 | ] 92 | }, 93 | { 94 | "cell_type": "markdown", 95 | "metadata": {}, 96 | "source": [ 97 | "Create Model and DataParallel\n-----------------------------\n\nThis is the core part of the tutorial. First, we need to make a model instance\nand check if we have multiple GPUs. If we have multiple GPUs, we can wrap\nour model using ``nn.DataParallel``. Then we can put our model on GPUs by\n``model.to(device)``\n\n\n" 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": null, 103 | "metadata": { 104 | "collapsed": false 105 | }, 106 | "outputs": [], 107 | "source": [ 108 | "model = Model(input_size, output_size)\nif torch.cuda.device_count() > 1:\n print(\"Let's use\", torch.cuda.device_count(), \"GPUs!\")\n # dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs\n model = nn.DataParallel(model)\n\nmodel.to(device)" 109 | ] 110 | }, 111 | { 112 | "cell_type": "markdown", 113 | "metadata": {}, 114 | "source": [ 115 | "Run the Model\n-------------\n\nNow we can see the sizes of input and output tensors.\n\n\n" 116 | ] 117 | }, 118 | { 119 | "cell_type": "code", 120 | "execution_count": null, 121 | "metadata": { 122 | "collapsed": false 123 | }, 124 | "outputs": [], 125 | "source": [ 126 | "for data in rand_loader:\n input = data.to(device)\n output = model(input)\n print(\"Outside: input size\", input.size(),\n \"output_size\", output.size())" 127 | ] 128 | }, 129 | { 130 | "cell_type": "markdown", 131 | "metadata": {}, 132 | "source": [ 133 | "Results\n-------\n\nIf you have no GPU or one GPU, when we batch 30 inputs and 30 outputs, the model gets 30 and outputs 30 as\nexpected. But if you have multiple GPUs, then you can get results like this.\n\n2 GPUs\n~~~~~~\n\nIf you have 2, you will see:\n\n.. code:: bash\n\n # on 2 GPUs\n Let's use 2 GPUs!\n In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])\n In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])\n Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])\n In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])\n Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])\n In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])\n Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n In Model: input size torch.Size([5, 5]) output size torch.Size([5, 2])\n In Model: input size torch.Size([5, 5]) output size torch.Size([5, 2])\n Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])\n\n3 GPUs\n~~~~~~\n\nIf you have 3 GPUs, you will see:\n\n.. code:: bash\n\n Let's use 3 GPUs!\n In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])\n Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])\n\n8 GPUs\n~~~~~~~~~~~~~~\n\nIf you have 8, you will see:\n\n.. code:: bash\n\n Let's use 8 GPUs!\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])\n In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])\n In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])\n Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])\n\n\n" 134 | ] 135 | }, 136 | { 137 | "cell_type": "markdown", 138 | "metadata": {}, 139 | "source": [ 140 | "Summary\n-------\n\nDataParallel splits your data automatically and sends job orders to multiple\nmodels on several GPUs. After each model finishes their job, DataParallel\ncollects and merges the results before returning it to you.\n\nFor more information, please check out\nhttps://pytorch.org/tutorials/beginner/former\\_torchies/parallelism\\_tutorial.html.\n\n\n" 141 | ] 142 | } 143 | ], 144 | "metadata": { 145 | "kernelspec": { 146 | "display_name": "Python 3", 147 | "language": "python", 148 | "name": "python3" 149 | }, 150 | "language_info": { 151 | "codemirror_mode": { 152 | "name": "ipython", 153 | "version": 3 154 | }, 155 | "file_extension": ".py", 156 | "mimetype": "text/x-python", 157 | "name": "python", 158 | "nbconvert_exporter": "python", 159 | "pygments_lexer": "ipython3", 160 | "version": "3.6.7" 161 | } 162 | }, 163 | "nbformat": 4, 164 | "nbformat_minor": 0 165 | } -------------------------------------------------------------------------------- /chapter1/neural_networks_tutorial.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%matplotlib inline" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "\nNeural Networks\n===============\n\nNeural networks can be constructed using the ``torch.nn`` package.\n\nNow that you had a glimpse of ``autograd``, ``nn`` depends on\n``autograd`` to define models and differentiate them.\nAn ``nn.Module`` contains layers, and a method ``forward(input)``\\ that\nreturns the ``output``.\n\nFor example, look at this network that classifies digit images:\n\n.. figure:: /_static/img/mnist.png\n :alt: convnet\n\n convnet\n\nIt is a simple feed-forward network. It takes the input, feeds it\nthrough several layers one after the other, and then finally gives the\noutput.\n\nA typical training procedure for a neural network is as follows:\n\n- Define the neural network that has some learnable parameters (or\n weights)\n- Iterate over a dataset of inputs\n- Process input through the network\n- Compute the loss (how far is the output from being correct)\n- Propagate gradients back into the network\u2019s parameters\n- Update the weights of the network, typically using a simple update rule:\n ``weight = weight - learning_rate * gradient``\n\nDefine the network\n------------------\n\nLet\u2019s define this network:\n\n" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": { 25 | "collapsed": false 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\n\nclass Net(nn.Module):\n\n def __init__(self):\n super(Net, self).__init__()\n # 1 input image channel, 6 output channels, 5x5 square convolution\n # kernel\n self.conv1 = nn.Conv2d(1, 6, 5)\n self.conv2 = nn.Conv2d(6, 16, 5)\n # an affine operation: y = Wx + b\n self.fc1 = nn.Linear(16 * 5 * 5, 120)\n self.fc2 = nn.Linear(120, 84)\n self.fc3 = nn.Linear(84, 10)\n\n def forward(self, x):\n # Max pooling over a (2, 2) window\n x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))\n # If the size is a square you can only specify a single number\n x = F.max_pool2d(F.relu(self.conv2(x)), 2)\n x = x.view(-1, self.num_flat_features(x))\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = self.fc3(x)\n return x\n\n def num_flat_features(self, x):\n size = x.size()[1:] # all dimensions except the batch dimension\n num_features = 1\n for s in size:\n num_features *= s\n return num_features\n\n\nnet = Net()\nprint(net)" 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "metadata": {}, 35 | "source": [ 36 | "You just have to define the ``forward`` function, and the ``backward``\nfunction (where gradients are computed) is automatically defined for you\nusing ``autograd``.\nYou can use any of the Tensor operations in the ``forward`` function.\n\nThe learnable parameters of a model are returned by ``net.parameters()``\n\n" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": null, 42 | "metadata": { 43 | "collapsed": false 44 | }, 45 | "outputs": [], 46 | "source": [ 47 | "params = list(net.parameters())\nprint(len(params))\nprint(params[0].size()) # conv1's .weight" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": {}, 53 | "source": [ 54 | "Let try a random 32x32 input\nNote: Expected input size to this net(LeNet) is 32x32. To use this net on\nMNIST dataset, please resize the images from the dataset to 32x32.\n\n" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": { 61 | "collapsed": false 62 | }, 63 | "outputs": [], 64 | "source": [ 65 | "input = torch.randn(1, 1, 32, 32)\nout = net(input)\nprint(out)" 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "metadata": {}, 71 | "source": [ 72 | "Zero the gradient buffers of all parameters and backprops with random\ngradients:\n\n" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": null, 78 | "metadata": { 79 | "collapsed": false 80 | }, 81 | "outputs": [], 82 | "source": [ 83 | "net.zero_grad()\nout.backward(torch.randn(1, 10))" 84 | ] 85 | }, 86 | { 87 | "cell_type": "markdown", 88 | "metadata": {}, 89 | "source": [ 90 | "

Note

``torch.nn`` only supports mini-batches. The entire ``torch.nn``\n package only supports inputs that are a mini-batch of samples, and not\n a single sample.\n\n For example, ``nn.Conv2d`` will take in a 4D Tensor of\n ``nSamples x nChannels x Height x Width``.\n\n If you have a single sample, just use ``input.unsqueeze(0)`` to add\n a fake batch dimension.

\n\nBefore proceeding further, let's recap all the classes you\u2019ve seen so far.\n\n**Recap:**\n - ``torch.Tensor`` - A *multi-dimensional array* with support for autograd\n operations like ``backward()``. Also *holds the gradient* w.r.t. the\n tensor.\n - ``nn.Module`` - Neural network module. *Convenient way of\n encapsulating parameters*, with helpers for moving them to GPU,\n exporting, loading, etc.\n - ``nn.Parameter`` - A kind of Tensor, that is *automatically\n registered as a parameter when assigned as an attribute to a*\n ``Module``.\n - ``autograd.Function`` - Implements *forward and backward definitions\n of an autograd operation*. Every ``Tensor`` operation, creates at\n least a single ``Function`` node, that connects to functions that\n created a ``Tensor`` and *encodes its history*.\n\n**At this point, we covered:**\n - Defining a neural network\n - Processing inputs and calling backward\n\n**Still Left:**\n - Computing the loss\n - Updating the weights of the network\n\nLoss Function\n-------------\nA loss function takes the (output, target) pair of inputs, and computes a\nvalue that estimates how far away the output is from the target.\n\nThere are several different\n`loss functions `_ under the\nnn package .\nA simple loss is: ``nn.MSELoss`` which computes the mean-squared error\nbetween the input and the target.\n\nFor example:\n\n" 91 | ] 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": null, 96 | "metadata": { 97 | "collapsed": false 98 | }, 99 | "outputs": [], 100 | "source": [ 101 | "output = net(input)\ntarget = torch.randn(10) # a dummy target, for example\ntarget = target.view(1, -1) # make it the same shape as output\ncriterion = nn.MSELoss()\n\nloss = criterion(output, target)\nprint(loss)" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "Now, if you follow ``loss`` in the backward direction, using its\n``.grad_fn`` attribute, you will see a graph of computations that looks\nlike this:\n\n::\n\n input -> conv2d -> relu -> maxpool2d -> conv2d -> relu -> maxpool2d\n -> view -> linear -> relu -> linear -> relu -> linear\n -> MSELoss\n -> loss\n\nSo, when we call ``loss.backward()``, the whole graph is differentiated\nw.r.t. the loss, and all Tensors in the graph that has ``requires_grad=True``\nwill have their ``.grad`` Tensor accumulated with the gradient.\n\nFor illustration, let us follow a few steps backward:\n\n" 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": null, 114 | "metadata": { 115 | "collapsed": false 116 | }, 117 | "outputs": [], 118 | "source": [ 119 | "print(loss.grad_fn) # MSELoss\nprint(loss.grad_fn.next_functions[0][0]) # Linear\nprint(loss.grad_fn.next_functions[0][0].next_functions[0][0]) # ReLU" 120 | ] 121 | }, 122 | { 123 | "cell_type": "markdown", 124 | "metadata": {}, 125 | "source": [ 126 | "Backprop\n--------\nTo backpropagate the error all we have to do is to ``loss.backward()``.\nYou need to clear the existing gradients though, else gradients will be\naccumulated to existing gradients.\n\n\nNow we shall call ``loss.backward()``, and have a look at conv1's bias\ngradients before and after the backward.\n\n" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": null, 132 | "metadata": { 133 | "collapsed": false 134 | }, 135 | "outputs": [], 136 | "source": [ 137 | "net.zero_grad() # zeroes the gradient buffers of all parameters\n\nprint('conv1.bias.grad before backward')\nprint(net.conv1.bias.grad)\n\nloss.backward()\n\nprint('conv1.bias.grad after backward')\nprint(net.conv1.bias.grad)" 138 | ] 139 | }, 140 | { 141 | "cell_type": "markdown", 142 | "metadata": {}, 143 | "source": [ 144 | "Now, we have seen how to use loss functions.\n\n**Read Later:**\n\n The neural network package contains various modules and loss functions\n that form the building blocks of deep neural networks. A full list with\n documentation is `here `_.\n\n**The only thing left to learn is:**\n\n - Updating the weights of the network\n\nUpdate the weights\n------------------\nThe simplest update rule used in practice is the Stochastic Gradient\nDescent (SGD):\n\n ``weight = weight - learning_rate * gradient``\n\nWe can implement this using simple python code:\n\n.. code:: python\n\n learning_rate = 0.01\n for f in net.parameters():\n f.data.sub_(f.grad.data * learning_rate)\n\nHowever, as you use neural networks, you want to use various different\nupdate rules such as SGD, Nesterov-SGD, Adam, RMSProp, etc.\nTo enable this, we built a small package: ``torch.optim`` that\nimplements all these methods. Using it is very simple:\n\n" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": null, 150 | "metadata": { 151 | "collapsed": false 152 | }, 153 | "outputs": [], 154 | "source": [ 155 | "import torch.optim as optim\n\n# create your optimizer\noptimizer = optim.SGD(net.parameters(), lr=0.01)\n\n# in your training loop:\noptimizer.zero_grad() # zero the gradient buffers\noutput = net(input)\nloss = criterion(output, target)\nloss.backward()\noptimizer.step() # Does the update" 156 | ] 157 | }, 158 | { 159 | "cell_type": "markdown", 160 | "metadata": {}, 161 | "source": [ 162 | ".. Note::\n\n Observe how gradient buffers had to be manually set to zero using\n ``optimizer.zero_grad()``. This is because gradients are accumulated\n as explained in `Backprop`_ section.\n\n" 163 | ] 164 | } 165 | ], 166 | "metadata": { 167 | "kernelspec": { 168 | "display_name": "Python 3", 169 | "language": "python", 170 | "name": "python3" 171 | }, 172 | "language_info": { 173 | "codemirror_mode": { 174 | "name": "ipython", 175 | "version": 3 176 | }, 177 | "file_extension": ".py", 178 | "mimetype": "text/x-python", 179 | "name": "python", 180 | "nbconvert_exporter": "python", 181 | "pygments_lexer": "ipython3", 182 | "version": "3.6.7" 183 | } 184 | }, 185 | "nbformat": 4, 186 | "nbformat_minor": 0 187 | } -------------------------------------------------------------------------------- /chapter1/pic1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter1/pic1.png -------------------------------------------------------------------------------- /chapter1/readme.md: -------------------------------------------------------------------------------- 1 | # PyTorch 中文手册第一章 ： PyTorch入门 2 | 3 | ## 目录 4 | 5 | 1. [PyTorch 简介](1.1-pytorch-introduction.md) 6 | 2. [PyTorch 环境搭建](1.2-pytorch-installation.md) 7 | 3. [PyTorch 深度学习：60分钟快速入门（官方）](1.3-deep-learning-with-pytorch-60-minute-blitz.md) 8 | - [张量](1_tensor_tutorial.ipynb) 9 | - [Autograd：自动求导](2_autograd_tutorial.ipynb) 10 | - [神经网络](3_neural_networks_tutorial.ipynb) 11 | - [训练一个分类器](4_cifar10_tutorial.ipynb) 12 | - [选读：数据并行处理（多GPU）](5_data_parallel_tutorial.ipynb) 13 | 4. [相关资源介绍](1.4-pytorch-resource.md) 14 | -------------------------------------------------------------------------------- /chapter1/tensor_tutorial.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%matplotlib inline" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "\nWhat is PyTorch?\n================\n\nIt\u2019s a Python-based scientific computing package targeted at two sets of\naudiences:\n\n- A replacement for NumPy to use the power of GPUs\n- a deep learning research platform that provides maximum flexibility\n and speed\n\nGetting Started\n---------------\n\nTensors\n^^^^^^^\n\nTensors are similar to NumPy\u2019s ndarrays, with the addition being that\nTensors can also be used on a GPU to accelerate computing.\n\n" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": { 25 | "collapsed": false 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "from __future__ import print_function\nimport torch" 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "metadata": {}, 35 | "source": [ 36 | "Construct a 5x3 matrix, uninitialized:\n\n" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": null, 42 | "metadata": { 43 | "collapsed": false 44 | }, 45 | "outputs": [], 46 | "source": [ 47 | "x = torch.empty(5, 3)\nprint(x)" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": {}, 53 | "source": [ 54 | "Construct a randomly initialized matrix:\n\n" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": { 61 | "collapsed": false 62 | }, 63 | "outputs": [], 64 | "source": [ 65 | "x = torch.rand(5, 3)\nprint(x)" 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "metadata": {}, 71 | "source": [ 72 | "Construct a matrix filled zeros and of dtype long:\n\n" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": null, 78 | "metadata": { 79 | "collapsed": false 80 | }, 81 | "outputs": [], 82 | "source": [ 83 | "x = torch.zeros(5, 3, dtype=torch.long)\nprint(x)" 84 | ] 85 | }, 86 | { 87 | "cell_type": "markdown", 88 | "metadata": {}, 89 | "source": [ 90 | "Construct a tensor directly from data:\n\n" 91 | ] 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": null, 96 | "metadata": { 97 | "collapsed": false 98 | }, 99 | "outputs": [], 100 | "source": [ 101 | "x = torch.tensor([5.5, 3])\nprint(x)" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "or create a tensor based on an existing tensor. These methods\nwill reuse properties of the input tensor, e.g. dtype, unless\nnew values are provided by user\n\n" 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": null, 114 | "metadata": { 115 | "collapsed": false 116 | }, 117 | "outputs": [], 118 | "source": [ 119 | "x = x.new_ones(5, 3, dtype=torch.double) # new_* methods take in sizes\nprint(x)\n\nx = torch.randn_like(x, dtype=torch.float) # override dtype!\nprint(x) # result has the same size" 120 | ] 121 | }, 122 | { 123 | "cell_type": "markdown", 124 | "metadata": {}, 125 | "source": [ 126 | "Get its size:\n\n" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": null, 132 | "metadata": { 133 | "collapsed": false 134 | }, 135 | "outputs": [], 136 | "source": [ 137 | "print(x.size())" 138 | ] 139 | }, 140 | { 141 | "cell_type": "markdown", 142 | "metadata": {}, 143 | "source": [ 144 | "

Note

``torch.Size`` is in fact a tuple, so it supports all tuple operations.

\n\nOperations\n^^^^^^^^^^\nThere are multiple syntaxes for operations. In the following\nexample, we will take a look at the addition operation.\n\nAddition: syntax 1\n\n" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": null, 150 | "metadata": { 151 | "collapsed": false 152 | }, 153 | "outputs": [], 154 | "source": [ 155 | "y = torch.rand(5, 3)\nprint(x + y)" 156 | ] 157 | }, 158 | { 159 | "cell_type": "markdown", 160 | "metadata": {}, 161 | "source": [ 162 | "Addition: syntax 2\n\n" 163 | ] 164 | }, 165 | { 166 | "cell_type": "code", 167 | "execution_count": null, 168 | "metadata": { 169 | "collapsed": false 170 | }, 171 | "outputs": [], 172 | "source": [ 173 | "print(torch.add(x, y))" 174 | ] 175 | }, 176 | { 177 | "cell_type": "markdown", 178 | "metadata": {}, 179 | "source": [ 180 | "Addition: providing an output tensor as argument\n\n" 181 | ] 182 | }, 183 | { 184 | "cell_type": "code", 185 | "execution_count": null, 186 | "metadata": { 187 | "collapsed": false 188 | }, 189 | "outputs": [], 190 | "source": [ 191 | "result = torch.empty(5, 3)\ntorch.add(x, y, out=result)\nprint(result)" 192 | ] 193 | }, 194 | { 195 | "cell_type": "markdown", 196 | "metadata": {}, 197 | "source": [ 198 | "Addition: in-place\n\n" 199 | ] 200 | }, 201 | { 202 | "cell_type": "code", 203 | "execution_count": null, 204 | "metadata": { 205 | "collapsed": false 206 | }, 207 | "outputs": [], 208 | "source": [ 209 | "# adds x to y\ny.add_(x)\nprint(y)" 210 | ] 211 | }, 212 | { 213 | "cell_type": "markdown", 214 | "metadata": {}, 215 | "source": [ 216 | "

Note

Any operation that mutates a tensor in-place is post-fixed with an ``_``.\n For example: ``x.copy_(y)``, ``x.t_()``, will change ``x``.

\n\nYou can use standard NumPy-like indexing with all bells and whistles!\n\n" 217 | ] 218 | }, 219 | { 220 | "cell_type": "code", 221 | "execution_count": null, 222 | "metadata": { 223 | "collapsed": false 224 | }, 225 | "outputs": [], 226 | "source": [ 227 | "print(x[:, 1])" 228 | ] 229 | }, 230 | { 231 | "cell_type": "markdown", 232 | "metadata": {}, 233 | "source": [ 234 | "Resizing: If you want to resize/reshape tensor, you can use ``torch.view``:\n\n" 235 | ] 236 | }, 237 | { 238 | "cell_type": "code", 239 | "execution_count": null, 240 | "metadata": { 241 | "collapsed": false 242 | }, 243 | "outputs": [], 244 | "source": [ 245 | "x = torch.randn(4, 4)\ny = x.view(16)\nz = x.view(-1, 8) # the size -1 is inferred from other dimensions\nprint(x.size(), y.size(), z.size())" 246 | ] 247 | }, 248 | { 249 | "cell_type": "markdown", 250 | "metadata": {}, 251 | "source": [ 252 | "If you have a one element tensor, use ``.item()`` to get the value as a\nPython number\n\n" 253 | ] 254 | }, 255 | { 256 | "cell_type": "code", 257 | "execution_count": null, 258 | "metadata": { 259 | "collapsed": false 260 | }, 261 | "outputs": [], 262 | "source": [ 263 | "x = torch.randn(1)\nprint(x)\nprint(x.item())" 264 | ] 265 | }, 266 | { 267 | "cell_type": "markdown", 268 | "metadata": {}, 269 | "source": [ 270 | "**Read later:**\n\n\n 100+ Tensor operations, including transposing, indexing, slicing,\n mathematical operations, linear algebra, random numbers, etc.,\n are described\n `here `_.\n\nNumPy Bridge\n------------\n\nConverting a Torch Tensor to a NumPy array and vice versa is a breeze.\n\nThe Torch Tensor and NumPy array will share their underlying memory\nlocations, and changing one will change the other.\n\nConverting a Torch Tensor to a NumPy Array\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n" 271 | ] 272 | }, 273 | { 274 | "cell_type": "code", 275 | "execution_count": null, 276 | "metadata": { 277 | "collapsed": false 278 | }, 279 | "outputs": [], 280 | "source": [ 281 | "a = torch.ones(5)\nprint(a)" 282 | ] 283 | }, 284 | { 285 | "cell_type": "code", 286 | "execution_count": null, 287 | "metadata": { 288 | "collapsed": false 289 | }, 290 | "outputs": [], 291 | "source": [ 292 | "b = a.numpy()\nprint(b)" 293 | ] 294 | }, 295 | { 296 | "cell_type": "markdown", 297 | "metadata": {}, 298 | "source": [ 299 | "See how the numpy array changed in value.\n\n" 300 | ] 301 | }, 302 | { 303 | "cell_type": "code", 304 | "execution_count": null, 305 | "metadata": { 306 | "collapsed": false 307 | }, 308 | "outputs": [], 309 | "source": [ 310 | "a.add_(1)\nprint(a)\nprint(b)" 311 | ] 312 | }, 313 | { 314 | "cell_type": "markdown", 315 | "metadata": {}, 316 | "source": [ 317 | "Converting NumPy Array to Torch Tensor\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSee how changing the np array changed the Torch Tensor automatically\n\n" 318 | ] 319 | }, 320 | { 321 | "cell_type": "code", 322 | "execution_count": null, 323 | "metadata": { 324 | "collapsed": false 325 | }, 326 | "outputs": [], 327 | "source": [ 328 | "import numpy as np\na = np.ones(5)\nb = torch.from_numpy(a)\nnp.add(a, 1, out=a)\nprint(a)\nprint(b)" 329 | ] 330 | }, 331 | { 332 | "cell_type": "markdown", 333 | "metadata": {}, 334 | "source": [ 335 | "All the Tensors on the CPU except a CharTensor support converting to\nNumPy and back.\n\nCUDA Tensors\n------------\n\nTensors can be moved onto any device using the ``.to`` method.\n\n" 336 | ] 337 | }, 338 | { 339 | "cell_type": "code", 340 | "execution_count": null, 341 | "metadata": { 342 | "collapsed": false 343 | }, 344 | "outputs": [], 345 | "source": [ 346 | "# let us run this cell only if CUDA is available\n# We will use ``torch.device`` objects to move tensors in and out of GPU\nif torch.cuda.is_available():\n device = torch.device(\"cuda\") # a CUDA device object\n y = torch.ones_like(x, device=device) # directly create a tensor on GPU\n x = x.to(device) # or just use strings ``.to(\"cuda\")``\n z = x + y\n print(z)\n print(z.to(\"cpu\", torch.double)) # ``.to`` can also change dtype together!" 347 | ] 348 | } 349 | ], 350 | "metadata": { 351 | "kernelspec": { 352 | "display_name": "Python 3", 353 | "language": "python", 354 | "name": "python3" 355 | }, 356 | "language_info": { 357 | "codemirror_mode": { 358 | "name": "ipython", 359 | "version": 3 360 | }, 361 | "file_extension": ".py", 362 | "mimetype": "text/x-python", 363 | "name": "python", 364 | "nbconvert_exporter": "python", 365 | "pygments_lexer": "ipython3", 366 | "version": "3.6.7" 367 | } 368 | }, 369 | "nbformat": 4, 370 | "nbformat_minor": 0 371 | } -------------------------------------------------------------------------------- /chapter2/1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/1.png -------------------------------------------------------------------------------- /chapter2/10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/10.png -------------------------------------------------------------------------------- /chapter2/11.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/11.png -------------------------------------------------------------------------------- /chapter2/12.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/12.png -------------------------------------------------------------------------------- /chapter2/2.1.2-pytorch-basics-autograd.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# PyTorch 基础 : 自动求导\n", 8 | "深度学习的算法本质上是通过反向传播求导数，而PyTorch的autograd模块则实现了此功能。在Tensor上的所有操作，autograd都能为它们自动提供微分，避免了手动计算导数的复杂过程。\n", 9 | "\n", 10 | "***从0.4起, Variable 正式合并入Tensor, Variable 本来实现的自动微分功能，Tensor就能支持。读者还是可以使用Variable(tensor), 但是这个操作其实什么都没做。***\n", 11 | "\n", 12 | "所以，以后的代码建议直接使用Tensor，因为官方文档中已经将Variable设置成过期模块\n", 13 | "\n", 14 | "要想使得Tensor使用autograd功能，只需要设置tensor.requries_grad=True" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": 1, 20 | "metadata": {}, 21 | "outputs": [ 22 | { 23 | "data": { 24 | "text/plain": [ 25 | "'1.0.0'" 26 | ] 27 | }, 28 | "execution_count": 1, 29 | "metadata": {}, 30 | "output_type": "execute_result" 31 | } 32 | ], 33 | "source": [ 34 | "# 首先要引入相关的包\n", 35 | "import torch\n", 36 | "#打印一下版本\n", 37 | "torch.__version__" 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "在张量创建时，通过设置 requires_grad 标识为Ture来告诉Pytorch需要对该张量进行自动的求导，PyTorch回记录该张量的每一步操作历史并自动计算" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": 2, 50 | "metadata": {}, 51 | "outputs": [ 52 | { 53 | "data": { 54 | "text/plain": [ 55 | "tensor([[0.7966, 0.6401, 0.1673, 0.4988, 0.3223],\n", 56 | " [0.8881, 0.4756, 0.4391, 0.8762, 0.5714],\n", 57 | " [0.9581, 0.3330, 0.0885, 0.5103, 0.7473],\n", 58 | " [0.9493, 0.4851, 0.8115, 0.0083, 0.2560],\n", 59 | " [0.3152, 0.0132, 0.0858, 0.0406, 0.1677]], requires_grad=True)" 60 | ] 61 | }, 62 | "execution_count": 2, 63 | "metadata": {}, 64 | "output_type": "execute_result" 65 | } 66 | ], 67 | "source": [ 68 | "x = torch.rand(5, 5, requires_grad=True)\n", 69 | "x" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 3, 75 | "metadata": {}, 76 | "outputs": [ 77 | { 78 | "data": { 79 | "text/plain": [ 80 | "tensor([[0.7349, 0.7580, 0.2178, 0.1207, 0.9278],\n", 81 | " [0.8113, 0.3937, 0.9049, 0.0681, 0.0810],\n", 82 | " [0.5836, 0.2981, 0.7601, 0.6309, 0.3913],\n", 83 | " [0.4454, 0.2274, 0.4062, 0.5707, 0.7378],\n", 84 | " [0.4096, 0.7709, 0.7267, 0.4186, 0.1841]], requires_grad=True)" 85 | ] 86 | }, 87 | "execution_count": 3, 88 | "metadata": {}, 89 | "output_type": "execute_result" 90 | } 91 | ], 92 | "source": [ 93 | "y = torch.rand(5, 5, requires_grad=True)\n", 94 | "y" 95 | ] 96 | }, 97 | { 98 | "cell_type": "markdown", 99 | "metadata": {}, 100 | "source": [ 101 | "我们看到 该张量的grad_fn已经被赋予了一个新的函数，下面我们来调用反向传播函数，计算其梯度" 102 | ] 103 | }, 104 | { 105 | "cell_type": "code", 106 | "execution_count": 4, 107 | "metadata": {}, 108 | "outputs": [ 109 | { 110 | "data": { 111 | "text/plain": [ 112 | "tensor(24.0248, grad_fn=)" 113 | ] 114 | }, 115 | "execution_count": 4, 116 | "metadata": {}, 117 | "output_type": "execute_result" 118 | } 119 | ], 120 | "source": [ 121 | "z=torch.sum(x+y)\n", 122 | "z" 123 | ] 124 | }, 125 | { 126 | "cell_type": "markdown", 127 | "metadata": {}, 128 | "source": [ 129 | "## 简单的自动求导" 130 | ] 131 | }, 132 | { 133 | "cell_type": "code", 134 | "execution_count": 5, 135 | "metadata": {}, 136 | "outputs": [], 137 | "source": [ 138 | "z.backward()" 139 | ] 140 | }, 141 | { 142 | "cell_type": "code", 143 | "execution_count": 6, 144 | "metadata": {}, 145 | "outputs": [ 146 | { 147 | "name": "stdout", 148 | "output_type": "stream", 149 | "text": [ 150 | "tensor([[1., 1., 1., 1., 1.],\n", 151 | " [1., 1., 1., 1., 1.],\n", 152 | " [1., 1., 1., 1., 1.],\n", 153 | " [1., 1., 1., 1., 1.],\n", 154 | " [1., 1., 1., 1., 1.]]) tensor([[1., 1., 1., 1., 1.],\n", 155 | " [1., 1., 1., 1., 1.],\n", 156 | " [1., 1., 1., 1., 1.],\n", 157 | " [1., 1., 1., 1., 1.],\n", 158 | " [1., 1., 1., 1., 1.]])\n" 159 | ] 160 | } 161 | ], 162 | "source": [ 163 | "#看一下x和y的梯度\n", 164 | "print(x.grad,y.grad)" 165 | ] 166 | }, 167 | { 168 | "cell_type": "markdown", 169 | "metadata": {}, 170 | "source": [ 171 | "## 复杂的自动求导" 172 | ] 173 | }, 174 | { 175 | "cell_type": "code", 176 | "execution_count": 7, 177 | "metadata": {}, 178 | "outputs": [ 179 | { 180 | "data": { 181 | "text/plain": [ 182 | "tensor([[0.7252, 1.7343, 0.0371, 0.3972, 0.7656],\n", 183 | " [0.1452, 0.6768, 0.7937, 0.4231, 0.5193],\n", 184 | " [1.6275, 0.2478, 0.0672, 0.8888, 0.5644],\n", 185 | " [1.3385, 0.1239, 1.0143, 0.0052, 1.0832],\n", 186 | " [0.9570, 0.2122, 0.5613, 0.1223, 0.3774]], grad_fn=)" 187 | ] 188 | }, 189 | "execution_count": 7, 190 | "metadata": {}, 191 | "output_type": "execute_result" 192 | } 193 | ], 194 | "source": [ 195 | "x = torch.rand(5, 5, requires_grad=True)\n", 196 | "y = torch.rand(5, 5, requires_grad=True)\n", 197 | "z= x**2+y**3\n", 198 | "z" 199 | ] 200 | }, 201 | { 202 | "cell_type": "code", 203 | "execution_count": 8, 204 | "metadata": {}, 205 | "outputs": [ 206 | { 207 | "name": "stdout", 208 | "output_type": "stream", 209 | "text": [ 210 | "tensor([[0.2330, 1.9195, 0.2359, 1.2472, 1.7473],\n", 211 | " [0.7365, 0.5059, 1.7816, 1.0653, 1.4377],\n", 212 | " [1.8439, 0.7431, 0.5183, 1.5588, 1.4149],\n", 213 | " [1.4076, 0.3347, 1.6417, 0.1141, 0.7594],\n", 214 | " [1.8638, 0.5341, 0.3941, 0.6932, 0.6859]])\n" 215 | ] 216 | } 217 | ], 218 | "source": [ 219 | "#我们的返回值不是一个scalar，所以需要输入一个大小相同的张量作为参数，这里我们用ones_like函数根据x生成一个张量\n", 220 | "z.backward(torch.ones_like(x))\n", 221 | "print(x.grad)" 222 | ] 223 | }, 224 | { 225 | "cell_type": "markdown", 226 | "metadata": {}, 227 | "source": [ 228 | "我们可以使用`with torch.no_grad()`禁止已经设置requires_grad=True的向量进行自动求导，这个方法在测试集测试准确率的时候回经常用到，例如" 229 | ] 230 | }, 231 | { 232 | "cell_type": "code", 233 | "execution_count": 9, 234 | "metadata": {}, 235 | "outputs": [ 236 | { 237 | "name": "stdout", 238 | "output_type": "stream", 239 | "text": [ 240 | "False\n" 241 | ] 242 | } 243 | ], 244 | "source": [ 245 | "with torch.no_grad():\n", 246 | " print((x +y*2).requires_grad)" 247 | ] 248 | } 249 | ], 250 | "metadata": { 251 | "kernelspec": { 252 | "display_name": "Pytorch for Deeplearning", 253 | "language": "python", 254 | "name": "pytorch" 255 | }, 256 | "language_info": { 257 | "codemirror_mode": { 258 | "name": "ipython", 259 | "version": 3 260 | }, 261 | "file_extension": ".py", 262 | "mimetype": "text/x-python", 263 | "name": "python", 264 | "nbconvert_exporter": "python", 265 | "pygments_lexer": "ipython3", 266 | "version": "3.6.7" 267 | } 268 | }, 269 | "nbformat": 4, 270 | "nbformat_minor": 2 271 | } 272 | -------------------------------------------------------------------------------- /chapter2/2.1.3-pytorch-basics-nerual-network.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# PyTorch 基础 : 神经网络包nn和优化器optm\n", 8 | "torch.nn是专门为神经网络设计的模块化接口。nn构建于 Autograd之上，可用来定义和运行神经网络。\n", 9 | "这里我们主要介绍几个一些常用的类" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "**约定：torch.nn 我们为了方便使用，会为他设置别名为nn,本章除nn以外还有其他的命名约定**" 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": 1, 22 | "metadata": {}, 23 | "outputs": [ 24 | { 25 | "data": { 26 | "text/plain": [ 27 | "'1.0.0'" 28 | ] 29 | }, 30 | "execution_count": 1, 31 | "metadata": {}, 32 | "output_type": "execute_result" 33 | } 34 | ], 35 | "source": [ 36 | "# 首先要引入相关的包\n", 37 | "import torch\n", 38 | "# 引入torch.nn并指定别名\n", 39 | "import torch.nn as nn\n", 40 | "#打印一下版本\n", 41 | "torch.__version__" 42 | ] 43 | }, 44 | { 45 | "cell_type": "markdown", 46 | "metadata": {}, 47 | "source": [ 48 | "除了nn别名以外，我们还引用了nn.functional，这个包中包含了神经网络中使用的一些常用函数，这些函数的特点是，不具有可学习的参数(如ReLU，pool，DropOut等)，这些函数可以放在构造函数中，也可以不放，但是这里建议不放。\n", 49 | "\n", 50 | "一般情况下我们会**将nn.functional 设置为大写的F**，这样缩写方便调用" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": 2, 56 | "metadata": {}, 57 | "outputs": [], 58 | "source": [ 59 | "import torch.nn.functional as F" 60 | ] 61 | }, 62 | { 63 | "cell_type": "markdown", 64 | "metadata": {}, 65 | "source": [ 66 | "## 定义一个网络\n", 67 | "PyTorch中已经为我们准备好了现成的网络模型，只要继承nn.Module，并实现它的forward方法，PyTorch会根据autograd，自动实现backward函数，在forward函数中可使用任何tensor支持的函数，还可以使用if、for循环、print、log等Python语法，写法和标准的Python写法一致。" 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": 3, 73 | "metadata": {}, 74 | "outputs": [ 75 | { 76 | "name": "stdout", 77 | "output_type": "stream", 78 | "text": [ 79 | "Net(\n", 80 | " (conv1): Conv2d(1, 6, kernel_size=(3, 3), stride=(1, 1))\n", 81 | " (fc1): Linear(in_features=1350, out_features=10, bias=True)\n", 82 | ")\n" 83 | ] 84 | } 85 | ], 86 | "source": [ 87 | "class Net(nn.Module):\n", 88 | " def __init__(self):\n", 89 | " # nn.Module子类的函数必须在构造函数中执行父类的构造函数\n", 90 | " super(Net, self).__init__()\n", 91 | " \n", 92 | " # 卷积层 '1'表示输入图片为单通道, '6'表示输出通道数，'3'表示卷积核为3*3\n", 93 | " self.conv1 = nn.Conv2d(1, 6, 3) \n", 94 | " #线性层，输入1350个特征，输出10个特征\n", 95 | " self.fc1 = nn.Linear(1350, 10) #这里的1350是如何计算的呢？这就要看后面的forward函数\n", 96 | " #正向传播 \n", 97 | " def forward(self, x): \n", 98 | " print(x.size()) # 结果：[1, 1, 32, 32]\n", 99 | " # 卷积 -> 激活 -> 池化 \n", 100 | " x = self.conv1(x) #根据卷积的尺寸计算公式，计算结果是30，具体计算公式后面第二张第四节 卷积神经网络 有详细介绍。\n", 101 | " x = F.relu(x)\n", 102 | " print(x.size()) # 结果：[1, 6, 30, 30]\n", 103 | " x = F.max_pool2d(x, (2, 2)) #我们使用池化层，计算结果是15\n", 104 | " x = F.relu(x)\n", 105 | " print(x.size()) # 结果：[1, 6, 15, 15]\n", 106 | " # reshape，‘-1’表示自适应\n", 107 | " #这里做的就是压扁的操作 就是把后面的[1, 6, 15, 15]压扁，变为 [1, 1350]\n", 108 | " x = x.view(x.size()[0], -1) \n", 109 | " print(x.size()) # 这里就是fc1层的的输入1350 \n", 110 | " x = self.fc1(x) \n", 111 | " return x\n", 112 | "\n", 113 | "net = Net()\n", 114 | "print(net)" 115 | ] 116 | }, 117 | { 118 | "cell_type": "markdown", 119 | "metadata": {}, 120 | "source": [ 121 | "网络的可学习参数通过net.parameters()返回" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": 4, 127 | "metadata": { 128 | "scrolled": true 129 | }, 130 | "outputs": [ 131 | { 132 | "name": "stdout", 133 | "output_type": "stream", 134 | "text": [ 135 | "Parameter containing:\n", 136 | "tensor([[[[ 0.2745, 0.2594, 0.0171],\n", 137 | " [ 0.0429, 0.3013, -0.0208],\n", 138 | " [ 0.1459, -0.3223, 0.1797]]],\n", 139 | "\n", 140 | "\n", 141 | " [[[ 0.1847, 0.0227, -0.1919],\n", 142 | " [-0.0210, -0.1336, -0.2176],\n", 143 | " [-0.2164, -0.1244, -0.2428]]],\n", 144 | "\n", 145 | "\n", 146 | " [[[ 0.1042, -0.0055, -0.2171],\n", 147 | " [ 0.3306, -0.2808, 0.2058],\n", 148 | " [ 0.2492, 0.2971, 0.2277]]],\n", 149 | "\n", 150 | "\n", 151 | " [[[ 0.2134, -0.0644, -0.3044],\n", 152 | " [ 0.0040, 0.0828, -0.2093],\n", 153 | " [ 0.0204, 0.1065, 0.1168]]],\n", 154 | "\n", 155 | "\n", 156 | " [[[ 0.1651, -0.2244, 0.3072],\n", 157 | " [-0.2301, 0.2443, -0.2340],\n", 158 | " [ 0.0685, 0.1026, 0.1754]]],\n", 159 | "\n", 160 | "\n", 161 | " [[[ 0.1691, -0.0790, 0.2617],\n", 162 | " [ 0.1956, 0.1477, 0.0877],\n", 163 | " [ 0.0538, -0.3091, 0.2030]]]], requires_grad=True)\n", 164 | "Parameter containing:\n", 165 | "tensor([ 0.2355, 0.2949, -0.1283, -0.0848, 0.2027, -0.3331],\n", 166 | " requires_grad=True)\n", 167 | "Parameter containing:\n", 168 | "tensor([[ 2.0555e-02, -2.1445e-02, -1.7981e-02, ..., -2.3864e-02,\n", 169 | " 8.5149e-03, -6.2071e-04],\n", 170 | " [-1.1755e-02, 1.0010e-02, 2.1978e-02, ..., 1.8433e-02,\n", 171 | " 7.1362e-03, -4.0951e-03],\n", 172 | " [ 1.6187e-02, 2.1623e-02, 1.1840e-02, ..., 5.7059e-03,\n", 173 | " -2.7165e-02, 1.3463e-03],\n", 174 | " ...,\n", 175 | " [-3.2552e-03, 1.7277e-02, -1.4907e-02, ..., 7.4232e-03,\n", 176 | " -2.7188e-02, -4.6431e-03],\n", 177 | " [-1.9786e-02, -3.7382e-03, 1.2259e-02, ..., 3.2471e-03,\n", 178 | " -1.2375e-02, -1.6372e-02],\n", 179 | " [-8.2350e-03, 4.1301e-03, -1.9192e-03, ..., -2.3119e-05,\n", 180 | " 2.0167e-03, 1.9528e-02]], requires_grad=True)\n", 181 | "Parameter containing:\n", 182 | "tensor([ 0.0162, -0.0146, -0.0218, 0.0212, -0.0119, -0.0142, -0.0079, 0.0171,\n", 183 | " 0.0205, 0.0164], requires_grad=True)\n" 184 | ] 185 | } 186 | ], 187 | "source": [ 188 | "for parameters in net.parameters():\n", 189 | " print(parameters)" 190 | ] 191 | }, 192 | { 193 | "cell_type": "markdown", 194 | "metadata": {}, 195 | "source": [ 196 | "net.named_parameters可同时返回可学习的参数及名称。" 197 | ] 198 | }, 199 | { 200 | "cell_type": "code", 201 | "execution_count": 5, 202 | "metadata": {}, 203 | "outputs": [ 204 | { 205 | "name": "stdout", 206 | "output_type": "stream", 207 | "text": [ 208 | "conv1.weight : torch.Size([6, 1, 3, 3])\n", 209 | "conv1.bias : torch.Size([6])\n", 210 | "fc1.weight : torch.Size([10, 1350])\n", 211 | "fc1.bias : torch.Size([10])\n" 212 | ] 213 | } 214 | ], 215 | "source": [ 216 | "for name,parameters in net.named_parameters():\n", 217 | " print(name,':',parameters.size())" 218 | ] 219 | }, 220 | { 221 | "cell_type": "markdown", 222 | "metadata": {}, 223 | "source": [ 224 | "forward函数的输入和输出都是Tensor" 225 | ] 226 | }, 227 | { 228 | "cell_type": "code", 229 | "execution_count": 6, 230 | "metadata": {}, 231 | "outputs": [ 232 | { 233 | "name": "stdout", 234 | "output_type": "stream", 235 | "text": [ 236 | "torch.Size([1, 1, 32, 32])\n", 237 | "torch.Size([1, 6, 30, 30])\n", 238 | "torch.Size([1, 6, 15, 15])\n", 239 | "torch.Size([1, 1350])\n" 240 | ] 241 | }, 242 | { 243 | "data": { 244 | "text/plain": [ 245 | "torch.Size([1, 10])" 246 | ] 247 | }, 248 | "execution_count": 6, 249 | "metadata": {}, 250 | "output_type": "execute_result" 251 | } 252 | ], 253 | "source": [ 254 | "input = torch.randn(1, 1, 32, 32) # 这里的对应前面fforward的输入是32\n", 255 | "out = net(input)\n", 256 | "out.size()" 257 | ] 258 | }, 259 | { 260 | "cell_type": "code", 261 | "execution_count": 7, 262 | "metadata": {}, 263 | "outputs": [ 264 | { 265 | "data": { 266 | "text/plain": [ 267 | "torch.Size([1, 1, 32, 32])" 268 | ] 269 | }, 270 | "execution_count": 7, 271 | "metadata": {}, 272 | "output_type": "execute_result" 273 | } 274 | ], 275 | "source": [ 276 | "input.size()" 277 | ] 278 | }, 279 | { 280 | "cell_type": "markdown", 281 | "metadata": {}, 282 | "source": [ 283 | "在反向传播前，先要将所有参数的梯度清零" 284 | ] 285 | }, 286 | { 287 | "cell_type": "code", 288 | "execution_count": 8, 289 | "metadata": {}, 290 | "outputs": [], 291 | "source": [ 292 | "net.zero_grad() \n", 293 | "out.backward(torch.ones(1,10)) # 反向传播的实现是PyTorch自动实现的，我们只要调用这个函数即可" 294 | ] 295 | }, 296 | { 297 | "cell_type": "markdown", 298 | "metadata": {}, 299 | "source": [ 300 | "**注意**:torch.nn只支持mini-batches，不支持一次只输入一个样本，即一次必须是一个batch。\n", 301 | "\n", 302 | "也就是说，就算我们输入一个样本，也会对样本进行分批，所以，所有的输入都会增加一个维度，我们对比下刚才的input,nn中定义为3维,但是我们人工创建时多增加了一个维度，变为了4维,最前面的1即为batch-size" 303 | ] 304 | }, 305 | { 306 | "cell_type": "markdown", 307 | "metadata": {}, 308 | "source": [ 309 | "## 损失函数\n", 310 | "在nn中PyTorch还预制了常用的损失函数，下面我们用MSELoss用来计算均方误差" 311 | ] 312 | }, 313 | { 314 | "cell_type": "code", 315 | "execution_count": 9, 316 | "metadata": {}, 317 | "outputs": [ 318 | { 319 | "name": "stdout", 320 | "output_type": "stream", 321 | "text": [ 322 | "28.92203712463379\n" 323 | ] 324 | } 325 | ], 326 | "source": [ 327 | "y = torch.arange(0,10).view(1,10).float()\n", 328 | "criterion = nn.MSELoss()\n", 329 | "loss = criterion(out, y)\n", 330 | "#loss是个scalar，我们可以直接用item获取到他的python类型的数值\n", 331 | "print(loss.item()) " 332 | ] 333 | }, 334 | { 335 | "cell_type": "markdown", 336 | "metadata": {}, 337 | "source": [ 338 | "## 优化器\n", 339 | "在反向传播计算完所有参数的梯度后，还需要使用优化方法来更新网络的权重和参数，例如随机梯度下降法(SGD)的更新策略如下：\n", 340 | "\n", 341 | "weight = weight - learning_rate * gradient\n", 342 | "\n", 343 | "在torch.optim中实现大多数的优化方法，例如RMSProp、Adam、SGD等，下面我们使用SGD做个简单的样例" 344 | ] 345 | }, 346 | { 347 | "cell_type": "code", 348 | "execution_count": 10, 349 | "metadata": {}, 350 | "outputs": [], 351 | "source": [ 352 | "import torch.optim" 353 | ] 354 | }, 355 | { 356 | "cell_type": "code", 357 | "execution_count": 11, 358 | "metadata": {}, 359 | "outputs": [ 360 | { 361 | "name": "stdout", 362 | "output_type": "stream", 363 | "text": [ 364 | "torch.Size([1, 1, 32, 32])\n", 365 | "torch.Size([1, 6, 30, 30])\n", 366 | "torch.Size([1, 6, 15, 15])\n", 367 | "torch.Size([1, 1350])\n" 368 | ] 369 | } 370 | ], 371 | "source": [ 372 | "out = net(input) # 这里调用的时候会打印出我们在forword函数中打印的x的大小\n", 373 | "criterion = nn.MSELoss()\n", 374 | "loss = criterion(out, y)\n", 375 | "#新建一个优化器，SGD只需要要调整的参数和学习率\n", 376 | "optimizer = torch.optim.SGD(net.parameters(), lr = 0.01)\n", 377 | "# 先梯度清零(与net.zero_grad()效果一样)\n", 378 | "optimizer.zero_grad() \n", 379 | "loss.backward()\n", 380 | "\n", 381 | "#更新参数\n", 382 | "optimizer.step()" 383 | ] 384 | }, 385 | { 386 | "cell_type": "markdown", 387 | "metadata": {}, 388 | "source": [ 389 | "这样，神经网络的数据的一个完整的传播就已经通过PyTorch实现了，下面一章将介绍PyTorch提供的数据加载和处理工具，使用这些工具可以方便的处理所需要的数据。\n", 390 | "\n", 391 | "看完这节，大家可能对神经网络模型里面的一些参数的计算方式还有疑惑，这部分会在第二张第四节 卷积神经网络 有详细介绍，并且在第三章 第二节 MNIST数据集手写数字识别 的实践代码中有详细的注释说明。" 392 | ] 393 | }, 394 | { 395 | "cell_type": "code", 396 | "execution_count": null, 397 | "metadata": {}, 398 | "outputs": [], 399 | "source": [] 400 | } 401 | ], 402 | "metadata": { 403 | "kernelspec": { 404 | "display_name": "pytorch 1.0", 405 | "language": "python", 406 | "name": "pytorch1" 407 | }, 408 | "language_info": { 409 | "codemirror_mode": { 410 | "name": "ipython", 411 | "version": 3 412 | }, 413 | "file_extension": ".py", 414 | "mimetype": "text/x-python", 415 | "name": "python", 416 | "nbconvert_exporter": "python", 417 | "pygments_lexer": "ipython3", 418 | "version": "3.6.6" 419 | } 420 | }, 421 | "nbformat": 4, 422 | "nbformat_minor": 2 423 | } 424 | -------------------------------------------------------------------------------- /chapter2/2.1.4-pytorch-basics-data-lorder.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# PyTorch 基础 :数据的加载和预处理\n", 8 | "PyTorch通过torch.utils.data对一般常用的数据加载进行了封装，可以很容易地实现多线程数据预读和批量加载。\n", 9 | "并且torchvision已经预先实现了常用图像数据集，包括前面使用过的CIFAR-10，ImageNet、COCO、MNIST、LSUN等数据集，可通过torchvision.datasets方便的调用" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 1, 15 | "metadata": {}, 16 | "outputs": [ 17 | { 18 | "data": { 19 | "text/plain": [ 20 | "'1.0.0'" 21 | ] 22 | }, 23 | "execution_count": 1, 24 | "metadata": {}, 25 | "output_type": "execute_result" 26 | } 27 | ], 28 | "source": [ 29 | "# 首先要引入相关的包\n", 30 | "import torch\n", 31 | "#打印一下版本\n", 32 | "torch.__version__" 33 | ] 34 | }, 35 | { 36 | "cell_type": "markdown", 37 | "metadata": {}, 38 | "source": [ 39 | "## Dataset\n", 40 | "Dataset是一个抽象类, 为了能够方便的读取，需要将要使用的数据包装为Dataset类。\n", 41 | "自定义的Dataset需要继承它并且实现两个成员方法：\n", 42 | "1. `__getitem__()` 该方法定义每次怎么获取数据\n", 43 | "2. `__len__()` 该方法返回数据集的总长度\n", 44 | "\n", 45 | "下面我们使用kaggle上的一个竞赛[bluebook for bulldozers](https://www.kaggle.com/c/bluebook-for-bulldozers/data)自定义一个数据集，为了方便介绍，我们使用里面的数据字典来做说明（因为条数少）" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 2, 51 | "metadata": {}, 52 | "outputs": [], 53 | "source": [ 54 | "#引用\n", 55 | "from torch.utils.data import Dataset\n", 56 | "import pandas as pd" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": 3, 62 | "metadata": {}, 63 | "outputs": [], 64 | "source": [ 65 | "#定义一个数据集\n", 66 | "class BulldozerDataset(Dataset):\n", 67 | " \"\"\" 数据集演示 \"\"\"\n", 68 | " def __init__(self, csv_file):\n", 69 | " \"\"\"实现初始化方法，在初始化的时候将数据读载入\"\"\"\n", 70 | " self.df=pd.read_csv(csv_file)\n", 71 | " def __len__(self):\n", 72 | " '''\n", 73 | " 返回df的长度\n", 74 | " '''\n", 75 | " return len(self.df)\n", 76 | " def __getitem__(self, idx):\n", 77 | " '''\n", 78 | " 根据IDX返回一列数据\n", 79 | " '''\n", 80 | " return self.df.iloc[idx].SalePrice" 81 | ] 82 | }, 83 | { 84 | "cell_type": "markdown", 85 | "metadata": {}, 86 | "source": [ 87 | "至此，我们的数据集已经定义完成了，我们可以实例话一个对象访问他" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": 4, 93 | "metadata": {}, 94 | "outputs": [], 95 | "source": [ 96 | "ds_demo= BulldozerDataset('median_benchmark.csv')" 97 | ] 98 | }, 99 | { 100 | "cell_type": "markdown", 101 | "metadata": {}, 102 | "source": [ 103 | "我们可以直接使用如下命令查看数据集数据\n" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": 5, 109 | "metadata": {}, 110 | "outputs": [ 111 | { 112 | "data": { 113 | "text/plain": [ 114 | "11573" 115 | ] 116 | }, 117 | "execution_count": 5, 118 | "metadata": {}, 119 | "output_type": "execute_result" 120 | } 121 | ], 122 | "source": [ 123 | "#实现了__len__ 方法所以可以直接使用len获取数据总数\n", 124 | "len(ds_demo)" 125 | ] 126 | }, 127 | { 128 | "cell_type": "code", 129 | "execution_count": 6, 130 | "metadata": {}, 131 | "outputs": [ 132 | { 133 | "data": { 134 | "text/plain": [ 135 | "24000.0" 136 | ] 137 | }, 138 | "execution_count": 6, 139 | "metadata": {}, 140 | "output_type": "execute_result" 141 | } 142 | ], 143 | "source": [ 144 | "#用索引可以直接访问对应的数据\n", 145 | "ds_demo[0]" 146 | ] 147 | }, 148 | { 149 | "cell_type": "markdown", 150 | "metadata": {}, 151 | "source": [ 152 | "自定义的数据集已经创建好了，下面我们使用官方提供的数据载入器，读取数据\n", 153 | "## Datalorder\n", 154 | "DataLoader为我们提供了对Dataset的读取操作，常用参数有：batch_size(每个batch的大小), shuffle(是否进行shuffle操作), num_workers(加载数据的时候使用几个子进程)，下面做一个简单的操作" 155 | ] 156 | }, 157 | { 158 | "cell_type": "code", 159 | "execution_count": 7, 160 | "metadata": {}, 161 | "outputs": [], 162 | "source": [ 163 | "dl = torch.utils.data.DataLoader(ds_demo, batch_size=10, shuffle=True, num_workers=0)" 164 | ] 165 | }, 166 | { 167 | "cell_type": "markdown", 168 | "metadata": {}, 169 | "source": [ 170 | "DataLoader返回的是一个迭代器，我们可以使用迭代器分次获取数据" 171 | ] 172 | }, 173 | { 174 | "cell_type": "code", 175 | "execution_count": 8, 176 | "metadata": {}, 177 | "outputs": [ 178 | { 179 | "name": "stdout", 180 | "output_type": "stream", 181 | "text": [ 182 | "tensor([24000., 24000., 24000., 24000., 24000., 24000., 24000., 24000., 24000.,\n", 183 | " 24000.], dtype=torch.float64)\n" 184 | ] 185 | } 186 | ], 187 | "source": [ 188 | "idata=iter(dl)\n", 189 | "print(next(idata))" 190 | ] 191 | }, 192 | { 193 | "cell_type": "markdown", 194 | "metadata": {}, 195 | "source": [ 196 | "或者直接使用for循环对其进行遍历" 197 | ] 198 | }, 199 | { 200 | "cell_type": "code", 201 | "execution_count": 9, 202 | "metadata": {}, 203 | "outputs": [ 204 | { 205 | "name": "stdout", 206 | "output_type": "stream", 207 | "text": [ 208 | "0 tensor([24000., 24000., 24000., 24000., 24000., 24000., 24000., 24000., 24000.,\n", 209 | " 24000.], dtype=torch.float64)\n" 210 | ] 211 | } 212 | ], 213 | "source": [ 214 | "for i, data in enumerate(dl):\n", 215 | " print(i,data)\n", 216 | " #这里只循环一遍\n", 217 | " break" 218 | ] 219 | }, 220 | { 221 | "cell_type": "markdown", 222 | "metadata": {}, 223 | "source": [ 224 | "我们已经可以通过dataset定义数据集，并使用Datalorder载入和遍历数据集，除了这些以外，PyTorch还提供能torcvision的计算机视觉扩展包，里面封装了\n", 225 | "## torchvision 包\n", 226 | "torchvision 是PyTorch中专门用来处理图像的库，PyTorch官网的安装教程中最后的pip install torchvision 就是安装这个包。" 227 | ] 228 | }, 229 | { 230 | "cell_type": "markdown", 231 | "metadata": {}, 232 | "source": [ 233 | "### torchvision.datasets\n", 234 | "torchvision.datasets 可以理解为PyTorch团队自定义的dataset，这些dataset帮我们提前处理好了很多的图片数据集，我们拿来就可以直接使用：\n", 235 | "- MNIST\n", 236 | "- COCO\n", 237 | "- Captions\n", 238 | "- Detection\n", 239 | "- LSUN\n", 240 | "- ImageFolder\n", 241 | "- Imagenet-12\n", 242 | "- CIFAR\n", 243 | "- STL10\n", 244 | "- SVHN\n", 245 | "- PhotoTour\n", 246 | "我们可以直接使用，示例如下：" 247 | ] 248 | }, 249 | { 250 | "cell_type": "code", 251 | "execution_count": 10, 252 | "metadata": {}, 253 | "outputs": [], 254 | "source": [ 255 | "import torchvision.datasets as datasets\n", 256 | "trainset = datasets.MNIST(root='./data', # 表示 MNIST 数据的加载的目录\n", 257 | " train=True, # 表示是否加载数据库的训练集，false的时候加载测试集\n", 258 | " download=True, # 表示是否自动下载 MNIST 数据集\n", 259 | " transform=None) # 表示是否需要对数据进行预处理，none为不进行预处理\n" 260 | ] 261 | }, 262 | { 263 | "cell_type": "markdown", 264 | "metadata": {}, 265 | "source": [ 266 | "### torchvision.models\n", 267 | "torchvision不仅提供了常用图片数据集，还提供了训练好的模型，可以加载之后，直接使用，或者在进行迁移学习\n", 268 | "torchvision.models模块的 子模块中包含以下模型结构。\n", 269 | "- AlexNet\n", 270 | "- VGG\n", 271 | "- ResNet\n", 272 | "- SqueezeNet\n", 273 | "- DenseNet" 274 | ] 275 | }, 276 | { 277 | "cell_type": "code", 278 | "execution_count": 11, 279 | "metadata": {}, 280 | "outputs": [], 281 | "source": [ 282 | "#我们直接可以使用训练好的模型，当然这个与datasets相同，都是需要从服务器下载的\n", 283 | "import torchvision.models as models\n", 284 | "resnet18 = models.resnet18(pretrained=True)" 285 | ] 286 | }, 287 | { 288 | "cell_type": "markdown", 289 | "metadata": {}, 290 | "source": [ 291 | "### torchvision.transforms\n", 292 | "transforms 模块提供了一般的图像转换操作类，用作数据的处理和增广" 293 | ] 294 | }, 295 | { 296 | "cell_type": "code", 297 | "execution_count": 12, 298 | "metadata": {}, 299 | "outputs": [], 300 | "source": [ 301 | "from torchvision import transforms as transforms\n", 302 | "transform = transforms.Compose([\n", 303 | " transforms.RandomCrop(32, padding=4), #先四周填充0，在吧图像随机裁剪成32*32\n", 304 | " transforms.RandomHorizontalFlip(), #图像一半的概率翻转，一半的概率不翻转\n", 305 | " transforms.RandomRotation((-45,45)), #随机旋转\n", 306 | " transforms.ToTensor(),\n", 307 | " transforms.Normalize((0.4914, 0.4822, 0.4465), (0.229, 0.224, 0.225)), #R,G,B每层的归一化用到的均值和方差\n", 308 | "])" 309 | ] 310 | }, 311 | { 312 | "cell_type": "markdown", 313 | "metadata": {}, 314 | "source": [ 315 | "肯定有人会问：(0.485, 0.456, 0.406), (0.2023, 0.1994, 0.2010) 这几个数字是什么意思？\n", 316 | "\n", 317 | "官方的这个帖子有详细的说明:\n", 318 | "https://discuss.pytorch.org/t/normalization-in-the-mnist-example/457/21\n", 319 | "这些都是根据ImageNet训练的归一化参数，可以直接使用，我们认为这个是固定值就可以" 320 | ] 321 | }, 322 | { 323 | "cell_type": "markdown", 324 | "metadata": {}, 325 | "source": [ 326 | "我们已经完成了Python的基本内容的介绍，下面我们要介绍神经网络的理论基础，里面的公式等内容我们都使用PyTorch来实现" 327 | ] 328 | }, 329 | { 330 | "cell_type": "code", 331 | "execution_count": null, 332 | "metadata": {}, 333 | "outputs": [], 334 | "source": [] 335 | } 336 | ], 337 | "metadata": { 338 | "kernelspec": { 339 | "display_name": "Pytorch for Deeplearning", 340 | "language": "python", 341 | "name": "pytorch" 342 | }, 343 | "language_info": { 344 | "codemirror_mode": { 345 | "name": "ipython", 346 | "version": 3 347 | }, 348 | "file_extension": ".py", 349 | "mimetype": "text/x-python", 350 | "name": "python", 351 | "nbconvert_exporter": "python", 352 | "pygments_lexer": "ipython3", 353 | "version": "3.6.7" 354 | } 355 | }, 356 | "nbformat": 4, 357 | "nbformat_minor": 2 358 | } 359 | -------------------------------------------------------------------------------- /chapter2/2.5-rnn.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "data": { 10 | "text/plain": [ 11 | "'1.0.0'" 12 | ] 13 | }, 14 | "execution_count": 1, 15 | "metadata": {}, 16 | "output_type": "execute_result" 17 | } 18 | ], 19 | "source": [ 20 | "import torch\n", 21 | "torch.__version__" 22 | ] 23 | }, 24 | { 25 | "cell_type": "markdown", 26 | "metadata": {}, 27 | "source": [ 28 | "# 2.5 循环神经网络\n", 29 | "## 2.5.1 RNN简介\n", 30 | "我们的大脑区别于机器的一个最大的特征就是我们有记忆，并且能够根据自己的记忆对未知的事务进行推导，我们的思想拥有持久性的。但是本教程目前所介绍的神经网络结构各个元素之间是相互独立的，输入与输出是独立的。\n", 31 | "### RNN的起因\n", 32 | "现实世界中，很多元素都是相互连接的，比如室外的温度是随着气候的变化而周期性的变化的、我们的语言也需要通过上下文的关系来确认所表达的含义。但是机器要做到这一步就相当得难了。因此，就有了现在的循环神经网络，他的本质是：拥有记忆的能力，并且会根据这些记忆的内容来进行推断。因此，他的输出就依赖于当前的输入和记忆。\n", 33 | "### 为什么需要RNN\n", 34 | "RNN背后的想法是利用顺序的信息。 在传统的神经网络中，我们假设所有输入（和输出）彼此独立。 如果你想预测句子中的下一个单词，你就要知道它前面有哪些单词，甚至要看到后面的单词才能够给出正确的答案。 \n", 35 | "RNN之所以称为循环，就是因为它们对序列的每个元素都会执行相同的任务，所有的输出都取决于先前的计算。\n", 36 | "从另一个角度讲RNN的它是有“记忆”的，可以捕获到目前为止计算的信息。 理论上，RNN可以在任意长的序列中使用信息，但实际上它们仅限于回顾几个步骤。\n", 37 | "循环神经网络的提出便是基于记忆模型的想法，期望网络能够记住前面出现的特征.并依据特征推断后面的结果，而且整体的网络结构不断循环，因为得名循环神经网络。\n", 38 | "\n", 39 | "\n", 40 | "### RNN都能做什么\n", 41 | "RNN在许多NLP任务中取得了巨大成功。 在这一点上，我应该提到最常用的RNN类型是LSTM，它在捕获长期依赖性方面要比RNN好得多。 但不要担心，LSTM与我们将在本教程中开发的RNN基本相同，它们只是采用不同的方式来计算隐藏状态。 我们将在后面更详细地介绍LSTM。 以下是RNN在NLP中的一些示例：\n", 42 | "**语言建模与生成文本**\n", 43 | "\n", 44 | "我们通过语言的建模，可以通过给定的单词生成人类可以理解的以假乱真的文本\n", 45 | "\n", 46 | "**机器翻译**\n", 47 | "\n", 48 | "机器翻译类似于语言建模，我们的输入源语言中的一系列单词，通过模型的计算可以输出目标语言与之对应的内容。 \n", 49 | "\n", 50 | "**语音识别**\n", 51 | "\n", 52 | "给定来自声波的声学信号的输入序列，我们可以预测一系列语音片段及其概率，并把语音转化成文字\n", 53 | "\n", 54 | "**生成图像描述**\n", 55 | "\n", 56 | "与卷积神经网络一起，RNN可以生成未标记图像的描述。" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "metadata": {}, 62 | "source": [ 63 | "## 2.5.2 RNN的网络结构及原理\n", 64 | "### RNN\n", 65 | "循环神经网络的基本结构特别简单，就是将网络的输出保存在一个记忆单元中，这个记忆单元和下一次的输入一起进入神经网络中。我们可以看到网络在输入的时候会联合记忆单元一起作为输入，网络不仅输出结果，还会将结果保存到记忆单元中，下图就是一个最简单的循环神经网络在输入时的结构示意图.[图片来源](https://medium.com/explore-artificial-intelligence/an-introduction-to-recurrent-neural-networks-72c97bf0912)\n", 66 | "\n", 67 | "![](10.png)\n", 68 | "RNN 可以被看做是同一神经网络的多次赋值，每个神经网络模块会把消息传递给下一个,我们将这个图的结构展开\n", 69 | "![](11.png)\n", 70 | "网络中具有循环结构，这也是循环神经网络名字的由来，同时根据循环神经网络的结构也可以看出它在处理序列类型的数据上具有天然的优势.因为网络本身就是 一个序列结构，这也是所有循环神经网络最本质的结构。\n", 71 | "\n", 72 | "循环神经网络具有特别好的记忆特性，能够将记忆内容应用到当前情景下，但是网络的记忆能力并没有想象的那么有效。记忆最大的问题在于它有遗忘性，我们总是更加清楚地记得最近发生的事情而遗忘很久之前发生的事情，循环神经网络同样有这样的问题。\n", 73 | "\n", 74 | "pytorch 中使用 nn.RNN 类来搭建基于序列的循环神经网络,它的构造函数有以下几个参数：\n", 75 | "- nput_size：输入数据X的特征值的数目。 \n", 76 | "- hidden_size：隐藏层的神经元数量，也就是隐藏层的特征数量。\n", 77 | "- num_layers：循环神经网络的层数，默认值是 1。 \n", 78 | "- bias：默认为 True，如果为 false 则表示神经元不使用 bias 偏移参数。\n", 79 | "- batch_first：如果设置为 True，则输入数据的维度中第一个维度就 是 batch 值，默认为 False。默认情况下第一个维度是序列的长度， 第二个维度才是 - - batch，第三个维度是特征数目。\n", 80 | "- dropout：如果不为空，则表示最后跟一个 dropout 层抛弃部分数据，抛弃数据的比例由该参数指定。\n", 81 | "\n", 82 | "RNN 中最主要的参数是 input_size 和 hidden_size，这两个参数务必要搞清楚。其余的参数通常不用设置，采用默认值就可以了。" 83 | ] 84 | }, 85 | { 86 | "cell_type": "code", 87 | "execution_count": 2, 88 | "metadata": {}, 89 | "outputs": [ 90 | { 91 | "name": "stdout", 92 | "output_type": "stream", 93 | "text": [ 94 | "torch.Size([100, 32, 50]) torch.Size([2, 32, 50])\n" 95 | ] 96 | } 97 | ], 98 | "source": [ 99 | "rnn = torch.nn.RNN(20,50,2)\n", 100 | "input = torch.randn(100 , 32 , 20)\n", 101 | "h_0 =torch.randn(2 , 32 , 50)\n", 102 | "output,hn=rnn(input ,h_0) \n", 103 | "print(output.size(),hn.size())" 104 | ] 105 | }, 106 | { 107 | "cell_type": "markdown", 108 | "metadata": {}, 109 | "source": [ 110 | "### LSTM\n", 111 | "LSTM 是 Long Short Term Memory Networks 的缩写，按字面翻译就是长的短时记忆网络。LSTM 的网络结构是 1997 年由 Hochreiter 和 Schmidhuber 提出的，随后这种网络结构变得非常流行，。\n", 112 | "LSTM虽然只解决了短期依赖的问题，并且它通过刻意的设计来避免长期依赖问题，这样的做法在实际应用中被证明还是十分有效的，有很多人跟进相关的工作解决了很多实际的问题，所以现在LSTM 仍然被广泛地使用。[图片来源](https://towardsdatascience.com/animated-rnn-lstm-and-gru-ef124d06cf45)\n", 113 | "![](lstm.gif)\n", 114 | "标准的循环神经网络内部只有一个简单的层结构，而 LSTM 内部有 4 个层结构：\n", 115 | "\n", 116 | "第一层是个忘记层：决定状态中丢弃什么信息\n", 117 | "\n", 118 | "第二层tanh层用来产生更新值的候选项，说明状态在某些维度上需要加强，在某些维度上需要减弱\n", 119 | "\n", 120 | "第三层sigmoid层（输入门层），它的输出值要乘到tanh层的输出上，起到一个缩放的作用，极端情况下sigmoid输出0说明相应维度上的状态不需要更新\n", 121 | "\n", 122 | "最后一层 决定输出什么,输出值跟状态有关。候选项中的哪些部分最终会被输出由一个sigmoid层来决定。\n", 123 | "\n", 124 | "\n", 125 | "pytorch 中使用 nn.LSTM 类来搭建基于序列的循环神经网络,他的参数基本与RNN类似，这里就不列出了。" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": 3, 131 | "metadata": {}, 132 | "outputs": [ 133 | { 134 | "name": "stdout", 135 | "output_type": "stream", 136 | "text": [ 137 | "torch.Size([5, 3, 20]) torch.Size([2, 3, 20]) torch.Size([2, 3, 20])\n" 138 | ] 139 | } 140 | ], 141 | "source": [ 142 | "lstm = torch.nn.LSTM(10, 20,2)\n", 143 | "input = torch.randn(5, 3, 10)\n", 144 | "h0 =torch.randn(2, 3, 20)\n", 145 | "c0 = torch.randn(2, 3, 20)\n", 146 | "output, hn = lstm(input, (h0, c0))\n", 147 | "print(output.size(),hn[0].size(),hn[1].size())" 148 | ] 149 | }, 150 | { 151 | "cell_type": "markdown", 152 | "metadata": {}, 153 | "source": [ 154 | "### GRU\n", 155 | "\n", 156 | "GRU 是 gated recurrent units 的缩写，由 Cho在 2014 年提出GRU 和 LSTM 最 的不同在于 GRU 将遗忘门和输入门合成了一个\"更新门\"，同时网络不再额外给出记忆状态，而是将输出结果作为记忆状态不断向后循环传递，网络的输人和输出都变得特别简单。\n", 157 | "\n", 158 | "![](gru.gif)" 159 | ] 160 | }, 161 | { 162 | "cell_type": "code", 163 | "execution_count": 4, 164 | "metadata": {}, 165 | "outputs": [ 166 | { 167 | "name": "stdout", 168 | "output_type": "stream", 169 | "text": [ 170 | "torch.Size([5, 3, 20]) torch.Size([2, 3, 20])\n" 171 | ] 172 | } 173 | ], 174 | "source": [ 175 | "rnn = torch.nn.GRU(10, 20, 2)\n", 176 | "input = torch.randn(5, 3, 10)\n", 177 | "h_0= torch.randn(2, 3, 20)\n", 178 | "output, hn = rnn(input, h0)\n", 179 | "print(output.size(),h0.size())" 180 | ] 181 | }, 182 | { 183 | "cell_type": "markdown", 184 | "metadata": {}, 185 | "source": [ 186 | "## 2.5.3 循环网络的向后传播（BPTT）\n", 187 | "在向前传播的情况下，RNN的输入随着每一个时间步前进。在反向传播的情况下，我们“回到过去”改变权重，因此我们叫它通过时间的反向传播（BPTT）。\n", 188 | "\n", 189 | "我们通常把整个序列（单词）看作一个训练样本，所以总的误差是每个时间步（字符）中误差的和。权重在每一个时间步长是相同的（所以可以计算总误差后一起更新）。\n", 190 | "1. 使用预测输出和实际输出计算交叉熵误差\n", 191 | "2. 网络按照时间步完全展开\n", 192 | "3. 对于展开的网络，对于每一个实践步计算权重的梯度\n", 193 | "4. 因为对于所有时间步来说，权重都一样，所以对于所有的时间步，可以一起得到梯度（而不是像神经网络一样对不同的隐藏层得到不同的梯度）\n", 194 | "5. 随后对循环神经元的权重进行升级\n", 195 | "\n", 196 | "RNN展开的网络看起来像一个普通的神经网络。反向传播也类似于普通的神经网络，只不过我们一次得到所有时间步的梯度。如果有100个时间步，那么网络展开后将变得非常巨大，所以为了解决这个问题才会出现LSTM和GRU这样的结构。\n" 197 | ] 198 | }, 199 | { 200 | "cell_type": "markdown", 201 | "metadata": {}, 202 | "source": [ 203 | "循环神经网络目前在自然语言处理中应用最为火热，所以后面的内容将介绍一下循环神经网络在处理NLP的时候需要用到的一些其他的知识\n", 204 | "\n", 205 | "## 2.5.4 词嵌入（word embedding）\n", 206 | "在我们人类交流过程中表征词汇是直接使用英文单词来进行表征的，但是对于计算机来说，是无法直接认识单词的。为了让计算机能够能更好地理解我们的语言，建立更好的语言模型，我们需要将词汇进行表征。\n", 207 | "\n", 208 | "在图像分类问题会使用 one-hot 编码.比如LeNet中一共有10个数字0-9，如果这个数字是2的话类，它的 编码就是 (0，0，1，0， 0，0 ，0，0，0，0)，对于分类问题这样表示十分的清楚，但是在自然语言处理中，因为单词的数目过多比如有 10000 个不同的词，那么使用 one-hot 这样的方式来定义，效率就特别低，每个单词都是 10000 维的向量.其中只有一位是 1 , 其余都是 0，特别占用内存,而且也不能体现单词的词性，因为每一个单词都是 one-hot，虽然有些单词在语义上会更加接近.但是 one-hot 没办法体现这个特点，所以 必须使用另外一种方式定义每一个单词。\n", 209 | "\n", 210 | "用不同的特征来对各个词汇进行表征，相对与不同的特征，不同的单词均有不同的值这就是词嵌入。下图还是来自吴恩达老师的课程截图\n", 211 | "![](12.png)\n", 212 | "\n", 213 | "词嵌入不仅对不同单词实现了特征化的表示，还能通过计算词与词之间的相似度，实际上是在多维空间中，寻找词向量之间各个维度的距离相似度，我们就可以实现类比推理，比如说夏天和热，冬天和冷，都是有关联关系的。\n", 214 | "\n", 215 | "在 PyTorch 中我们用 nn.Embedding 层来做嵌入词袋模型，Embedding层第一个输入表示我们有多少个词，第二个输入表示每一个词使用多少个向量表示。" 216 | ] 217 | }, 218 | { 219 | "cell_type": "code", 220 | "execution_count": 5, 221 | "metadata": {}, 222 | "outputs": [ 223 | { 224 | "name": "stdout", 225 | "output_type": "stream", 226 | "text": [ 227 | "torch.Size([2, 4, 3])\n" 228 | ] 229 | } 230 | ], 231 | "source": [ 232 | "# an Embedding module containing 10 tensors of size 3\n", 233 | "embedding = torch.nn.Embedding(10, 3)\n", 234 | "# a batch of 2 samples of 4 indices each\n", 235 | "input = torch.LongTensor([[1,2,4,5],[4,3,2,9]])\n", 236 | "output=embedding(input)\n", 237 | "print(output.size())" 238 | ] 239 | }, 240 | { 241 | "cell_type": "markdown", 242 | "metadata": {}, 243 | "source": [ 244 | "## 2.5.5 其他重要概念\n", 245 | "\n", 246 | "### Beam search\n", 247 | "在生成第一个词的分布后，可以使用贪心搜索会根据我们的条件语言模型挑选出最有可能输出的第一个词语，但是对于贪心搜索算法来说，我们的单词库中有成百到千万的词汇，去计算每一种单词的组合的可能性是不可行的。所以我们使用近似的搜索办法，使得条件概率最大化或者近似最大化的句子，而不是通过单词去实现。\n", 248 | "\n", 249 | "Beam Search（集束搜索）是一种启发式图搜索算法，通常用在图的解空间比较大的情况下，为了减少搜索所占用的空间和时间，在每一步深度扩展的时候，剪掉一些质量比较差的结点，保留下一些质量较高的结点。虽然Beam Search算法是不完全的，但是用于了解空间较大的系统中，可以减少空间占用和时间。\n", 250 | "\n", 251 | "beam search可以看做是做了约束优化的广度优先搜索，首先使用广度优先策略建立搜索树，树的每层，按照启发代价对节点进行排序，然后仅留下预先确定的个数（Beam width-集束宽度）的节点，仅这些节点在下一层次继续扩展，其他节点被剪切掉。\n", 252 | "1. 将初始节点插入到list中\n", 253 | "2. 将给节点出堆，如果该节点是目标节点，则算法结束；\n", 254 | "3. 否则扩展该节点，取集束宽度的节点入堆。然后到第二步继续循环。\n", 255 | "4. 算法结束的条件是找到最优解或者堆为空。\n", 256 | "\n", 257 | "在使用上，集束宽度可以是预先约定的，也可以是变化的，具体可以根据实际场景调整设定。\n" 258 | ] 259 | }, 260 | { 261 | "cell_type": "markdown", 262 | "metadata": {}, 263 | "source": [ 264 | "### 注意力模型\n", 265 | "对于使用编码和解码的RNN模型，我们能够实现较为准确度机器翻译结果。对于短句子来说，其性能是十分良好的，但是如果是很长的句子，翻译的结果就会变差。\n", 266 | "我们人类进行人工翻译的时候，都是一部分一部分地进行翻译，引入的注意力机制，和人类的翻译过程非常相似，其也是一部分一部分地进行长句子的翻译。\n", 267 | "\n", 268 | "具体的内容在这里就不详细介绍了\n" 269 | ] 270 | }, 271 | { 272 | "cell_type": "code", 273 | "execution_count": null, 274 | "metadata": {}, 275 | "outputs": [], 276 | "source": [] 277 | } 278 | ], 279 | "metadata": { 280 | "kernelspec": { 281 | "display_name": "pytorch 1.0", 282 | "language": "python", 283 | "name": "pytorch1" 284 | }, 285 | "language_info": { 286 | "codemirror_mode": { 287 | "name": "ipython", 288 | "version": 3 289 | }, 290 | "file_extension": ".py", 291 | "mimetype": "text/x-python", 292 | "name": "python", 293 | "nbconvert_exporter": "python", 294 | "pygments_lexer": "ipython3", 295 | "version": "3.6.7" 296 | } 297 | }, 298 | "nbformat": 4, 299 | "nbformat_minor": 2 300 | } 301 | -------------------------------------------------------------------------------- /chapter2/2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/2.png -------------------------------------------------------------------------------- /chapter2/3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/3.png -------------------------------------------------------------------------------- /chapter2/4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/4.png -------------------------------------------------------------------------------- /chapter2/5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/5.png -------------------------------------------------------------------------------- /chapter2/6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/6.png -------------------------------------------------------------------------------- /chapter2/7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/7.png -------------------------------------------------------------------------------- /chapter2/8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/8.png -------------------------------------------------------------------------------- /chapter2/9.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/9.gif -------------------------------------------------------------------------------- /chapter2/alexnet.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/alexnet.png -------------------------------------------------------------------------------- /chapter2/cnn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/cnn.png -------------------------------------------------------------------------------- /chapter2/googlenet.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/googlenet.png -------------------------------------------------------------------------------- /chapter2/gru.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/gru.gif -------------------------------------------------------------------------------- /chapter2/lenet5.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/lenet5.jpg -------------------------------------------------------------------------------- /chapter2/lstm.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/lstm.gif -------------------------------------------------------------------------------- /chapter2/readme.md: -------------------------------------------------------------------------------- 1 | # Pytorch 中文手册第二章 ： 基础 2 | 3 | ## 目录 4 | 5 | ### 第一节 PyTorch 基础 6 | 1. [张量](2.1.1.pytorch-basics-tensor.ipynb) 7 | 2. [自动求导](2.1.2-pytorch-basics-autograd.ipynb) 8 | 3. [神经网络包nn和优化器optm](2.1.3-pytorch-basics-nerual-network.ipynb) 9 | 4. [数据的加载和预处理](2.1.4-pytorch-basics-data-lorder.ipynb) 10 | 11 | ### 第二节 深度学习基础及数学原理 12 | [深度学习基础及数学原理](2.2-deep-learning-basic-mathematics.ipynb) 13 | 14 | ### 第三节 神经网络简介 15 | [神经网络简介](2.3-deep-learning-neural-network-introduction.ipynb) 16 | 17 | ### 第四节 卷积神经网络 18 | [卷积神经网络](2.4-cnn.ipynb) 19 | 20 | ### 第五节 循环神经网络 21 | [循环神经网络](2.5-rnn.ipynb) -------------------------------------------------------------------------------- /chapter2/resnet.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/resnet.png -------------------------------------------------------------------------------- /chapter2/resnet18.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/resnet18.jpg -------------------------------------------------------------------------------- /chapter2/vgg16.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter2/vgg16.png -------------------------------------------------------------------------------- /chapter3/3.1-logistic-regression.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "data": { 10 | "text/plain": [ 11 | "'1.0.0'" 12 | ] 13 | }, 14 | "execution_count": 1, 15 | "metadata": {}, 16 | "output_type": "execute_result" 17 | } 18 | ], 19 | "source": [ 20 | "import torch\n", 21 | "import torch.nn as nn\n", 22 | "import numpy as np\n", 23 | "torch.__version__" 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "# 3.1 logistic回归实战\n", 31 | "在这一章里面，我们将处理一下结构化数据，并使用logistic回归对结构化数据进行简单的分类。\n", 32 | "## 3.1.1 logistic回归介绍\n", 33 | "logistic回归是一种广义线性回归（generalized linear model），与多重线性回归分析有很多相同之处。它们的模型形式基本上相同，都具有 wx + b，其中w和b是待求参数，其区别在于他们的因变量不同，多重线性回归直接将wx+b作为因变量，即y =wx+b,而logistic回归则通过函数L将wx+b对应一个隐状态p，p =L(wx+b),然后根据p 与1-p的大小决定因变量的值。如果L是logistic函数，就是logistic回归，如果L是多项式函数就是多项式回归。\n", 34 | "\n", 35 | "说的更通俗一点，就是logistic回归会在线性回归后再加一层logistic函数的调用。\n", 36 | "\n", 37 | "logistic回归主要是进行二分类预测，我们在激活函数时候讲到过 Sigmod函数，Sigmod函数是最常见的logistic函数，因为Sigmod函数的输出的是是对于0~1之间的概率值，当概率大于0.5预测为1，小于0.5预测为0。\n", 38 | "\n", 39 | "下面我们就来使用公开的数据来进行介绍" 40 | ] 41 | }, 42 | { 43 | "cell_type": "markdown", 44 | "metadata": {}, 45 | "source": [ 46 | "## 3.1.2 UCI German Credit 数据集\n", 47 | "\n", 48 | "UCI German Credit是UCI的德国信用数据集，里面有原数据和数值化后的数据。\n", 49 | "\n", 50 | "German Credit数据是根据个人的银行贷款信息和申请客户贷款逾期发生情况来预测贷款违约倾向的数据集，数据集包含24个维度的，1000条数据，\n", 51 | "\n", 52 | "在这里我们直接使用处理好的数值化的数据，作为展示。\n", 53 | "\n", 54 | "[地址](https://archive.ics.uci.edu/ml/machine-learning-databases/statlog/german/)" 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "## 3.2 代码实战\n", 62 | "我们这里使用的 german.data-numeric是numpy处理好数值化数据，我们直接使用numpy的load方法读取即可" 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": 2, 68 | "metadata": {}, 69 | "outputs": [], 70 | "source": [ 71 | "data=np.loadtxt(\"german.data-numeric\")" 72 | ] 73 | }, 74 | { 75 | "cell_type": "markdown", 76 | "metadata": {}, 77 | "source": [ 78 | "数据读取完成后我们要对数据做一下归一化的处理" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": 3, 84 | "metadata": {}, 85 | "outputs": [], 86 | "source": [ 87 | "n,l=data.shape\n", 88 | "for j in range(l-1):\n", 89 | " meanVal=np.mean(data[:,j])\n", 90 | " stdVal=np.std(data[:,j])\n", 91 | " data[:,j]=(data[:,j]-meanVal)/stdVal" 92 | ] 93 | }, 94 | { 95 | "cell_type": "markdown", 96 | "metadata": {}, 97 | "source": [ 98 | "打乱数据" 99 | ] 100 | }, 101 | { 102 | "cell_type": "code", 103 | "execution_count": 4, 104 | "metadata": {}, 105 | "outputs": [], 106 | "source": [ 107 | "np.random.shuffle(data)" 108 | ] 109 | }, 110 | { 111 | "cell_type": "markdown", 112 | "metadata": {}, 113 | "source": [ 114 | "区分训练集和测试集，由于这里没有验证集，所以我们直接使用测试集的准确度作为评判好坏的标准\n", 115 | "\n", 116 | "区分规则：900条用于训练，100条作为测试\n", 117 | "\n", 118 | "german.data-numeric的格式为，前24列为24个维度，最后一个为要打的标签（0，1），所以我们将数据和标签一起区分出来" 119 | ] 120 | }, 121 | { 122 | "cell_type": "code", 123 | "execution_count": 5, 124 | "metadata": {}, 125 | "outputs": [], 126 | "source": [ 127 | "train_data=data[:900,:l-1]\n", 128 | "train_lab=data[:900,l-1]-1\n", 129 | "test_data=data[900:,:l-1]\n", 130 | "test_lab=data[900:,l-1]-1" 131 | ] 132 | }, 133 | { 134 | "cell_type": "markdown", 135 | "metadata": {}, 136 | "source": [ 137 | "下面我们定义模型，模型很简单" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 6, 143 | "metadata": {}, 144 | "outputs": [], 145 | "source": [ 146 | "class LR(nn.Module):\n", 147 | " def __init__(self):\n", 148 | " super(LR,self).__init__()\n", 149 | " self.fc=nn.Linear(24,2) # 由于24个维度已经固定了，所以这里写24\n", 150 | " def forward(self,x):\n", 151 | " out=self.fc(x)\n", 152 | " out=torch.sigmoid(out)\n", 153 | " return out\n", 154 | " " 155 | ] 156 | }, 157 | { 158 | "cell_type": "markdown", 159 | "metadata": {}, 160 | "source": [ 161 | "测试集上的准确率" 162 | ] 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": 7, 167 | "metadata": {}, 168 | "outputs": [], 169 | "source": [ 170 | "def test(pred,lab):\n", 171 | " t=pred.max(-1)[1]==lab\n", 172 | " return torch.mean(t.float())" 173 | ] 174 | }, 175 | { 176 | "cell_type": "markdown", 177 | "metadata": {}, 178 | "source": [ 179 | "下面就是对一些设置" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": 8, 185 | "metadata": {}, 186 | "outputs": [], 187 | "source": [ 188 | "net=LR() \n", 189 | "criterion=nn.CrossEntropyLoss() # 使用CrossEntropyLoss损失\n", 190 | "optm=torch.optim.Adam(net.parameters()) # Adam优化\n", 191 | "epochs=1000 # 训练10000次\n" 192 | ] 193 | }, 194 | { 195 | "cell_type": "markdown", 196 | "metadata": {}, 197 | "source": [ 198 | "下面开始训练了" 199 | ] 200 | }, 201 | { 202 | "cell_type": "code", 203 | "execution_count": 9, 204 | "metadata": {}, 205 | "outputs": [ 206 | { 207 | "name": "stdout", 208 | "output_type": "stream", 209 | "text": [ 210 | "Epoch:100,Loss:0.6313,Accuracy：0.76\n", 211 | "Epoch:200,Loss:0.6065,Accuracy：0.79\n", 212 | "Epoch:300,Loss:0.5909,Accuracy：0.80\n", 213 | "Epoch:400,Loss:0.5801,Accuracy：0.81\n", 214 | "Epoch:500,Loss:0.5720,Accuracy：0.82\n", 215 | "Epoch:600,Loss:0.5657,Accuracy：0.81\n", 216 | "Epoch:700,Loss:0.5606,Accuracy：0.81\n", 217 | "Epoch:800,Loss:0.5563,Accuracy：0.81\n", 218 | "Epoch:900,Loss:0.5527,Accuracy：0.81\n", 219 | "Epoch:1000,Loss:0.5496,Accuracy：0.80\n" 220 | ] 221 | } 222 | ], 223 | "source": [ 224 | "for i in range(epochs):\n", 225 | " # 指定模型为训练模式，计算梯度\n", 226 | " net.train()\n", 227 | " # 输入值都需要转化成torch的Tensor\n", 228 | " x=torch.from_numpy(train_data).float()\n", 229 | " y=torch.from_numpy(train_lab).long()\n", 230 | " y_hat=net(x)\n", 231 | " loss=criterion(y_hat,y) # 计算损失\n", 232 | " optm.zero_grad() # 前一步的损失清零\n", 233 | " loss.backward() # 反响传播\n", 234 | " optm.step() # 优化\n", 235 | " if (i+1)%100 ==0 : # 这里我们每100次输出相关的信息\n", 236 | " # 指定模型为计算模式\n", 237 | " net.eval()\n", 238 | " test_in=torch.from_numpy(test_data).float()\n", 239 | " test_l=torch.from_numpy(test_lab).long()\n", 240 | " test_out=net(test_in)\n", 241 | " # 使用我们的测试函数计算准确率\n", 242 | " accu=test(test_out,test_l)\n", 243 | " print(\"Epoch:{},Loss:{:.4f},Accuracy：{:.2f}\".format(i+1,loss.item(),accu))" 244 | ] 245 | }, 246 | { 247 | "cell_type": "markdown", 248 | "metadata": {}, 249 | "source": [ 250 | "训练完成了，我们的准确度达到了80%" 251 | ] 252 | }, 253 | { 254 | "cell_type": "code", 255 | "execution_count": null, 256 | "metadata": {}, 257 | "outputs": [], 258 | "source": [] 259 | } 260 | ], 261 | "metadata": { 262 | "kernelspec": { 263 | "display_name": "pytorch 1.0", 264 | "language": "python", 265 | "name": "pytorch1" 266 | }, 267 | "language_info": { 268 | "codemirror_mode": { 269 | "name": "ipython", 270 | "version": 3 271 | }, 272 | "file_extension": ".py", 273 | "mimetype": "text/x-python", 274 | "name": "python", 275 | "nbconvert_exporter": "python", 276 | "pygments_lexer": "ipython3", 277 | "version": "3.6.7" 278 | } 279 | }, 280 | "nbformat": 4, 281 | "nbformat_minor": 2 282 | } 283 | -------------------------------------------------------------------------------- /chapter3/3.2-mnist.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "data": { 10 | "text/plain": [ 11 | "'1.0.0'" 12 | ] 13 | }, 14 | "execution_count": 1, 15 | "metadata": {}, 16 | "output_type": "execute_result" 17 | } 18 | ], 19 | "source": [ 20 | "import torch\n", 21 | "import torch.nn as nn\n", 22 | "import torch.nn.functional as F\n", 23 | "import torch.optim as optim\n", 24 | "from torchvision import datasets, transforms\n", 25 | "torch.__version__" 26 | ] 27 | }, 28 | { 29 | "cell_type": "markdown", 30 | "metadata": {}, 31 | "source": [ 32 | "# 3.2 MNIST数据集手写数字识别\n", 33 | "\n", 34 | "## 3.2.1 数据集介绍\n", 35 | "MNIST 包括6万张28x28的训练样本，1万张测试样本，很多教程都会对它”下手”几乎成为一个 “典范”，可以说他就是计算机视觉里面的Hello World。所以我们这里也会使用MNIST来进行实战。\n", 36 | "\n", 37 | "我们在介绍卷积神经网络的时候说到过LeNet-5，LeNet-5之所以强大就是因为在当时的环境下将MNIST数据的识别率提高到了99%，这里我们也自己从头搭建一个卷积神经网络，也达到99%的准确率" 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "## 3.2.2 手写数字识别\n", 45 | "首先，我们定义一些超参数" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 2, 51 | "metadata": {}, 52 | "outputs": [], 53 | "source": [ 54 | "BATCH_SIZE=512 #大概需要2G的显存\n", 55 | "EPOCHS=20 # 总共训练批次\n", 56 | "DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\") # 让torch判断是否使用GPU，建议使用GPU环境，因为会快很多" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "metadata": {}, 62 | "source": [ 63 | "因为Pytorch里面包含了MNIST的数据集，所以我们这里直接使用即可。\n", 64 | "如果第一次执行会生成data文件夹，并且需要一些时间下载，如果以前下载过就不会再次下载了\n", 65 | "\n", 66 | "由于官方已经实现了dataset，所以这里可以直接使用DataLoader来对数据进行读取" 67 | ] 68 | }, 69 | { 70 | "cell_type": "code", 71 | "execution_count": 3, 72 | "metadata": {}, 73 | "outputs": [ 74 | { 75 | "name": "stdout", 76 | "output_type": "stream", 77 | "text": [ 78 | "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\n", 79 | "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz\n", 80 | "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz\n", 81 | "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz\n", 82 | "Processing...\n", 83 | "Done!\n" 84 | ] 85 | } 86 | ], 87 | "source": [ 88 | "train_loader = torch.utils.data.DataLoader(\n", 89 | " datasets.MNIST('data', train=True, download=True, \n", 90 | " transform=transforms.Compose([\n", 91 | " transforms.ToTensor(),\n", 92 | " transforms.Normalize((0.1307,), (0.3081,))\n", 93 | " ])),\n", 94 | " batch_size=BATCH_SIZE, shuffle=True)" 95 | ] 96 | }, 97 | { 98 | "cell_type": "markdown", 99 | "metadata": {}, 100 | "source": [ 101 | "测试集" 102 | ] 103 | }, 104 | { 105 | "cell_type": "code", 106 | "execution_count": 4, 107 | "metadata": {}, 108 | "outputs": [], 109 | "source": [ 110 | "test_loader = torch.utils.data.DataLoader(\n", 111 | " datasets.MNIST('data', train=False, transform=transforms.Compose([\n", 112 | " transforms.ToTensor(),\n", 113 | " transforms.Normalize((0.1307,), (0.3081,))\n", 114 | " ])),\n", 115 | " batch_size=BATCH_SIZE, shuffle=True)" 116 | ] 117 | }, 118 | { 119 | "cell_type": "markdown", 120 | "metadata": {}, 121 | "source": [ 122 | "下面我们定义一个网络，网络包含两个卷积层，conv1和conv2，然后紧接着两个线性层作为输出，最后输出10个维度，这10个维度我们作为0-9的标识来确定识别出的是那个数字\n", 123 | "\n", 124 | "在这里建议大家将每一层的输入和输出都显作为注释标注出来，这样后面阅读代码的会方便很多" 125 | ] 126 | }, 127 | { 128 | "cell_type": "code", 129 | "execution_count": 5, 130 | "metadata": {}, 131 | "outputs": [], 132 | "source": [ 133 | "class ConvNet(nn.Module):\n", 134 | " def __init__(self):\n", 135 | " super().__init__()\n", 136 | " # 1,28x28\n", 137 | " self.conv1=nn.Conv2d(1,10,5) # 10, 24x24\n", 138 | " self.conv2=nn.Conv2d(10,20,3) # 128, 10x10\n", 139 | " self.fc1 = nn.Linear(20*10*10,500)\n", 140 | " self.fc2 = nn.Linear(500,10)\n", 141 | " def forward(self,x):\n", 142 | " in_size = x.size(0)\n", 143 | " out = self.conv1(x) #24\n", 144 | " out = F.relu(out)\n", 145 | " out = F.max_pool2d(out, 2, 2) #12\n", 146 | " out = self.conv2(out) #10\n", 147 | " out = F.relu(out)\n", 148 | " out = out.view(in_size,-1)\n", 149 | " out = self.fc1(out)\n", 150 | " out = F.relu(out)\n", 151 | " out = self.fc2(out)\n", 152 | " out = F.log_softmax(out,dim=1)\n", 153 | " return out" 154 | ] 155 | }, 156 | { 157 | "cell_type": "markdown", 158 | "metadata": {}, 159 | "source": [ 160 | "我们实例化一个网络，实例化后使用.to方法将网络移动到GPU\n", 161 | "\n", 162 | "优化器我们也直接选择简单暴力的Adam" 163 | ] 164 | }, 165 | { 166 | "cell_type": "code", 167 | "execution_count": 6, 168 | "metadata": {}, 169 | "outputs": [], 170 | "source": [ 171 | "model = ConvNet().to(DEVICE)\n", 172 | "optimizer = optim.Adam(model.parameters())" 173 | ] 174 | }, 175 | { 176 | "cell_type": "markdown", 177 | "metadata": {}, 178 | "source": [ 179 | "下面定义一下训练的函数，我们将训练的所有操作都封装到这个函数中" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": 7, 185 | "metadata": {}, 186 | "outputs": [], 187 | "source": [ 188 | "def train(model, device, train_loader, optimizer, epoch):\n", 189 | " model.train()\n", 190 | " for batch_idx, (data, target) in enumerate(train_loader):\n", 191 | " data, target = data.to(device), target.to(device)\n", 192 | " optimizer.zero_grad()\n", 193 | " output = model(data)\n", 194 | " loss = F.nll_loss(output, target)\n", 195 | " loss.backward()\n", 196 | " optimizer.step()\n", 197 | " if(batch_idx+1)%30 == 0: \n", 198 | " print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n", 199 | " epoch, batch_idx * len(data), len(train_loader.dataset),\n", 200 | " 100. * batch_idx / len(train_loader), loss.item()))" 201 | ] 202 | }, 203 | { 204 | "cell_type": "markdown", 205 | "metadata": {}, 206 | "source": [ 207 | "测试的操作也一样封装成一个函数" 208 | ] 209 | }, 210 | { 211 | "cell_type": "code", 212 | "execution_count": 8, 213 | "metadata": {}, 214 | "outputs": [], 215 | "source": [ 216 | "def test(model, device, test_loader):\n", 217 | " model.eval()\n", 218 | " test_loss = 0\n", 219 | " correct = 0\n", 220 | " with torch.no_grad():\n", 221 | " for data, target in test_loader:\n", 222 | " data, target = data.to(device), target.to(device)\n", 223 | " output = model(data)\n", 224 | " test_loss += F.nll_loss(output, target, reduction='sum').item() # 将一批的损失相加\n", 225 | " pred = output.max(1, keepdim=True)[1] # 找到概率最大的下标\n", 226 | " correct += pred.eq(target.view_as(pred)).sum().item()\n", 227 | "\n", 228 | " test_loss /= len(test_loader.dataset)\n", 229 | " print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n", 230 | " test_loss, correct, len(test_loader.dataset),\n", 231 | " 100. * correct / len(test_loader.dataset)))" 232 | ] 233 | }, 234 | { 235 | "cell_type": "markdown", 236 | "metadata": {}, 237 | "source": [ 238 | "下面开始训练，封装起来的好处这里就体现出来了，只要谢两行就可以了" 239 | ] 240 | }, 241 | { 242 | "cell_type": "code", 243 | "execution_count": 9, 244 | "metadata": {}, 245 | "outputs": [ 246 | { 247 | "name": "stdout", 248 | "output_type": "stream", 249 | "text": [ 250 | "Train Epoch: 1 [14848/60000 (25%)]\tLoss: 0.272529\n", 251 | "Train Epoch: 1 [30208/60000 (50%)]\tLoss: 0.235455\n", 252 | "Train Epoch: 1 [45568/60000 (75%)]\tLoss: 0.101858\n", 253 | "\n", 254 | "Test set: Average loss: 0.1018, Accuracy: 9695/10000 (97%)\n", 255 | "\n", 256 | "Train Epoch: 2 [14848/60000 (25%)]\tLoss: 0.057989\n", 257 | "Train Epoch: 2 [30208/60000 (50%)]\tLoss: 0.083935\n", 258 | "Train Epoch: 2 [45568/60000 (75%)]\tLoss: 0.051921\n", 259 | "\n", 260 | "Test set: Average loss: 0.0523, Accuracy: 9825/10000 (98%)\n", 261 | "\n", 262 | "Train Epoch: 3 [14848/60000 (25%)]\tLoss: 0.045383\n", 263 | "Train Epoch: 3 [30208/60000 (50%)]\tLoss: 0.049402\n", 264 | "Train Epoch: 3 [45568/60000 (75%)]\tLoss: 0.061366\n", 265 | "\n", 266 | "Test set: Average loss: 0.0408, Accuracy: 9866/10000 (99%)\n", 267 | "\n", 268 | "Train Epoch: 4 [14848/60000 (25%)]\tLoss: 0.035253\n", 269 | "Train Epoch: 4 [30208/60000 (50%)]\tLoss: 0.038444\n", 270 | "Train Epoch: 4 [45568/60000 (75%)]\tLoss: 0.036877\n", 271 | "\n", 272 | "Test set: Average loss: 0.0433, Accuracy: 9859/10000 (99%)\n", 273 | "\n", 274 | "Train Epoch: 5 [14848/60000 (25%)]\tLoss: 0.038996\n", 275 | "Train Epoch: 5 [30208/60000 (50%)]\tLoss: 0.020670\n", 276 | "Train Epoch: 5 [45568/60000 (75%)]\tLoss: 0.034658\n", 277 | "\n", 278 | "Test set: Average loss: 0.0339, Accuracy: 9885/10000 (99%)\n", 279 | "\n", 280 | "Train Epoch: 6 [14848/60000 (25%)]\tLoss: 0.067320\n", 281 | "Train Epoch: 6 [30208/60000 (50%)]\tLoss: 0.016328\n", 282 | "Train Epoch: 6 [45568/60000 (75%)]\tLoss: 0.017037\n", 283 | "\n", 284 | "Test set: Average loss: 0.0348, Accuracy: 9881/10000 (99%)\n", 285 | "\n", 286 | "Train Epoch: 7 [14848/60000 (25%)]\tLoss: 0.022150\n", 287 | "Train Epoch: 7 [30208/60000 (50%)]\tLoss: 0.009608\n", 288 | "Train Epoch: 7 [45568/60000 (75%)]\tLoss: 0.012742\n", 289 | "\n", 290 | "Test set: Average loss: 0.0346, Accuracy: 9895/10000 (99%)\n", 291 | "\n", 292 | "Train Epoch: 8 [14848/60000 (25%)]\tLoss: 0.010173\n", 293 | "Train Epoch: 8 [30208/60000 (50%)]\tLoss: 0.019482\n", 294 | "Train Epoch: 8 [45568/60000 (75%)]\tLoss: 0.012159\n", 295 | "\n", 296 | "Test set: Average loss: 0.0323, Accuracy: 9886/10000 (99%)\n", 297 | "\n", 298 | "Train Epoch: 9 [14848/60000 (25%)]\tLoss: 0.007792\n", 299 | "Train Epoch: 9 [30208/60000 (50%)]\tLoss: 0.006970\n", 300 | "Train Epoch: 9 [45568/60000 (75%)]\tLoss: 0.004989\n", 301 | "\n", 302 | "Test set: Average loss: 0.0294, Accuracy: 9909/10000 (99%)\n", 303 | "\n", 304 | "Train Epoch: 10 [14848/60000 (25%)]\tLoss: 0.003764\n", 305 | "Train Epoch: 10 [30208/60000 (50%)]\tLoss: 0.005944\n", 306 | "Train Epoch: 10 [45568/60000 (75%)]\tLoss: 0.001866\n", 307 | "\n", 308 | "Test set: Average loss: 0.0361, Accuracy: 9902/10000 (99%)\n", 309 | "\n", 310 | "Train Epoch: 11 [14848/60000 (25%)]\tLoss: 0.002737\n", 311 | "Train Epoch: 11 [30208/60000 (50%)]\tLoss: 0.014134\n", 312 | "Train Epoch: 11 [45568/60000 (75%)]\tLoss: 0.001365\n", 313 | "\n", 314 | "Test set: Average loss: 0.0309, Accuracy: 9905/10000 (99%)\n", 315 | "\n", 316 | "Train Epoch: 12 [14848/60000 (25%)]\tLoss: 0.003344\n", 317 | "Train Epoch: 12 [30208/60000 (50%)]\tLoss: 0.003090\n", 318 | "Train Epoch: 12 [45568/60000 (75%)]\tLoss: 0.004847\n", 319 | "\n", 320 | "Test set: Average loss: 0.0318, Accuracy: 9902/10000 (99%)\n", 321 | "\n", 322 | "Train Epoch: 13 [14848/60000 (25%)]\tLoss: 0.001278\n", 323 | "Train Epoch: 13 [30208/60000 (50%)]\tLoss: 0.003016\n", 324 | "Train Epoch: 13 [45568/60000 (75%)]\tLoss: 0.001328\n", 325 | "\n", 326 | "Test set: Average loss: 0.0358, Accuracy: 9906/10000 (99%)\n", 327 | "\n", 328 | "Train Epoch: 14 [14848/60000 (25%)]\tLoss: 0.002219\n", 329 | "Train Epoch: 14 [30208/60000 (50%)]\tLoss: 0.003487\n", 330 | "Train Epoch: 14 [45568/60000 (75%)]\tLoss: 0.014429\n", 331 | "\n", 332 | "Test set: Average loss: 0.0376, Accuracy: 9896/10000 (99%)\n", 333 | "\n", 334 | "Train Epoch: 15 [14848/60000 (25%)]\tLoss: 0.003042\n", 335 | "Train Epoch: 15 [30208/60000 (50%)]\tLoss: 0.002974\n", 336 | "Train Epoch: 15 [45568/60000 (75%)]\tLoss: 0.000871\n", 337 | "\n", 338 | "Test set: Average loss: 0.0346, Accuracy: 9909/10000 (99%)\n", 339 | "\n", 340 | "Train Epoch: 16 [14848/60000 (25%)]\tLoss: 0.000618\n", 341 | "Train Epoch: 16 [30208/60000 (50%)]\tLoss: 0.003164\n", 342 | "Train Epoch: 16 [45568/60000 (75%)]\tLoss: 0.007245\n", 343 | "\n", 344 | "Test set: Average loss: 0.0357, Accuracy: 9905/10000 (99%)\n", 345 | "\n", 346 | "Train Epoch: 17 [14848/60000 (25%)]\tLoss: 0.001874\n", 347 | "Train Epoch: 17 [30208/60000 (50%)]\tLoss: 0.013951\n", 348 | "Train Epoch: 17 [45568/60000 (75%)]\tLoss: 0.000729\n", 349 | "\n", 350 | "Test set: Average loss: 0.0322, Accuracy: 9922/10000 (99%)\n", 351 | "\n", 352 | "Train Epoch: 18 [14848/60000 (25%)]\tLoss: 0.002581\n", 353 | "Train Epoch: 18 [30208/60000 (50%)]\tLoss: 0.001396\n", 354 | "Train Epoch: 18 [45568/60000 (75%)]\tLoss: 0.015521\n", 355 | "\n", 356 | "Test set: Average loss: 0.0389, Accuracy: 9914/10000 (99%)\n", 357 | "\n", 358 | "Train Epoch: 19 [14848/60000 (25%)]\tLoss: 0.000283\n", 359 | "Train Epoch: 19 [30208/60000 (50%)]\tLoss: 0.001385\n", 360 | "Train Epoch: 19 [45568/60000 (75%)]\tLoss: 0.011184\n", 361 | "\n", 362 | "Test set: Average loss: 0.0383, Accuracy: 9901/10000 (99%)\n", 363 | "\n", 364 | "Train Epoch: 20 [14848/60000 (25%)]\tLoss: 0.000472\n", 365 | "Train Epoch: 20 [30208/60000 (50%)]\tLoss: 0.003306\n", 366 | "Train Epoch: 20 [45568/60000 (75%)]\tLoss: 0.018017\n", 367 | "\n", 368 | "Test set: Average loss: 0.0393, Accuracy: 9899/10000 (99%)\n", 369 | "\n" 370 | ] 371 | } 372 | ], 373 | "source": [ 374 | "for epoch in range(1, EPOCHS + 1):\n", 375 | " train(model, DEVICE, train_loader, optimizer, epoch)\n", 376 | " test(model, DEVICE, test_loader)" 377 | ] 378 | }, 379 | { 380 | "cell_type": "markdown", 381 | "metadata": {}, 382 | "source": [ 383 | "我们看一下结果，准确率99%，没问题" 384 | ] 385 | }, 386 | { 387 | "cell_type": "markdown", 388 | "metadata": {}, 389 | "source": [ 390 | "如果你的模型连MNIST都搞不定，那么你的模型没有任何的价值\n", 391 | "\n", 392 | "如果你的模型搞定了MNIST，那么你的模型也可能没有任何的价值\n", 393 | "\n", 394 | "MNIST是一个很简单的数据集，但是因为他的局限性只能作为研究来使用，对于实际应用中带来的价值非常有限，但是通过这个例子，我们可以完全了解一个实际项目的工作流程\n", 395 | "\n", 396 | "我们找到数据集，对数据做预处理，定义我们的模型，调整超参数，测试训练，再通过训练结果对超参数进行调整或者对模型进行调整。\n", 397 | "\n", 398 | "并且通过这个实战我们已经有了一个很好的模板，以后的项目都可以以这个模板为样例" 399 | ] 400 | }, 401 | { 402 | "cell_type": "code", 403 | "execution_count": null, 404 | "metadata": {}, 405 | "outputs": [], 406 | "source": [] 407 | } 408 | ], 409 | "metadata": { 410 | "kernelspec": { 411 | "display_name": "pytorch 1.0", 412 | "language": "python", 413 | "name": "pytorch1" 414 | }, 415 | "language_info": { 416 | "codemirror_mode": { 417 | "name": "ipython", 418 | "version": 3 419 | }, 420 | "file_extension": ".py", 421 | "mimetype": "text/x-python", 422 | "name": "python", 423 | "nbconvert_exporter": "python", 424 | "pygments_lexer": "ipython3", 425 | "version": "3.6.7" 426 | } 427 | }, 428 | "nbformat": 4, 429 | "nbformat_minor": 2 430 | } 431 | -------------------------------------------------------------------------------- /chapter3/readme.md: -------------------------------------------------------------------------------- 1 | # Pytorch 中文手册第三章 ： 实践 2 | 3 | ## 目录 4 | 5 | ### 3.1 logistic回归 6 | [logistic回归二元分类](3.1-logistic-regression.ipynb) 7 | 8 | ### 3.2 CNN:MNIST数据集手写数字识别 9 | [CNN:MNIST数据集手写数字识别](3.2-mnist.ipynb) 10 | 11 | ### 3.3 CNN:MNIST数据集手写数字识别 12 | [RNN实例：通过Sin预测Cos](3.3-rnn.ipynb) 13 | -------------------------------------------------------------------------------- /chapter4/1.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter4/1.PNG -------------------------------------------------------------------------------- /chapter4/1280px-Felis_silvestris_catus_lying_on_rice_straw.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter4/1280px-Felis_silvestris_catus_lying_on_rice_straw.jpg -------------------------------------------------------------------------------- /chapter4/2.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter4/2.PNG -------------------------------------------------------------------------------- /chapter4/3.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter4/3.PNG -------------------------------------------------------------------------------- /chapter4/4.2.1-visdom.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "data": { 10 | "text/plain": [ 11 | "'1.0.0'" 12 | ] 13 | }, 14 | "execution_count": 1, 15 | "metadata": {}, 16 | "output_type": "execute_result" 17 | } 18 | ], 19 | "source": [ 20 | "import torch\n", 21 | "import math\n", 22 | "import numpy as np\n", 23 | "from visdom import Visdom\n", 24 | "import time\n", 25 | "torch.__version__" 26 | ] 27 | }, 28 | { 29 | "cell_type": "markdown", 30 | "metadata": {}, 31 | "source": [ 32 | "# 4.2.1 使用Visdom在 PyTorch 中进行可视化\n", 33 | "Visdom是Facebook在2017年发布的一款针对PyTorch的可视化工具。[官网](https://github.com/facebookresearch/visdom),visdom由于其功能简单，一般会被定义为服务器端的matplot，也就是说我们可以直接使用python的控制台模式进行开发并在服务器上执行，将一些可视化的数据传送到Visdom服务上，通过Visdom服务进行可视化" 34 | ] 35 | }, 36 | { 37 | "cell_type": "markdown", 38 | "metadata": {}, 39 | "source": [ 40 | "## 安装\n", 41 | "Visdom的安装很简单，直接使用命令`pip install visdom`安装即可。\n", 42 | "在安装完成后，使用命令`python -m visdom.server` 在本地启动服务器，启动后会提示`It's Alive! You can navigate to http://localhost:8097` 这就说明服务已经可用，我们打开浏览器，输入`http://localhost:8097` 即可看到页面。\n", 43 | "\n", 44 | "端口8097是默认的端口可以在启动命令后加 `-port`参数指定端口，常用的参数还有 `--hostname`，`-base_url`等" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": {}, 50 | "source": [ 51 | "## 坑\n", 52 | "Visdom的服务在启动时会自动下载一些静态文件，这里坑就来了，因为某些无法描述的原因，导致下载会失败，比如类似这样的提示 `ERROR:root:Error 404 while downloading https://unpkg.com/layout-bin-packer@1.4.0` 就说明静态文件没有下载完全，这样有可能就会打不开或者页面中没有菜单栏，那么需要手动进行下载，这里我打包了一份正常的静态文件，直接复制到`Lib\\site-packages\\visdom`中即可。\n", 53 | "\n", 54 | "如果不知道conda的环境目录在哪里，可以使用`conda env list` 查看\n", 55 | "\n", 56 | "感谢CSDN的伙伴提供的缺失文件，原文[这里](https://blog.csdn.net/qq_36941368/article/details/82288154)" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "metadata": {}, 62 | "source": [ 63 | "## 基本概念\n", 64 | "### Environments\n", 65 | "Environments的作用是对可视化区域进行分区，每个用户都会有一个叫做main的默认分区，如图所示:\n", 66 | "![](1.PNG)\n", 67 | "在程序指定的情况下，默认的图表都会放到这里面" 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "metadata": {}, 73 | "source": [ 74 | "## Panes\n", 75 | "Panes是作为每一个可视化图表的容器，可以使用生成的图表，图片，文本进行填充，我们可以对Panes进行拖放，删除，调整大小和销毁等操作：\n", 76 | "![](2.PNG)\n", 77 | "Panes和Environments是一对多的关系，即一个Environments可以包含多个Panes\n" 78 | ] 79 | }, 80 | { 81 | "cell_type": "markdown", 82 | "metadata": {}, 83 | "source": [ 84 | "### VIEW\n", 85 | "在对Panes进行调整后，可以通过VIEW对状态进行管理：\n", 86 | "![](3.PNG)" 87 | ] 88 | }, 89 | { 90 | "cell_type": "markdown", 91 | "metadata": {}, 92 | "source": [ 93 | "## 可视化接口\n", 94 | "Visdom是由Plotly 提供的可视化支持，所以提供一下可视化的接口:\n", 95 | "- vis.scatter : 2D 或 3D 散点图\n", 96 | "- vis.line : 线图\n", 97 | "- vis.stem : 茎叶图\n", 98 | "- vis.heatmap : 热力图\n", 99 | "- vis.bar : 条形图\n", 100 | "- vis.histogram: 直方图\n", 101 | "- vis.boxplot : 箱型图\n", 102 | "- vis.surf : 表面图\n", 103 | "- vis.contour : 轮廓图\n", 104 | "- vis.quiver : 绘出二维矢量场\n", 105 | "- vis.image : 图片\n", 106 | "- vis.text : 文本\n", 107 | "- vis.mesh : 网格图\n", 108 | "- vis.save : 序列化状态" 109 | ] 110 | }, 111 | { 112 | "cell_type": "markdown", 113 | "metadata": {}, 114 | "source": [ 115 | "## 使用\n", 116 | "### 绘制简单的图形\n", 117 | "这里我们使用官方的DEMO来做样例" 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": 2, 123 | "metadata": {}, 124 | "outputs": [], 125 | "source": [ 126 | "env = Visdom() \n", 127 | "assert env.check_connection() #测试一下链接，链接错误的话会报错" 128 | ] 129 | }, 130 | { 131 | "cell_type": "markdown", 132 | "metadata": {}, 133 | "source": [ 134 | "这里生成sin和cos两条曲线数据" 135 | ] 136 | }, 137 | { 138 | "cell_type": "code", 139 | "execution_count": 3, 140 | "metadata": {}, 141 | "outputs": [], 142 | "source": [ 143 | "Y = np.linspace(0, 2 * math.pi, 70)\n", 144 | "X = np.column_stack((np.sin(Y), np.cos(Y)))" 145 | ] 146 | }, 147 | { 148 | "cell_type": "markdown", 149 | "metadata": {}, 150 | "source": [ 151 | "使用茎叶图展示" 152 | ] 153 | }, 154 | { 155 | "cell_type": "code", 156 | "execution_count": 4, 157 | "metadata": { 158 | "scrolled": true 159 | }, 160 | "outputs": [ 161 | { 162 | "data": { 163 | "text/plain": [ 164 | "'window_36f18bc34b4992'" 165 | ] 166 | }, 167 | "execution_count": 4, 168 | "metadata": {}, 169 | "output_type": "execute_result" 170 | } 171 | ], 172 | "source": [ 173 | "env.stem(\n", 174 | " X=X,\n", 175 | " Y=Y,\n", 176 | " opts=dict(legend=['Sine', 'Cosine'])\n", 177 | " )" 178 | ] 179 | }, 180 | { 181 | "cell_type": "markdown", 182 | "metadata": {}, 183 | "source": [ 184 | "可以通过env参数指定Environments，如果名称包含了下划线`_`那么visdom会跟根据下划线分割并自动分组" 185 | ] 186 | }, 187 | { 188 | "cell_type": "code", 189 | "execution_count": 5, 190 | "metadata": { 191 | "scrolled": true 192 | }, 193 | "outputs": [], 194 | "source": [ 195 | "envtest = Visdom(env='test_mesh')\n", 196 | "assert envtest.check_connection()" 197 | ] 198 | }, 199 | { 200 | "cell_type": "markdown", 201 | "metadata": {}, 202 | "source": [ 203 | "生成一个网格图" 204 | ] 205 | }, 206 | { 207 | "cell_type": "code", 208 | "execution_count": 6, 209 | "metadata": {}, 210 | "outputs": [ 211 | { 212 | "data": { 213 | "text/plain": [ 214 | "'window_36f18bc533e990'" 215 | ] 216 | }, 217 | "execution_count": 6, 218 | "metadata": {}, 219 | "output_type": "execute_result" 220 | } 221 | ], 222 | "source": [ 223 | "x = [0, 0, 1, 1, 0, 0, 1, 1]\n", 224 | "y = [0, 1, 1, 0, 0, 1, 1, 0]\n", 225 | "z = [0, 0, 0, 0, 1, 1, 1, 1]\n", 226 | "X = np.c_[x, y, z]\n", 227 | "i = [7, 0, 0, 0, 4, 4, 6, 6, 4, 0, 3, 2]\n", 228 | "j = [3, 4, 1, 2, 5, 6, 5, 2, 0, 1, 6, 3]\n", 229 | "k = [0, 7, 2, 3, 6, 7, 1, 1, 5, 5, 7, 6]\n", 230 | "Y = np.c_[i, j, k]\n", 231 | "envtest.mesh(X=X, Y=Y, opts=dict(opacity=0.5))" 232 | ] 233 | }, 234 | { 235 | "cell_type": "markdown", 236 | "metadata": {}, 237 | "source": [ 238 | "## 更新损失函数" 239 | ] 240 | }, 241 | { 242 | "cell_type": "markdown", 243 | "metadata": {}, 244 | "source": [ 245 | "在训练的时候我们每一批次都会打印一下训练的损失和测试的准确率，这样展示的图表是需要动态增加数据的，下面我们来模拟一下这种情况" 246 | ] 247 | }, 248 | { 249 | "cell_type": "code", 250 | "execution_count": 7, 251 | "metadata": {}, 252 | "outputs": [], 253 | "source": [ 254 | "x,y=0,0\n", 255 | "env2 = Visdom()\n", 256 | "pane1= env2.line(\n", 257 | " X=np.array([x]),\n", 258 | " Y=np.array([y]),\n", 259 | " opts=dict(title='dynamic data'))" 260 | ] 261 | }, 262 | { 263 | "cell_type": "code", 264 | "execution_count": 8, 265 | "metadata": {}, 266 | "outputs": [ 267 | { 268 | "name": "stdout", 269 | "output_type": "stream", 270 | "text": [ 271 | "0 0.0\n", 272 | "1 1.5\n", 273 | "3 5.25\n", 274 | "6 12.375\n", 275 | "10 24.5625\n", 276 | "15 44.34375\n", 277 | "21 75.515625\n", 278 | "28 123.7734375\n", 279 | "36 197.66015625\n", 280 | "45 309.990234375\n" 281 | ] 282 | } 283 | ], 284 | "source": [ 285 | "for i in range(10):\n", 286 | " time.sleep(1) #每隔一秒钟打印一次数据\n", 287 | " x+=i\n", 288 | " y=(y+i)*1.5\n", 289 | " print(x,y)\n", 290 | " env2.line(\n", 291 | " X=np.array([x]),\n", 292 | " Y=np.array([y]),\n", 293 | " win=pane1,#win参数确认使用哪一个pane\n", 294 | " update='append') #我们做的动作是追加，除了追加意外还有其他方式，这里我们不做介绍了" 295 | ] 296 | }, 297 | { 298 | "cell_type": "markdown", 299 | "metadata": {}, 300 | "source": [ 301 | "在运行完上述程序时，切换到visdom，看看效果吧\n", 302 | "\n", 303 | "visdom的基本用法介绍完毕，下一节介绍更加强大的 tensorboardx" 304 | ] 305 | }, 306 | { 307 | "cell_type": "code", 308 | "execution_count": null, 309 | "metadata": {}, 310 | "outputs": [], 311 | "source": [] 312 | } 313 | ], 314 | "metadata": { 315 | "kernelspec": { 316 | "display_name": "pytorch 1.0", 317 | "language": "python", 318 | "name": "pytorch1" 319 | }, 320 | "language_info": { 321 | "codemirror_mode": { 322 | "name": "ipython", 323 | "version": 3 324 | }, 325 | "file_extension": ".py", 326 | "mimetype": "text/x-python", 327 | "name": "python", 328 | "nbconvert_exporter": "python", 329 | "pygments_lexer": "ipython3", 330 | "version": "3.6.7" 331 | } 332 | }, 333 | "nbformat": 4, 334 | "nbformat_minor": 2 335 | } 336 | -------------------------------------------------------------------------------- /chapter4/4.2.2-tensorboardx.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "data": { 10 | "text/plain": [ 11 | "'1.0.0'" 12 | ] 13 | }, 14 | "execution_count": 1, 15 | "metadata": {}, 16 | "output_type": "execute_result" 17 | } 18 | ], 19 | "source": [ 20 | "import torch\n", 21 | "import numpy as np\n", 22 | "import torch.nn as nn\n", 23 | "import torch.nn.functional as F\n", 24 | "from PIL import Image\n", 25 | "from torchvision import transforms\n", 26 | "from torchvision import models,datasets\n", 27 | "torch.__version__" 28 | ] 29 | }, 30 | { 31 | "cell_type": "markdown", 32 | "metadata": {}, 33 | "source": [ 34 | "# 4.2.2 使用Tensorboard在 PyTorch 中进行可视化 \n", 35 | "\n", 36 | "## Tensorboard 简介\n", 37 | "Tensorboard是tensorflow内置的一个可视化工具，它通过将tensorflow程序输出的日志文件的信息可视化使得tensorflow程序的理解、调试和优化更加简单高效。\n", 38 | "Tensorboard的可视化依赖于tensorflow程序运行输出的日志文件，因而tensorboard和tensorflow程序在不同的进程中运行。\n", 39 | "TensorBoard给我们提供了极其方便而强大的可视化环境。它可以帮助我们理解整个神经网络的学习过程、数据的分布、性能瓶颈等等。\n", 40 | "\n", 41 | "tensorboard虽然是tensorflow内置的可视化工具，但是他们跑在不同的进程中，所以Github上已经有大神将tensorboard应用到Pytorch中 [链接在这里]( https://github.com/lanpa/tensorboardX)" 42 | ] 43 | }, 44 | { 45 | "cell_type": "markdown", 46 | "metadata": {}, 47 | "source": [ 48 | "## Tensorboard 安装\n", 49 | "首先需要安装tensorboard\n", 50 | "\n", 51 | "`pip install tensorboard`\n", 52 | "\n", 53 | "然后再安装tensorboardx\n", 54 | "\n", 55 | "`pip install tensorboardx`\n", 56 | "\n", 57 | "安装完成后与 visdom一样执行独立的命令\n", 58 | "`tensorboard --logdir logs` 即可启动，默认的端口是 6006,在浏览器中打开 `http://localhost:6006/` 即可看到web页面。\n", 59 | "\n", 60 | "这里要说明的是 微软的Edge浏览器css会无法加载，使用chrome正常显示" 61 | ] 62 | }, 63 | { 64 | "cell_type": "markdown", 65 | "metadata": {}, 66 | "source": [ 67 | "## 页面\n", 68 | "与visdom不同，tensorboard针对不同的类型人为的区分多个标签，每一个标签页面代表不同的类型。\n", 69 | "下面我们根据不同的页面功能做个简单的介绍，更多详细内容请参考官网\n", 70 | "### SCALAR\n", 71 | "对标量数据进行汇总和记录，通常用来可视化训练过程中随着迭代次数准确率(val acc)、损失值(train/test loss)、学习率(learning rate)、每一层的权重和偏置的统计量(mean、std、max/min)等的变化曲线\n", 72 | "### IMAGES\n", 73 | "可视化当前轮训练使用的训练/测试图片或者 feature maps\n", 74 | "### GRAPHS\n", 75 | "可视化计算图的结构及计算图上的信息，通常用来展示网络的结构\n", 76 | "### HISTOGRAMS\n", 77 | "可视化张量的取值分布，记录变量的直方图(统计张量随着迭代轮数的变化情况）\n", 78 | "### PROJECTOR\n", 79 | "全称Embedding Projector 高维向量进行可视化" 80 | ] 81 | }, 82 | { 83 | "cell_type": "markdown", 84 | "metadata": {}, 85 | "source": [ 86 | "## 使用\n", 87 | "在使用前请先去确认执行`tensorboard --logdir logs` 并保证 `http://localhost:6006/` 页面能够正常打开\n", 88 | "\n", 89 | "### 图像展示\n", 90 | "首先介绍比较简单的功能，查看我们训练集和数据集中的图像，这里我们使用现成的图像作为展示。这里使用wikipedia上的一张猫的图片[这里](https://en.wikipedia.org/wiki/Cat#/media/File:Felis_silvestris_catus_lying_on_rice_straw.jpg)\n", 91 | "\n", 92 | "引入 tensorboardX 包" 93 | ] 94 | }, 95 | { 96 | "cell_type": "code", 97 | "execution_count": 2, 98 | "metadata": {}, 99 | "outputs": [], 100 | "source": [ 101 | "from tensorboardX import SummaryWriter" 102 | ] 103 | }, 104 | { 105 | "cell_type": "code", 106 | "execution_count": 3, 107 | "metadata": {}, 108 | "outputs": [ 109 | { 110 | "data": { 111 | "text/plain": [ 112 | "(1280, 853)" 113 | ] 114 | }, 115 | "execution_count": 3, 116 | "metadata": {}, 117 | "output_type": "execute_result" 118 | } 119 | ], 120 | "source": [ 121 | "cat_img = Image.open('./1280px-Felis_silvestris_catus_lying_on_rice_straw.jpg')\n", 122 | "cat_img.size" 123 | ] 124 | }, 125 | { 126 | "cell_type": "markdown", 127 | "metadata": {}, 128 | "source": [ 129 | "这是一张1280x853的图，我们先把她变成224x224的图片，因为后面要使用的是vgg16" 130 | ] 131 | }, 132 | { 133 | "cell_type": "code", 134 | "execution_count": 4, 135 | "metadata": {}, 136 | "outputs": [], 137 | "source": [ 138 | "transform_224 = transforms.Compose([\n", 139 | " transforms.Resize(224), # 这里要说明下 Scale 已经过期了，使用Resize\n", 140 | " transforms.CenterCrop(224),\n", 141 | " transforms.ToTensor(),\n", 142 | " ])\n", 143 | "cat_img_224=transform_224(cat_img)" 144 | ] 145 | }, 146 | { 147 | "cell_type": "markdown", 148 | "metadata": {}, 149 | "source": [ 150 | "将图片展示在tebsorboard中：" 151 | ] 152 | }, 153 | { 154 | "cell_type": "code", 155 | "execution_count": 5, 156 | "metadata": {}, 157 | "outputs": [], 158 | "source": [ 159 | "writer = SummaryWriter(log_dir='./logs', comment='cat image') # 这里的logs要与--logdir的参数一样\n", 160 | "writer.add_image(\"cat\",cat_img_224)\n", 161 | "writer.close()# 执行close立即刷新，否则将每120秒自动刷新" 162 | ] 163 | }, 164 | { 165 | "cell_type": "markdown", 166 | "metadata": {}, 167 | "source": [ 168 | "浏览器访问 `http://localhost:6006/#images` 即可看到猫的图片 \n", 169 | "### 更新损失函数\n", 170 | "更新损失函数和训练批次我们与visdom一样使用模拟展示，这里用到的是tensorboard的SCALAR页面" 171 | ] 172 | }, 173 | { 174 | "cell_type": "code", 175 | "execution_count": 6, 176 | "metadata": {}, 177 | "outputs": [], 178 | "source": [ 179 | "x = torch.FloatTensor([100])\n", 180 | "y = torch.FloatTensor([500])\n", 181 | "\n", 182 | "for epoch in range(100):\n", 183 | " x /= 1.5\n", 184 | " y /= 1.5\n", 185 | " loss = y - x\n", 186 | " with SummaryWriter(log_dir='./logs', comment='train') as writer: #可以直接使用python的with语法，自动调用close方法\n", 187 | " writer.add_histogram('his/x', x, epoch)\n", 188 | " writer.add_histogram('his/y', y, epoch)\n", 189 | " writer.add_scalar('data/x', x, epoch)\n", 190 | " writer.add_scalar('data/y', y, epoch)\n", 191 | " writer.add_scalar('data/loss', loss, epoch)\n", 192 | " writer.add_scalars('data/data_group', {'x': x,\n", 193 | " 'y': y,\n", 194 | " 'loss': loss}, epoch)\n", 195 | "\n", 196 | " " 197 | ] 198 | }, 199 | { 200 | "cell_type": "markdown", 201 | "metadata": {}, 202 | "source": [ 203 | "浏览器访问 `http://localhost:6006/#scalars` 即可看到图形\n", 204 | "### 使用PROJECTOR对高维向量可视化\n", 205 | "PROJECTOR的的原理是通过PCA，T-SNE等方法将高维向量投影到三维坐标系（降维度）。Embedding Projector从模型运行过程中保存的checkpoint文件中读取数据，默认使用主成分分析法（PCA）将高维数据投影到3D空间中，也可以通过设置设置选择T-SNE投影方法，这里做一个简单的展示。\n", 206 | "\n", 207 | "我们还是用第三章的mnist代码" 208 | ] 209 | }, 210 | { 211 | "cell_type": "code", 212 | "execution_count": 7, 213 | "metadata": {}, 214 | "outputs": [], 215 | "source": [ 216 | "BATCH_SIZE=512 \n", 217 | "EPOCHS=20 \n", 218 | "train_loader = torch.utils.data.DataLoader(\n", 219 | " datasets.MNIST('data', train=True, download=True, \n", 220 | " transform=transforms.Compose([\n", 221 | " transforms.ToTensor(),\n", 222 | " transforms.Normalize((0.1307,), (0.3081,))\n", 223 | " ])),\n", 224 | " batch_size=BATCH_SIZE, shuffle=True)" 225 | ] 226 | }, 227 | { 228 | "cell_type": "code", 229 | "execution_count": 8, 230 | "metadata": {}, 231 | "outputs": [], 232 | "source": [ 233 | "class ConvNet(nn.Module):\n", 234 | " def __init__(self):\n", 235 | " super().__init__()\n", 236 | " # 1,28x28\n", 237 | " self.conv1=nn.Conv2d(1,10,5) # 10, 24x24\n", 238 | " self.conv2=nn.Conv2d(10,20,3) # 128, 10x10\n", 239 | " self.fc1 = nn.Linear(20*10*10,500)\n", 240 | " self.fc2 = nn.Linear(500,10)\n", 241 | " def forward(self,x):\n", 242 | " in_size = x.size(0)\n", 243 | " out = self.conv1(x) #24\n", 244 | " out = F.relu(out)\n", 245 | " out = F.max_pool2d(out, 2, 2) #12\n", 246 | " out = self.conv2(out) #10\n", 247 | " out = F.relu(out)\n", 248 | " out = out.view(in_size,-1)\n", 249 | " out = self.fc1(out)\n", 250 | " out = F.relu(out)\n", 251 | " out = self.fc2(out)\n", 252 | " out = F.log_softmax(out,dim=1)\n", 253 | " return out\n", 254 | "model = ConvNet()\n", 255 | "optimizer = torch.optim.Adam(model.parameters())" 256 | ] 257 | }, 258 | { 259 | "cell_type": "code", 260 | "execution_count": 9, 261 | "metadata": {}, 262 | "outputs": [], 263 | "source": [ 264 | "def train(model, train_loader, optimizer, epoch):\n", 265 | " n_iter=0\n", 266 | " model.train()\n", 267 | " for batch_idx, (data, target) in enumerate(train_loader):\n", 268 | " optimizer.zero_grad()\n", 269 | " output = model(data)\n", 270 | " loss = F.nll_loss(output, target)\n", 271 | " loss.backward()\n", 272 | " optimizer.step()\n", 273 | " if(batch_idx+1)%30 == 0: \n", 274 | " n_iter=n_iter+1\n", 275 | " print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n", 276 | " epoch, batch_idx * len(data), len(train_loader.dataset),\n", 277 | " 100. * batch_idx / len(train_loader), loss.item()))\n", 278 | " #主要增加了一下内容\n", 279 | " out = torch.cat((output.data, torch.ones(len(output), 1)), 1) # 因为是投影到3D的空间，所以我们只需要3个维度\n", 280 | " with SummaryWriter(log_dir='./logs', comment='mnist') as writer: \n", 281 | " #使用add_embedding方法进行可视化展示\n", 282 | " writer.add_embedding(\n", 283 | " out,\n", 284 | " metadata=target.data,\n", 285 | " label_img=data.data,\n", 286 | " global_step=n_iter)" 287 | ] 288 | }, 289 | { 290 | "cell_type": "markdown", 291 | "metadata": {}, 292 | "source": [ 293 | "这里节省时间，只训练一次" 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": 10, 299 | "metadata": {}, 300 | "outputs": [ 301 | { 302 | "name": "stdout", 303 | "output_type": "stream", 304 | "text": [ 305 | "Train Epoch: 0 [14848/60000 (25%)]\tLoss: 0.405838\n", 306 | "Train Epoch: 0 [30208/60000 (50%)]\tLoss: 0.206041\n", 307 | "Train Epoch: 0 [45568/60000 (75%)]\tLoss: 0.144166\n" 308 | ] 309 | } 310 | ], 311 | "source": [ 312 | "train(model, train_loader, optimizer, 0)" 313 | ] 314 | }, 315 | { 316 | "cell_type": "markdown", 317 | "metadata": {}, 318 | "source": [ 319 | "打开 `http://localhost:6006/#projector` 即可看到效果。\n", 320 | "\n", 321 | "### 绘制网络结构\n", 322 | "在pytorch中我们可以使用print直接打印出网络的结构，但是这种方法可视化效果不好，这里使用tensorboard的GRAPHS来实现网络结构的可视化。\n", 323 | "由于pytorch使用的是动态图计算，所以我们这里要手动进行一次前向的传播.\n", 324 | "\n", 325 | "使用Pytorch已经构建好的模型进行展示" 326 | ] 327 | }, 328 | { 329 | "cell_type": "code", 330 | "execution_count": 11, 331 | "metadata": {}, 332 | "outputs": [ 333 | { 334 | "name": "stdout", 335 | "output_type": "stream", 336 | "text": [ 337 | "VGG(\n", 338 | " (features): Sequential(\n", 339 | " (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 340 | " (1): ReLU(inplace)\n", 341 | " (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 342 | " (3): ReLU(inplace)\n", 343 | " (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n", 344 | " (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 345 | " (6): ReLU(inplace)\n", 346 | " (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 347 | " (8): ReLU(inplace)\n", 348 | " (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n", 349 | " (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 350 | " (11): ReLU(inplace)\n", 351 | " (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 352 | " (13): ReLU(inplace)\n", 353 | " (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 354 | " (15): ReLU(inplace)\n", 355 | " (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n", 356 | " (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 357 | " (18): ReLU(inplace)\n", 358 | " (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 359 | " (20): ReLU(inplace)\n", 360 | " (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 361 | " (22): ReLU(inplace)\n", 362 | " (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n", 363 | " (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 364 | " (25): ReLU(inplace)\n", 365 | " (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 366 | " (27): ReLU(inplace)\n", 367 | " (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 368 | " (29): ReLU(inplace)\n", 369 | " (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n", 370 | " )\n", 371 | " (classifier): Sequential(\n", 372 | " (0): Linear(in_features=25088, out_features=4096, bias=True)\n", 373 | " (1): ReLU(inplace)\n", 374 | " (2): Dropout(p=0.5)\n", 375 | " (3): Linear(in_features=4096, out_features=4096, bias=True)\n", 376 | " (4): ReLU(inplace)\n", 377 | " (5): Dropout(p=0.5)\n", 378 | " (6): Linear(in_features=4096, out_features=1000, bias=True)\n", 379 | " )\n", 380 | ")\n" 381 | ] 382 | } 383 | ], 384 | "source": [ 385 | "vgg16 = models.vgg16() # 这里下载预训练好的模型\n", 386 | "print(vgg16) # 打印一下这个模型" 387 | ] 388 | }, 389 | { 390 | "cell_type": "markdown", 391 | "metadata": {}, 392 | "source": [ 393 | "在前向传播前，先要把图片做一些调整" 394 | ] 395 | }, 396 | { 397 | "cell_type": "code", 398 | "execution_count": 12, 399 | "metadata": {}, 400 | "outputs": [], 401 | "source": [ 402 | "transform_2 = transforms.Compose([\n", 403 | " transforms.Resize(224), \n", 404 | " transforms.CenterCrop(224),\n", 405 | " transforms.ToTensor(),\n", 406 | " # convert RGB to BGR\n", 407 | " # from \n", 408 | " transforms.Lambda(lambda x: torch.index_select(x, 0, torch.LongTensor([2, 1, 0]))),\n", 409 | " transforms.Lambda(lambda x: x*255),\n", 410 | " transforms.Normalize(mean = [103.939, 116.779, 123.68],\n", 411 | " std = [ 1, 1, 1 ]),\n", 412 | "])" 413 | ] 414 | }, 415 | { 416 | "cell_type": "markdown", 417 | "metadata": {}, 418 | "source": [ 419 | "使用上一张猫的图片进行前向传播" 420 | ] 421 | }, 422 | { 423 | "cell_type": "code", 424 | "execution_count": 13, 425 | "metadata": {}, 426 | "outputs": [ 427 | { 428 | "data": { 429 | "text/plain": [ 430 | "torch.Size([1, 3, 224, 224])" 431 | ] 432 | }, 433 | "execution_count": 13, 434 | "metadata": {}, 435 | "output_type": "execute_result" 436 | } 437 | ], 438 | "source": [ 439 | "vgg16_input=transform_2(cat_img)[np.newaxis]# 因为pytorch的是分批次进行的，所以我们这里建立一个批次为1的数据集\n", 440 | "vgg16_input.shape" 441 | ] 442 | }, 443 | { 444 | "cell_type": "markdown", 445 | "metadata": {}, 446 | "source": [ 447 | "开始前向传播，打印输出值" 448 | ] 449 | }, 450 | { 451 | "cell_type": "code", 452 | "execution_count": 14, 453 | "metadata": {}, 454 | "outputs": [ 455 | { 456 | "name": "stdout", 457 | "output_type": "stream", 458 | "text": [ 459 | "(1, 1000) 931\n" 460 | ] 461 | } 462 | ], 463 | "source": [ 464 | "raw_score = vgg16(vgg16_input)\n", 465 | "raw_score_numpy = raw_score.data.numpy()\n", 466 | "print(raw_score_numpy.shape, np.argmax(raw_score_numpy.ravel()))" 467 | ] 468 | }, 469 | { 470 | "cell_type": "markdown", 471 | "metadata": {}, 472 | "source": [ 473 | "将结构图在tensorboard进行展示" 474 | ] 475 | }, 476 | { 477 | "cell_type": "code", 478 | "execution_count": 15, 479 | "metadata": {}, 480 | "outputs": [], 481 | "source": [ 482 | "with SummaryWriter(log_dir='./logs', comment='vgg16') as writer:\n", 483 | " writer.add_graph(vgg16, (vgg16_input,))" 484 | ] 485 | }, 486 | { 487 | "cell_type": "markdown", 488 | "metadata": {}, 489 | "source": [ 490 | "打开tensorboard找到graphs 看看效果吧" 491 | ] 492 | }, 493 | { 494 | "cell_type": "code", 495 | "execution_count": null, 496 | "metadata": {}, 497 | "outputs": [], 498 | "source": [] 499 | } 500 | ], 501 | "metadata": { 502 | "kernelspec": { 503 | "display_name": "pytorch 1.0", 504 | "language": "python", 505 | "name": "pytorch1" 506 | }, 507 | "language_info": { 508 | "codemirror_mode": { 509 | "name": "ipython", 510 | "version": 3 511 | }, 512 | "file_extension": ".py", 513 | "mimetype": "text/x-python", 514 | "name": "python", 515 | "nbconvert_exporter": "python", 516 | "pygments_lexer": "ipython3", 517 | "version": "3.6.6" 518 | } 519 | }, 520 | "nbformat": 4, 521 | "nbformat_minor": 2 522 | } 523 | -------------------------------------------------------------------------------- /chapter4/readme.md: -------------------------------------------------------------------------------- 1 | # Pytorch 中文手册第四章 ： 提高 2 | 3 | ## 目录 4 | 5 | ## 第一节 Fine-tuning 6 | 7 | [Fine-tuning](4.1-fine-tuning.ipynb) 8 | 9 | ## 第二节 可视化 10 | 11 | [visdom](4.2.1-visdom.ipynb) 12 | 13 | [tensorboardx](4.2.2-tensorboardx.ipynb) -------------------------------------------------------------------------------- /chapter4/static.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HazeDT/pytorch-handbook/6fff3a73230c0b44efba40826efa88d29cc08b83/chapter4/static.zip --------------------------------------------------------------------------------