├── dataset └── train │ ├── ants_label │ ├── 0013035.txt │ ├── 5650366_e22b7e1065.txt │ ├── 6240329_72c01e663e.txt │ ├── 6240338_93729615ec.txt │ ├── 6743948_2b8c096dda.txt │ ├── 7759525_1363d24e88.txt │ └── 9715481_b3cb4114ff.txt │ ├── bees_label │ ├── 16838648_415acd9e3f.txt │ ├── 17209602_fe5a5a746f.txt │ ├── 21399619_3e61e5bb6f.txt │ ├── 29494643_e3410f0d37.txt │ ├── 36900412_92b81831ad.txt │ ├── 39672681_1302d204d1.txt │ ├── 39747887_42df2855ee.txt │ ├── 85112639_6e860b0469.txt │ ├── 90179376_abc234e5f4.txt │ └── 92663402_37f379e57a.txt │ ├── ants_image │ ├── 0013035.jpg │ ├── 5650366_e22b7e1065.jpg │ ├── 6240329_72c01e663e.jpg │ ├── 6240338_93729615ec.jpg │ ├── 6743948_2b8c096dda.jpg │ ├── 7759525_1363d24e88.jpg │ └── 9715481_b3cb4114ff.jpg │ └── bees_image │ ├── 16838648_415acd9e3f.jpg │ ├── 17209602_fe5a5a746f.jpg │ ├── 21399619_3e61e5bb6f.jpg │ ├── 29494643_e3410f0d37.jpg │ ├── 36900412_92b81831ad.jpg │ ├── 39672681_1302d204d1.jpg │ ├── 39747887_42df2855ee.jpg │ ├── 85112639_6e860b0469.jpg │ ├── 90179376_abc234e5f4.jpg │ └── 92663402_37f379e57a.jpg ├── imgs └── dog.png ├── assets ├── cifar10.png └── neural.png ├── note.md ├── scripts_liu ├── p5_linear.py ├── p2_linearModel.py ├── p2_exercise.py ├── p4_exercise.py ├── p3_gradientDescent.py ├── p3_SGD.py ├── p4_backPropagation.py └── bingdundun.py ├── scripts_xiaotudui ├── CallTest.py ├── nn_Module.py ├── rename_dataset.py ├── test_tensorboard.py ├── test_transforms_1.py ├── p26_model_save.py ├── p23_nn_loss_network.py ├── p25_model_pretrained.py ├── nn_conv.py ├── p23_nn_loss.py ├── model.py ├── UsefulTransform_2.py ├── dataLoader.py ├── test.py ├── dataset_transform.py ├── read_data.py ├── p26_model_load.py ├── p24_nn_optim.py ├── nn_linear.py ├── nn_relu.py ├── nn_maxpool.py ├── p22_nn_seq.py ├── train.py └── train_gpu.py ├── README.md ├── .gitignore └── 60-Minute-Guide ├── 2_autograd.ipynb ├── neural_networks_tutorial_en.ipynb ├── tensor_tutorial_en.ipynb ├── 1_tensors.ipynb ├── 3_neural-networks.ipynb ├── autograd_tutorial_en.ipynb └── cifar10_tutorial_en.ipynb /dataset/train/ants_label/0013035.txt: -------------------------------------------------------------------------------- 1 | ants -------------------------------------------------------------------------------- /dataset/train/ants_label/5650366_e22b7e1065.txt: -------------------------------------------------------------------------------- 1 | ants -------------------------------------------------------------------------------- /dataset/train/ants_label/6240329_72c01e663e.txt: -------------------------------------------------------------------------------- 1 | ants -------------------------------------------------------------------------------- /dataset/train/ants_label/6240338_93729615ec.txt: -------------------------------------------------------------------------------- 1 | ants -------------------------------------------------------------------------------- /dataset/train/ants_label/6743948_2b8c096dda.txt: -------------------------------------------------------------------------------- 1 | ants -------------------------------------------------------------------------------- /dataset/train/ants_label/7759525_1363d24e88.txt: -------------------------------------------------------------------------------- 1 | ants -------------------------------------------------------------------------------- /dataset/train/ants_label/9715481_b3cb4114ff.txt: -------------------------------------------------------------------------------- 1 | ants -------------------------------------------------------------------------------- /dataset/train/bees_label/16838648_415acd9e3f.txt: -------------------------------------------------------------------------------- 1 | bees -------------------------------------------------------------------------------- /dataset/train/bees_label/17209602_fe5a5a746f.txt: -------------------------------------------------------------------------------- 1 | bees -------------------------------------------------------------------------------- /dataset/train/bees_label/21399619_3e61e5bb6f.txt: -------------------------------------------------------------------------------- 1 | bees -------------------------------------------------------------------------------- /dataset/train/bees_label/29494643_e3410f0d37.txt: -------------------------------------------------------------------------------- 1 | bees -------------------------------------------------------------------------------- /dataset/train/bees_label/36900412_92b81831ad.txt: -------------------------------------------------------------------------------- 1 | bees -------------------------------------------------------------------------------- /dataset/train/bees_label/39672681_1302d204d1.txt: -------------------------------------------------------------------------------- 1 | bees -------------------------------------------------------------------------------- /dataset/train/bees_label/39747887_42df2855ee.txt: -------------------------------------------------------------------------------- 1 | bees -------------------------------------------------------------------------------- /dataset/train/bees_label/85112639_6e860b0469.txt: -------------------------------------------------------------------------------- 1 | bees -------------------------------------------------------------------------------- /dataset/train/bees_label/90179376_abc234e5f4.txt: -------------------------------------------------------------------------------- 1 | bees -------------------------------------------------------------------------------- /dataset/train/bees_label/92663402_37f379e57a.txt: -------------------------------------------------------------------------------- 1 | bees -------------------------------------------------------------------------------- /imgs/dog.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/imgs/dog.png -------------------------------------------------------------------------------- /assets/cifar10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/assets/cifar10.png -------------------------------------------------------------------------------- /assets/neural.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/assets/neural.png -------------------------------------------------------------------------------- /note.md: -------------------------------------------------------------------------------- 1 | python中的量大法宝函数： 2 | > dir() 3 | > help() 4 | 5 | # p5 -- pytorch加载数据初认识 6 | 7 | dataset 8 | dataloader 9 | -------------------------------------------------------------------------------- /dataset/train/ants_image/0013035.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/dataset/train/ants_image/0013035.jpg -------------------------------------------------------------------------------- /scripts_liu/p5_linear.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/2/8下午2:34 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | 6 | import torch 7 | 8 | -------------------------------------------------------------------------------- /dataset/train/ants_image/5650366_e22b7e1065.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/dataset/train/ants_image/5650366_e22b7e1065.jpg -------------------------------------------------------------------------------- /dataset/train/ants_image/6240329_72c01e663e.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/dataset/train/ants_image/6240329_72c01e663e.jpg -------------------------------------------------------------------------------- /dataset/train/ants_image/6240338_93729615ec.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/dataset/train/ants_image/6240338_93729615ec.jpg -------------------------------------------------------------------------------- /dataset/train/ants_image/6743948_2b8c096dda.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/dataset/train/ants_image/6743948_2b8c096dda.jpg -------------------------------------------------------------------------------- /dataset/train/ants_image/7759525_1363d24e88.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/dataset/train/ants_image/7759525_1363d24e88.jpg -------------------------------------------------------------------------------- /dataset/train/ants_image/9715481_b3cb4114ff.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/dataset/train/ants_image/9715481_b3cb4114ff.jpg -------------------------------------------------------------------------------- /dataset/train/bees_image/16838648_415acd9e3f.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/dataset/train/bees_image/16838648_415acd9e3f.jpg -------------------------------------------------------------------------------- /dataset/train/bees_image/17209602_fe5a5a746f.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/dataset/train/bees_image/17209602_fe5a5a746f.jpg -------------------------------------------------------------------------------- /dataset/train/bees_image/21399619_3e61e5bb6f.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/dataset/train/bees_image/21399619_3e61e5bb6f.jpg -------------------------------------------------------------------------------- /dataset/train/bees_image/29494643_e3410f0d37.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/dataset/train/bees_image/29494643_e3410f0d37.jpg -------------------------------------------------------------------------------- /dataset/train/bees_image/36900412_92b81831ad.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/dataset/train/bees_image/36900412_92b81831ad.jpg -------------------------------------------------------------------------------- /dataset/train/bees_image/39672681_1302d204d1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/dataset/train/bees_image/39672681_1302d204d1.jpg -------------------------------------------------------------------------------- /dataset/train/bees_image/39747887_42df2855ee.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/dataset/train/bees_image/39747887_42df2855ee.jpg -------------------------------------------------------------------------------- /dataset/train/bees_image/85112639_6e860b0469.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/dataset/train/bees_image/85112639_6e860b0469.jpg -------------------------------------------------------------------------------- /dataset/train/bees_image/90179376_abc234e5f4.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/dataset/train/bees_image/90179376_abc234e5f4.jpg -------------------------------------------------------------------------------- /dataset/train/bees_image/92663402_37f379e57a.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/juliusyang97/pytorch_learn/HEAD/dataset/train/bees_image/92663402_37f379e57a.jpg -------------------------------------------------------------------------------- /scripts_xiaotudui/CallTest.py: -------------------------------------------------------------------------------- 1 | class Person: 2 | def __call__(self, name): 3 | print("__call__" + " Hello " + name) 4 | 5 | def hello(self, name): 6 | print("hello" + name) 7 | 8 | person = Person() 9 | person("zhangsan") 10 | person.hello("lisi") 11 | 12 | -------------------------------------------------------------------------------- /scripts_xiaotudui/nn_Module.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from torch import nn 3 | 4 | 5 | class Tudui(nn.Module): 6 | def __init__(self): 7 | super().__init__() 8 | 9 | def forward(self, input): 10 | output = input + 1 11 | return output 12 | 13 | tudui = Tudui() 14 | x = torch.tensor(1.0) 15 | output = tudui(x) 16 | print(output) 17 | 18 | 19 | 20 | -------------------------------------------------------------------------------- /scripts_xiaotudui/rename_dataset.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | root_dir = "../dataset/train" 4 | # target_dir = "ants_image" 5 | target_dir = "bees_image" 6 | img_path = os.listdir(os.path.join(root_dir, target_dir)) 7 | label = target_dir.split("_")[0] 8 | # out_dir = "ants_label" 9 | out_dir = "bees_label" 10 | 11 | for i in img_path: 12 | file_name = i.split(".")[0] 13 | with open(os.path.join(root_dir, out_dir, "{}.txt".format(file_name)), "w") as f: 14 | f.write(label) 15 | 16 | 17 | -------------------------------------------------------------------------------- /scripts_xiaotudui/test_tensorboard.py: -------------------------------------------------------------------------------- 1 | from torch.utils.tensorboard import SummaryWriter 2 | import numpy as np 3 | from PIL import Image 4 | 5 | writer = SummaryWriter("../logs") 6 | image_path = "../dataset/train/ants_image/6240329_72c01e663e.jpg" 7 | img_PIL = Image.open(image_path) 8 | img_array = np.array(img_PIL) 9 | print(type(img_array)) 10 | print(img_array.shape) 11 | 12 | 13 | writer.add_image("test", img_array, 1, dataformats='HWC') 14 | # writer.add_scalar() 15 | # y = 2 * x 16 | for i in range(100): 17 | writer.add_scalar("y=2x", 4*i**2, i) 18 | writer.close() 19 | 20 | 21 | -------------------------------------------------------------------------------- /scripts_xiaotudui/test_transforms_1.py: -------------------------------------------------------------------------------- 1 | from PIL import Image 2 | from torchvision import transforms 3 | from torch.utils.tensorboard import SummaryWriter 4 | import cv2 5 | 6 | # python的用法 --> tensor数据类型 7 | # 通过 transforms.ToTensor 去看两个问题 8 | # 1. transforms 该如何使用（python） 9 | # 2. 为什么我们需要Tensor数据类型 10 | 11 | img_path = "../dataset/train/ants_image/0013035.jpg" 12 | img = Image.open(img_path) 13 | # print(img) 14 | 15 | writer = SummaryWriter("../logs/transforms") 16 | 17 | # 1. transforms 该如何使用（python） 18 | tensor_trans = transforms.ToTensor() 19 | tensor_img = tensor_trans(img) 20 | # print(tensor_img) 21 | 22 | writer.add_image("Tensor_img", tensor_img) 23 | writer.close() 24 | 25 | -------------------------------------------------------------------------------- /scripts_xiaotudui/p26_model_save.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/7下午11:13 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | import torch 6 | import torchvision 7 | from torch import nn 8 | 9 | vgg16 = torchvision.models.vgg16(pretrained=False) 10 | 11 | # 保存方式1 -- 模型结构+模型参数 12 | # torch.save(vgg16, "../models/vgg16_method1.pth") 13 | 14 | # 保存方式2 -- 模型参数（官方推荐） 15 | torch.save(vgg16.state_dict(), "../models/vgg16_method2.pth") 16 | 17 | # 陷阱 -- 使用方式1保存的陷阱 18 | class Tudui(nn.Module): 19 | def __init__(self): 20 | super(Tudui, self).__init__() 21 | self.conv1 = nn.Conv2d(3, 64, kernel_size=3) 22 | 23 | def forward(self, x): 24 | x = self.conv1(x) 25 | return x 26 | 27 | tudui = Tudui() 28 | torch.save(tudui, "../models/tudui_method1.pth") 29 | 30 | -------------------------------------------------------------------------------- /scripts_xiaotudui/p23_nn_loss_network.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/7下午9:40 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | import torch 6 | import torchvision 7 | from torch.utils.data import DataLoader 8 | from p22_nn_seq import Tudui 9 | from torch import nn 10 | 11 | 12 | dataset = torchvision.datasets.CIFAR10(root="../dataset", train=True, 13 | transform=torchvision.transforms.ToTensor(), 14 | download=True) 15 | dataloader = DataLoader(dataset, batch_size=1) 16 | 17 | tudui = Tudui() 18 | 19 | loss = nn.CrossEntropyLoss() 20 | 21 | for data in dataloader: 22 | imgs, targets = data 23 | outputs = tudui(imgs) 24 | result_loss = loss(outputs, targets) 25 | result_loss.backward() 26 | print(result_loss) 27 | 28 | -------------------------------------------------------------------------------- /scripts_xiaotudui/p25_model_pretrained.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/7下午10:43 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | import torch 6 | import torchvision 7 | from torch.utils.data import DataLoader 8 | from torch import nn 9 | 10 | train_dataset = torchvision.datasets.CIFAR10(root="../dataset", train=True, download=True, 11 | transform=torchvision.transforms.ToTensor()) 12 | 13 | dataloader = DataLoader(train_dataset, batch_size=64) 14 | 15 | vgg16_false = torchvision.models.vgg16(pretrained=False) 16 | vgg16_true = torchvision.models.vgg16(pretrained=True) 17 | 18 | print(vgg16_false) 19 | 20 | vgg16_false.classifier.add_module("add_linear", nn.Linear(1000, 10)) 21 | print(vgg16_false) 22 | 23 | vgg16_false.classifier[6] = nn.Linear(4096, 10) 24 | print(vgg16_false) 25 | -------------------------------------------------------------------------------- /scripts_xiaotudui/nn_conv.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/6下午11:13 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | 6 | import torch 7 | import torch.nn.functional as F 8 | 9 | input = torch.tensor([[1,2,0,3,1], 10 | [0,1,2,3,1], 11 | [1,2,1,0,0,], 12 | [5,2,3,1,1], 13 | [2,1,0,1,1]]) 14 | 15 | kernel = torch.tensor([[1,2,1], 16 | [0,1,0], 17 | [2,1,0]]) 18 | 19 | input = torch.reshape(input, (1, 1, 5, 5)) 20 | kernel = torch.reshape(kernel, (1, 1, 3, 3)) 21 | 22 | print(input.shape) 23 | print(kernel.shape) 24 | 25 | output = F.conv2d(input, kernel, stride=1) 26 | print(output) 27 | 28 | output2 = F.conv2d(input, kernel, stride=2) 29 | print(output2) 30 | 31 | output3 = F.conv2d(input, kernel, stride=1, padding=1) 32 | print(output3) -------------------------------------------------------------------------------- /scripts_xiaotudui/p23_nn_loss.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/7下午9:12 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | import torch 6 | from torch.nn import L1Loss 7 | from torch import nn 8 | 9 | input = torch.tensor([1, 2, 3], dtype=torch.float32) 10 | target = torch.tensor([1, 2, 5], dtype=torch.float32) 11 | 12 | input = torch.reshape(input, (1, 1, 1, 3)) 13 | target = torch.reshape(target, (1, 1, 1, 3)) 14 | 15 | # L1Loss 16 | loss = L1Loss(reduction="sum") 17 | result = loss(input, target) 18 | 19 | # MSELoss 20 | loss_mse = nn.MSELoss() 21 | result_mse = loss_mse(input, target) 22 | 23 | print(result) 24 | print(result_mse) 25 | 26 | # CrossEntropyLoss 27 | x = torch.tensor([0.1, 0.2, 0.3]) 28 | y = torch.tensor([1]) 29 | x = torch.reshape(x, (1, 3)) 30 | loss_cross = nn.CrossEntropyLoss() 31 | result_cross = loss_cross(x, y) 32 | print(result_cross) 33 | 34 | 35 | 36 | -------------------------------------------------------------------------------- /scripts_xiaotudui/model.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/8上午11:26 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | import torch 6 | from torch import nn 7 | 8 | 9 | class Tudui(nn.Module): 10 | def __init__(self): 11 | super(Tudui, self).__init__() 12 | self.model1 = nn.Sequential( 13 | nn.Conv2d(3, 32, 5, 1, 2), 14 | nn.MaxPool2d(2), 15 | nn.Conv2d(32, 32, 5, 1, 2), 16 | nn.MaxPool2d(2), 17 | nn.Conv2d(32, 64, 5, 1, 2), 18 | nn.MaxPool2d(2), 19 | nn.Flatten(), 20 | nn.Linear(64 * 4 * 4, 64), 21 | nn.Linear(64, 10) 22 | ) 23 | 24 | def forward(self, x): 25 | x = self.model1(x) 26 | return x 27 | 28 | if __name__ == "__main__": 29 | tudui = Tudui() 30 | input = torch.ones((64, 3, 32, 32)) 31 | output = tudui(input) 32 | print(output.shape) 33 | -------------------------------------------------------------------------------- /scripts_xiaotudui/UsefulTransform_2.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torchvision 3 | from PIL import Image 4 | from torchvision import transforms 5 | from torch.utils.tensorboard import SummaryWriter 6 | 7 | writer = SummaryWriter("../logs/transforms") 8 | img = Image.open("../dataset/train/ants_image/0013035.jpg") 9 | print(img) 10 | 11 | # ToTensor的使用 12 | trans_totensor = transforms.ToTensor() 13 | img_tensor = trans_totensor(img) 14 | writer.add_image("ToTensor", img_tensor) 15 | 16 | # Normalize 17 | # output[channel] = (input[channel] - mean[channel]) / std[channel] 18 | print(img_tensor[0][0][0]) 19 | # trans_norm = transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5]) 20 | # trans_norm = transforms.Normalize([1, 3, 5], [3, 2, 1]) 21 | trans_norm = transforms.Normalize([3, 6, 8], [2, 5, 8]) 22 | img_norm = trans_norm(img_tensor) 23 | print(img_norm[0][0][0]) 24 | writer.add_image("Normalize", img_norm, 2) 25 | 26 | writer.close() 27 | 28 | -------------------------------------------------------------------------------- /scripts_xiaotudui/dataLoader.py: -------------------------------------------------------------------------------- 1 | import torchvision 2 | from torch.utils.data import DataLoader 3 | from torch.utils.tensorboard import SummaryWriter 4 | 5 | # 准备的测试数据集 6 | test_dataset = torchvision.datasets.CIFAR10("../dataset", train=False, transform=torchvision.transforms.ToTensor()) 7 | test_loader = DataLoader(dataset=test_dataset, batch_size=64, shuffle=True, num_workers=0, drop_last=False) 8 | 9 | # 测试数据集中的第一张图片及target 10 | # img, target = test_dataset[0] 11 | # print(img.shape) 12 | # print(target) 13 | # print(test_dataset.classes[target]) 14 | 15 | # tensorboard 查看数据 16 | writer = SummaryWriter("../logs") 17 | for epoch in range(2): 18 | step = 0 19 | for data in test_loader: 20 | imgs, targets = data 21 | # print(imgs) 22 | # print(targets) 23 | # writer.add_images("test_loader", imgs, step) 24 | writer.add_images("epoch: {}".format(epoch), imgs, step) 25 | step = step + 1 26 | 27 | writer.close() 28 | 29 | -------------------------------------------------------------------------------- /scripts_xiaotudui/test.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/8下午5:22 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | 6 | import torch 7 | import torchvision 8 | from PIL import Image 9 | from torch import nn 10 | from model import * 11 | 12 | img_path = "../imgs/dog.png" 13 | img = Image.open(img_path) 14 | print(img) 15 | img = img.convert("RGB") # png格式图片有四个通道，除了RGB还有一个透明度；调用这步可以保留其颜色通道。 16 | # 如果图片本来就是三通道，加上此操作，不影响。以后可以直接加上，以适应所有图片。 17 | transform = torchvision.transforms.Compose([torchvision.transforms.Resize((32, 32)), 18 | torchvision.transforms.ToTensor()]) 19 | 20 | img = transform(img) 21 | print(img.shape) 22 | 23 | # 导入模型 -- 由于模型保存使用的方式1，所以需要导入模型 24 | # from model import * 25 | 26 | model = torch.load("../models/tudui_4.pth", map_location=torch.device("cpu")) 27 | print(model) 28 | 29 | img = torch.reshape(img, (1, 3, 32, 32)) 30 | 31 | model.eval() 32 | with torch.no_grad(): 33 | output = model(img) 34 | print(output) 35 | 36 | print(output.argmax(1)) 37 | 38 | -------------------------------------------------------------------------------- /scripts_xiaotudui/dataset_transform.py: -------------------------------------------------------------------------------- 1 | # p14 torchvision 中的数据集使用 2 | import cv2 3 | import torchvision 4 | from PIL import Image 5 | from torch.utils.tensorboard import SummaryWriter 6 | 7 | 8 | dataset_transform = torchvision.transforms.Compose([ 9 | torchvision.transforms.ToTensor() 10 | ]) 11 | 12 | train_daset = torchvision.datasets.CIFAR10(root="../dataset", train=True, download=True, transform=dataset_transform) 13 | test_daset = torchvision.datasets.CIFAR10(root="../dataset", train=False, download=True, transform=dataset_transform) 14 | 15 | # print(test_daset[0]) 16 | # print(test_daset.classes) 17 | # 18 | # img, target = test_daset[0] 19 | # print(img) 20 | # print(target) 21 | # print(test_daset.classes[target]) 22 | # img.show() # PIL 23 | # 24 | # print(test_daset[0]) 25 | 26 | writer = SummaryWriter(log_dir="../logs") 27 | for i in range(10): 28 | img, target = test_daset[i] 29 | # writer.add_image(tag="test_dataset", img_tensor=img, global_step=i) 30 | writer.add_image("test_dataset", img, i) 31 | 32 | writer.close() 33 | 34 | -------------------------------------------------------------------------------- /scripts_xiaotudui/read_data.py: -------------------------------------------------------------------------------- 1 | from torch.utils.data import Dataset 2 | # import cv2 3 | from PIL import Image 4 | import os 5 | 6 | 7 | 8 | class MyData(Dataset): 9 | # class MyData(): 10 | 11 | def __init__(self, root_dir, label_dir): # 初始化类 12 | self.root_dir = root_dir 13 | self.label_dir = label_dir 14 | self.path = os.path.join(self.root_dir, self.label_dir) 15 | self.img_path = os.listdir(self.path) 16 | 17 | def __getitem__(self, idx): 18 | img_name = self.img_path[idx] 19 | img_item_path = os.path.join(self.root_dir, self.label_dir, img_name) 20 | img = Image.open(img_item_path) 21 | label = self.label_dir 22 | return img, label 23 | 24 | def __len__(self): 25 | return len(self.img_path) 26 | 27 | 28 | root_dir = "./dataset/train" 29 | ants_label_dir = "ants" 30 | bees_label_dir = "bees" 31 | ants_dataset = MyData(root_dir, ants_label_dir) 32 | bees_dataset = MyData(root_dir, bees_label_dir) 33 | 34 | 35 | train_dataset = ants_dataset + bees_dataset # 数据集拼接在一起 36 | 37 | -------------------------------------------------------------------------------- /scripts_xiaotudui/p26_model_load.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/8上午9:55 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | import torch 6 | import torchvision 7 | from p26_model_save import * 8 | 9 | 10 | # 方式1 -> 使用保存方式1来加载模型 11 | model = torch.load("../models/vgg16_method1.pth") 12 | # print(model) 13 | 14 | # 方式2 -> 使用保存方式2来加载模型 15 | vgg16 = torchvision.models.vgg16(pretrained=False) 16 | vgg16.load_state_dict(torch.load("../models/vgg16_method2.pth")) 17 | # model = torch.load("../models/vgg_method2.pth") 18 | # print(vgg16) 19 | 20 | 21 | # 陷阱 -- 使用方式1加载 22 | # 这种方式需要加载时把 Tudui 这个类加载进来，但是不需要再创建（tudui = Tudui()） 23 | 24 | # 解决方式1 -- 把class Tudui(模型的定义) 复制过来 25 | # class Tudui(nn.Module): 26 | # def __init__(self): 27 | # super(Tudui, self).__init__() 28 | # self.conv1 = nn.Conv2d(3, 64, kernel_size=3) 29 | # 30 | # def forward(self, x): 31 | # x = self.conv1(x) 32 | # return x 33 | 34 | # 解决方式2 -- 直接从源文件中导入class 35 | # from p26_model_save import * 36 | 37 | 38 | model = torch.load("../models/tudui_method1.pth") 39 | print(model) 40 | 41 | -------------------------------------------------------------------------------- /scripts_xiaotudui/p24_nn_optim.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/7下午10:13 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | import torch 6 | import torchvision 7 | from torch.utils.data import DataLoader 8 | from p22_nn_seq import Tudui 9 | from torch import nn 10 | 11 | 12 | dataset = torchvision.datasets.CIFAR10(root="../dataset", train=True, 13 | transform=torchvision.transforms.ToTensor(), 14 | download=True) 15 | dataloader = DataLoader(dataset, batch_size=1) 16 | 17 | tudui = Tudui() 18 | 19 | loss = nn.CrossEntropyLoss() 20 | optimer = torch.optim.SGD(tudui.parameters(), lr=0.01, momentum=0.9) # 定义优化器 21 | for epoch in range(20): 22 | running_loss = 0.0 23 | for data in dataloader: 24 | imgs, targets = data 25 | outputs = tudui(imgs) 26 | result_loss = loss(outputs, targets) 27 | optimer.zero_grad() # 优化器中每一个梯度参数清零 28 | result_loss.backward() # 反向传播求梯度 29 | optimer.step() # 对参数进行调优 30 | running_loss = running_loss + result_loss 31 | print(running_loss) -------------------------------------------------------------------------------- /scripts_liu/p2_linearModel.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/20下午9:28 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | 6 | # 1、函数forward()中，有一个变量w。这个变量最终的值是从for循环中传入的。 7 | # 2、for循环中，使用了np.arange()。若对numpy不太熟悉，传送门Numpy数据计算从入门到实战(https://www.bilibili.com/video/BV1U7411x76j?p=5) 8 | # 3、python中zip()函数的用法 9 | 10 | import numpy as np 11 | import matplotlib.pyplot as plt 12 | 13 | x_data = [1.0, 2.0, 3.0] 14 | y_data = [2.0, 4.0, 6.0] 15 | 16 | 17 | def forward(x): 18 | return x * w 19 | 20 | 21 | def loss(x, y): 22 | y_pred = forward(x) 23 | return (y_pred - y) * (y_pred - y) 24 | 25 | 26 | w_list = [] 27 | mse_list = [] 28 | for w in np.arange(0.0, 4.1, 0.1): 29 | print("w=", w) 30 | l_sum = 0 31 | for x_val, y_val in zip(x_data, y_data): 32 | y_pred_val = forward(x_val) 33 | loss_val = loss(x_val, y_val) 34 | l_sum += loss_val 35 | print('\t', x_val, y_val, y_pred_val, loss_val) 36 | print('MSE=', l_sum / 3) 37 | w_list.append(w) 38 | mse_list.append(l_sum / 3) 39 | 40 | plt.plot(w_list, mse_list) 41 | plt.ylabel("Loss") 42 | plt.xlabel("w") 43 | plt.show() 44 | -------------------------------------------------------------------------------- /scripts_xiaotudui/nn_linear.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/7下午6:24 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | import torch 6 | import torchvision 7 | from torch.utils.data import DataLoader 8 | from torch import nn 9 | from torch.nn import Linear 10 | 11 | dataset = torchvision.datasets.CIFAR10("../dataset", train=False, download=True, 12 | transform=torchvision.transforms.ToTensor()) 13 | dataloader = DataLoader(dataset, batch_size=64) 14 | 15 | # for data in dataloader: 16 | # imgs, targets = data 17 | # print(imgs.shape) 18 | # output = torch.reshape(imgs, (1, 1, 1, -1)) 19 | # print(output.shape) 20 | 21 | class Tudui(nn.Module): 22 | def __init__(self): 23 | super(Tudui, self).__init__() 24 | self.linear1 = Linear(196608, 10) 25 | 26 | def forward(self, input): 27 | output = self.linear1(input) 28 | return output 29 | 30 | tudui = Tudui() 31 | 32 | for data in dataloader: 33 | imgs, targets = data 34 | print(imgs.shape) 35 | output = torch.flatten(imgs) 36 | print(output.shape) 37 | output = tudui(output) 38 | print(output.shape) 39 | -------------------------------------------------------------------------------- /scripts_liu/p2_exercise.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/20下午10:11 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | 6 | # Numpy 7 | import numpy 8 | # For plotting 9 | import matplotlib.pyplot as plt 10 | from matplotlib import cm 11 | from mpl_toolkits.mplot3d import Axes3D 12 | 13 | x_data = [1.0, 2.0, 3.0] 14 | y_data = [2.0, 4.0, 6.0] 15 | 16 | 17 | def forward(w: numpy.ndarray, b: numpy.ndarray, x: float) -> numpy.ndarray: 18 | return w * x + b 19 | 20 | 21 | def loss(y_hat: numpy.ndarray, y: float) -> numpy.ndarray: 22 | return (y_hat - y) ** 2 23 | 24 | 25 | w_cor = numpy.arange(0.0, 4.0, 0.1) 26 | b_cor = numpy.arange(-2.0, 2.1, 0.1) 27 | 28 | # 此处直接使用矩阵进行计算 29 | w, b = numpy.meshgrid(w_cor, b_cor) 30 | mse = numpy.zeros(w.shape) 31 | 32 | for x, y in zip(x_data, y_data): 33 | _y = forward(w, b, x) 34 | mse += loss(_y, y) 35 | mse /= len(x_data) 36 | 37 | h = plt.contourf(w, b, mse) 38 | 39 | fig = plt.figure() 40 | ax = Axes3D(fig) 41 | plt.xlabel(r'w', fontsize=20, color='cyan') 42 | plt.ylabel(r'b', fontsize=20, color='cyan') 43 | ax.plot_surface(w, b, mse, rstride=1, cstride=1, cmap=plt.get_cmap('rainbow')) 44 | plt.show() 45 | 46 | 47 | 48 | 49 | 50 | -------------------------------------------------------------------------------- /scripts_xiaotudui/nn_relu.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/7上午10:00 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | import torch 6 | import torchvision 7 | from torch.nn import ReLU, Sigmoid 8 | from torch.utils.data import DataLoader 9 | from torch import nn 10 | from torch.utils.tensorboard import SummaryWriter 11 | 12 | input = torch.tensor([[1, -0.5], 13 | [-1, 3]]) 14 | 15 | input = torch.reshape(input, (-1, 1, 2, 2)) 16 | print(input.shape) 17 | 18 | dataset = torchvision.datasets.CIFAR10(root="../dataset", train=False, download=True, 19 | transform=torchvision.transforms.ToTensor()) 20 | 21 | dataloader = DataLoader(dataset, batch_size=64) 22 | 23 | class Tudui(nn.Module): 24 | def __init__(self): 25 | super(Tudui, self).__init__() 26 | self.relu1 = ReLU() 27 | self.sigmoid1 = Sigmoid() 28 | 29 | def forward(self, input): 30 | output = self.sigmoid1(input) 31 | return output 32 | 33 | tudui = Tudui() 34 | 35 | writer = SummaryWriter("../logs") 36 | step = 0 37 | for data in dataloader: 38 | imgs, targets = data 39 | writer.add_images("input", imgs, global_step=step) 40 | output = tudui(imgs) 41 | writer.add_images("output", output, step) 42 | step += 1 43 | 44 | writer.close() 45 | 46 | 47 | -------------------------------------------------------------------------------- /scripts_liu/p4_exercise.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/21上午10:24 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | 6 | import numpy as np 7 | import torch 8 | import matplotlib.pyplot as plt 9 | 10 | x_data = [1.0, 2.0, 3.0] 11 | y_data = [2.0, 4.0, 6.0] 12 | 13 | w1 = torch.tensor([1.0]) 14 | w1.requires_grad = True 15 | w2 = torch.tensor([1.0]) 16 | w2.requires_grad = True 17 | b = torch.tensor([1.0]) 18 | b.requires_grad = True 19 | 20 | 21 | def forward(x): 22 | return w1 * x ** 2 + w2 * x + b 23 | 24 | 25 | def loss(x, y): 26 | y_pred = forward(x) 27 | return (y_pred - y) ** 2 28 | 29 | 30 | print('Predit (before training)', 4, forward(4)) 31 | 32 | for epoch in range(1000): 33 | l = loss(1, 2) # 为了在for循环之前定义l,以便之后的输出，无实际意义 34 | for x, y in zip(x_data, y_data): 35 | l = loss(x, y) 36 | l.backward() 37 | print('\tgrad', x, y, w1.grad.item(), w2.grad.item(), b.grad.item()) 38 | w1.data = w1.data - 0.01 * w1.grad.data # 注意这里的grad是一个tensor，所以要取他的data 39 | w2.data = w2.data - 0.01 * w2.grad.data 40 | b.data = b.data - 0.01 * b.grad.data 41 | w1.grad.data.zero_() # 释放之前计算的梯度 42 | w2.grad.data.zero_() 43 | b.grad.data.zero_() 44 | 45 | print('Epoch:', epoch, l.item()) 46 | 47 | print('Predit (after training', 4, forward(4).item()) 48 | 49 | -------------------------------------------------------------------------------- /scripts_liu/p3_gradientDescent.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/20下午10:42 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | 6 | import matplotlib.pyplot as plt 7 | 8 | 9 | # prepare the training set 10 | x_data = [1., 2., 3.] 11 | y_data = [2., 4., 6.] 12 | 13 | # initial guess of weight 14 | w = 1 15 | 16 | 17 | # define the model linear model y = w*x 18 | def forward(x): 19 | return x * w 20 | 21 | 22 | # define the cost function MSE 23 | def cost(xs, ys): 24 | cost = 0 25 | for x, y in zip(xs, ys): 26 | y_pred = forward(x) 27 | cost += (y_pred - y) ** 2 28 | return cost / len(xs) 29 | 30 | 31 | # define the gradient function gd 32 | def gradient(xs, ys): 33 | grad = 0 34 | for x, y in zip(xs, ys): 35 | grad += 2 * x * (x * w - y) 36 | return grad / len(xs) 37 | 38 | 39 | epoch_list = [] 40 | cost_list = [] 41 | 42 | print("Predit (before training)", 4, forward(4)) 43 | for epoch in range(100): 44 | cost_val = cost(x_data, y_data) 45 | grad_val = gradient(x_data, y_data) 46 | w -= 0.01 * grad_val 47 | print('Epoch:', epoch, "w=", w, 'loss=', cost_val) 48 | 49 | epoch_list.append(epoch) 50 | cost_list.append(cost_val) 51 | 52 | print('Predit (after training)', 4, forward(4)) 53 | 54 | plt.plot(epoch_list, cost_list) 55 | plt.ylabel('cost') 56 | plt.xlabel('epoch') 57 | plt.show() -------------------------------------------------------------------------------- /scripts_liu/p3_SGD.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/20下午11:00 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | 6 | # 随机梯度下降法和梯度下降法的主要区别在于： 7 | # 1、损失函数由cost()更改为loss()。cost是计算所有训练数据的损失，loss是计算一个训练函数的损失。对应于源代码则是少了两个for循环。 8 | # 2、梯度函数gradient()由计算所有训练数据的梯度更改为计算一个训练数据的梯度。 9 | # 3、本算法中的随机梯度主要是指，每次拿一个训练数据来训练，然后更新梯度参数。本算法中梯度总共更新100(epoch)x3 = 300次。梯度下降法中梯度总共更新100(epoch)次。 10 | # 4. 综合梯度下降和随机梯度下降算法，折中：batch（mini-patch）-- 小批量随机梯度下降 11 | import matplotlib.pyplot as plt 12 | 13 | x_data = [1.0, 2.0, 3.0] 14 | y_data = [2.0, 4.0, 6.0] 15 | 16 | w = 1.0 17 | 18 | 19 | def forward(x): 20 | return x * w 21 | 22 | 23 | # calculate loss function 24 | def loss(x, y): 25 | y_pred = forward(x) 26 | return (y_pred - y) ** 2 27 | 28 | 29 | # define the gradient function sgd 30 | def gradient(x, y): 31 | return 2 * x * (x * w - y) 32 | 33 | 34 | epoch_list = [] 35 | loss_list = [] 36 | print('predict (before training)', 4, forward(4)) 37 | for epoch in range(100): 38 | for x, y in zip(x_data, y_data): 39 | grad = gradient(x, y) 40 | w = w - 0.01 * grad # update weight by every grad of sample of training set 41 | print("\tgrad:", x, y, grad) 42 | l = loss(x, y) 43 | print("progress:", epoch, "w=", w, "loss=", l) 44 | epoch_list.append(epoch) 45 | loss_list.append(l) 46 | 47 | print('predict (after training)', 4, forward(4)) 48 | plt.plot(epoch_list, loss_list) 49 | plt.ylabel('loss') 50 | plt.xlabel('epoch') 51 | plt.show() -------------------------------------------------------------------------------- /scripts_xiaotudui/nn_maxpool.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/6下午11:42 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | 6 | import torch 7 | import torchvision 8 | from torch import nn 9 | from torch.nn import MaxPool2d 10 | from torch.utils.data import DataLoader 11 | from torch.utils.tensorboard import SummaryWriter 12 | 13 | dataset = torchvision.datasets.CIFAR10(root="../dataset", train=False, download=True, 14 | transform=torchvision.transforms.ToTensor()) 15 | 16 | dataloader = DataLoader(dataset, batch_size=64) 17 | 18 | # input = torch.tensor([[1,2,0,3,1], 19 | # [0,1,2,3,1], 20 | # [1,2,1,0,0], 21 | # [5,2,3,1,1], 22 | # [2,1,0,1,1]],dtype=torch.float32) 23 | # 24 | # input = torch.reshape(input, (-1, 1, 5, 5)) 25 | # print(input.shape) 26 | 27 | class Tudui(nn.Module): 28 | def __init__(self): 29 | super(Tudui, self).__init__() 30 | self.maxpool1 = MaxPool2d(kernel_size=3, ceil_mode=False) 31 | 32 | def forward(self, input): 33 | output = self.maxpool1(input) 34 | return output 35 | 36 | tudui = Tudui() 37 | # output = tudui(input) 38 | # print('输出是：', output) 39 | 40 | writer = SummaryWriter("../logs/maxpools") 41 | step = 0 42 | for data in dataloader: 43 | imgs, targets = data 44 | writer.add_images("input", imgs, step) 45 | output = tudui(imgs) 46 | writer.add_images("output", output, step) 47 | step = step + 1 48 | 49 | writer.close() 50 | writer.add_image() -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # 这是我的pytorch学习笔记 2 | 3 | ## 一、 学习进度 4 | **pytroch学习进度** 5 | > 1. pytorch官方60分钟入门中文翻译版； 6 | > -- 已完成 7 | > 2. B站：小土堆，PyTorch深度学习快速入门教程 8 | > -- 进行中 --> 完成：2022年01月08日17:52:46 9 | > 3. B站：Pytorch深度学习实战教学_唐宇迪 10 | > -- 进行中 11 | > start：2022年01月20日17:20:12 -- p1-p4 12 | > 看评论好像不友好对新手，暂时搁浅。。。 13 | > 4. B站：《PyTorch深度学习实践》完结合集 -- 刘二大人 14 | > -- 进行中 15 | > -- start：2022年01月20日18:30:15 16 | > -- 2022年01月20日23:13:58 --> p1-p3 17 | 18 | 19 | 20 | 21 | ## 二、 资源链接 22 | ### 1. Deep Learning with PyTorch: A 60 Minute Blitz 23 | > https://github.com/fengdu78/Data-Science-Notes/tree/master/8.deep-learning/PyTorch_beginner 24 | > 从零开始的Pytorch官方入门新手教程:https://www.bilibili.com/video/BV13Q4y1T7CD 25 | 26 | ### 2. PyTorch深度学习快速入门教程（绝对通俗易懂！）【小土堆】 27 | 资料来源于B站：小土堆 28 | 学习视频地址：https://www.bilibili.com/video/BV1hE411t7RN 29 | CSDN笔记：https://blog.csdn.net/sinat_39448069/article/details/120866541 30 | 31 | ### 3. pytorch快速入门 32 | > https://www.bilibili.com/video/BV1iv41117Zg 33 | 34 | ### 4. Pytorch深度学习实战教学 - 唐宇迪 35 | > https://www.bilibili.com/video/BV1Zv4y1o7uG?p=2 36 | 37 | ### 5. 《PyTorch深度学习实践》完结合集 - Hongpu Liu 38 | > https://www.bilibili.com/video/BV1Y7411d7Ys 39 | > 课后练习：https://blog.csdn.net/bit452/category_10569531.html 40 | 41 | ## 三、番外篇 42 | **唐宇迪聊工程师** 43 | 入门 -- python 44 | - 1. python快速入门 45 | - 2. 科学计算库 - Numpy 46 | - 3. 数据分析处理库 - Pandas 47 | - 4. 可视化库 - Matplotlib 48 | - 5. 可视化库 - Seaborn 49 | 50 | - opencv -- 工具，，知道有折磨回事就好 51 | 目标检测实践项目 52 | 图像分割 53 | 行为识别 -- slowfast 54 | 55 | transformer -- 可能是未来的趋势； cv & nlp 56 | 缺点：无法解决数据少的场景; 57 | 58 | 知识图谱 ： cv & nlp -- 可学 -------------------------------------------------------------------------------- /scripts_xiaotudui/p22_nn_seq.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/7下午7:07 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | 6 | import torch 7 | from torch import nn 8 | from torch.nn import Conv2d, MaxPool2d, Flatten, Linear, Sequential 9 | from torch.utils.tensorboard import SummaryWriter 10 | 11 | 12 | class Tudui(nn.Module): 13 | def __init__(self): 14 | super(Tudui, self).__init__() 15 | # self.conv1 = Conv2d(3, 32, 5, padding=2) 16 | # self.maxpool1 = MaxPool2d(2) 17 | # self.conv2 = Conv2d(32, 32, 5, padding=2) 18 | # self.maxpool2 = MaxPool2d(2) 19 | # self.conv3 = Conv2d(32, 64, 5, padding=2) 20 | # self.maxpool3 = MaxPool2d(2) 21 | # self.flatten = Flatten() 22 | # self.linear1 = Linear(1024, 64) 23 | # self.linear2 = Linear(64, 10) 24 | 25 | self.model1 = Sequential( 26 | Conv2d(3, 32, 5, padding=2), 27 | MaxPool2d(2), 28 | Conv2d(32, 32, 5, padding=2), 29 | MaxPool2d(2), 30 | Conv2d(32, 64, 5, padding=2), 31 | MaxPool2d(2), 32 | Flatten(), 33 | Linear(1024, 64), 34 | Linear(64, 10) 35 | ) 36 | def forward(self, x): 37 | # x = self.conv1(x) 38 | # x = self.maxpool1(x) 39 | # x = self.conv2(x) 40 | # x = self.maxpool2(x) 41 | # x = self.conv3(x) 42 | # x = self.maxpool3(x) 43 | # x = self.flatten(x) 44 | # x = self.linear1(x) 45 | # x = self.linear2(x) 46 | 47 | x = self.model1(x) 48 | return x 49 | 50 | tudui = Tudui() 51 | 52 | print(tudui) 53 | input = torch.ones((64, 3, 32, 32)) 54 | output = tudui(input) 55 | print(output.shape) 56 | 57 | # tensorboard 58 | writer = SummaryWriter(log_dir="../logs") 59 | writer.add_graph(tudui, input) 60 | writer.close() -------------------------------------------------------------------------------- /scripts_liu/p4_backPropagation.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/21上午9:33 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | 6 | # 【PyTorch】Tensor和tensor的区别:https://blog.csdn.net/tfcy694/article/details/85338745 7 | 8 | # 1. w是Tensor(张量类型)，Tensor中包含data和grad，data和grad也是Tensor。\ 9 | # grad初始为None，调用l.backward()方法后w.grad为Tensor，故更新w.data时需使用w.grad.data。\ 10 | # 如果w需要计算梯度，那构建的计算图中，跟w相关的tensor都默认需要计算梯度。 11 | 12 | # a = torch.tensor([1.0]) 13 | # a.requires_grad = True # 或者 a.requires_grad_() 14 | # print(a) 15 | # print(a.data) 16 | # print(a.type()) # a的类型是tensor 17 | # print(a.data.type()) # a.data 的类型是tensor 18 | # print(a.grad) 19 | # print(type(a.grad)) 20 | 21 | 22 | # 2. w是Tensor， forward函数的返回值也是Tensor，loss函数的返回值也是Tensor; 23 | # 3. 本算法中反向传播主要体现在，l.backward()。调用该方法后w.grad由None更新为Tensor类型，\ 24 | # 且w.grad.data的值用于后续w.data的更新。 25 | # l.backward()会把计算图中所有需要梯度(grad)的地方都会求出来，然后把梯度都存在对应的待求的参数中，最终计算图被释放。 26 | # 取tensor中的data是不会构建计算图的。 27 | 28 | 29 | import torch 30 | 31 | x_data = [1.0, 2.0, 3.0] 32 | y_data = [2.0, 4.0, 6.0] 33 | 34 | w = torch.tensor([1.0]) # w的初始值为1.0 35 | w.requires_grad = True # 计算梯度,默认不计算 36 | 37 | 38 | def forward(x): 39 | return x * w # w 是一个tensor 40 | 41 | 42 | def loss(x, y): # 构建计算图 43 | y_pred = forward(x) 44 | return (y_pred - y) ** 2 45 | 46 | 47 | print('Predit (before training)', 4, forward(4).item()) 48 | 49 | for epoch in range(100): 50 | for x, y in zip(x_data, y_data): 51 | l = loss(x, y) # l是一个张量，tensor主要是在建立计算图 forward, compute the loss 52 | l.backward() # backward,compute grad for Tensor whose requires_grad set to True 53 | print('\t grad:', x, y, w.grad.item()) 54 | w.data = w.data - 0.01 * w.grad.data # 权重更新时，注意这里的grad是一个tensor，所以要取他的data 55 | 56 | w.grad.data.zero_() # 释放之前计算的梯度 57 | 58 | print('progress:', epoch, l.item()) # 取出loss使用l.item，不要直接使用l（l是tensor会构建计算图） 59 | 60 | print('Predit (after training)', 4, forward(4).item()) 61 | 62 | 63 | 64 | 65 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | pip-wheel-metadata/ 24 | share/python-wheels/ 25 | *.egg-info/ 26 | .installed.cfg 27 | *.egg 28 | MANIFEST 29 | 30 | # PyInstaller 31 | # Usually these files are written by a python script from a template 32 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 33 | *.manifest 34 | *.spec 35 | 36 | # Installer logs 37 | pip-log.txt 38 | pip-delete-this-directory.txt 39 | 40 | # Unit test / coverage reports 41 | htmlcov/ 42 | .tox/ 43 | .nox/ 44 | .coverage 45 | .coverage.* 46 | .cache 47 | nosetests.xml 48 | coverage.xml 49 | *.cover 50 | *.py,cover 51 | .hypothesis/ 52 | .pytest_cache/ 53 | 54 | # Translations 55 | *.mo 56 | *.pot 57 | 58 | # Django stuff: 59 | *.log 60 | local_settings.py 61 | db.sqlite3 62 | db.sqlite3-journal 63 | 64 | # Flask stuff: 65 | instance/ 66 | .webassets-cache 67 | 68 | # Scrapy stuff: 69 | .scrapy 70 | 71 | # Sphinx documentation 72 | docs/_build/ 73 | 74 | # PyBuilder 75 | target/ 76 | 77 | # Jupyter Notebook 78 | .ipynb_checkpoints 79 | 80 | # IPython 81 | profile_default/ 82 | ipython_config.py 83 | 84 | # pyenv 85 | .python-version 86 | 87 | # pipenv 88 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 89 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 90 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 91 | # install all needed dependencies. 92 | #Pipfile.lock 93 | 94 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow 95 | __pypackages__/ 96 | 97 | # Celery stuff 98 | celerybeat-schedule 99 | celerybeat.pid 100 | 101 | # SageMath parsed files 102 | *.sage.py 103 | 104 | # Environments 105 | .env 106 | .venv 107 | env/ 108 | venv/ 109 | ENV/ 110 | env.bak/ 111 | venv.bak/ 112 | 113 | # Spyder project settings 114 | .spyderproject 115 | .spyproject 116 | 117 | # Rope project settings 118 | .ropeproject 119 | 120 | # mkdocs documentation 121 | /site 122 | 123 | # mypy 124 | .mypy_cache/ 125 | .dmypy.json 126 | dmypy.json 127 | 128 | # Pyre type checker 129 | .pyre/ 130 | 131 | 132 | .idea/ 133 | .ipynb_checkpoints 134 | dataset/cifar* 135 | 136 | logs 137 | models -------------------------------------------------------------------------------- /scripts_xiaotudui/train.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/8上午10:35 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | 6 | import torch 7 | import torchvision 8 | from torch import nn 9 | from torch.utils.data import DataLoader 10 | from torch.utils.tensorboard import SummaryWriter 11 | 12 | from model import * 13 | 14 | 15 | # 准备数据集 16 | train_dataset = torchvision.datasets.CIFAR10(root="../dataset", train=True, download=True, 17 | transform=torchvision.transforms.ToTensor()) 18 | test_dataset = torchvision.datasets.CIFAR10(root="../dataset", train=False, download=True, 19 | transform=torchvision.transforms.ToTensor()) 20 | 21 | # 使用DataLoader来加载数据集 22 | train_dataloader = DataLoader(train_dataset, batch_size=64) 23 | test_dataloader = DataLoader(test_dataset, batch_size=64) 24 | 25 | # 数据集的长度 -- length 26 | train_data_size = len(train_dataset) 27 | test_data_size = len(test_dataset) 28 | print("训练数据集的长度为：{}".format(train_data_size)) 29 | print("测试数据集的长度为：{}".format(test_data_size)) 30 | 31 | # 创建网络模型 32 | # 方式1：从模型文件导入（推荐） 33 | # from model import * 34 | 35 | # 方式2：直接定义网络模型 36 | class Tudui(nn.Module): 37 | def __init__(self): 38 | super(Tudui, self).__init__() 39 | self.model1 = nn.Sequential( 40 | nn.Conv2d(3, 32, 5, 1, 2), 41 | nn.MaxPool2d(2), 42 | nn.Conv2d(32, 32, 5, 1, 2), 43 | nn.MaxPool2d(2), 44 | nn.Conv2d(32, 64, 5, 1, 2), 45 | nn.MaxPool2d(2), 46 | nn.Flatten(), 47 | nn.Linear(64 * 4 * 4, 64), 48 | nn.Linear(64, 10) 49 | ) 50 | 51 | def forward(self, x): 52 | x= self.model1(x) 53 | return x 54 | 55 | tudui= Tudui() 56 | 57 | # 损失函数 58 | loss_fn = nn.CrossEntropyLoss() 59 | 60 | # 优化器 61 | learning_rate = 1e-2 62 | optimizer = torch.optim.SGD(tudui.parameters(), lr=learning_rate) 63 | 64 | # 设置训练网络的一些参数 65 | # 记录训练次数 66 | total_train_step = 0 67 | # 记录测试次数 68 | total_test_step = 0 69 | # 训练轮数 70 | epoch = 5 71 | 72 | # add tensorboard logs 73 | writer = SummaryWriter(log_dir="../logs/train") 74 | 75 | for i in range(epoch): 76 | print("--------第{}轮训练开始------".format(i+1)) 77 | 78 | # 训练步骤开始 79 | tudui.train() # 网络模型模式设置，pytorch官网有讲解，对Dropout、BN等起作用。 80 | for data in train_dataloader: 81 | imgs, targets = data 82 | outputs = tudui(imgs) 83 | loss = loss_fn(outputs, targets) 84 | 85 | # 优化器优化模型 86 | optimizer.zero_grad() 87 | loss.backward() 88 | optimizer.step() 89 | 90 | total_train_step = total_train_step + 1 91 | if total_train_step % 100 == 0: 92 | print("训练次数：{}, loss:{}".format(total_train_step, loss.item())) 93 | writer.add_scalar("train_loss", loss.item(), global_step=total_train_step) 94 | 95 | # 测试步骤开始 96 | tudui.eval() 97 | total_test_loss = 0 98 | total_accuracy = 0 99 | with torch.no_grad(): 100 | for data in test_dataloader: 101 | imgs, targets = data 102 | outputs = tudui(imgs) 103 | loss = loss_fn(outputs, targets) 104 | total_test_loss = total_test_loss + loss.item() 105 | accuracy = (outputs.argmax(1) == targets).sum() 106 | total_accuracy = total_accuracy + accuracy 107 | 108 | print("整体测试集上的Loss：{}".format(total_test_loss)) 109 | print("整体测试集上的准确率：{}".format(total_accuracy / test_data_size)) 110 | writer.add_scalar(tag="test_loss", scalar_value=total_test_loss, global_step=total_test_step) 111 | writer.add_scalar("test_accuracy", total_accuracy / test_data_size, total_test_step) 112 | total_test_loss += 1 113 | 114 | torch.save(tudui, "../models/tudui_{}.pth".format(i)) 115 | # torch.save(tudui.state_dict(), "../models/tudui_{}.pth".format(i)) 116 | print("模型已保存") 117 | writer.close() 118 | 119 | print("训练测试完成") 120 | 121 | # 出现问题： 122 | # 1.RuntimeError: Integer division of tensors using div or / is no longer supported, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead. 123 | # 2.测试准确率一直为零 -------------------------------------------------------------------------------- /scripts_xiaotudui/train_gpu.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/1/8上午10:35 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | 6 | # 可以放在cuda上的数据有：网络模型、数据（输入和标注）、损失函数 7 | 8 | 9 | import time 10 | import torch 11 | import torchvision 12 | from torch import nn 13 | from torch.utils.data import DataLoader 14 | from torch.utils.tensorboard import SummaryWriter 15 | 16 | # from model import * 17 | 18 | # 定义训练设备 19 | # device = torch.device("cpu") 20 | device = torch.device("cuda:0") 21 | # device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") 22 | print(device) 23 | 24 | # 准备数据集 25 | train_dataset = torchvision.datasets.CIFAR10(root="../dataset", train=True, download=True, 26 | transform=torchvision.transforms.ToTensor()) 27 | test_dataset = torchvision.datasets.CIFAR10(root="../dataset", train=False, download=True, 28 | transform=torchvision.transforms.ToTensor()) 29 | 30 | # 使用DataLoader来加载数据集 31 | train_dataloader = DataLoader(train_dataset, batch_size=64) 32 | test_dataloader = DataLoader(test_dataset, batch_size=64) 33 | 34 | # 数据集的长度 -- length 35 | train_data_size = len(train_dataset) 36 | test_data_size = len(test_dataset) 37 | print("训练数据集的长度为：{}".format(train_data_size)) 38 | print("测试数据集的长度为：{}".format(test_data_size)) 39 | 40 | # 创建网络模型 41 | # 方式1：从模型文件导入（推荐） 42 | # from model import * 43 | 44 | # 方式2：直接定义网络模型 45 | class Tudui(nn.Module): 46 | def __init__(self): 47 | super(Tudui, self).__init__() 48 | self.model1 = nn.Sequential( 49 | nn.Conv2d(3, 32, 5, 1, 2), 50 | nn.MaxPool2d(2), 51 | nn.Conv2d(32, 32, 5, 1, 2), 52 | nn.MaxPool2d(2), 53 | nn.Conv2d(32, 64, 5, 1, 2), 54 | nn.MaxPool2d(2), 55 | nn.Flatten(), 56 | nn.Linear(64 * 4 * 4, 64), 57 | nn.Linear(64, 10) 58 | ) 59 | 60 | def forward(self, x): 61 | x= self.model1(x) 62 | return x 63 | 64 | tudui = Tudui() 65 | # 方法1： 66 | # if torch.cuda.is_available(): 67 | # tudui = tudui.cuda() 68 | # print("------GPU可用，正在使用GPU训练------") 69 | # 方法2： 70 | tudui = tudui.to(device) 71 | 72 | 73 | # 损失函数 74 | loss_fn = nn.CrossEntropyLoss() 75 | # if torch.cuda.is_available(): 76 | # loss_fn = loss_fn.cuda() 77 | loss_fn = loss_fn.to(device) 78 | 79 | # 优化器 80 | learning_rate = 1e-2 81 | optimizer = torch.optim.SGD(tudui.parameters(), lr=learning_rate) 82 | 83 | # 设置训练网络的一些参数 84 | # 记录训练次数 85 | total_train_step = 0 86 | # 记录测试次数 87 | total_test_step = 0 88 | # 训练轮数 89 | epoch = 5 90 | 91 | # add tensorboard logs 92 | writer = SummaryWriter(log_dir="../logs/train") 93 | 94 | start_time = time.time() 95 | for i in range(epoch): 96 | print("--------第{}轮训练开始------".format(i+1)) 97 | 98 | # 训练步骤开始 99 | tudui.train() # 网络模型模式设置，pytorch官网有讲解，对Dropout、BN等起作用。 100 | for data in train_dataloader: 101 | imgs, targets = data 102 | # if torch.cuda.is_available(): 103 | # imgs = imgs.cuda() 104 | # targets = targets.cuda() 105 | imgs = imgs.to(device) 106 | targets = targets.to(device) 107 | 108 | outputs = tudui(imgs) 109 | loss = loss_fn(outputs, targets) 110 | 111 | # 优化器优化模型 112 | optimizer.zero_grad() 113 | loss.backward() 114 | optimizer.step() 115 | 116 | total_train_step = total_train_step + 1 117 | if total_train_step % 100 == 0: 118 | end_time = time.time() 119 | print(end_time - start_time) 120 | print("训练次数：{}, loss:{}".format(total_train_step, loss.item())) 121 | writer.add_scalar("train_loss", loss.item(), global_step=total_train_step) 122 | 123 | # 测试步骤开始 124 | tudui.eval() 125 | total_test_loss = 0 126 | total_accuracy = 0 127 | with torch.no_grad(): 128 | for data in test_dataloader: 129 | imgs, targets = data 130 | # if torch.cuda.is_available(): 131 | # imgs = imgs.cuda() 132 | # targets = targets.cuda() 133 | imgs = imgs.to(device) 134 | targets = targets.to(device) 135 | outputs = tudui(imgs) 136 | loss = loss_fn(outputs, targets) 137 | total_test_loss = total_test_loss + loss.item() 138 | accuracy = (outputs.argmax(1) == targets).sum() 139 | total_accuracy = total_accuracy + accuracy 140 | 141 | print("整体测试集上的Loss：{}".format(total_test_loss)) 142 | print("整体测试集上的准确率：{}".format(total_accuracy // test_data_size)) 143 | writer.add_scalar(tag="test_loss", scalar_value=total_test_loss, global_step=total_test_step) 144 | writer.add_scalar("test_accuracy", total_accuracy // test_data_size, total_test_step) 145 | total_test_loss += 1 146 | 147 | torch.save(tudui, "../models/tudui_{}.pth".format(i)) 148 | # torch.save(tudui.state_dict(), "../models/tudui_{}.pth".format(i)) 149 | print("模型已保存") 150 | writer.close() 151 | 152 | print("训练测试完成") 153 | 154 | -------------------------------------------------------------------------------- /scripts_liu/bingdundun.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # @Time:2022/2/11下午4:21 3 | # @Author: 放羊Wa 4 | # @Github: juliusyang97 5 | 6 | import turtle 7 | 8 | # 速度 9 | turtle.delay(3) 10 | 11 | # 左手 12 | turtle.penup() 13 | turtle.goto(177, 112) 14 | turtle.pencolor("lightgray") 15 | turtle.pensize(3) 16 | turtle.fillcolor("white") 17 | turtle.begin_fill() 18 | turtle.pendown() 19 | turtle.setheading(80) 20 | turtle.circle(-45, 200) 21 | turtle.circle(-300, 23) 22 | turtle.end_fill() 23 | 24 | # 左手内 25 | turtle.penup() 26 | turtle.goto(182, 95) 27 | turtle.pencolor("black") 28 | turtle.pensize(1) 29 | turtle.fillcolor("black") 30 | turtle.begin_fill() 31 | turtle.setheading(95) 32 | turtle.pendown() 33 | turtle.circle(-37, 160) 34 | turtle.circle(-20, 50) 35 | turtle.circle(-200, 30) 36 | turtle.end_fill() 37 | # 轮廓 38 | # 头顶 39 | turtle.penup() 40 | turtle.goto(-73, 230) 41 | turtle.pencolor("lightgray") 42 | turtle.pensize(3) 43 | turtle.fillcolor("white") 44 | turtle.begin_fill() 45 | turtle.pendown() 46 | turtle.setheading(20) 47 | turtle.circle(-250, 35) 48 | # 左耳 49 | turtle.setheading(50) 50 | turtle.circle(-42, 180) 51 | # 左侧 52 | turtle.setheading(-50) 53 | turtle.circle(-190, 30) 54 | turtle.circle(-320, 45) 55 | # 左腿 56 | turtle.circle(120, 30) 57 | turtle.circle(200, 12) 58 | turtle.circle(-18, 85) 59 | turtle.circle(-180, 23) 60 | turtle.circle(-20, 110) 61 | turtle.circle(15, 115) 62 | turtle.circle(100, 12) 63 | # 右腿 64 | turtle.circle(15, 120) 65 | turtle.circle(-15, 110) 66 | turtle.circle(-150, 30) 67 | turtle.circle(-15, 70) 68 | turtle.circle(-150, 10) 69 | turtle.circle(200, 35) 70 | turtle.circle(-150, 20) 71 | # 右手 72 | turtle.setheading(-120) 73 | turtle.circle(50, 30) 74 | turtle.circle(-35, 200) 75 | turtle.circle(-300, 23) 76 | # 右侧 77 | turtle.setheading(86) 78 | turtle.circle(-300, 26) 79 | # 右耳 80 | turtle.setheading(122) 81 | turtle.circle(-53, 160) 82 | turtle.end_fill() 83 | 84 | # 右耳内 85 | turtle.penup() 86 | turtle.goto(-130, 180) 87 | turtle.pencolor("black") 88 | turtle.pensize(1) 89 | turtle.fillcolor("black") 90 | turtle.begin_fill() 91 | turtle.pendown() 92 | turtle.setheading(120) 93 | turtle.circle(-28, 160) 94 | turtle.setheading(210) 95 | turtle.circle(150, 20) 96 | turtle.end_fill() 97 | 98 | # 左耳内 99 | turtle.penup() 100 | turtle.goto(90, 230) 101 | turtle.setheading(40) 102 | turtle.begin_fill() 103 | turtle.pendown() 104 | turtle.circle(-30, 170) 105 | turtle.setheading(125) 106 | turtle.circle(150, 23) 107 | turtle.end_fill() 108 | 109 | # 右手内 110 | turtle.penup() 111 | turtle.goto(-180, -55) 112 | turtle.fillcolor("black") 113 | turtle.begin_fill() 114 | turtle.setheading(-120) 115 | turtle.pendown() 116 | turtle.circle(50, 30) 117 | turtle.circle(-27, 200) 118 | turtle.circle(-300, 20) 119 | turtle.setheading(-90) 120 | turtle.circle(300, 14) 121 | turtle.end_fill() 122 | 123 | # 左腿内 124 | turtle.penup() 125 | turtle.goto(108, -168) 126 | turtle.fillcolor("black") 127 | turtle.begin_fill() 128 | turtle.pendown() 129 | turtle.setheading(-115) 130 | turtle.circle(110, 15) 131 | turtle.circle(200, 10) 132 | turtle.circle(-18, 80) 133 | turtle.circle(-180, 13) 134 | turtle.circle(-20, 90) 135 | turtle.circle(15, 60) 136 | turtle.setheading(42) 137 | turtle.circle(-200, 29) 138 | turtle.end_fill() 139 | # 右腿内 140 | turtle.penup() 141 | turtle.goto(-38, -210) 142 | turtle.fillcolor("black") 143 | turtle.begin_fill() 144 | turtle.pendown() 145 | turtle.setheading(-155) 146 | turtle.circle(15, 100) 147 | turtle.circle(-10, 110) 148 | turtle.circle(-100, 30) 149 | turtle.circle(-15, 65) 150 | turtle.circle(-100, 10) 151 | turtle.circle(200, 15) 152 | turtle.setheading(-14) 153 | turtle.circle(-200, 27) 154 | turtle.end_fill() 155 | 156 | # 右眼 157 | # 眼圈 158 | turtle.penup() 159 | turtle.goto(-64, 120) 160 | turtle.begin_fill() 161 | turtle.pendown() 162 | turtle.setheading(40) 163 | turtle.circle(-35, 152) 164 | turtle.circle(-100, 50) 165 | turtle.circle(-35, 130) 166 | turtle.circle(-100, 50) 167 | turtle.end_fill() 168 | # 眼珠 169 | turtle.penup() 170 | turtle.goto(-47, 55) 171 | turtle.fillcolor("white") 172 | turtle.begin_fill() 173 | turtle.pendown() 174 | turtle.setheading(0) 175 | turtle.circle(25, 360) 176 | turtle.end_fill() 177 | turtle.penup() 178 | turtle.goto(-45, 62) 179 | turtle.pencolor("darkslategray") 180 | turtle.fillcolor("darkslategray") 181 | turtle.begin_fill() 182 | turtle.pendown() 183 | turtle.setheading(0) 184 | turtle.circle(19, 360) 185 | turtle.end_fill() 186 | turtle.penup() 187 | turtle.goto(-45, 68) 188 | turtle.fillcolor("black") 189 | turtle.begin_fill() 190 | turtle.pendown() 191 | turtle.setheading(0) 192 | turtle.circle(10, 360) 193 | turtle.end_fill() 194 | turtle.penup() 195 | turtle.goto(-47, 86) 196 | turtle.pencolor("white") 197 | turtle.fillcolor("white") 198 | turtle.begin_fill() 199 | turtle.pendown() 200 | turtle.setheading(0) 201 | turtle.circle(5, 360) 202 | turtle.end_fill() 203 | 204 | # 左眼 205 | # 眼圈 206 | turtle.penup() 207 | turtle.goto(51, 82) 208 | turtle.fillcolor("black") 209 | turtle.begin_fill() 210 | turtle.pendown() 211 | turtle.setheading(120) 212 | turtle.circle(-32, 152) 213 | turtle.circle(-100, 55) 214 | turtle.circle(-25, 120) 215 | turtle.circle(-120, 45) 216 | turtle.end_fill() 217 | # 眼珠 218 | turtle.penup() 219 | turtle.goto(79, 60) 220 | turtle.fillcolor("white") 221 | turtle.begin_fill() 222 | turtle.pendown() 223 | turtle.setheading(0) 224 | turtle.circle(24, 360) 225 | turtle.end_fill() 226 | turtle.penup() 227 | turtle.goto(79, 64) 228 | turtle.pencolor("darkslategray") 229 | turtle.fillcolor("darkslategray") 230 | turtle.begin_fill() 231 | turtle.pendown() 232 | turtle.setheading(0) 233 | turtle.circle(19, 360) 234 | turtle.end_fill() 235 | turtle.penup() 236 | turtle.goto(79, 70) 237 | turtle.fillcolor("black") 238 | turtle.begin_fill() 239 | turtle.pendown() 240 | turtle.setheading(0) 241 | turtle.circle(10, 360) 242 | turtle.end_fill() 243 | turtle.penup() 244 | turtle.goto(79, 88) 245 | turtle.pencolor("white") 246 | turtle.fillcolor("white") 247 | turtle.begin_fill() 248 | turtle.pendown() 249 | turtle.setheading(0) 250 | turtle.circle(5, 360) 251 | turtle.end_fill() 252 | 253 | # 鼻子 254 | turtle.penup() 255 | turtle.goto(37, 80) 256 | turtle.fillcolor("black") 257 | turtle.begin_fill() 258 | turtle.pendown() 259 | turtle.circle(-8, 130) 260 | turtle.circle(-22, 100) 261 | turtle.circle(-8, 130) 262 | turtle.end_fill() 263 | 264 | # 嘴 265 | turtle.penup() 266 | turtle.goto(-15, 48) 267 | turtle.setheading(-36) 268 | turtle.begin_fill() 269 | turtle.pendown() 270 | turtle.circle(60, 70) 271 | turtle.setheading(-132) 272 | turtle.circle(-45, 100) 273 | turtle.end_fill() 274 | 275 | # 彩虹圈 276 | turtle.penup() 277 | turtle.goto(-135, 120) 278 | turtle.pensize(5) 279 | turtle.pencolor("cyan") 280 | turtle.pendown() 281 | turtle.setheading(60) 282 | turtle.circle(-165, 150) 283 | turtle.circle(-130, 78) 284 | turtle.circle(-250, 30) 285 | turtle.circle(-138, 105) 286 | turtle.penup() 287 | turtle.goto(-131, 116) 288 | turtle.pencolor("slateblue") 289 | turtle.pendown() 290 | turtle.setheading(60) 291 | turtle.circle(-160, 144) 292 | turtle.circle(-120, 78) 293 | turtle.circle(-242, 30) 294 | turtle.circle(-135, 105) 295 | turtle.penup() 296 | turtle.goto(-127, 112) 297 | turtle.pencolor("orangered") 298 | turtle.pendown() 299 | turtle.setheading(60) 300 | turtle.circle(-155, 136) 301 | turtle.circle(-116, 86) 302 | turtle.circle(-220, 30) 303 | turtle.circle(-134, 103) 304 | turtle.penup() 305 | turtle.goto(-123, 108) 306 | turtle.pencolor("gold") 307 | turtle.pendown() 308 | turtle.setheading(60) 309 | turtle.circle(-150, 136) 310 | turtle.circle(-104, 86) 311 | turtle.circle(-220, 30) 312 | turtle.circle(-126, 102) 313 | turtle.penup() 314 | turtle.goto(-120, 104) 315 | turtle.pencolor("greenyellow") 316 | turtle.pendown() 317 | turtle.setheading(60) 318 | turtle.circle(-145, 136) 319 | turtle.circle(-90, 83) 320 | turtle.circle(-220, 30) 321 | turtle.circle(-120, 100) 322 | turtle.penup() 323 | 324 | # 爱心 325 | turtle.penup() 326 | turtle.goto(220, 115) 327 | turtle.pencolor("brown") 328 | turtle.pensize(1) 329 | turtle.fillcolor("brown") 330 | turtle.begin_fill() 331 | turtle.pendown() 332 | turtle.setheading(36) 333 | turtle.circle(-8, 180) 334 | turtle.circle(-60, 24) 335 | turtle.setheading(110) 336 | turtle.circle(-60, 24) 337 | turtle.circle(-8, 180) 338 | turtle.end_fill() 339 | 340 | # 五环 341 | turtle.penup() 342 | turtle.goto(-5, -170) 343 | turtle.pendown() 344 | turtle.pencolor("blue") 345 | turtle.circle(6) 346 | turtle.penup() 347 | turtle.goto(10, -170) 348 | turtle.pendown() 349 | turtle.pencolor("black") 350 | turtle.circle(6) 351 | turtle.penup() 352 | turtle.goto(25, -170) 353 | turtle.pendown() 354 | turtle.pencolor("brown") 355 | turtle.circle(6) 356 | turtle.penup() 357 | turtle.goto(2, -175) 358 | turtle.pendown() 359 | turtle.pencolor("lightgoldenrod") 360 | turtle.circle(6) 361 | turtle.penup() 362 | turtle.goto(16, -175) 363 | turtle.pendown() 364 | turtle.pencolor("green") 365 | turtle.circle(6) 366 | turtle.penup() 367 | 368 | turtle.pencolor("black") 369 | turtle.goto(-16, -160) 370 | turtle.write("BEIJING 2022", font=('Arial', 10, 'bold italic')) 371 | turtle.hideturtle() 372 | 373 | turtle.done() 374 | -------------------------------------------------------------------------------- /60-Minute-Guide/2_autograd.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "bea78130", 6 | "metadata": {}, 7 | "source": [ 8 | "# AutoGrad : 自动求导 \n", 9 | "torch.autograd是pytorch自动求导的工具，也是所有神经网络的核心。我们首先先简单了解一下这个包如何训练神经网络。" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "id": "9390f58a", 15 | "metadata": {}, 16 | "source": [ 17 | "## 背景介绍 \n", 18 | "\n", 19 | "神经网络(NNs)是作用在输入数据上的一系列嵌套函数的集合，这些函数由权重和误差来定义，被存储在PyTorch中的tensors中。\n", 20 | "\n", 21 | "神经网络训练的两个步骤：\n", 22 | "\n", 23 | "**前向传播：** 在前向传播中，神经网络通过将接收到的数据与每一层对应的权重和误差进行运算来对正确的输出做出最好的预测。\n", 24 | "\n", 25 | "**反向传播：** 在反向传播中，神经网络调整其参数使得其与输出误差成比例。反向传播基于梯度下降策略，是链式求导法则的一个应用，以目标的负梯度方向对参数进行调整。\n", 26 | "\n", 27 | "更加详细的介绍可以参照下述地址：[video from 3Blue1Brown](https://www.youtube.com/watch?v=tIeHLnjs5U8)\n", 28 | "\n" 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "id": "c2c8dc1d", 34 | "metadata": {}, 35 | "source": [ 36 | "## 在Pytorch 中的应用 \n", 37 | "\n", 38 | "来看一个简单的示例，我们从torchvision加载一个预先训练好的resnet18模型，接着创建一个随机数据tensor来表示一有3个通道、高度和宽度为64的图像，其对应的标签初始化为一些随机值。" 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": 1, 44 | "id": "aeacae86", 45 | "metadata": {}, 46 | "outputs": [], 47 | "source": [ 48 | "%matplotlib inline" 49 | ] 50 | }, 51 | { 52 | "cell_type": "code", 53 | "execution_count": 3, 54 | "id": "6c84e3e7", 55 | "metadata": {}, 56 | "outputs": [], 57 | "source": [ 58 | "import torch, torchvision\n", 59 | "model = torchvision.models.resnet18(pretrained=True)\n", 60 | "data = torch.rand(1, 3, 64, 64)\n", 61 | "labels = torch.rand(1, 1000)" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "id": "7bee168f", 67 | "metadata": {}, 68 | "source": [ 69 | "接下来，我们将输入数据向输出方向传播到模型的每一层中来预测输出，这就是**前向传播** 。" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 4, 75 | "id": "8d44cba0", 76 | "metadata": {}, 77 | "outputs": [], 78 | "source": [ 79 | "prediction = model(data) # 前向传播" 80 | ] 81 | }, 82 | { 83 | "cell_type": "markdown", 84 | "id": "7d21ee1c", 85 | "metadata": {}, 86 | "source": [ 87 | "我们利用模型的预测和对应的标签来计算误差（loss），然后反向传播误差。完成计算后，您可以调用`.backward()`并自动计算所有梯度。此张量的梯度将累积到`.grad`属性中。" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": 5, 93 | "id": "700f81ac", 94 | "metadata": {}, 95 | "outputs": [], 96 | "source": [ 97 | "loss = (prediction - labels).sum()\n", 98 | "loss.backward() # 反向传播" 99 | ] 100 | }, 101 | { 102 | "cell_type": "markdown", 103 | "id": "8c63f55e", 104 | "metadata": {}, 105 | "source": [ 106 | "接着，我们加载一个优化器，在本例中，SGD的学习率为0.01，momentum 为0.9。我们在优化器中注册模型的所有参数。" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": 6, 112 | "id": "2d49947d", 113 | "metadata": {}, 114 | "outputs": [], 115 | "source": [ 116 | "optim = torch.optim.SGD(model.parameters(), lr= 0.001, momentum= 0.9)" 117 | ] 118 | }, 119 | { 120 | "cell_type": "markdown", 121 | "id": "6d431e68", 122 | "metadata": {}, 123 | "source": [ 124 | "最后，我们调用`.step()`来执行梯度下降，优化器通过存储在`.grad`中的梯度来调整每个参数。" 125 | ] 126 | }, 127 | { 128 | "cell_type": "code", 129 | "execution_count": 7, 130 | "id": "9324ae8b", 131 | "metadata": {}, 132 | "outputs": [], 133 | "source": [ 134 | "optim.step() # 梯度下降" 135 | ] 136 | }, 137 | { 138 | "cell_type": "markdown", 139 | "id": "3e86a697", 140 | "metadata": {}, 141 | "source": [ 142 | "现在，你已经具备了训练神经网络所需所有条件。下面几节详细介绍了Autograd包的工作原理——可以跳过它们。\n", 143 | "\n", 144 | "\n", 145 | "-----" 146 | ] 147 | }, 148 | { 149 | "cell_type": "markdown", 150 | "id": "15bf953a", 151 | "metadata": {}, 152 | "source": [ 153 | "## Autograd中的求导 \n", 154 | "\n", 155 | "先来看一下autograd是如何收集梯度的。我们创建两个张量a和b并设置requires_grad = True以跟踪它的计算。" 156 | ] 157 | }, 158 | { 159 | "cell_type": "code", 160 | "execution_count": 9, 161 | "id": "c2e1ed9a", 162 | "metadata": {}, 163 | "outputs": [], 164 | "source": [ 165 | "import torch" 166 | ] 167 | }, 168 | { 169 | "cell_type": "code", 170 | "execution_count": 15, 171 | "id": "672f92a7", 172 | "metadata": {}, 173 | "outputs": [], 174 | "source": [ 175 | "a = torch.tensor([2., 3.], requires_grad= True)\n", 176 | "b = torch.tensor([6., 4.], requires_grad= True)" 177 | ] 178 | }, 179 | { 180 | "cell_type": "markdown", 181 | "id": "17aceea4", 182 | "metadata": {}, 183 | "source": [ 184 | "接着在a和b的基础上创建张量Q\n", 185 | "\n", 186 | "\\begin{align}Q = 3a^3 - b^2\\end{align}" 187 | ] 188 | }, 189 | { 190 | "cell_type": "code", 191 | "execution_count": 17, 192 | "id": "a913c135", 193 | "metadata": {}, 194 | "outputs": [], 195 | "source": [ 196 | "Q = 3*a**3 - b**2" 197 | ] 198 | }, 199 | { 200 | "cell_type": "markdown", 201 | "id": "870fe269", 202 | "metadata": {}, 203 | "source": [ 204 | "假设a和b是一个神经网络的权重，Q是它的误差，在神经网络训练中，我们需要w.r.t参数的误差梯度，即 \n", 205 | "\n", 206 | "\\begin{align}\\frac{\\partial Q}{\\partial a} = 9a^2\\end{align}\n", 207 | "\n", 208 | "\\begin{align}\\frac{\\partial Q}{\\partial b} = -2b\\end{align}\n", 209 | "\n", 210 | "当我们调用Q的.backward()时，autograd计算这些梯度并把它们存储在张量的 .grad属性中。\n", 211 | "\n", 212 | "我们需要在Q.backward()中显式传递gradient，gradient是一个与Q相同形状的张量，它表示Q w.r.t本身的梯度，即 \n", 213 | "\n", 214 | "\\begin{align}\\frac{dQ}{dQ} = 1\\end{align}\n", 215 | "\n", 216 | "同样，我们也可以将Q聚合为一个标量并隐式向后调用，如Q.sum().backward()。 \n" 217 | ] 218 | }, 219 | { 220 | "cell_type": "code", 221 | "execution_count": 18, 222 | "id": "75408a33", 223 | "metadata": {}, 224 | "outputs": [], 225 | "source": [ 226 | "external_grad = torch.tensor([1., 1,])\n", 227 | "Q.backward(gradient=external_grad)" 228 | ] 229 | }, 230 | { 231 | "cell_type": "markdown", 232 | "id": "2c71f727", 233 | "metadata": {}, 234 | "source": [ 235 | "现在梯度都被存放在a.grad和b.grad中" 236 | ] 237 | }, 238 | { 239 | "cell_type": "code", 240 | "execution_count": 19, 241 | "id": "a5a81ae1", 242 | "metadata": {}, 243 | "outputs": [ 244 | { 245 | "name": "stdout", 246 | "output_type": "stream", 247 | "text": [ 248 | "tensor([True, True])\n", 249 | "tensor([True, True])\n" 250 | ] 251 | } 252 | ], 253 | "source": [ 254 | "# 检查一下存储的梯度都正确\n", 255 | "print(9*a**2 == a.grad)\n", 256 | "print(-2*b == b.grad)" 257 | ] 258 | }, 259 | { 260 | "cell_type": "markdown", 261 | "id": "6f0f1d75", 262 | "metadata": {}, 263 | "source": [ 264 | "## 选读----用autograd进行向量计算 \n", 265 | "\n", 266 | "在数学上，如果你有一个向量值函数𝑦⃗ =𝑓(𝑥⃗ ) ，则𝑦⃗ 相对于𝑥⃗ 的梯度是雅可比矩阵： \n", 267 | "\n", 268 | "\\begin{align}J\n", 269 | " =\n", 270 | " \\left(\\begin{array}{cc}\n", 271 | " \\frac{\\partial \\bf{y}}{\\partial x_{1}} &\n", 272 | " ... &\n", 273 | " \\frac{\\partial \\bf{y}}{\\partial x_{n}}\n", 274 | " \\end{array}\\right)\n", 275 | " =\n", 276 | " \\left(\\begin{array}{ccc}\n", 277 | " \\frac{\\partial y_{1}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{1}}{\\partial x_{n}}\\\\\n", 278 | " \\vdots & \\ddots & \\vdots\\\\\n", 279 | " \\frac{\\partial y_{m}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{n}}\n", 280 | " \\end{array}\\right)\\end{align}\n", 281 | " \n", 282 | "\n", 283 | "一般来说，torch.autograd是一个计算雅可比向量积的引擎。 也就是说，给定任何向量𝑣=(𝑣1𝑣2...𝑣𝑚)𝑇，计算乘积$J^{T}\\cdot \\vec{v}$。 \n", 284 | "\n", 285 | "如果𝑣恰好是标量函数的梯度𝑙=𝑔(𝑦⃗ )，即\n", 286 | " \n", 287 | "\\begin{align}\\vec{v}\n", 288 | " =\n", 289 | " \\left(\\begin{array}{ccc}\\frac{\\partial l}{\\partial y_{1}} & \\cdots & \\frac{\\partial l}{\\partial y_{m}}\\end{array}\\right)^{T}\\end{align}\n", 290 | " \n", 291 | " 然后根据链式法则，雅可比向量乘积将是𝑙相对于𝑥⃗ 的梯度 \n", 292 | " \n", 293 | "\\begin{align}J^{T}\\cdot \\vec{v}=\\left(\\begin{array}{ccc}\n", 294 | " \\frac{\\partial y_{1}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{1}}\\\\\n", 295 | " \\vdots & \\ddots & \\vdots\\\\\n", 296 | " \\frac{\\partial y_{1}}{\\partial x_{n}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{n}}\n", 297 | " \\end{array}\\right)\\left(\\begin{array}{c}\n", 298 | " \\frac{\\partial l}{\\partial y_{1}}\\\\\n", 299 | " \\vdots\\\\\n", 300 | " \\frac{\\partial l}{\\partial y_{m}}\n", 301 | " \\end{array}\\right)=\\left(\\begin{array}{c}\n", 302 | " \\frac{\\partial l}{\\partial x_{1}}\\\\\n", 303 | " \\vdots\\\\\n", 304 | " \\frac{\\partial l}{\\partial x_{n}}\n", 305 | " \\end{array}\\right)\\end{align}\n", 306 | " \n", 307 | " \n", 308 | " \n", 309 | " \n", 310 | "雅可比向量积的这种特性使得将外部梯度馈送到具有非标量输出的模型中非常方便。external_grad 代表$\\vec{v}$\n", 311 | "." 312 | ] 313 | }, 314 | { 315 | "cell_type": "code", 316 | "execution_count": null, 317 | "id": "2beed98c", 318 | "metadata": {}, 319 | "outputs": [], 320 | "source": [] 321 | } 322 | ], 323 | "metadata": { 324 | "kernelspec": { 325 | "display_name": "Python 3", 326 | "language": "python", 327 | "name": "python3" 328 | }, 329 | "language_info": { 330 | "codemirror_mode": { 331 | "name": "ipython", 332 | "version": 3 333 | }, 334 | "file_extension": ".py", 335 | "mimetype": "text/x-python", 336 | "name": "python", 337 | "nbconvert_exporter": "python", 338 | "pygments_lexer": "ipython3", 339 | "version": "3.6.13" 340 | } 341 | }, 342 | "nbformat": 4, 343 | "nbformat_minor": 5 344 | } 345 | -------------------------------------------------------------------------------- /60-Minute-Guide/neural_networks_tutorial_en.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%matplotlib inline" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "\nNeural Networks\n===============\n\nNeural networks can be constructed using the ``torch.nn`` package.\n\nNow that you had a glimpse of ``autograd``, ``nn`` depends on\n``autograd`` to define models and differentiate them.\nAn ``nn.Module`` contains layers, and a method ``forward(input)`` that\nreturns the ``output``.\n\nFor example, look at this network that classifies digit images:\n\n.. figure:: /_static/img/mnist.png\n :alt: convnet\n\n convnet\n\nIt is a simple feed-forward network. It takes the input, feeds it\nthrough several layers one after the other, and then finally gives the\noutput.\n\nA typical training procedure for a neural network is as follows:\n\n- Define the neural network that has some learnable parameters (or\n weights)\n- Iterate over a dataset of inputs\n- Process input through the network\n- Compute the loss (how far is the output from being correct)\n- Propagate gradients back into the network\u2019s parameters\n- Update the weights of the network, typically using a simple update rule:\n ``weight = weight - learning_rate * gradient``\n\nDefine the network\n------------------\n\nLet\u2019s define this network:\n\n" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": { 25 | "collapsed": false 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\n\nclass Net(nn.Module):\n\n def __init__(self):\n super(Net, self).__init__()\n # 1 input image channel, 6 output channels, 5x5 square convolution\n # kernel\n self.conv1 = nn.Conv2d(1, 6, 5)\n self.conv2 = nn.Conv2d(6, 16, 5)\n # an affine operation: y = Wx + b\n self.fc1 = nn.Linear(16 * 5 * 5, 120) # 5*5 from image dimension \n self.fc2 = nn.Linear(120, 84)\n self.fc3 = nn.Linear(84, 10)\n\n def forward(self, x):\n # Max pooling over a (2, 2) window\n x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))\n # If the size is a square, you can specify with a single number\n x = F.max_pool2d(F.relu(self.conv2(x)), 2)\n x = torch.flatten(x, 1) # flatten all dimensions except the batch dimension\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = self.fc3(x)\n return x\n\n\nnet = Net()\nprint(net)" 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "metadata": {}, 35 | "source": [ 36 | "You just have to define the ``forward`` function, and the ``backward``\nfunction (where gradients are computed) is automatically defined for you\nusing ``autograd``.\nYou can use any of the Tensor operations in the ``forward`` function.\n\nThe learnable parameters of a model are returned by ``net.parameters()``\n\n" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": null, 42 | "metadata": { 43 | "collapsed": false 44 | }, 45 | "outputs": [], 46 | "source": [ 47 | "params = list(net.parameters())\nprint(len(params))\nprint(params[0].size()) # conv1's .weight" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": {}, 53 | "source": [ 54 | "Let's try a random 32x32 input.\nNote: expected input size of this net (LeNet) is 32x32. To use this net on\nthe MNIST dataset, please resize the images from the dataset to 32x32.\n\n" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": { 61 | "collapsed": false 62 | }, 63 | "outputs": [], 64 | "source": [ 65 | "input = torch.randn(1, 1, 32, 32)\nout = net(input)\nprint(out)" 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "metadata": {}, 71 | "source": [ 72 | "Zero the gradient buffers of all parameters and backprops with random\ngradients:\n\n" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": null, 78 | "metadata": { 79 | "collapsed": false 80 | }, 81 | "outputs": [], 82 | "source": [ 83 | "net.zero_grad()\nout.backward(torch.randn(1, 10))" 84 | ] 85 | }, 86 | { 87 | "cell_type": "markdown", 88 | "metadata": {}, 89 | "source": [ 90 | "

Note

``torch.nn`` only supports mini-batches. The entire ``torch.nn``\n package only supports inputs that are a mini-batch of samples, and not\n a single sample.\n\n For example, ``nn.Conv2d`` will take in a 4D Tensor of\n ``nSamples x nChannels x Height x Width``.\n\n If you have a single sample, just use ``input.unsqueeze(0)`` to add\n a fake batch dimension.

\n\nBefore proceeding further, let's recap all the classes you\u2019ve seen so far.\n\n**Recap:**\n - ``torch.Tensor`` - A *multi-dimensional array* with support for autograd\n operations like ``backward()``. Also *holds the gradient* w.r.t. the\n tensor.\n - ``nn.Module`` - Neural network module. *Convenient way of\n encapsulating parameters*, with helpers for moving them to GPU,\n exporting, loading, etc.\n - ``nn.Parameter`` - A kind of Tensor, that is *automatically\n registered as a parameter when assigned as an attribute to a*\n ``Module``.\n - ``autograd.Function`` - Implements *forward and backward definitions\n of an autograd operation*. Every ``Tensor`` operation creates at\n least a single ``Function`` node that connects to functions that\n created a ``Tensor`` and *encodes its history*.\n\n**At this point, we covered:**\n - Defining a neural network\n - Processing inputs and calling backward\n\n**Still Left:**\n - Computing the loss\n - Updating the weights of the network\n\nLoss Function\n-------------\nA loss function takes the (output, target) pair of inputs, and computes a\nvalue that estimates how far away the output is from the target.\n\nThere are several different\n`loss functions `_ under the\nnn package .\nA simple loss is: ``nn.MSELoss`` which computes the mean-squared error\nbetween the input and the target.\n\nFor example:\n\n" 91 | ] 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": null, 96 | "metadata": { 97 | "collapsed": false 98 | }, 99 | "outputs": [], 100 | "source": [ 101 | "output = net(input)\ntarget = torch.randn(10) # a dummy target, for example\ntarget = target.view(1, -1) # make it the same shape as output\ncriterion = nn.MSELoss()\n\nloss = criterion(output, target)\nprint(loss)" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "Now, if you follow ``loss`` in the backward direction, using its\n``.grad_fn`` attribute, you will see a graph of computations that looks\nlike this:\n\n::\n\n input -> conv2d -> relu -> maxpool2d -> conv2d -> relu -> maxpool2d\n -> flatten -> linear -> relu -> linear -> relu -> linear\n -> MSELoss\n -> loss\n\nSo, when we call ``loss.backward()``, the whole graph is differentiated\nw.r.t. the neural net parameters, and all Tensors in the graph that have\n``requires_grad=True`` will have their ``.grad`` Tensor accumulated with the\ngradient.\n\nFor illustration, let us follow a few steps backward:\n\n" 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": null, 114 | "metadata": { 115 | "collapsed": false 116 | }, 117 | "outputs": [], 118 | "source": [ 119 | "print(loss.grad_fn) # MSELoss\nprint(loss.grad_fn.next_functions[0][0]) # Linear\nprint(loss.grad_fn.next_functions[0][0].next_functions[0][0]) # ReLU" 120 | ] 121 | }, 122 | { 123 | "cell_type": "markdown", 124 | "metadata": {}, 125 | "source": [ 126 | "Backprop\n--------\nTo backpropagate the error all we have to do is to ``loss.backward()``.\nYou need to clear the existing gradients though, else gradients will be\naccumulated to existing gradients.\n\n\nNow we shall call ``loss.backward()``, and have a look at conv1's bias\ngradients before and after the backward.\n\n" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": null, 132 | "metadata": { 133 | "collapsed": false 134 | }, 135 | "outputs": [], 136 | "source": [ 137 | "net.zero_grad() # zeroes the gradient buffers of all parameters\n\nprint('conv1.bias.grad before backward')\nprint(net.conv1.bias.grad)\n\nloss.backward()\n\nprint('conv1.bias.grad after backward')\nprint(net.conv1.bias.grad)" 138 | ] 139 | }, 140 | { 141 | "cell_type": "markdown", 142 | "metadata": {}, 143 | "source": [ 144 | "Now, we have seen how to use loss functions.\n\n**Read Later:**\n\n The neural network package contains various modules and loss functions\n that form the building blocks of deep neural networks. A full list with\n documentation is `here `_.\n\n**The only thing left to learn is:**\n\n - Updating the weights of the network\n\nUpdate the weights\n------------------\nThe simplest update rule used in practice is the Stochastic Gradient\nDescent (SGD):\n\n ``weight = weight - learning_rate * gradient``\n\nWe can implement this using simple Python code:\n\n.. code:: python\n\n learning_rate = 0.01\n for f in net.parameters():\n f.data.sub_(f.grad.data * learning_rate)\n\nHowever, as you use neural networks, you want to use various different\nupdate rules such as SGD, Nesterov-SGD, Adam, RMSProp, etc.\nTo enable this, we built a small package: ``torch.optim`` that\nimplements all these methods. Using it is very simple:\n\n" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": null, 150 | "metadata": { 151 | "collapsed": false 152 | }, 153 | "outputs": [], 154 | "source": [ 155 | "import torch.optim as optim\n\n# create your optimizer\noptimizer = optim.SGD(net.parameters(), lr=0.01)\n\n# in your training loop:\noptimizer.zero_grad() # zero the gradient buffers\noutput = net(input)\nloss = criterion(output, target)\nloss.backward()\noptimizer.step() # Does the update" 156 | ] 157 | }, 158 | { 159 | "cell_type": "markdown", 160 | "metadata": {}, 161 | "source": [ 162 | ".. Note::\n\n Observe how gradient buffers had to be manually set to zero using\n ``optimizer.zero_grad()``. This is because gradients are accumulated\n as explained in the `Backprop`_ section.\n\n" 163 | ] 164 | } 165 | ], 166 | "metadata": { 167 | "kernelspec": { 168 | "display_name": "Python 3", 169 | "language": "python", 170 | "name": "python3" 171 | }, 172 | "language_info": { 173 | "codemirror_mode": { 174 | "name": "ipython", 175 | "version": 3 176 | }, 177 | "file_extension": ".py", 178 | "mimetype": "text/x-python", 179 | "name": "python", 180 | "nbconvert_exporter": "python", 181 | "pygments_lexer": "ipython3", 182 | "version": "3.6.13" 183 | } 184 | }, 185 | "nbformat": 4, 186 | "nbformat_minor": 0 187 | } -------------------------------------------------------------------------------- /60-Minute-Guide/tensor_tutorial_en.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%matplotlib inline" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "\nTensors\n--------------------------------------------\n\nTensors are a specialized data structure that are very similar to arrays\nand matrices. In PyTorch, we use tensors to encode the inputs and\noutputs of a model, as well as the model\u2019s parameters.\n\nTensors are similar to NumPy\u2019s ndarrays, except that tensors can run on\nGPUs or other specialized hardware to accelerate computing. If you\u2019re familiar with ndarrays, you\u2019ll\nbe right at home with the Tensor API. If not, follow along in this quick\nAPI walkthrough.\n\n\n" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": { 25 | "collapsed": false 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "import torch\nimport numpy as np" 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "metadata": {}, 35 | "source": [ 36 | "Tensor Initialization\n~~~~~~~~~~~~~~~~~~~~~\n\nTensors can be initialized in various ways. Take a look at the following examples:\n\n**Directly from data**\n\nTensors can be created directly from data. The data type is automatically inferred.\n\n" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": null, 42 | "metadata": { 43 | "collapsed": false 44 | }, 45 | "outputs": [], 46 | "source": [ 47 | "data = [[1, 2], [3, 4]]\nx_data = torch.tensor(data)" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": {}, 53 | "source": [ 54 | "**From a NumPy array**\n\nTensors can be created from NumPy arrays (and vice versa - see `bridge-to-np-label`).\n\n" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": { 61 | "collapsed": false 62 | }, 63 | "outputs": [], 64 | "source": [ 65 | "np_array = np.array(data)\nx_np = torch.from_numpy(np_array)" 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "metadata": {}, 71 | "source": [ 72 | "**From another tensor:**\n\nThe new tensor retains the properties (shape, datatype) of the argument tensor, unless explicitly overridden.\n\n" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": null, 78 | "metadata": { 79 | "collapsed": false 80 | }, 81 | "outputs": [], 82 | "source": [ 83 | "x_ones = torch.ones_like(x_data) # retains the properties of x_data\nprint(f\"Ones Tensor: \\n {x_ones} \\n\")\n\nx_rand = torch.rand_like(x_data, dtype=torch.float) # overrides the datatype of x_data\nprint(f\"Random Tensor: \\n {x_rand} \\n\")" 84 | ] 85 | }, 86 | { 87 | "cell_type": "markdown", 88 | "metadata": {}, 89 | "source": [ 90 | "**With random or constant values:**\n\n``shape`` is a tuple of tensor dimensions. In the functions below, it determines the dimensionality of the output tensor.\n\n" 91 | ] 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": null, 96 | "metadata": { 97 | "collapsed": false 98 | }, 99 | "outputs": [], 100 | "source": [ 101 | "shape = (2, 3,)\nrand_tensor = torch.rand(shape)\nones_tensor = torch.ones(shape)\nzeros_tensor = torch.zeros(shape)\n\nprint(f\"Random Tensor: \\n {rand_tensor} \\n\")\nprint(f\"Ones Tensor: \\n {ones_tensor} \\n\")\nprint(f\"Zeros Tensor: \\n {zeros_tensor}\")" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "--------------\n\n\n" 109 | ] 110 | }, 111 | { 112 | "cell_type": "markdown", 113 | "metadata": {}, 114 | "source": [ 115 | "Tensor Attributes\n~~~~~~~~~~~~~~~~~\n\nTensor attributes describe their shape, datatype, and the device on which they are stored.\n\n" 116 | ] 117 | }, 118 | { 119 | "cell_type": "code", 120 | "execution_count": null, 121 | "metadata": { 122 | "collapsed": false 123 | }, 124 | "outputs": [], 125 | "source": [ 126 | "tensor = torch.rand(3, 4)\n\nprint(f\"Shape of tensor: {tensor.shape}\")\nprint(f\"Datatype of tensor: {tensor.dtype}\")\nprint(f\"Device tensor is stored on: {tensor.device}\")" 127 | ] 128 | }, 129 | { 130 | "cell_type": "markdown", 131 | "metadata": {}, 132 | "source": [ 133 | "--------------\n\n\n" 134 | ] 135 | }, 136 | { 137 | "cell_type": "markdown", 138 | "metadata": {}, 139 | "source": [ 140 | "Tensor Operations\n~~~~~~~~~~~~~~~~~\n\nOver 100 tensor operations, including transposing, indexing, slicing,\nmathematical operations, linear algebra, random sampling, and more are\ncomprehensively described\n`here `__.\n\nEach of them can be run on the GPU (at typically higher speeds than on a\nCPU). If you\u2019re using Colab, allocate a GPU by going to Edit > Notebook\nSettings.\n\n\n" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": null, 146 | "metadata": { 147 | "collapsed": false 148 | }, 149 | "outputs": [], 150 | "source": [ 151 | "# We move our tensor to the GPU if available\nif torch.cuda.is_available():\n tensor = tensor.to('cuda')\n print(f\"Device tensor is stored on: {tensor.device}\")" 152 | ] 153 | }, 154 | { 155 | "cell_type": "markdown", 156 | "metadata": {}, 157 | "source": [ 158 | "Try out some of the operations from the list.\nIf you're familiar with the NumPy API, you'll find the Tensor API a breeze to use.\n\n\n" 159 | ] 160 | }, 161 | { 162 | "cell_type": "markdown", 163 | "metadata": {}, 164 | "source": [ 165 | "**Standard numpy-like indexing and slicing:**\n\n" 166 | ] 167 | }, 168 | { 169 | "cell_type": "code", 170 | "execution_count": null, 171 | "metadata": { 172 | "collapsed": false 173 | }, 174 | "outputs": [], 175 | "source": [ 176 | "tensor = torch.ones(4, 4)\ntensor[:,1] = 0\nprint(tensor)" 177 | ] 178 | }, 179 | { 180 | "cell_type": "markdown", 181 | "metadata": {}, 182 | "source": [ 183 | "**Joining tensors** You can use ``torch.cat`` to concatenate a sequence of tensors along a given dimension.\nSee also `torch.stack `__,\nanother tensor joining op that is subtly different from ``torch.cat``.\n\n" 184 | ] 185 | }, 186 | { 187 | "cell_type": "code", 188 | "execution_count": null, 189 | "metadata": { 190 | "collapsed": false 191 | }, 192 | "outputs": [], 193 | "source": [ 194 | "t1 = torch.cat([tensor, tensor, tensor], dim=1)\nprint(t1)" 195 | ] 196 | }, 197 | { 198 | "cell_type": "markdown", 199 | "metadata": {}, 200 | "source": [ 201 | "**Multiplying tensors**\n\n" 202 | ] 203 | }, 204 | { 205 | "cell_type": "code", 206 | "execution_count": null, 207 | "metadata": { 208 | "collapsed": false 209 | }, 210 | "outputs": [], 211 | "source": [ 212 | "# This computes the element-wise product\nprint(f\"tensor.mul(tensor) \\n {tensor.mul(tensor)} \\n\")\n# Alternative syntax:\nprint(f\"tensor * tensor \\n {tensor * tensor}\")" 213 | ] 214 | }, 215 | { 216 | "cell_type": "markdown", 217 | "metadata": {}, 218 | "source": [ 219 | "This computes the matrix multiplication between two tensors\n\n" 220 | ] 221 | }, 222 | { 223 | "cell_type": "code", 224 | "execution_count": null, 225 | "metadata": { 226 | "collapsed": false 227 | }, 228 | "outputs": [], 229 | "source": [ 230 | "print(f\"tensor.matmul(tensor.T) \\n {tensor.matmul(tensor.T)} \\n\")\n# Alternative syntax:\nprint(f\"tensor @ tensor.T \\n {tensor @ tensor.T}\")" 231 | ] 232 | }, 233 | { 234 | "cell_type": "markdown", 235 | "metadata": {}, 236 | "source": [ 237 | "**In-place operations**\nOperations that have a ``_`` suffix are in-place. For example: ``x.copy_(y)``, ``x.t_()``, will change ``x``.\n\n" 238 | ] 239 | }, 240 | { 241 | "cell_type": "code", 242 | "execution_count": null, 243 | "metadata": { 244 | "collapsed": false 245 | }, 246 | "outputs": [], 247 | "source": [ 248 | "print(tensor, \"\\n\")\ntensor.add_(5)\nprint(tensor)" 249 | ] 250 | }, 251 | { 252 | "cell_type": "markdown", 253 | "metadata": {}, 254 | "source": [ 255 | "

Note

In-place operations save some memory, but can be problematic when computing derivatives because of an immediate loss\n of history. Hence, their use is discouraged.

\n\n" 256 | ] 257 | }, 258 | { 259 | "cell_type": "markdown", 260 | "metadata": {}, 261 | "source": [ 262 | "--------------\n\n\n" 263 | ] 264 | }, 265 | { 266 | "cell_type": "markdown", 267 | "metadata": {}, 268 | "source": [ 269 | "\nBridge with NumPy\n~~~~~~~~~~~~~~~~~\nTensors on the CPU and NumPy arrays can share their underlying memory\nlocations, and changing one will change\tthe other.\n\n" 270 | ] 271 | }, 272 | { 273 | "cell_type": "markdown", 274 | "metadata": {}, 275 | "source": [ 276 | "Tensor to NumPy array\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n" 277 | ] 278 | }, 279 | { 280 | "cell_type": "code", 281 | "execution_count": null, 282 | "metadata": { 283 | "collapsed": false 284 | }, 285 | "outputs": [], 286 | "source": [ 287 | "t = torch.ones(5)\nprint(f\"t: {t}\")\nn = t.numpy()\nprint(f\"n: {n}\")" 288 | ] 289 | }, 290 | { 291 | "cell_type": "markdown", 292 | "metadata": {}, 293 | "source": [ 294 | "A change in the tensor reflects in the NumPy array.\n\n" 295 | ] 296 | }, 297 | { 298 | "cell_type": "code", 299 | "execution_count": null, 300 | "metadata": { 301 | "collapsed": false 302 | }, 303 | "outputs": [], 304 | "source": [ 305 | "t.add_(1)\nprint(f\"t: {t}\")\nprint(f\"n: {n}\")" 306 | ] 307 | }, 308 | { 309 | "cell_type": "markdown", 310 | "metadata": {}, 311 | "source": [ 312 | "NumPy array to Tensor\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n" 313 | ] 314 | }, 315 | { 316 | "cell_type": "code", 317 | "execution_count": null, 318 | "metadata": { 319 | "collapsed": false 320 | }, 321 | "outputs": [], 322 | "source": [ 323 | "n = np.ones(5)\nt = torch.from_numpy(n)" 324 | ] 325 | }, 326 | { 327 | "cell_type": "markdown", 328 | "metadata": {}, 329 | "source": [ 330 | "Changes in the NumPy array reflects in the tensor.\n\n" 331 | ] 332 | }, 333 | { 334 | "cell_type": "code", 335 | "execution_count": null, 336 | "metadata": { 337 | "collapsed": false 338 | }, 339 | "outputs": [], 340 | "source": [ 341 | "np.add(n, 1, out=n)\nprint(f\"t: {t}\")\nprint(f\"n: {n}\")" 342 | ] 343 | } 344 | ], 345 | "metadata": { 346 | "kernelspec": { 347 | "display_name": "Python 3", 348 | "language": "python", 349 | "name": "python3" 350 | }, 351 | "language_info": { 352 | "codemirror_mode": { 353 | "name": "ipython", 354 | "version": 3 355 | }, 356 | "file_extension": ".py", 357 | "mimetype": "text/x-python", 358 | "name": "python", 359 | "nbconvert_exporter": "python", 360 | "pygments_lexer": "ipython3", 361 | "version": "3.6.13" 362 | } 363 | }, 364 | "nbformat": 4, 365 | "nbformat_minor": 0 366 | } -------------------------------------------------------------------------------- /60-Minute-Guide/1_tensors.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "collapsed": true 7 | }, 8 | "source": [ 9 | "什么是 PyTorch？ \n", 10 | "\n", 11 | "PyTorch 是一个基于 Python 的科学计算包，有两大用途：\n", 12 | "\n", 13 | "- NumPy 的替代品，可使用 GPU 和其他加速器的强大功能。\n", 14 | "- 一个用于实现神经网络的自动微分库。" 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "本教程的目标：\n", 22 | "- 高层次理解 PyTorch 的 Tensor 库和神经网络。\n", 23 | "- 训练一个小型神经网络来对图像进行分类\n", 24 | "\n", 25 | "> note：确保您安装了torch和torchvision软件包。" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": 1, 31 | "metadata": {}, 32 | "outputs": [], 33 | "source": [ 34 | "%matplotlib inline" 35 | ] 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "metadata": {}, 40 | "source": [ 41 | "# Tensors\n", 42 | "\n", 43 | "张量是一种特殊的数据结构，与数组和矩阵非常相似。在 PyTorch 中，我们使用张量对模型的输入和输出以及模型的参数进行编码。 \n", 44 | "\n", 45 | "张量类似于 NumPy 的 ndarray，不同之处在于张量可以在 GPU 或其他专用硬件上运行以加速计算。如果您熟悉 ndarrays，那么您将熟悉 Tensor API。如果没有，请按照此快速 API 演练进行操作。" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 2, 51 | "metadata": { 52 | "pycharm": { 53 | "name": "#%%\n" 54 | } 55 | }, 56 | "outputs": [], 57 | "source": [ 58 | "import torch\n", 59 | "import numpy as np" 60 | ] 61 | }, 62 | { 63 | "cell_type": "markdown", 64 | "metadata": {}, 65 | "source": [ 66 | "## Tensor 初始化\n", 67 | "创建Tensor有多种方法，如： \n", 68 | "\n", 69 | "1. **直接从数据创建** \n", 70 | "可以直接利用数据创建tensor,数据类型会被自动推断出." 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": 3, 76 | "metadata": {}, 77 | "outputs": [ 78 | { 79 | "data": { 80 | "text/plain": "tensor([[1, 2],\n [3, 4]])" 81 | }, 82 | "execution_count": 3, 83 | "metadata": {}, 84 | "output_type": "execute_result" 85 | } 86 | ], 87 | "source": [ 88 | "data = [[1,2], [3,4]]\n", 89 | "x_data = torch.tensor(data)\n", 90 | "x_data" 91 | ] 92 | }, 93 | { 94 | "cell_type": "markdown", 95 | "metadata": {}, 96 | "source": [ 97 | "2. **来自一个Numpy 数组** \n", 98 | "\n", 99 | "Tensor 可以直接从numpy的array创建（反之亦然-参见 [Bridge with NumPy](https://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html#bridge-to-np-label)）" 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": 4, 105 | "metadata": {}, 106 | "outputs": [ 107 | { 108 | "data": { 109 | "text/plain": "tensor([[1, 2],\n [3, 4]])" 110 | }, 111 | "execution_count": 4, 112 | "metadata": {}, 113 | "output_type": "execute_result" 114 | } 115 | ], 116 | "source": [ 117 | "np_array = np.array(data)\n", 118 | "x_np = torch.from_numpy(np_array)\n", 119 | "x_np" 120 | ] 121 | }, 122 | { 123 | "cell_type": "markdown", 124 | "metadata": {}, 125 | "source": [ 126 | "3. **来自另一个tensor：** \n", 127 | "新的tensor保留了参数tensor的一些属性（形状，数据类型），除非显式覆盖" 128 | ] 129 | }, 130 | { 131 | "cell_type": "code", 132 | "execution_count": 5, 133 | "metadata": {}, 134 | "outputs": [ 135 | { 136 | "name": "stdout", 137 | "output_type": "stream", 138 | "text": [ 139 | "Ones Tensor: \n", 140 | " tensor([[1, 1],\n", 141 | " [1, 1]]) \n", 142 | "\n", 143 | "Random Tensor: \n", 144 | " tensor([[0.8782, 0.0328],\n", 145 | " [0.0179, 0.8443]]) \n", 146 | "\n" 147 | ] 148 | } 149 | ], 150 | "source": [ 151 | "x_ones = torch.ones_like(x_data) # 保留x_data的属性\n", 152 | "print(f\"Ones Tensor: \\n {x_ones} \\n\")\n", 153 | "\n", 154 | "\n", 155 | "x_rand = torch.rand_like(x_data, dtype=torch.float) # 覆盖x_data的数据类型\n", 156 | "print(f\"Random Tensor: \\n {x_rand} \\n\")\n" 157 | ] 158 | }, 159 | { 160 | "cell_type": "markdown", 161 | "metadata": {}, 162 | "source": [ 163 | "4. **使用随机数或常数创建** \n", 164 | "shape是关于tensor维度的一个元组，在下面的函数中，它决定了输出tensor的维数。" 165 | ] 166 | }, 167 | { 168 | "cell_type": "code", 169 | "execution_count": 6, 170 | "metadata": {}, 171 | "outputs": [ 172 | { 173 | "name": "stdout", 174 | "output_type": "stream", 175 | "text": [ 176 | "Random Tensor: \n", 177 | " tensor([[0.0575, 0.9938, 0.7409],\n", 178 | " [0.3115, 0.4109, 0.0989]]) \n", 179 | "\n", 180 | "Ones Tensor: \n", 181 | " tensor([[1., 1., 1.],\n", 182 | " [1., 1., 1.]]) \n", 183 | " \n", 184 | "Zeros Tensor: \n", 185 | " tensor([[0., 0., 0.],\n", 186 | " [0., 0., 0.]]) \n", 187 | " \n", 188 | " tensor([[0., 0., 0.],\n", 189 | " [0., 0., 0.]]) \n", 190 | "\n" 191 | ] 192 | } 193 | ], 194 | "source": [ 195 | "shape = (2,3)\n", 196 | "rand_tensor = torch.rand(shape)\n", 197 | "ones_tensor = torch.ones(shape) \n", 198 | "zeros_tensor = torch.zeros(shape)\n", 199 | "print(f\"Random Tensor: \\n {rand_tensor} \\n\")\n", 200 | "print(f\"Ones Tensor: \\n {ones_tensor} \\n \")\n", 201 | "print(\"Zeros Tensor: \\n \", zeros_tensor, f\"\\n \\n {zeros_tensor} \\n\")\n" 202 | ] 203 | }, 204 | { 205 | "cell_type": "markdown", 206 | "metadata": {}, 207 | "source": [ 208 | "## Tensor 属性 \n", 209 | "张量属性描述了它们的形状、数据类型和存储它们的设备。\n" 210 | ] 211 | }, 212 | { 213 | "cell_type": "code", 214 | "execution_count": 7, 215 | "metadata": {}, 216 | "outputs": [ 217 | { 218 | "name": "stdout", 219 | "output_type": "stream", 220 | "text": [ 221 | "Shape of tensor:torch.Size([2, 3])\n", 222 | "Datatype of tensor: torch.float32\n", 223 | "Datatype of tensor: torch.float32\n", 224 | "Device tensor is stored on: cpu\n", 225 | "Device tensor is stored on: cpu\n" 226 | ] 227 | } 228 | ], 229 | "source": [ 230 | "tensor = torch.rand(shape)\n", 231 | "\n", 232 | "print(f\"Shape of tensor:{tensor.shape}\")\n", 233 | "print(\"Datatype of tensor: \", tensor.dtype)\n", 234 | "print(f\"Datatype of tensor: {tensor.dtype}\")\n", 235 | "print(\"Device tensor is stored on:\", tensor.device)\n", 236 | "print(f\"Device tensor is stored on: {tensor.device}\")" 237 | ] 238 | }, 239 | { 240 | "cell_type": "markdown", 241 | "metadata": {}, 242 | "source": [ 243 | "## Tensor 运算 \n", 244 | "Tensor有超过100个操作，包括 transposing, indexing, slicing, mathematical operations, linear algebra, random sampling,更多详细的介绍请点击[这里](https://pytorch.org/docs/stable/torch.html) \n", 245 | "\n", 246 | "它们都可以在GPU上运行（速度通常比CPU快），如果你使用的是Colab，通过编辑>笔记本设置来分配一个GPU。" 247 | ] 248 | }, 249 | { 250 | "cell_type": "code", 251 | "execution_count": 8, 252 | "metadata": {}, 253 | "outputs": [ 254 | { 255 | "name": "stdout", 256 | "output_type": "stream", 257 | "text": [ 258 | "Device tensor is stored on: cuda:0\n" 259 | ] 260 | } 261 | ], 262 | "source": [ 263 | "# 如果有GPU的话，我们把Tensor移到GPU上\n", 264 | "\n", 265 | "if torch.cuda.is_available(): \n", 266 | " tensor = tensor.to(\"cuda\")\n", 267 | " print(f\"Device tensor is stored on: {tensor.device}\")" 268 | ] 269 | }, 270 | { 271 | "cell_type": "markdown", 272 | "metadata": {}, 273 | "source": [ 274 | "尝试列表中的一些操作。如果你熟悉NumPy API，你会发现tensor的API很容易使用。" 275 | ] 276 | }, 277 | { 278 | "cell_type": "markdown", 279 | "metadata": {}, 280 | "source": [ 281 | "**标准的numpy类索引和切片:**" 282 | ] 283 | }, 284 | { 285 | "cell_type": "code", 286 | "execution_count": 9, 287 | "metadata": {}, 288 | "outputs": [ 289 | { 290 | "name": "stdout", 291 | "output_type": "stream", 292 | "text": [ 293 | "tensor([[1., 0., 1., 1.],\n", 294 | " [1., 0., 1., 1.],\n", 295 | " [1., 0., 1., 1.],\n", 296 | " [1., 0., 1., 1.]])\n" 297 | ] 298 | } 299 | ], 300 | "source": [ 301 | "tensor = torch.ones(4, 4)\n", 302 | "tensor[:, 1] = 0 \n", 303 | "print(tensor)\n" 304 | ] 305 | }, 306 | { 307 | "cell_type": "markdown", 308 | "metadata": {}, 309 | "source": [ 310 | "**合并 Tensor** \n", 311 | "可以使用torch.cat来沿着特定维数连接一系列张量。 torch.stack另一个加入op的张量与torch.cat有细微的不同\n" 312 | ] 313 | }, 314 | { 315 | "cell_type": "code", 316 | "execution_count": 10, 317 | "metadata": {}, 318 | "outputs": [ 319 | { 320 | "data": { 321 | "text/plain": "tensor([[1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.],\n [1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.],\n [1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.],\n [1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.]])" 322 | }, 323 | "execution_count": 10, 324 | "metadata": {}, 325 | "output_type": "execute_result" 326 | } 327 | ], 328 | "source": [ 329 | "t1 = torch.cat([tensor, tensor, tensor], dim=1) # dim=0\n", 330 | "t1" 331 | ] 332 | }, 333 | { 334 | "cell_type": "markdown", 335 | "metadata": {}, 336 | "source": [ 337 | "**Tensor 乘积** \n" 338 | ] 339 | }, 340 | { 341 | "cell_type": "code", 342 | "execution_count": 11, 343 | "metadata": {}, 344 | "outputs": [ 345 | { 346 | "name": "stdout", 347 | "output_type": "stream", 348 | "text": [ 349 | "tensor.mul(tensor): \n", 350 | " tensor([[1., 0., 1., 1.],\n", 351 | " [1., 0., 1., 1.],\n", 352 | " [1., 0., 1., 1.],\n", 353 | " [1., 0., 1., 1.]]) \n", 354 | "\n" 355 | ] 356 | }, 357 | { 358 | "data": { 359 | "text/plain": "tensor([[1., 0., 1., 1.],\n [1., 0., 1., 1.],\n [1., 0., 1., 1.],\n [1., 0., 1., 1.]])" 360 | }, 361 | "execution_count": 11, 362 | "metadata": {}, 363 | "output_type": "execute_result" 364 | } 365 | ], 366 | "source": [ 367 | "# 这将计算元素的乘积\n", 368 | "print(f\"tensor.mul(tensor): \\n {tensor.mul(tensor)} \\n\")\n", 369 | "# 替代语法：\n", 370 | "tensor * tensor" 371 | ] 372 | }, 373 | { 374 | "cell_type": "markdown", 375 | "metadata": {}, 376 | "source": [ 377 | "下面示例计算两个tensor之间的矩阵乘法" 378 | ] 379 | }, 380 | { 381 | "cell_type": "code", 382 | "execution_count": 12, 383 | "metadata": {}, 384 | "outputs": [ 385 | { 386 | "name": "stdout", 387 | "output_type": "stream", 388 | "text": [ 389 | "tensor([[3., 3., 3., 3.],\n", 390 | " [3., 3., 3., 3.],\n", 391 | " [3., 3., 3., 3.],\n", 392 | " [3., 3., 3., 3.]]) \n", 393 | "\n", 394 | "tensor([[3., 3., 3., 3.],\n", 395 | " [3., 3., 3., 3.],\n", 396 | " [3., 3., 3., 3.],\n", 397 | " [3., 3., 3., 3.]])\n" 398 | ] 399 | } 400 | ], 401 | "source": [ 402 | "print(tensor.matmul(tensor.T), f'\\n')\n", 403 | "# 替代语法：\n", 404 | "print(tensor @ tensor.T)" 405 | ] 406 | }, 407 | { 408 | "cell_type": "markdown", 409 | "metadata": {}, 410 | "source": [ 411 | "**原地操作** \n", 412 | "带有后缀_的操作表示的是原地操作，例如： x.copy_(y), x.t_()将改变 x." 413 | ] 414 | }, 415 | { 416 | "cell_type": "code", 417 | "execution_count": 13, 418 | "metadata": {}, 419 | "outputs": [ 420 | { 421 | "name": "stdout", 422 | "output_type": "stream", 423 | "text": [ 424 | "tensor([[1., 0., 1., 1.],\n", 425 | " [1., 0., 1., 1.],\n", 426 | " [1., 0., 1., 1.],\n", 427 | " [1., 0., 1., 1.]]) \n", 428 | "\n" 429 | ] 430 | }, 431 | { 432 | "data": { 433 | "text/plain": "tensor([[6., 5., 6., 6.],\n [6., 5., 6., 6.],\n [6., 5., 6., 6.],\n [6., 5., 6., 6.]])" 434 | }, 435 | "execution_count": 13, 436 | "metadata": {}, 437 | "output_type": "execute_result" 438 | } 439 | ], 440 | "source": [ 441 | "print(tensor, \"\\n\")\n", 442 | "tensor.add_(5)\n", 443 | "tensor\n" 444 | ] 445 | }, 446 | { 447 | "cell_type": "markdown", 448 | "metadata": {}, 449 | "source": [ 450 | "- 注意 \n", 451 | "原地操作虽然会节省许多空间，但是由于会立刻清除历史记录所以在计算导数时可能会有问题，因此不建议使用" 452 | ] 453 | }, 454 | { 455 | "cell_type": "markdown", 456 | "metadata": {}, 457 | "source": [ 458 | "## Tensor 与 Numpy 桥梁 \n", 459 | "在CPU上的 Tensor和 NumPy arrays 可以共享它们的底层内存位置，改变一个会改变另一个。" 460 | ] 461 | }, 462 | { 463 | "cell_type": "markdown", 464 | "metadata": {}, 465 | "source": [ 466 | "**Tensor转换为Numpt 数组** \n" 467 | ] 468 | }, 469 | { 470 | "cell_type": "code", 471 | "execution_count": 14, 472 | "metadata": {}, 473 | "outputs": [ 474 | { 475 | "name": "stdout", 476 | "output_type": "stream", 477 | "text": [ 478 | "t: tensor([1., 1., 1., 1., 1.]) \n", 479 | " \n", 480 | "n: [1. 1. 1. 1. 1.]\n" 481 | ] 482 | } 483 | ], 484 | "source": [ 485 | "t = torch.ones(5)\n", 486 | "print(f\"t: {t} \\n \")\n", 487 | "\n", 488 | "n = t.numpy()\n", 489 | "print(f\"n: {n}\")" 490 | ] 491 | }, 492 | { 493 | "cell_type": "markdown", 494 | "metadata": {}, 495 | "source": [ 496 | "Tensor的变化反映在NumPy数组中。" 497 | ] 498 | }, 499 | { 500 | "cell_type": "code", 501 | "execution_count": 15, 502 | "metadata": {}, 503 | "outputs": [ 504 | { 505 | "name": "stdout", 506 | "output_type": "stream", 507 | "text": [ 508 | "t: tensor([2., 2., 2., 2., 2.]) \n", 509 | "\n", 510 | "n: [2. 2. 2. 2. 2.]\n" 511 | ] 512 | } 513 | ], 514 | "source": [ 515 | "t.add_(1)\n", 516 | "print(f\"t: {t} \\n\")\n", 517 | "print(f\"n: {n}\")\n" 518 | ] 519 | }, 520 | { 521 | "cell_type": "markdown", 522 | "metadata": {}, 523 | "source": [ 524 | "**Numpy数组转换为Tensor**" 525 | ] 526 | }, 527 | { 528 | "cell_type": "code", 529 | "execution_count": 16, 530 | "metadata": {}, 531 | "outputs": [ 532 | { 533 | "data": { 534 | "text/plain": "tensor([1., 1., 1., 1., 1.], dtype=torch.float64)" 535 | }, 536 | "execution_count": 16, 537 | "metadata": {}, 538 | "output_type": "execute_result" 539 | } 540 | ], 541 | "source": [ 542 | "n = np.ones(5)\n", 543 | "t = torch.from_numpy(n)\n", 544 | "t" 545 | ] 546 | }, 547 | { 548 | "cell_type": "markdown", 549 | "metadata": {}, 550 | "source": [ 551 | "NumPy数组的变化反映在tensor中" 552 | ] 553 | }, 554 | { 555 | "cell_type": "code", 556 | "execution_count": 17, 557 | "metadata": {}, 558 | "outputs": [ 559 | { 560 | "name": "stdout", 561 | "output_type": "stream", 562 | "text": [ 563 | "t: tensor([2., 2., 2., 2., 2.], dtype=torch.float64)\n", 564 | "n: [2. 2. 2. 2. 2.]\n" 565 | ] 566 | } 567 | ], 568 | "source": [ 569 | "np.add(n, 1, out=n)\n", 570 | "print(f\"t: {t}\")\n", 571 | "print(f\"n: {n}\")" 572 | ] 573 | }, 574 | { 575 | "cell_type": "code", 576 | "execution_count": 17, 577 | "metadata": {}, 578 | "outputs": [], 579 | "source": [] 580 | } 581 | ], 582 | "metadata": { 583 | "kernelspec": { 584 | "display_name": "Python 3", 585 | "language": "python", 586 | "name": "python3" 587 | }, 588 | "language_info": { 589 | "codemirror_mode": { 590 | "name": "ipython", 591 | "version": 3 592 | }, 593 | "file_extension": ".py", 594 | "mimetype": "text/x-python", 595 | "name": "python", 596 | "nbconvert_exporter": "python", 597 | "pygments_lexer": "ipython3", 598 | "version": "3.6.13" 599 | } 600 | }, 601 | "nbformat": 4, 602 | "nbformat_minor": 1 603 | } -------------------------------------------------------------------------------- /60-Minute-Guide/3_neural-networks.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "id": "32052676", 7 | "metadata": {}, 8 | "outputs": [], 9 | "source": [ 10 | "%matplotlib inline" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "id": "7149acdb", 16 | "metadata": {}, 17 | "source": [ 18 | "# 神经网络 \n", 19 | "\n", 20 | "可以使用torch.nn包来构建神经网络. \n", 21 | "\n", 22 | "你已知道autograd包,nn包依赖autograd包来定义模型并求导.一个nn.Module包含各个层和一个forward(input)方法,该方法返回output. \n", 23 | "\n", 24 | "例如,我们来看一下下面这个分类数字图像的网络.\n", 25 | "\n", 26 | "![neural](../assets/neural.png)\n", 27 | "\n", 28 | "他是一个简单的前馈神经网络,它接受一个输入,然后一层接着一层的输入,直到最后得到结果。\n", 29 | "\n", 30 | "神经网络的典型训练过程如下:\n", 31 | "\n", 32 | "- 定义神经网络模型,它有一些可学习的参数(或者权重); \n", 33 | "- 在数据集上迭代; \n", 34 | "* 通过神经网络处理输入; \n", 35 | "- 计算损失(输出结果和正确值的差距大小) \n", 36 | "- 将梯度反向传播回网络参数; \n", 37 | "- 更新网络的权重,主要使用如下简单的更新原则:weight = weight - learning_rate * gradient " 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "id": "87968ff2", 43 | "metadata": {}, 44 | "source": [ 45 | "## 定义网络 \n", 46 | "\n", 47 | "定义一个网络" 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": 11, 53 | "id": "ea350197", 54 | "metadata": {}, 55 | "outputs": [ 56 | { 57 | "name": "stdout", 58 | "output_type": "stream", 59 | "text": [ 60 | "Net(\n", 61 | " (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))\n", 62 | " (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n", 63 | " (fc1): Linear(in_features=400, out_features=120, bias=True)\n", 64 | " (fc2): Linear(in_features=120, out_features=84, bias=True)\n", 65 | " (fc3): Linear(in_features=84, out_features=10, bias=True)\n", 66 | ")\n" 67 | ] 68 | } 69 | ], 70 | "source": [ 71 | "import torch\n", 72 | "import torch.nn as nn \n", 73 | "import torch.nn.functional as F\n", 74 | "\n", 75 | "class Net(nn.Module):\n", 76 | " \n", 77 | " def __init__(self):\n", 78 | " super(Net, self).__init__()\n", 79 | " # 1 -- 输入图像通道数， 6 -- 输出通道数， 5x5 -- square convolution（平方卷积）\n", 80 | " # kernel\n", 81 | " self.conv1 = nn.Conv2d(1, 6, 5)\n", 82 | " self.conv2 = nn.Conv2d(6, 16, 5)\n", 83 | " # 仿射运算 -- an affine operation: y = Wx + b\n", 84 | " self.fc1 = nn.Linear(16 * 5 * 5, 120) # 5*5 from image dimension(图像维度)\n", 85 | " self.fc2 = nn.Linear(120, 84)\n", 86 | " self.fc3 = nn.Linear(84, 10)\n", 87 | " \n", 88 | " \n", 89 | " def forward(self, x):\n", 90 | " # Max pooling over a (2, 2) window\n", 91 | " x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))\n", 92 | " # If the size is a square, you can specify with a single number\n", 93 | " #如果大小为正方形，则可以使用单个数字指定\n", 94 | " x = F.max_pool2d(F.relu(self.conv2(x)), 2)\n", 95 | " x = torch.flatten(x, 1) # flatten all dimensions except the batch dimension\n", 96 | " # 展开除批次维度以外的所有维度\n", 97 | " x = F.relu(self.fc1(x))\n", 98 | " x = F.relu(self.fc2(x))\n", 99 | " x = self.fc3(x)\n", 100 | " return x\n", 101 | " \n", 102 | "net = Net()\n", 103 | "print(net)\n", 104 | "\n", 105 | " \n" 106 | ] 107 | }, 108 | { 109 | "cell_type": "markdown", 110 | "id": "d813e4e4", 111 | "metadata": {}, 112 | "source": [ 113 | "你只需定义forward函数,backward函数(计算梯度)在使用autograd时自动为你创建.你可以在forward函数中使用Tensor的任何操作。\n", 114 | "\n", 115 | "net.parameters()返回模型需要学习的参数。" 116 | ] 117 | }, 118 | { 119 | "cell_type": "code", 120 | "execution_count": 12, 121 | "id": "007deca0", 122 | "metadata": {}, 123 | "outputs": [ 124 | { 125 | "name": "stdout", 126 | "output_type": "stream", 127 | "text": [ 128 | "10\n", 129 | "torch.Size([6, 1, 5, 5])\n" 130 | ] 131 | } 132 | ], 133 | "source": [ 134 | "params = list(net.parameters())\n", 135 | "print(len(params))\n", 136 | "print(params[0].size()) # conv1's .weight" 137 | ] 138 | }, 139 | { 140 | "cell_type": "markdown", 141 | "id": "0982487d", 142 | "metadata": {}, 143 | "source": [ 144 | "构造一个随机的32*32的输入，注意:这个网络(LeNet)期望的输入大小是32*32.如果使用MNIST数据集来训练这个网络,请把图片大小重新调整到32*32." 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": 16, 150 | "id": "c456efd3", 151 | "metadata": {}, 152 | "outputs": [ 153 | { 154 | "data": { 155 | "text/plain": [ 156 | "tensor([[-0.0395, -0.0515, 0.1344, 0.0207, -0.0588, -0.0194, 0.0373, 0.0187,\n", 157 | " 0.0234, -0.0002]], grad_fn=)" 158 | ] 159 | }, 160 | "execution_count": 16, 161 | "metadata": {}, 162 | "output_type": "execute_result" 163 | } 164 | ], 165 | "source": [ 166 | "input = torch.randn(1, 1, 32, 32)\n", 167 | "out = net(input)\n", 168 | "out\n" 169 | ] 170 | }, 171 | { 172 | "cell_type": "markdown", 173 | "id": "dc54b2b1", 174 | "metadata": {}, 175 | "source": [ 176 | "将所有参数的梯度缓存清零,然后进行随机梯度的的反向传播." 177 | ] 178 | }, 179 | { 180 | "cell_type": "code", 181 | "execution_count": 17, 182 | "id": "fad2d828", 183 | "metadata": {}, 184 | "outputs": [], 185 | "source": [ 186 | "net.zero_grad()\n", 187 | "out.backward(torch.randn(1, 10))" 188 | ] 189 | }, 190 | { 191 | "cell_type": "markdown", 192 | "id": "58ae1490", 193 | "metadata": {}, 194 | "source": [ 195 | "> 注意： \n", 196 | "torch.nn仅支持小批量。整个torch.nn 包仅支持小批量样本的输入，而不支持单个样本。 \n", 197 | "例如，nn.Conv2d将采用 4D 张量 。nSamples x nChannels x Height x Width \n", 198 | "如果您只有一个样本，只需使用input.unsqueeze(0)添加一个伪批次维度即可。\n", 199 | "\n", 200 | "\n", 201 | "在继续之前，让我们回顾一下您目前看到的所有类。" 202 | ] 203 | }, 204 | { 205 | "cell_type": "markdown", 206 | "id": "964471f7", 207 | "metadata": {}, 208 | "source": [ 209 | "**回顾** \n", 210 | "\n", 211 | "- torch.Tensor - 一个多维数组，支持像backward(). 还持有张量的梯度w.r.t.\n", 212 | "- nn.Module - 神经网络模块。封装参数的便捷方式，带有将它们移动到 GPU、导出、加载等的帮助程序。\n", 213 | "- nn.Parameter - 一种张量，当作为属性分配给 Module.\n", 214 | "- autograd.Function - 实现autograd 操作的向前和向后定义。每个Tensor操作至少创建一个Function节点，该节点连接到创建一个 Tensor并编码其历史的函数。\n", 215 | "\n", 216 | "在这一节，我们介绍了：\n", 217 | "- 定义神经网络\n", 218 | "- 处理输入并向后传递\n", 219 | "还剩：\n", 220 | "- 计算损失\n", 221 | "- 更新网络的权重\n", 222 | "\n" 223 | ] 224 | }, 225 | { 226 | "cell_type": "markdown", 227 | "id": "48ae5c7f", 228 | "metadata": {}, 229 | "source": [ 230 | "## 损失函数 \n", 231 | "\n", 232 | "损失函数采用（输出，目标）对输入，并计算一个值，该值估计输出与目标的距离。\n", 233 | "\n", 234 | "nn 包下有几个不同的 损失函数。一个简单的损失是：`nn.MSELoss`它计算输入和目标之间的均方误差。" 235 | ] 236 | }, 237 | { 238 | "cell_type": "code", 239 | "execution_count": 18, 240 | "id": "a4588259", 241 | "metadata": {}, 242 | "outputs": [ 243 | { 244 | "name": "stdout", 245 | "output_type": "stream", 246 | "text": [ 247 | "tensor(1.9755, grad_fn=)\n" 248 | ] 249 | } 250 | ], 251 | "source": [ 252 | "output = net(input)\n", 253 | "target = torch.randn(10)\n", 254 | "target = target.view(1, -1)\n", 255 | "criterion = nn.MSELoss()\n", 256 | "\n", 257 | "loss = criterion(output, target)\n", 258 | "print(loss)" 259 | ] 260 | }, 261 | { 262 | "cell_type": "markdown", 263 | "id": "437a5896", 264 | "metadata": {}, 265 | "source": [ 266 | "现在,你反向跟踪loss,使用它的.grad_fn属性,你会看到向下面这样的一个计算图: \n", 267 | "\n", 268 | "```\n", 269 | "input -> conv2d -> relu -> maxpool2d -> conv2d -> relu -> maxpool2d\n", 270 | " -> flatten -> linear -> relu -> linear -> relu -> linear\n", 271 | " -> MSELoss\n", 272 | " -> loss\n", 273 | " \n", 274 | "```" 275 | ] 276 | }, 277 | { 278 | "cell_type": "markdown", 279 | "id": "24a6dc58", 280 | "metadata": {}, 281 | "source": [ 282 | "所以, 当你调用loss.backward(),整个图被区分为损失以及图中所有具有requires_grad = True的张量，并且其.grad 张量的梯度累积。\n", 283 | "\n", 284 | "为了说明,我们反向跟踪几步:" 285 | ] 286 | }, 287 | { 288 | "cell_type": "code", 289 | "execution_count": 21, 290 | "id": "68a7fd26", 291 | "metadata": {}, 292 | "outputs": [ 293 | { 294 | "name": "stdout", 295 | "output_type": "stream", 296 | "text": [ 297 | "\n", 298 | "\n", 299 | "\n" 300 | ] 301 | } 302 | ], 303 | "source": [ 304 | "print(loss.grad_fn) # MSELoss\n", 305 | "print(loss.grad_fn.next_functions[0][0]) # Linear\n", 306 | "print(loss.grad_fn.next_functions[0][0].next_functions[0][0]) # ReLU" 307 | ] 308 | }, 309 | { 310 | "cell_type": "markdown", 311 | "id": "91a64445", 312 | "metadata": {}, 313 | "source": [ 314 | "## Backprop -- 反向传播 \n", 315 | "\n", 316 | "为了反向传播误差,我们所需做的是调用loss.backward().你需要清除已存在的梯度,否则梯度将被累加到已存在的梯度。\n", 317 | "\n", 318 | "现在,我们将调用loss.backward(),并查看conv1层的偏置项在反向传播前后的梯度。" 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": 24, 324 | "id": "3060dd17", 325 | "metadata": {}, 326 | "outputs": [ 327 | { 328 | "name": "stdout", 329 | "output_type": "stream", 330 | "text": [ 331 | "conv1.bias.grad before backward\n", 332 | "tensor([0., 0., 0., 0., 0., 0.])\n" 333 | ] 334 | }, 335 | { 336 | "ename": "RuntimeError", 337 | "evalue": "Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.", 338 | "output_type": "error", 339 | "traceback": [ 340 | "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", 341 | "\u001b[0;31mRuntimeError\u001b[0m Traceback (most recent call last)", 342 | "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnet\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconv1\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mloss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'conv1.bias.grad after backward'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 343 | "\u001b[0;32m~/miniconda3/envs/fsdet/lib/python3.6/site-packages/torch/tensor.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(self, gradient, retain_graph, create_graph)\u001b[0m\n\u001b[1;32m 183\u001b[0m \u001b[0mproducts\u001b[0m\u001b[0;34m.\u001b[0m \u001b[0mDefaults\u001b[0m \u001b[0mto\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 184\u001b[0m \"\"\"\n\u001b[0;32m--> 185\u001b[0;31m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgradient\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 186\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 187\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mregister_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 344 | "\u001b[0;32m~/miniconda3/envs/fsdet/lib/python3.6/site-packages/torch/autograd/__init__.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(tensors, grad_tensors, retain_graph, create_graph, grad_variables)\u001b[0m\n\u001b[1;32m 125\u001b[0m Variable._execution_engine.run_backward(\n\u001b[1;32m 126\u001b[0m \u001b[0mtensors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad_tensors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 127\u001b[0;31m allow_unreachable=True) # allow_unreachable flag\n\u001b[0m\u001b[1;32m 128\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 129\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", 345 | "\u001b[0;31mRuntimeError\u001b[0m: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time." 346 | ] 347 | } 348 | ], 349 | "source": [ 350 | "net.zero_grad() #将所有参数的渐变缓冲区归零 \n", 351 | "\n", 352 | "print(\"conv1.bias.grad before backward\")\n", 353 | "print(net.conv1.bias.grad)\n", 354 | "\n", 355 | "loss.backward()\n", 356 | "\n", 357 | "print('conv1.bias.grad after backward')\n", 358 | "print(net.conv1.bias.grad)" 359 | ] 360 | }, 361 | { 362 | "cell_type": "markdown", 363 | "id": "898c62c3", 364 | "metadata": {}, 365 | "source": [ 366 | "现在，我们已经看到了如何使用损失函数。\n", 367 | "\n", 368 | "稍后阅读:\n", 369 | "神经网络包包含了各种用来构成深度神经网络构建块的模块和损失函数,一份完整的文档查看[这里](https://pytorch.org/docs/nn)" 370 | ] 371 | }, 372 | { 373 | "cell_type": "markdown", 374 | "id": "160ef1d7", 375 | "metadata": {}, 376 | "source": [ 377 | "## 更新权重\n", 378 | "\n", 379 | "实践中使用的最简单的更新规则是随机梯度下降 (SGD)：\n", 380 | "\n", 381 | "```\n", 382 | "weight = weight - learning_rate * gradient\n", 383 | "```\n", 384 | "我们可以使用简单的 Python 代码来实现：" 385 | ] 386 | }, 387 | { 388 | "cell_type": "code", 389 | "execution_count": 25, 390 | "id": "e59d0bb2", 391 | "metadata": {}, 392 | "outputs": [], 393 | "source": [ 394 | "learning_rate = 0.01\n", 395 | "for f in net.parameters():\n", 396 | " f.data.sub_(f.grad.data * learning_rate)" 397 | ] 398 | }, 399 | { 400 | "cell_type": "markdown", 401 | "id": "7177733b", 402 | "metadata": {}, 403 | "source": [ 404 | "但是，当您使用神经网络时，您希望使用各种不同的更新规则，例如 SGD、Nesterov-SGD、Adam、RMSProp 等。为了实现这一点，我们构建了一个小包：torch.optim它实现了所有这些方法。使用它非常简单：" 405 | ] 406 | }, 407 | { 408 | "cell_type": "code", 409 | "execution_count": 26, 410 | "id": "3cd1d6d3", 411 | "metadata": {}, 412 | "outputs": [], 413 | "source": [ 414 | "import torch.optim as optim\n", 415 | "\n", 416 | "# 创建你的优化器\n", 417 | "optimizer = optim.SGD(net.parameters(), lr=0.01)\n", 418 | "\n", 419 | "# in your training loop:\n", 420 | "optimizer.zero_grad() # zero the gradient buffers -- 梯度缓冲区归零\n", 421 | "output = net(input)\n", 422 | "loss = criterion(output, target)\n", 423 | "loss.backward() \n", 424 | "optimizer.step() # Does the update" 425 | ] 426 | }, 427 | { 428 | "cell_type": "markdown", 429 | "id": "9a07d654", 430 | "metadata": {}, 431 | "source": [ 432 | "**注意**\n", 433 | "> 观察如何使用optimizer.zero_grad()手动将梯度缓冲区设置为零。 这是因为梯度是反向传播部分中的说明那样是累积的。" 434 | ] 435 | }, 436 | { 437 | "cell_type": "code", 438 | "execution_count": null, 439 | "id": "bedf8f86", 440 | "metadata": {}, 441 | "outputs": [], 442 | "source": [] 443 | } 444 | ], 445 | "metadata": { 446 | "kernelspec": { 447 | "display_name": "Python 3", 448 | "language": "python", 449 | "name": "python3" 450 | }, 451 | "language_info": { 452 | "codemirror_mode": { 453 | "name": "ipython", 454 | "version": 3 455 | }, 456 | "file_extension": ".py", 457 | "mimetype": "text/x-python", 458 | "name": "python", 459 | "nbconvert_exporter": "python", 460 | "pygments_lexer": "ipython3", 461 | "version": "3.6.13" 462 | } 463 | }, 464 | "nbformat": 4, 465 | "nbformat_minor": 5 466 | } 467 | -------------------------------------------------------------------------------- /60-Minute-Guide/autograd_tutorial_en.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%matplotlib inline" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "\nA Gentle Introduction to ``torch.autograd``\n---------------------------------\n\n``torch.autograd`` is PyTorch\u2019s automatic differentiation engine that powers\nneural network training. In this section, you will get a conceptual\nunderstanding of how autograd helps a neural network train.\n\nBackground\n~~~~~~~~~~\nNeural networks (NNs) are a collection of nested functions that are\nexecuted on some input data. These functions are defined by *parameters*\n(consisting of weights and biases), which in PyTorch are stored in\ntensors.\n\nTraining a NN happens in two steps:\n\n**Forward Propagation**: In forward prop, the NN makes its best guess\nabout the correct output. It runs the input data through each of its\nfunctions to make this guess.\n\n**Backward Propagation**: In backprop, the NN adjusts its parameters\nproportionate to the error in its guess. It does this by traversing\nbackwards from the output, collecting the derivatives of the error with\nrespect to the parameters of the functions (*gradients*), and optimizing\nthe parameters using gradient descent. For a more detailed walkthrough\nof backprop, check out this `video from\n3Blue1Brown `__.\n\n\n\n\nUsage in PyTorch\n~~~~~~~~~~~\nLet's take a look at a single training step.\nFor this example, we load a pretrained resnet18 model from ``torchvision``.\nWe create a random data tensor to represent a single image with 3 channels, and height & width of 64,\nand its corresponding ``label`` initialized to some random values.\n\n" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": { 25 | "collapsed": false 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "import torch, torchvision\nmodel = torchvision.models.resnet18(pretrained=True)\ndata = torch.rand(1, 3, 64, 64)\nlabels = torch.rand(1, 1000)" 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "metadata": {}, 35 | "source": [ 36 | "Next, we run the input data through the model through each of its layers to make a prediction.\nThis is the **forward pass**.\n\n\n" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": null, 42 | "metadata": { 43 | "collapsed": false 44 | }, 45 | "outputs": [], 46 | "source": [ 47 | "prediction = model(data) # forward pass" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": {}, 53 | "source": [ 54 | "We use the model's prediction and the corresponding label to calculate the error (``loss``).\nThe next step is to backpropagate this error through the network.\nBackward propagation is kicked off when we call ``.backward()`` on the error tensor.\nAutograd then calculates and stores the gradients for each model parameter in the parameter's ``.grad`` attribute.\n\n\n" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": { 61 | "collapsed": false 62 | }, 63 | "outputs": [], 64 | "source": [ 65 | "loss = (prediction - labels).sum()\nloss.backward() # backward pass" 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "metadata": {}, 71 | "source": [ 72 | "Next, we load an optimizer, in this case SGD with a learning rate of 0.01 and momentum of 0.9.\nWe register all the parameters of the model in the optimizer.\n\n\n" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": null, 78 | "metadata": { 79 | "collapsed": false 80 | }, 81 | "outputs": [], 82 | "source": [ 83 | "optim = torch.optim.SGD(model.parameters(), lr=1e-2, momentum=0.9)" 84 | ] 85 | }, 86 | { 87 | "cell_type": "markdown", 88 | "metadata": {}, 89 | "source": [ 90 | "Finally, we call ``.step()`` to initiate gradient descent. The optimizer adjusts each parameter by its gradient stored in ``.grad``.\n\n\n" 91 | ] 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": null, 96 | "metadata": { 97 | "collapsed": false 98 | }, 99 | "outputs": [], 100 | "source": [ 101 | "optim.step() #gradient descent" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "At this point, you have everything you need to train your neural network.\nThe below sections detail the workings of autograd - feel free to skip them.\n\n\n" 109 | ] 110 | }, 111 | { 112 | "cell_type": "markdown", 113 | "metadata": {}, 114 | "source": [ 115 | "--------------\n\n\n" 116 | ] 117 | }, 118 | { 119 | "cell_type": "markdown", 120 | "metadata": {}, 121 | "source": [ 122 | "Differentiation in Autograd\n~~~~~~~~~~~~~~~~~~~~~~~~~~~\nLet's take a look at how ``autograd`` collects gradients. We create two tensors ``a`` and ``b`` with\n``requires_grad=True``. This signals to ``autograd`` that every operation on them should be tracked.\n\n\n" 123 | ] 124 | }, 125 | { 126 | "cell_type": "code", 127 | "execution_count": null, 128 | "metadata": { 129 | "collapsed": false 130 | }, 131 | "outputs": [], 132 | "source": [ 133 | "import torch\n\na = torch.tensor([2., 3.], requires_grad=True)\nb = torch.tensor([6., 4.], requires_grad=True)" 134 | ] 135 | }, 136 | { 137 | "cell_type": "markdown", 138 | "metadata": {}, 139 | "source": [ 140 | "We create another tensor ``Q`` from ``a`` and ``b``.\n\n\\begin{align}Q = 3a^3 - b^2\\end{align}\n\n" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": null, 146 | "metadata": { 147 | "collapsed": false 148 | }, 149 | "outputs": [], 150 | "source": [ 151 | "Q = 3*a**3 - b**2" 152 | ] 153 | }, 154 | { 155 | "cell_type": "markdown", 156 | "metadata": {}, 157 | "source": [ 158 | "Let's assume ``a`` and ``b`` to be parameters of an NN, and ``Q``\nto be the error. In NN training, we want gradients of the error\nw.r.t. parameters, i.e.\n\n\\begin{align}\\frac{\\partial Q}{\\partial a} = 9a^2\\end{align}\n\n\\begin{align}\\frac{\\partial Q}{\\partial b} = -2b\\end{align}\n\n\nWhen we call ``.backward()`` on ``Q``, autograd calculates these gradients\nand stores them in the respective tensors' ``.grad`` attribute.\n\nWe need to explicitly pass a ``gradient`` argument in ``Q.backward()`` because it is a vector.\n``gradient`` is a tensor of the same shape as ``Q``, and it represents the\ngradient of Q w.r.t. itself, i.e.\n\n\\begin{align}\\frac{dQ}{dQ} = 1\\end{align}\n\nEquivalently, we can also aggregate Q into a scalar and call backward implicitly, like ``Q.sum().backward()``.\n\n\n" 159 | ] 160 | }, 161 | { 162 | "cell_type": "code", 163 | "execution_count": null, 164 | "metadata": { 165 | "collapsed": false 166 | }, 167 | "outputs": [], 168 | "source": [ 169 | "external_grad = torch.tensor([1., 1.])\nQ.backward(gradient=external_grad)" 170 | ] 171 | }, 172 | { 173 | "cell_type": "markdown", 174 | "metadata": {}, 175 | "source": [ 176 | "Gradients are now deposited in ``a.grad`` and ``b.grad``\n\n" 177 | ] 178 | }, 179 | { 180 | "cell_type": "code", 181 | "execution_count": null, 182 | "metadata": { 183 | "collapsed": false 184 | }, 185 | "outputs": [], 186 | "source": [ 187 | "# check if collected gradients are correct\nprint(9*a**2 == a.grad)\nprint(-2*b == b.grad)" 188 | ] 189 | }, 190 | { 191 | "cell_type": "markdown", 192 | "metadata": {}, 193 | "source": [ 194 | "Optional Reading - Vector Calculus using ``autograd``\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nMathematically, if you have a vector valued function\n$\\vec{y}=f(\\vec{x})$, then the gradient of $\\vec{y}$ with\nrespect to $\\vec{x}$ is a Jacobian matrix $J$:\n\n\\begin{align}J\n =\n \\left(\\begin{array}{cc}\n \\frac{\\partial \\bf{y}}{\\partial x_{1}} &\n ... &\n \\frac{\\partial \\bf{y}}{\\partial x_{n}}\n \\end{array}\\right)\n =\n \\left(\\begin{array}{ccc}\n \\frac{\\partial y_{1}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{1}}{\\partial x_{n}}\\\\\n \\vdots & \\ddots & \\vdots\\\\\n \\frac{\\partial y_{m}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{n}}\n \\end{array}\\right)\\end{align}\n\nGenerally speaking, ``torch.autograd`` is an engine for computing\nvector-Jacobian product. That is, given any vector $\\vec{v}$, compute the product\n$J^{T}\\cdot \\vec{v}$\n\nIf $\\vec{v}$ happens to be the gradient of a scalar function $l=g\\left(\\vec{y}\\right)$:\n\n\\begin{align}\\vec{v}\n =\n \\left(\\begin{array}{ccc}\\frac{\\partial l}{\\partial y_{1}} & \\cdots & \\frac{\\partial l}{\\partial y_{m}}\\end{array}\\right)^{T}\\end{align}\n\nthen by the chain rule, the vector-Jacobian product would be the\ngradient of $l$ with respect to $\\vec{x}$:\n\n\\begin{align}J^{T}\\cdot \\vec{v}=\\left(\\begin{array}{ccc}\n \\frac{\\partial y_{1}}{\\partial x_{1}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{1}}\\\\\n \\vdots & \\ddots & \\vdots\\\\\n \\frac{\\partial y_{1}}{\\partial x_{n}} & \\cdots & \\frac{\\partial y_{m}}{\\partial x_{n}}\n \\end{array}\\right)\\left(\\begin{array}{c}\n \\frac{\\partial l}{\\partial y_{1}}\\\\\n \\vdots\\\\\n \\frac{\\partial l}{\\partial y_{m}}\n \\end{array}\\right)=\\left(\\begin{array}{c}\n \\frac{\\partial l}{\\partial x_{1}}\\\\\n \\vdots\\\\\n \\frac{\\partial l}{\\partial x_{n}}\n \\end{array}\\right)\\end{align}\n\nThis characteristic of vector-Jacobian product is what we use in the above example;\n``external_grad`` represents $\\vec{v}$.\n\n\n" 195 | ] 196 | }, 197 | { 198 | "cell_type": "markdown", 199 | "metadata": {}, 200 | "source": [ 201 | "Computational Graph\n~~~~~~~~~~~~~~~~~~~\n\nConceptually, autograd keeps a record of data (tensors) & all executed\noperations (along with the resulting new tensors) in a directed acyclic\ngraph (DAG) consisting of\n`Function `__\nobjects. In this DAG, leaves are the input tensors, roots are the output\ntensors. By tracing this graph from roots to leaves, you can\nautomatically compute the gradients using the chain rule.\n\nIn a forward pass, autograd does two things simultaneously:\n\n- run the requested operation to compute a resulting tensor, and\n- maintain the operation\u2019s *gradient function* in the DAG.\n\nThe backward pass kicks off when ``.backward()`` is called on the DAG\nroot. ``autograd`` then:\n\n- computes the gradients from each ``.grad_fn``,\n- accumulates them in the respective tensor\u2019s ``.grad`` attribute, and\n- using the chain rule, propagates all the way to the leaf tensors.\n\nBelow is a visual representation of the DAG in our example. In the graph,\nthe arrows are in the direction of the forward pass. The nodes represent the backward functions\nof each operation in the forward pass. The leaf nodes in blue represent our leaf tensors ``a`` and ``b``.\n\n.. figure:: /_static/img/dag_autograd.png\n\n

Note

**DAGs are dynamic in PyTorch**\n An important thing to note is that the graph is recreated from scratch; after each\n ``.backward()`` call, autograd starts populating a new graph. This is\n exactly what allows you to use control flow statements in your model;\n you can change the shape, size and operations at every iteration if\n needed.

\n\nExclusion from the DAG\n^^^^^^^^^^^^^^^^^^^^^^\n\n``torch.autograd`` tracks operations on all tensors which have their\n``requires_grad`` flag set to ``True``. For tensors that don\u2019t require\ngradients, setting this attribute to ``False`` excludes it from the\ngradient computation DAG.\n\nThe output tensor of an operation will require gradients even if only a\nsingle input tensor has ``requires_grad=True``.\n\n\n" 202 | ] 203 | }, 204 | { 205 | "cell_type": "code", 206 | "execution_count": null, 207 | "metadata": { 208 | "collapsed": false 209 | }, 210 | "outputs": [], 211 | "source": [ 212 | "x = torch.rand(5, 5)\ny = torch.rand(5, 5)\nz = torch.rand((5, 5), requires_grad=True)\n\na = x + y\nprint(f\"Does `a` require gradients? : {a.requires_grad}\")\nb = x + z\nprint(f\"Does `b` require gradients?: {b.requires_grad}\")" 213 | ] 214 | }, 215 | { 216 | "cell_type": "markdown", 217 | "metadata": {}, 218 | "source": [ 219 | "In a NN, parameters that don't compute gradients are usually called **frozen parameters**.\nIt is useful to \"freeze\" part of your model if you know in advance that you won't need the gradients of those parameters\n(this offers some performance benefits by reducing autograd computations).\n\nAnother common usecase where exclusion from the DAG is important is for\n`finetuning a pretrained network `__\n\nIn finetuning, we freeze most of the model and typically only modify the classifier layers to make predictions on new labels.\nLet's walk through a small example to demonstrate this. As before, we load a pretrained resnet18 model, and freeze all the parameters.\n\n" 220 | ] 221 | }, 222 | { 223 | "cell_type": "code", 224 | "execution_count": null, 225 | "metadata": { 226 | "collapsed": false 227 | }, 228 | "outputs": [], 229 | "source": [ 230 | "from torch import nn, optim\n\nmodel = torchvision.models.resnet18(pretrained=True)\n\n# Freeze all the parameters in the network\nfor param in model.parameters():\n param.requires_grad = False" 231 | ] 232 | }, 233 | { 234 | "cell_type": "markdown", 235 | "metadata": {}, 236 | "source": [ 237 | "Let's say we want to finetune the model on a new dataset with 10 labels.\nIn resnet, the classifier is the last linear layer ``model.fc``.\nWe can simply replace it with a new linear layer (unfrozen by default)\nthat acts as our classifier.\n\n" 238 | ] 239 | }, 240 | { 241 | "cell_type": "code", 242 | "execution_count": null, 243 | "metadata": { 244 | "collapsed": false 245 | }, 246 | "outputs": [], 247 | "source": [ 248 | "model.fc = nn.Linear(512, 10)" 249 | ] 250 | }, 251 | { 252 | "cell_type": "markdown", 253 | "metadata": {}, 254 | "source": [ 255 | "Now all parameters in the model, except the parameters of ``model.fc``, are frozen.\nThe only parameters that compute gradients are the weights and bias of ``model.fc``.\n\n" 256 | ] 257 | }, 258 | { 259 | "cell_type": "code", 260 | "execution_count": null, 261 | "metadata": { 262 | "collapsed": false 263 | }, 264 | "outputs": [], 265 | "source": [ 266 | "# Optimize only the classifier\noptimizer = optim.SGD(model.parameters(), lr=1e-2, momentum=0.9)" 267 | ] 268 | }, 269 | { 270 | "cell_type": "markdown", 271 | "metadata": {}, 272 | "source": [ 273 | "Notice although we register all the parameters in the optimizer,\nthe only parameters that are computing gradients (and hence updated in gradient descent)\nare the weights and bias of the classifier.\n\nThe same exclusionary functionality is available as a context manager in\n`torch.no_grad() `__\n\n\n" 274 | ] 275 | }, 276 | { 277 | "cell_type": "markdown", 278 | "metadata": {}, 279 | "source": [ 280 | "--------------\n\n\n" 281 | ] 282 | }, 283 | { 284 | "cell_type": "markdown", 285 | "metadata": {}, 286 | "source": [ 287 | "Further readings:\n~~~~~~~~~~~~~~~~~~~\n\n- `In-place operations & Multithreaded Autograd `__\n- `Example implementation of reverse-mode autodiff `__\n\n" 288 | ] 289 | } 290 | ], 291 | "metadata": { 292 | "kernelspec": { 293 | "display_name": "Python 3", 294 | "language": "python", 295 | "name": "python3" 296 | }, 297 | "language_info": { 298 | "codemirror_mode": { 299 | "name": "ipython", 300 | "version": 3 301 | }, 302 | "file_extension": ".py", 303 | "mimetype": "text/x-python", 304 | "name": "python", 305 | "nbconvert_exporter": "python", 306 | "pygments_lexer": "ipython3", 307 | "version": "3.6.13" 308 | } 309 | }, 310 | "nbformat": 4, 311 | "nbformat_minor": 0 312 | } -------------------------------------------------------------------------------- /60-Minute-Guide/cifar10_tutorial_en.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%matplotlib inline" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "\nTraining a Classifier\n=====================\n\nThis is it. You have seen how to define neural networks, compute loss and make\nupdates to the weights of the network.\n\nNow you might be thinking,\n\nWhat about data?\n----------------\n\nGenerally, when you have to deal with image, text, audio or video data,\nyou can use standard python packages that load data into a numpy array.\nThen you can convert this array into a ``torch.*Tensor``.\n\n- For images, packages such as Pillow, OpenCV are useful\n- For audio, packages such as scipy and librosa\n- For text, either raw Python or Cython based loading, or NLTK and\n SpaCy are useful\n\nSpecifically for vision, we have created a package called\n``torchvision``, that has data loaders for common datasets such as\nImageNet, CIFAR10, MNIST, etc. and data transformers for images, viz.,\n``torchvision.datasets`` and ``torch.utils.data.DataLoader``.\n\nThis provides a huge convenience and avoids writing boilerplate code.\n\nFor this tutorial, we will use the CIFAR10 dataset.\nIt has the classes: \u2018airplane\u2019, \u2018automobile\u2019, \u2018bird\u2019, \u2018cat\u2019, \u2018deer\u2019,\n\u2018dog\u2019, \u2018frog\u2019, \u2018horse\u2019, \u2018ship\u2019, \u2018truck\u2019. The images in CIFAR-10 are of\nsize 3x32x32, i.e. 3-channel color images of 32x32 pixels in size.\n\n.. figure:: /_static/img/cifar10.png\n :alt: cifar10\n\n cifar10\n\n\nTraining an image classifier\n----------------------------\n\nWe will do the following steps in order:\n\n1. Load and normalize the CIFAR10 training and test datasets using\n ``torchvision``\n2. Define a Convolutional Neural Network\n3. Define a loss function\n4. Train the network on the training data\n5. Test the network on the test data\n\n1. Load and normalize CIFAR10\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nUsing ``torchvision``, it\u2019s extremely easy to load CIFAR10.\n\n" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": { 25 | "collapsed": false 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "import torch\nimport torchvision\nimport torchvision.transforms as transforms" 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "metadata": {}, 35 | "source": [ 36 | "The output of torchvision datasets are PILImage images of range [0, 1].\nWe transform them to Tensors of normalized range [-1, 1].\n\n" 37 | ] 38 | }, 39 | { 40 | "cell_type": "markdown", 41 | "metadata": {}, 42 | "source": [ 43 | "

Note

If running on Windows and you get a BrokenPipeError, try setting\n the num_worker of torch.utils.data.DataLoader() to 0.

\n\n" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": null, 49 | "metadata": { 50 | "collapsed": false 51 | }, 52 | "outputs": [], 53 | "source": [ 54 | "transform = transforms.Compose(\n [transforms.ToTensor(),\n transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])\n\nbatch_size = 4\n\ntrainset = torchvision.datasets.CIFAR10(root='./data', train=True,\n download=True, transform=transform)\ntrainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,\n shuffle=True, num_workers=2)\n\ntestset = torchvision.datasets.CIFAR10(root='./data', train=False,\n download=True, transform=transform)\ntestloader = torch.utils.data.DataLoader(testset, batch_size=batch_size,\n shuffle=False, num_workers=2)\n\nclasses = ('plane', 'car', 'bird', 'cat',\n 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')" 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "Let us show some of the training images, for fun.\n\n" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": null, 67 | "metadata": { 68 | "collapsed": false 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "import matplotlib.pyplot as plt\nimport numpy as np\n\n# functions to show an image\n\n\ndef imshow(img):\n img = img / 2 + 0.5 # unnormalize\n npimg = img.numpy()\n plt.imshow(np.transpose(npimg, (1, 2, 0)))\n plt.show()\n\n\n# get some random training images\ndataiter = iter(trainloader)\nimages, labels = dataiter.next()\n\n# show images\nimshow(torchvision.utils.make_grid(images))\n# print labels\nprint(' '.join('%5s' % classes[labels[j]] for j in range(batch_size)))" 73 | ] 74 | }, 75 | { 76 | "cell_type": "markdown", 77 | "metadata": {}, 78 | "source": [ 79 | "2. Define a Convolutional Neural Network\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nCopy the neural network from the Neural Networks section before and modify it to\ntake 3-channel images (instead of 1-channel images as it was defined).\n\n" 80 | ] 81 | }, 82 | { 83 | "cell_type": "code", 84 | "execution_count": null, 85 | "metadata": { 86 | "collapsed": false 87 | }, 88 | "outputs": [], 89 | "source": [ 90 | "import torch.nn as nn\nimport torch.nn.functional as F\n\n\nclass Net(nn.Module):\n def __init__(self):\n super().__init__()\n self.conv1 = nn.Conv2d(3, 6, 5)\n self.pool = nn.MaxPool2d(2, 2)\n self.conv2 = nn.Conv2d(6, 16, 5)\n self.fc1 = nn.Linear(16 * 5 * 5, 120)\n self.fc2 = nn.Linear(120, 84)\n self.fc3 = nn.Linear(84, 10)\n\n def forward(self, x):\n x = self.pool(F.relu(self.conv1(x)))\n x = self.pool(F.relu(self.conv2(x)))\n x = torch.flatten(x, 1) # flatten all dimensions except batch\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = self.fc3(x)\n return x\n\n\nnet = Net()" 91 | ] 92 | }, 93 | { 94 | "cell_type": "markdown", 95 | "metadata": {}, 96 | "source": [ 97 | "3. Define a Loss function and optimizer\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nLet's use a Classification Cross-Entropy loss and SGD with momentum.\n\n" 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": null, 103 | "metadata": { 104 | "collapsed": false 105 | }, 106 | "outputs": [], 107 | "source": [ 108 | "import torch.optim as optim\n\ncriterion = nn.CrossEntropyLoss()\noptimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)" 109 | ] 110 | }, 111 | { 112 | "cell_type": "markdown", 113 | "metadata": {}, 114 | "source": [ 115 | "4. Train the network\n^^^^^^^^^^^^^^^^^^^^\n\nThis is when things start to get interesting.\nWe simply have to loop over our data iterator, and feed the inputs to the\nnetwork and optimize.\n\n" 116 | ] 117 | }, 118 | { 119 | "cell_type": "code", 120 | "execution_count": null, 121 | "metadata": { 122 | "collapsed": false 123 | }, 124 | "outputs": [], 125 | "source": [ 126 | "for epoch in range(2): # loop over the dataset multiple times\n\n running_loss = 0.0\n for i, data in enumerate(trainloader, 0):\n # get the inputs; data is a list of [inputs, labels]\n inputs, labels = data\n\n # zero the parameter gradients\n optimizer.zero_grad()\n\n # forward + backward + optimize\n outputs = net(inputs)\n loss = criterion(outputs, labels)\n loss.backward()\n optimizer.step()\n\n # print statistics\n running_loss += loss.item()\n if i % 2000 == 1999: # print every 2000 mini-batches\n print('[%d, %5d] loss: %.3f' %\n (epoch + 1, i + 1, running_loss / 2000))\n running_loss = 0.0\n\nprint('Finished Training')" 127 | ] 128 | }, 129 | { 130 | "cell_type": "markdown", 131 | "metadata": {}, 132 | "source": [ 133 | "Let's quickly save our trained model:\n\n" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": null, 139 | "metadata": { 140 | "collapsed": false 141 | }, 142 | "outputs": [], 143 | "source": [ 144 | "PATH = './cifar_net.pth'\ntorch.save(net.state_dict(), PATH)" 145 | ] 146 | }, 147 | { 148 | "cell_type": "markdown", 149 | "metadata": {}, 150 | "source": [ 151 | "See `here `_\nfor more details on saving PyTorch models.\n\n5. Test the network on the test data\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWe have trained the network for 2 passes over the training dataset.\nBut we need to check if the network has learnt anything at all.\n\nWe will check this by predicting the class label that the neural network\noutputs, and checking it against the ground-truth. If the prediction is\ncorrect, we add the sample to the list of correct predictions.\n\nOkay, first step. Let us display an image from the test set to get familiar.\n\n" 152 | ] 153 | }, 154 | { 155 | "cell_type": "code", 156 | "execution_count": null, 157 | "metadata": { 158 | "collapsed": false 159 | }, 160 | "outputs": [], 161 | "source": [ 162 | "dataiter = iter(testloader)\nimages, labels = dataiter.next()\n\n# print images\nimshow(torchvision.utils.make_grid(images))\nprint('GroundTruth: ', ' '.join('%5s' % classes[labels[j]] for j in range(4)))" 163 | ] 164 | }, 165 | { 166 | "cell_type": "markdown", 167 | "metadata": {}, 168 | "source": [ 169 | "Next, let's load back in our saved model (note: saving and re-loading the model\nwasn't necessary here, we only did it to illustrate how to do so):\n\n" 170 | ] 171 | }, 172 | { 173 | "cell_type": "code", 174 | "execution_count": null, 175 | "metadata": { 176 | "collapsed": false 177 | }, 178 | "outputs": [], 179 | "source": [ 180 | "net = Net()\nnet.load_state_dict(torch.load(PATH))" 181 | ] 182 | }, 183 | { 184 | "cell_type": "markdown", 185 | "metadata": {}, 186 | "source": [ 187 | "Okay, now let us see what the neural network thinks these examples above are:\n\n" 188 | ] 189 | }, 190 | { 191 | "cell_type": "code", 192 | "execution_count": null, 193 | "metadata": { 194 | "collapsed": false 195 | }, 196 | "outputs": [], 197 | "source": [ 198 | "outputs = net(images)" 199 | ] 200 | }, 201 | { 202 | "cell_type": "markdown", 203 | "metadata": {}, 204 | "source": [ 205 | "The outputs are energies for the 10 classes.\nThe higher the energy for a class, the more the network\nthinks that the image is of the particular class.\nSo, let's get the index of the highest energy:\n\n" 206 | ] 207 | }, 208 | { 209 | "cell_type": "code", 210 | "execution_count": null, 211 | "metadata": { 212 | "collapsed": false 213 | }, 214 | "outputs": [], 215 | "source": [ 216 | "_, predicted = torch.max(outputs, 1)\n\nprint('Predicted: ', ' '.join('%5s' % classes[predicted[j]]\n for j in range(4)))" 217 | ] 218 | }, 219 | { 220 | "cell_type": "markdown", 221 | "metadata": {}, 222 | "source": [ 223 | "The results seem pretty good.\n\nLet us look at how the network performs on the whole dataset.\n\n" 224 | ] 225 | }, 226 | { 227 | "cell_type": "code", 228 | "execution_count": null, 229 | "metadata": { 230 | "collapsed": false 231 | }, 232 | "outputs": [], 233 | "source": [ 234 | "correct = 0\ntotal = 0\n# since we're not training, we don't need to calculate the gradients for our outputs\nwith torch.no_grad():\n for data in testloader:\n images, labels = data\n # calculate outputs by running images through the network \n outputs = net(images)\n # the class with the highest energy is what we choose as prediction\n _, predicted = torch.max(outputs.data, 1)\n total += labels.size(0)\n correct += (predicted == labels).sum().item()\n\nprint('Accuracy of the network on the 10000 test images: %d %%' % (\n 100 * correct / total))" 235 | ] 236 | }, 237 | { 238 | "cell_type": "markdown", 239 | "metadata": {}, 240 | "source": [ 241 | "That looks way better than chance, which is 10% accuracy (randomly picking\na class out of 10 classes).\nSeems like the network learnt something.\n\nHmmm, what are the classes that performed well, and the classes that did\nnot perform well:\n\n" 242 | ] 243 | }, 244 | { 245 | "cell_type": "code", 246 | "execution_count": null, 247 | "metadata": { 248 | "collapsed": false 249 | }, 250 | "outputs": [], 251 | "source": [ 252 | "# prepare to count predictions for each class\ncorrect_pred = {classname: 0 for classname in classes}\ntotal_pred = {classname: 0 for classname in classes}\n\n# again no gradients needed\nwith torch.no_grad():\n for data in testloader:\n images, labels = data \n outputs = net(images) \n _, predictions = torch.max(outputs, 1)\n # collect the correct predictions for each class\n for label, prediction in zip(labels, predictions):\n if label == prediction:\n correct_pred[classes[label]] += 1\n total_pred[classes[label]] += 1\n\n \n# print accuracy for each class\nfor classname, correct_count in correct_pred.items():\n accuracy = 100 * float(correct_count) / total_pred[classname]\n print(\"Accuracy for class {:5s} is: {:.1f} %\".format(classname, \n accuracy))" 253 | ] 254 | }, 255 | { 256 | "cell_type": "markdown", 257 | "metadata": {}, 258 | "source": [ 259 | "Okay, so what next?\n\nHow do we run these neural networks on the GPU?\n\nTraining on GPU\n----------------\nJust like how you transfer a Tensor onto the GPU, you transfer the neural\nnet onto the GPU.\n\nLet's first define our device as the first visible cuda device if we have\nCUDA available:\n\n" 260 | ] 261 | }, 262 | { 263 | "cell_type": "code", 264 | "execution_count": null, 265 | "metadata": { 266 | "collapsed": false 267 | }, 268 | "outputs": [], 269 | "source": [ 270 | "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n\n# Assuming that we are on a CUDA machine, this should print a CUDA device:\n\nprint(device)" 271 | ] 272 | }, 273 | { 274 | "cell_type": "markdown", 275 | "metadata": {}, 276 | "source": [ 277 | "The rest of this section assumes that ``device`` is a CUDA device.\n\nThen these methods will recursively go over all modules and convert their\nparameters and buffers to CUDA tensors:\n\n.. code:: python\n\n net.to(device)\n\n\nRemember that you will have to send the inputs and targets at every step\nto the GPU too:\n\n.. code:: python\n\n inputs, labels = data[0].to(device), data[1].to(device)\n\nWhy don't I notice MASSIVE speedup compared to CPU? Because your network\nis really small.\n\n**Exercise:** Try increasing the width of your network (argument 2 of\nthe first ``nn.Conv2d``, and argument 1 of the second ``nn.Conv2d`` \u2013\nthey need to be the same number), see what kind of speedup you get.\n\n**Goals achieved**:\n\n- Understanding PyTorch's Tensor library and neural networks at a high level.\n- Train a small neural network to classify images\n\nTraining on multiple GPUs\n-------------------------\nIf you want to see even more MASSIVE speedup using all of your GPUs,\nplease check out :doc:`data_parallel_tutorial`.\n\nWhere do I go next?\n-------------------\n\n- :doc:`Train neural nets to play video games `\n- `Train a state-of-the-art ResNet network on imagenet`_\n- `Train a face generator using Generative Adversarial Networks`_\n- `Train a word-level language model using Recurrent LSTM networks`_\n- `More examples`_\n- `More tutorials`_\n- `Discuss PyTorch on the Forums`_\n- `Chat with other users on Slack`_\n\n\n" 278 | ] 279 | }, 280 | { 281 | "cell_type": "code", 282 | "execution_count": null, 283 | "metadata": { 284 | "collapsed": false 285 | }, 286 | "outputs": [], 287 | "source": [ 288 | ] 289 | } 290 | ], 291 | "metadata": { 292 | "kernelspec": { 293 | "display_name": "Python 3", 294 | "language": "python", 295 | "name": "python3" 296 | }, 297 | "language_info": { 298 | "codemirror_mode": { 299 | "name": "ipython", 300 | "version": 3 301 | }, 302 | "file_extension": ".py", 303 | "mimetype": "text/x-python", 304 | "name": "python", 305 | "nbconvert_exporter": "python", 306 | "pygments_lexer": "ipython3", 307 | "version": "3.6.13" 308 | } 309 | }, 310 | "nbformat": 4, 311 | "nbformat_minor": 0 312 | } --------------------------------------------------------------------------------