├── README.md ├── Deep Learning with PyTorch ├── DL-with-pytorch - 2 [MLP].ipynb ├── DL-with-pytorch - 1 [MLP].ipynb ├── DL-with-pytorch - 3 [MLP].ipynb ├── DL-with-pytorch - 4 [CNN].ipynb ├── DL-with-pytorch - 10 [CNN-RNN].ipynb ├── DL-with-pytorch - 5 [CNN].ipynb ├── DL-with-pytorch - 8 [RNN].ipynb ├── DL-with-pytorch - 6 [CNN].ipynb ├── DL-with-pytorch - 9 [RNN].ipynb ├── DL-with-pytorch - 7 [CNN].ipynb └── DL_with_pytorch_11_[Multi_Inputs].ipynb └── PyTorch Basics ├── pytorch-model-basics-2 [linear model].ipynb ├── pytorch-model-basics-1 [nn.Module].ipynb ├── pytorch-tensors.ipynb ├── pytorch-model-basics-3 [CNN].ipynb ├── pytorch-datasets-1.ipynb ├── pytorch-model-basics-4 [RNN].ipynb └── pytorch-datasets-2.ipynb /README.md: -------------------------------------------------------------------------------- 1 | # PyTorch-learners-tutorial 2 | - PyTorch tutorial for learners 3 | - Codes are compatible with torch version 1.0.0 4 | - Written in Google Colaboratory (.ipynb files) 5 | 6 | ![](https://pytorch.org/tutorials/_images/pytorch-logo-flat.png) 7 | 8 | ## PyTorch Basics 9 | 10 | ### [1. PyTorch Tensors](https://github.com/buomsoo-kim/PyTorch-learners-tutorial/blob/master/PyTorch%20Basics/pytorch-tensors.ipynb) 11 | - Tensors 12 | - Creating Tensors 13 | - Tensor Data Types 14 | - Size (shape) of Tensors 15 | 16 | ### [2. PyTorch datasets - Part 1](https://github.com/buomsoo-kim/PyTorch-learners-tutorial/blob/master/PyTorch%20Basics/pytorch-datasets-1.ipynb) 17 | - Generating data from NumPy array 18 | - Generating data using custom DataSet and DataLoaders 19 | 20 | ### [3. PyTorch datasets - Part 2](https://github.com/buomsoo-kim/PyTorch-learners-tutorial/blob/master/PyTorch%20Basics/pytorch-datasets-2.ipynb) 21 | - Train-test split 22 | - k-fold Cross-Validation 23 | 24 | ### [4. PyTorch Model Basics - nn.Module](https://github.com/buomsoo-kim/PyTorch-learners-tutorial/blob/master/PyTorch%20Basics/pytorch-model-basics-1%20%5Bnn.Module%5D.ipynb) 25 | - nn.Module 26 | - Data Types 27 | - nn.Sequential 28 | 29 | ### [5. PyTorch Model Basics - Building Blocks of Models](https://github.com/buomsoo-kim/PyTorch-learners-tutorial/blob/master/PyTorch%20Basics/pytorch-model-basics-2%20%5Blinear%20model%5D.ipynb) 30 | - nn.Linear 31 | - Nonlinear Activations 32 | - Loss functions 33 | - Optimizers 34 | 35 | ### [6. PyTorch Model Basics - Building Blocks of Models (CNN)](https://github.com/buomsoo-kim/PyTorch-learners-tutorial/blob/master/PyTorch%20Basics/pytorch-model-basics-3%20%5BCNN%5D.ipynb) 36 | - Convolution & Pooling 37 | - Padding 38 | 39 | ### [7. PyTorch Model Basics - Building Blocks of Models (RNN)](https://github.com/buomsoo-kim/PyTorch-learners-tutorial/blob/master/PyTorch%20Basics/pytorch-model-basics-4%20%5BRNN%5D.ipynb) 40 | - Vanilla RNN 41 | - Gated Recurrent Units 42 | - Long Short Term Memory 43 | 44 | 45 | ## Deep Learning with PyTorch 46 | 47 | ### [1. Multi Layer Perceptrons - Part 1](https://github.com/buomsoo-kim/PyTorch-learners-tutorial/blob/master/Deep%20Learning%20with%20PyTorch/DL-with-pytorch%20-%201%20%5BMLP%5D.ipynb) 48 | - Breast cancer prediction with MLP 49 | 50 | ### [2. Multi Layer Perceptrons - Part 2](https://github.com/buomsoo-kim/PyTorch-learners-tutorial/blob/master/Deep%20Learning%20with%20PyTorch/DL-with-pytorch%20-%202%20%5BMLP%5D.ipynb) 51 | - CIFAR-10 image classification with MLP 52 | 53 | ### [3. Multi Layer Perceptrons - Part 3](https://github.com/buomsoo-kim/PyTorch-learners-tutorial/blob/master/Deep%20Learning%20with%20PyTorch/DL-with-pytorch%20-%203%20%5BMLP%5D.ipynb) 54 | - CIFAR-10 image classification with deeper MLP 55 | 56 | ### [4. Convolutional Neural Networks - Part 1](https://github.com/buomsoo-kim/PyTorch-learners-tutorial/blob/master/Deep%20Learning%20with%20PyTorch/DL-with-pytorch%20-%204%20%5BCNN%5D.ipynb) 57 | - CIFAR-10 image classification with CNN 58 | 59 | ### [5. Convolutional Neural Networks - Part 2](https://github.com/buomsoo-kim/PyTorch-learners-tutorial/blob/master/Deep%20Learning%20with%20PyTorch/DL-with-pytorch%20-%205%20%5BCNN%5D.ipynb) 60 | - CIFAR-10 image classification with CNN 61 | 62 | ### [6. Convolutional Neural Networks - Part 3](https://github.com/buomsoo-kim/PyTorch-learners-tutorial/blob/master/Deep%20Learning%20with%20PyTorch/DL-with-pytorch%20-%206%20%5BCNN%5D.ipynb) 63 | - IMDB review sentiment classification with CNN 64 | 65 | ### [7. Convolutional Neural Networks - Part 4](https://github.com/buomsoo-kim/PyTorch-learners-tutorial/blob/master/Deep%20Learning%20with%20PyTorch/DL-with-pytorch%20-%207%20%5BCNN%5D.ipynb) 66 | - IMDB review sentiment classification with CNN 67 | 68 | ### [8. Recurrent Neural Networks - Part 1](https://github.com/buomsoo-kim/PyTorch-learners-tutorial/blob/master/Deep%20Learning%20with%20PyTorch/DL-with-pytorch%20-%208%20%5BRNN%5D.ipynb) 69 | -IMDB review sentiment classification with RNN 70 | 71 | ### [9. Recurrent Neural Networks - Part 2](https://github.com/buomsoo-kim/PyTorch-learners-tutorial/blob/master/Deep%20Learning%20with%20PyTorch/DL-with-pytorch%20-%209%20%5BRNN%5D.ipynb) 72 | - IMDB review sentiment classification with RNN 73 | 74 | ### [10. CNN-RNN network](https://github.com/buomsoo-kim/PyTorch-learners-tutorial/blob/master/Deep%20Learning%20with%20PyTorch/DL-with-pytorch%20-%2010%20%5BCNN-RNN%5D.ipynb) 75 | - Fashion MNIST classification with CNN-RNN 76 | -------------------------------------------------------------------------------- /Deep Learning with PyTorch/DL-with-pytorch - 2 [MLP].ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"DL-with-pytorch - 2 [MLP].ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Multi Layer Perceptrons\n","- CIFAR-10 image classification with MLP\n"]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","colab":{}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":0,"outputs":[]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","outputId":"4434d2e8-c5e8-4dce-8316-7f9230dc1b75","executionInfo":{"status":"ok","timestamp":1545794710219,"user_tz":420,"elapsed":1060,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","import torch.nn as nn\n","import torch.nn.functional as F\n","torch.__version__"],"execution_count":1,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'1.0.0'"]},"metadata":{"tags":[]},"execution_count":1}]},{"metadata":{"id":"ewrw93tt2BfV","colab_type":"text"},"cell_type":"markdown","source":["## 1. Import & process dataset\n","- CIFAR10 dataset can be downloaded by ```torchvision```\n"," - [torchvision.datasets](https://pytorch.org/docs/stable/torchvision/datasets.html)"]},{"metadata":{"id":"W5anlYa01w3w","colab_type":"code","outputId":"dc43d91b-ad85-423f-fb18-1a0d2af22f25","executionInfo":{"status":"ok","timestamp":1545795192346,"user_tz":420,"elapsed":2530,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":52}},"cell_type":"code","source":["from torchvision import datasets\n","import torchvision.transforms as transforms\n","\n","train_dataset = datasets.CIFAR10(root = \"/\", train = True, download = True, transform = transforms.ToTensor())\n","test_dataset = datasets.CIFAR10(root = \"/\", train = False, download = True, transform = transforms.ToTensor())"],"execution_count":25,"outputs":[{"output_type":"stream","text":["Files already downloaded and verified\n","Files already downloaded and verified\n"],"name":"stdout"}]},{"metadata":{"id":"9nznVMdo5edZ","colab_type":"text"},"cell_type":"markdown","source":["## 2. Creating MLP model and training\n","\n","- Create and train MLP with 1 hidden layer\n","\n","![](http://cs231n.github.io/assets/nn1/neural_net.jpeg)"]},{"metadata":{"id":"YEoNQr10kGUW","colab_type":"code","colab":{}},"cell_type":"code","source":["# create data loaders \n","train_loader = torch.utils.data.DataLoader(train_dataset, batch_size = 128, shuffle = True)\n","test_loader = torch.utils.data.DataLoader(test_dataset, batch_size = 128, shuffle = False)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"AQawpMRPI7jm","colab_type":"code","colab":{}},"cell_type":"code","source":["# create MLP with one hidden layer\n","class net(nn.Module):\n"," def __init__(self, input_dim, hidden_dim, output_dim):\n"," super(net, self).__init__()\n"," self.input_dim = input_dim\n"," self.dense1 = nn.Linear(input_dim, hidden_dim)\n"," self.relu = nn.ReLU()\n"," self.dense2 = nn.Linear(hidden_dim, output_dim)\n"," \n"," def forward(self, x):\n"," x = x.view(x.size(0), self.input_dim)\n"," x = self.dense1(x)\n"," x = self.relu(x)\n"," x = self.dense2(x)\n"," return x"],"execution_count":0,"outputs":[]},{"metadata":{"id":"rP0Gt5E9ajmd","colab_type":"code","colab":{}},"cell_type":"code","source":["# hyperparameters\n","DEVICE = torch.device('cuda')\n","INPUT_DIM = 32 * 32 * 3\n","HIDDEN_DIM = 50\n","OUTPUT_DIM = 10\n","LEARNING_RATE = 1e-3\n","NUM_EPOCHS = 10"],"execution_count":0,"outputs":[]},{"metadata":{"id":"cPBm8qDrSWsi","colab_type":"code","colab":{}},"cell_type":"code","source":["model = net(INPUT_DIM, HIDDEN_DIM, OUTPUT_DIM).to(DEVICE)\n","criterion = nn.CrossEntropyLoss() # do not need softmax layer when using CEloss criterion\n","optimizer = torch.optim.RMSprop(model.parameters(), lr = LEARNING_RATE)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"SEBtAPYCFeic","colab_type":"code","outputId":"edda354d-30d8-40cb-fef0-9c2a4fd968eb","executionInfo":{"status":"ok","timestamp":1545795315447,"user_tz":420,"elapsed":89814,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":193}},"cell_type":"code","source":["# training for NUM_EPOCHS\n","for i in range(NUM_EPOCHS):\n"," temp_loss = []\n"," for (x, y) in train_loader:\n"," x, y = x.float().to(DEVICE), y.to(DEVICE)\n"," outputs = model(x)\n"," loss = criterion(outputs, y)\n"," temp_loss.append(loss.item())\n"," \n"," optimizer.zero_grad()\n"," loss.backward()\n"," optimizer.step()\n"," \n"," print(\"Loss at {}th epoch: {}\".format(i, np.mean(temp_loss)))"],"execution_count":34,"outputs":[{"output_type":"stream","text":["Loss at 0th epoch: 2.0718233820117646\n","Loss at 1th epoch: 1.8863357893951105\n","Loss at 2th epoch: 1.8327037266758093\n","Loss at 3th epoch: 1.7953610965967788\n","Loss at 4th epoch: 1.764086505336225\n","Loss at 5th epoch: 1.7412422986896448\n","Loss at 6th epoch: 1.7202257174055289\n","Loss at 7th epoch: 1.6997354250124959\n","Loss at 8th epoch: 1.6853232405069845\n","Loss at 9th epoch: 1.6730405244680926\n"],"name":"stdout"}]},{"metadata":{"id":"qpAJUiHm529m","colab_type":"text"},"cell_type":"markdown","source":["## 3. Evaluation\n","- Evaluate the trained MLP model with ROC AUC score \n"," - Store probability of each instance to a list and compare it with true y label"]},{"metadata":{"id":"txXH3dknFpSx","colab_type":"code","outputId":"9330f014-2687-4464-b495-1ba861511e38","executionInfo":{"status":"ok","timestamp":1545795466013,"user_tz":420,"elapsed":2278,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":72}},"cell_type":"code","source":["y_pred, y_true = [], []\n","with torch.no_grad():\n"," for x, y in test_loader:\n"," x, y = x.float().to(DEVICE), y.to(DEVICE)\n"," outputs = F.softmax(model(x)).max(1)[-1] # predicted label\n"," y_true += list(y.cpu().numpy()) # true label\n"," y_pred += list(outputs.cpu().numpy()) "],"execution_count":39,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n"," \"\"\"\n"],"name":"stderr"}]},{"metadata":{"id":"HV1s3xf5Frkl","colab_type":"code","outputId":"cfe3f879-b673-4d29-ed3b-f1f6b6ee7452","executionInfo":{"status":"ok","timestamp":1545795467960,"user_tz":420,"elapsed":742,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# evaluation result\n","from sklearn.metrics import accuracy_score\n","accuracy_score(y_true, y_pred)"],"execution_count":40,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.369"]},"metadata":{"tags":[]},"execution_count":40}]}]} -------------------------------------------------------------------------------- /Deep Learning with PyTorch/DL-with-pytorch - 1 [MLP].ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"DL-with-pytorch - 1 [MLP].ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Multi Layer Perceptrons\n","- Breast cancer prediction with MLP\n"," - [Breast cancer dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html#sklearn.datasets.load_breast_cancer)"]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","outputId":"7bd3dd6e-db5b-431b-f695-02353c5e4338","executionInfo":{"status":"ok","timestamp":1545794590415,"user_tz":420,"elapsed":5680,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":125}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":1,"outputs":[{"output_type":"stream","text":["Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (1.0.0)\n","Requirement already satisfied: torchvision in /usr/local/lib/python3.6/dist-packages (0.2.1)\n","Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n","Requirement already satisfied: pillow>=4.1.1 in /usr/local/lib/python3.6/dist-packages (from torchvision) (5.3.0)\n","Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n"],"name":"stdout"}]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","outputId":"8b62db35-c199-4e53-95e5-f40c0dd6b14c","executionInfo":{"status":"ok","timestamp":1545794593243,"user_tz":420,"elapsed":718,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","import torch.nn as nn\n","import torch.nn.functional as F\n","torch.__version__"],"execution_count":2,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'1.0.0'"]},"metadata":{"tags":[]},"execution_count":2}]},{"metadata":{"id":"ewrw93tt2BfV","colab_type":"text"},"cell_type":"markdown","source":["## 1. Import & process dataset\n"]},{"metadata":{"id":"fCyPxp6QFe59","colab_type":"code","colab":{}},"cell_type":"code","source":["from sklearn.datasets import load_breast_cancer\n","# define dataset\n","class breastCancerDataset(torch.utils.data.Dataset):\n"," def __init__(self):\n"," data = load_breast_cancer()\n","\n"," self.X = data.data\n"," self.y = data.target\n"," \n"," def __getitem__(self, idx):\n"," return self.X[idx], self.y[idx]\n"," \n"," def __len__(self):\n"," return len(self.X)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"sRWTaxYiYweF","colab_type":"code","outputId":"ad1b6590-6533-40ac-8ca0-28585b2e0283","executionInfo":{"status":"ok","timestamp":1545794605379,"user_tz":420,"elapsed":710,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# create dataset instance and train-test split\n","dataset = breastCancerDataset()\n","\n","NUM_INSTANCES = len(dataset)\n","TEST_RATIO = 0.4\n","TEST_SIZE = int(NUM_INSTANCES * TEST_RATIO)\n","TRAIN_SIZE = NUM_INSTANCES - TEST_SIZE\n","\n","print(TRAIN_SIZE, TEST_SIZE)"],"execution_count":4,"outputs":[{"output_type":"stream","text":["342 227\n"],"name":"stdout"}]},{"metadata":{"id":"qCJfxbYbFnY3","colab_type":"code","colab":{}},"cell_type":"code","source":["train_data, test_data = torch.utils.data.random_split(dataset, (TRAIN_SIZE, TEST_SIZE))\n","\n","train_loader = torch.utils.data.DataLoader(train_data, batch_size = 128, shuffle = True)\n","test_loader = torch.utils.data.DataLoader(test_data, batch_size = 128, shuffle = False)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"9nznVMdo5edZ","colab_type":"text"},"cell_type":"markdown","source":["## 2. Creating MLP model and training\n","\n","- Create and train MLP with 1 hidden layer\n","\n","![](http://cs231n.github.io/assets/nn1/neural_net.jpeg)"]},{"metadata":{"id":"AQawpMRPI7jm","colab_type":"code","colab":{}},"cell_type":"code","source":["# create MLP with one hidden layer\n","class net(nn.Module):\n"," def __init__(self, input_dim, hidden_dim):\n"," super(net, self).__init__()\n"," self.dense1 = nn.Linear(input_dim, hidden_dim) # hidden layer\n"," self.relu = nn.ReLU()\n"," self.dense2 = nn.Linear(hidden_dim, 2) # output layer\n"," \n"," def forward(self, x):\n"," x = self.dense1(x) \n"," x = self.relu(x)\n"," x = self.dense2(x)\n"," return x"],"execution_count":0,"outputs":[]},{"metadata":{"id":"rP0Gt5E9ajmd","colab_type":"code","colab":{}},"cell_type":"code","source":["# hyperparameters\n","DEVICE = torch.device('cuda')\n","INPUT_DIM = load_breast_cancer().data.shape[-1]\n","HIDDEN_DIM = 5\n","LEARNING_RATE = 1e-3\n","NUM_EPOCHS = 10"],"execution_count":0,"outputs":[]},{"metadata":{"id":"cPBm8qDrSWsi","colab_type":"code","colab":{}},"cell_type":"code","source":["model = net(INPUT_DIM, HIDDEN_DIM).to(DEVICE)\n","criterion = nn.CrossEntropyLoss() # do not need softmax layer when using CEloss criterion\n","optimizer = torch.optim.RMSprop(model.parameters(), lr = LEARNING_RATE)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"SEBtAPYCFeic","colab_type":"code","outputId":"4280ef9b-4d56-4445-84ca-48fc2d828f67","executionInfo":{"status":"ok","timestamp":1545794653669,"user_tz":420,"elapsed":2454,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":193}},"cell_type":"code","source":["# training for NUM_EPOCHS\n","for i in range(NUM_EPOCHS):\n"," temp_loss = []\n"," for (x, y) in train_loader:\n"," x, y = x.float().to(DEVICE), y.to(DEVICE)\n"," outputs = model(x)\n"," loss = criterion(outputs, y)\n"," temp_loss.append(loss.item())\n"," \n"," optimizer.zero_grad()\n"," loss.backward()\n"," optimizer.step()\n"," \n"," print(\"Loss at {}th epoch: {}\".format(i, np.mean(temp_loss)))"],"execution_count":10,"outputs":[{"output_type":"stream","text":["Loss at 0th epoch: 12.068120320638021\n","Loss at 1th epoch: 2.8435821533203125\n","Loss at 2th epoch: 1.4054580926895142\n","Loss at 3th epoch: 1.2583297888437908\n","Loss at 4th epoch: 1.3461804389953613\n","Loss at 5th epoch: 1.047558069229126\n","Loss at 6th epoch: 1.0004488627115886\n","Loss at 7th epoch: 0.8730776309967041\n","Loss at 8th epoch: 0.8132720788319906\n","Loss at 9th epoch: 0.7596675952275594\n"],"name":"stdout"}]},{"metadata":{"id":"qpAJUiHm529m","colab_type":"text"},"cell_type":"markdown","source":["## 3. Evaluation\n","- Evaluate the trained MLP model with ROC AUC score \n"," - Store probability of each instance to a list and compare it with true y label"]},{"metadata":{"id":"txXH3dknFpSx","colab_type":"code","outputId":"96536291-ce13-4162-b60b-915484702ec7","executionInfo":{"status":"ok","timestamp":1545794679211,"user_tz":420,"elapsed":707,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":72}},"cell_type":"code","source":["# record probability and true label \n","y_prob, y_true = [], []\n","with torch.no_grad():\n"," for x, y in test_loader:\n"," x, y = x.float().to(DEVICE), y.to(DEVICE)\n"," outputs = F.softmax(model(x))\n"," y_true += list(y.cpu().numpy()) # true label\n"," y_prob += list(outputs.cpu().numpy()[:, -1]) # predicted probability "],"execution_count":11,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n"," \"\"\"\n"],"name":"stderr"}]},{"metadata":{"id":"HV1s3xf5Frkl","colab_type":"code","outputId":"8318627b-018d-4637-cdc5-99d931807156","executionInfo":{"status":"ok","timestamp":1545794680588,"user_tz":420,"elapsed":745,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# evaluation result\n","from sklearn.metrics import roc_auc_score\n","roc_auc_score(y_true, y_prob)"],"execution_count":12,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.5968810601163542"]},"metadata":{"tags":[]},"execution_count":12}]}]} -------------------------------------------------------------------------------- /PyTorch Basics/pytorch-model-basics-2 [linear model].ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"pytorch-model-basics-2.ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Building Blocks of Models\n","- ```nn.Linear```\n","- Nonlinear Activations\n","- Loss functions\n","- Optimizers"]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","outputId":"6a1f2dd8-7ff5-4a3e-f78b-0f89eda954b3","executionInfo":{"status":"ok","timestamp":1538894131201,"user_tz":420,"elapsed":171284,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":336}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Collecting torch\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/49/0e/e382bcf1a6ae8225f50b99cc26effa2d4cc6d66975ccf3fa9590efcbedce/torch-0.4.1-cp36-cp36m-manylinux1_x86_64.whl (519.5MB)\n","\u001b[K 100% |████████████████████████████████| 519.5MB 33kB/s \n","tcmalloc: large alloc 1073750016 bytes == 0x590bc000 @ 0x7fd7ce2351c4 0x46d6a4 0x5fcbcc 0x4c494d 0x54f3c4 0x553aaf 0x54e4c8 0x54f4f6 0x553aaf 0x54efc1 0x54f24d 0x553aaf 0x54efc1 0x54f24d 0x553aaf 0x54efc1 0x54f24d 0x551ee0 0x54e4c8 0x54f4f6 0x553aaf 0x54efc1 0x54f24d 0x551ee0 0x54efc1 0x54f24d 0x551ee0 0x54e4c8 0x54f4f6 0x553aaf 0x54e4c8\n","\u001b[?25hCollecting torchvision\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl (54kB)\n","\u001b[K 100% |████████████████████████████████| 61kB 20.8MB/s \n","\u001b[?25hCollecting pillow>=4.1.1 (from torchvision)\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/62/94/5430ebaa83f91cc7a9f687ff5238e26164a779cca2ef9903232268b0a318/Pillow-5.3.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n","\u001b[K 100% |████████████████████████████████| 2.0MB 4.3MB/s \n","\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n","Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n","Installing collected packages: torch, pillow, torchvision\n"," Found existing installation: Pillow 4.0.0\n"," Uninstalling Pillow-4.0.0:\n"," Successfully uninstalled Pillow-4.0.0\n","Successfully installed pillow-5.3.0 torch-0.4.1 torchvision-0.2.1\n"],"name":"stdout"}]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","outputId":"7031d993-5a68-47dc-bfae-68c70ebb15d3","executionInfo":{"status":"ok","timestamp":1538894145751,"user_tz":420,"elapsed":7742,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","torch.__version__"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'0.4.1'"]},"metadata":{"tags":[]},"execution_count":3}]},{"metadata":{"id":"gyv2Sy5WO8lK","colab_type":"code","colab":{}},"cell_type":"code","source":["import torch.nn as nn"],"execution_count":0,"outputs":[]},{"metadata":{"id":"ewrw93tt2BfV","colab_type":"text"},"cell_type":"markdown","source":["## 1. nn.Linear\n","```nn.Linear()``` is one of the basic building blocks of any neural network (NN) model\n"," - Performs linear (or affine) transformation in the form of ```Wx (+ b)```. In NN terminology, generates a fully connected, or dense, layer.\n"," - Two parameters, ```in_features``` and ```out_features``` should be specified\n"," - Documentation: [linear_layers](https://pytorch.org/docs/stable/nn.html#linear-layers)\n"," \n","```python\n","torch.nn.Linear(in_features, # size of each input sample\n"," out_features, # size of each output sample\n"," bias = True) # whether bias (b) will be added or not\n"," \n","```"]},{"metadata":{"id":"MZ30Xe3qFEgG","colab_type":"code","outputId":"1ba5f2b4-0789-426c-df9d-ea0ea7116869","executionInfo":{"status":"ok","timestamp":1538894184746,"user_tz":420,"elapsed":1829,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":87}},"cell_type":"code","source":["linear = nn.Linear(5, 1) # input dim = 5, output dim = 1\n","x = torch.FloatTensor([1, 2, 3, 4, 5]) # 1d tensor\n","print(linear(x)) \n","y = torch.ones(3, 5) # 2d tensor\n","print(linear(y))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor([3.5314], grad_fn=)\n","tensor([[1.1357],\n"," [1.1357],\n"," [1.1357]], grad_fn=)\n"],"name":"stdout"}]},{"metadata":{"id":"TIH5QLu0Pm6h","colab_type":"text"},"cell_type":"markdown","source":["## 2. Nonlinear activations\n","PyTorch provides a number of nonlinear activation functions. Most commonly used ones are:\n","```python\n","torch.nn.ReLU() # relu\n","torch.nn.Sigmoid() # sigmoid\n","torch.nn.Tanh() # tangent hyperbolic\n","torch.nn.Softmax() # softmax\n","```\n"," - Documentation: [nonlinear_activations](https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity)"]},{"metadata":{"id":"fCyPxp6QFe59","colab_type":"code","outputId":"b9ab9745-ac50-4b26-d642-0038f70febf7","executionInfo":{"status":"ok","timestamp":1538894578108,"user_tz":420,"elapsed":929,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":105}},"cell_type":"code","source":["relu = torch.nn.ReLU()\n","sigmoid = torch.nn.Sigmoid()\n","tanh = torch.nn.Tanh()\n","softmax = torch.nn.Softmax(dim = 0) # when using softmax, explicitly designate dimension\n","\n","x = torch.randn(5) # five random numbers\n","print(x)\n","print(relu(x)) \n","print(sigmoid(x))\n","print(tanh(x))\n","print(softmax(x))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor([ 0.9238, -0.9667, -0.4237, -1.7181, 2.3956])\n","tensor([0.9238, 0.0000, 0.0000, 0.0000, 2.3956])\n","tensor([0.7158, 0.2755, 0.3956, 0.1521, 0.9165])\n","tensor([ 0.7277, -0.7472, -0.4000, -0.9376, 0.9835])\n","tensor([0.1713, 0.0259, 0.0445, 0.0122, 0.7462])\n"],"name":"stdout"}]},{"metadata":{"id":"pxfoFJoqUWio","colab_type":"text"},"cell_type":"markdown","source":["## 3. Loss Functions\n","There are a number of loss functions that are already implemented in PyTorch. Common ones include:\n","- ```nn.MSELoss```: Mean squared error. Commonly used in regression tasks.\n","- ```nn.CrossEntropyLoss```: Cross entropy loss. Commonly used in classification tasks"]},{"metadata":{"id":"Bq8fI8SgVmLX","colab_type":"code","outputId":"8be47b62-7f57-4708-c1b7-1a594a915136","executionInfo":{"status":"ok","timestamp":1538895848482,"user_tz":420,"elapsed":792,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["a = torch.FloatTensor([2, 4, 5])\n","b = torch.FloatTensor([1, 3, 2])\n","\n","mse = nn.MSELoss()\n","print(mse(a, b))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor(3.6667)\n"],"name":"stdout"}]},{"metadata":{"id":"OmEnzUaOVyDD","colab_type":"code","outputId":"d91b197e-07d0-48a9-a275-8cd1d9942843","executionInfo":{"status":"ok","timestamp":1538896097420,"user_tz":420,"elapsed":912,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# note that when using CrossEntropyLoss, input has to have (N, C) shape, where\n","# N is the batch size\n","# C is the number of classes\n","a = torch.FloatTensor([[0.5, 0], [4.5, 0], [0, 0.4], [0, 0.1]]) # input\n","b = torch.LongTensor([1, 1, 1, 0]) # target\n","\n","ce = nn.CrossEntropyLoss()\n","print(ce(a,b))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor(1.6856)\n"],"name":"stdout"}]},{"metadata":{"id":"Rj9Dv9CbXINA","colab_type":"text"},"cell_type":"markdown","source":["## 4. Optimizers\n","- ```torch.optim``` provides various optimization algorithms that are commonly used. Some of them are: \n","```python\n","optim.Adagrad \n","optim.Adam\n","optim.RMSprop\n","optim.SGD\n","```\n","- As arguments, (model) parameters and (optionally) learning rate are passed\n","- Model training process\n"," - ```optimizer.zero_grad()```: sets all gradients to zero (for every training batches)\n"," - ```loss_fn.backward()```: back propagate with respect to the loss function\n"," - ```optimizer.step()```: update model parameters"]},{"metadata":{"id":"8HI6YPvDXuli","colab_type":"code","colab":{}},"cell_type":"code","source":["## how pytorch models are trained with loss function and optimizers\n","\n","# input and output data\n","x = torch.randn(5)\n","y = torch.ones(1)\n","\n","model = nn.Linear(5, 1) # generate model\n","loss_fn = nn.MSELoss() # define loss function\n","optimizer = torch.optim.RMSprop(model.parameters(), lr = 0.01) # create optimizer \n","optimizer.zero_grad() # setting gradients to zero\n","loss_fn(model(x), y).backward() # back propagation\n","optimizer.step() # update parameters based on gradients computed"],"execution_count":0,"outputs":[]}]} -------------------------------------------------------------------------------- /PyTorch Basics/pytorch-model-basics-1 [nn.Module].ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"pytorch-model-basics-1.ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Creating models\n","- nn.Module\n","- Data Types\n","- nn.Sequential"]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","colab":{}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":0,"outputs":[]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","outputId":"23bdc443-6327-48ab-ad81-049f8d61b70a","executionInfo":{"status":"ok","timestamp":1538806371082,"user_tz":420,"elapsed":1901,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","torch.__version__"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'0.4.1'"]},"metadata":{"tags":[]},"execution_count":1}]},{"metadata":{"id":"ewrw93tt2BfV","colab_type":"text"},"cell_type":"markdown","source":["## 1. nn.Module\n","- When using Pytorch, machine learning (ML) models are constructed using ```torch.nn```, especially ```torch.nn.Module```.\n"," - Create a subclass of ```nn.Module```, and a model instance from that class\n"," - Documentation: [```torch.nn```](https://pytorch.org/docs/stable/nn.html)\n"," \n","- When implementing a ML model using ```torch.nn```, at least two functions should be specified in detail\n"," - ```__init__()```: define structure of the model and its parameters\n"," - ```forward```: define operation between layers (e.g., convolution, pooling, activation, ...)\n"," \n"," ```python\n","class MyModel(nn.Module):\n"," def __init__(self):\n"," super(MyModel, self).__init__()\n"," define model structure\n"," def forward(self, x):\n"," define operation\n","```"]},{"metadata":{"id":"MZ30Xe3qFEgG","colab_type":"code","colab":{}},"cell_type":"code","source":["import torch.nn as nn\n","\n","# creating a simplest linear regression model\n","class linear(nn.Module):\n"," def __init__(self):\n"," super(linear, self).__init__()\n"," self.dense = nn.Linear(5, 1) # input features = 5, output features = 1\n"," \n"," def forward(self, x):\n"," x = self.dense(x)\n"," return x "],"execution_count":0,"outputs":[]},{"metadata":{"id":"fCyPxp6QFe59","colab_type":"code","outputId":"9f8d68f5-7a83-4103-c34a-a1e700ab6ec0","executionInfo":{"status":"ok","timestamp":1538807935019,"user_tz":420,"elapsed":1945,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":70}},"cell_type":"code","source":["model = linear()\n","model"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["linear(\n"," (dense): Linear(in_features=5, out_features=1, bias=True)\n",")"]},"metadata":{"tags":[]},"execution_count":13}]},{"metadata":{"id":"qCJfxbYbFnY3","colab_type":"code","outputId":"783035e0-adb0-4d60-8ee9-d52b7a6619e5","executionInfo":{"status":"ok","timestamp":1538807937773,"user_tz":420,"elapsed":2675,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":87}},"cell_type":"code","source":["# model parameters can be viewed with parameters()\n","# usually, parameters are passed onto optimizers (e.g., Adam) to perform gradient descent\n","for p in model.parameters():\n"," print(p)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Parameter containing:\n","tensor([[ 0.0432, -0.0955, 0.1572, 0.1421, -0.1961]], requires_grad=True)\n","Parameter containing:\n","tensor([0.1085], requires_grad=True)\n"],"name":"stdout"}]},{"metadata":{"id":"AQawpMRPI7jm","colab_type":"code","outputId":"c6624e5c-d402-407c-a781-7a5dd5f042e6","executionInfo":{"status":"ok","timestamp":1538808614048,"user_tz":420,"elapsed":1716,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":87}},"cell_type":"code","source":["# c.f. if you want to save parameters in a list...\n","nn.ParameterList(model.parameters())"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["ParameterList(\n"," (0): Parameter containing: [torch.cuda.FloatTensor of size 1x5 (GPU 0)]\n"," (1): Parameter containing: [torch.cuda.FloatTensor of size 1 (GPU 0)]\n",")"]},"metadata":{"tags":[]},"execution_count":29}]},{"metadata":{"id":"cPBm8qDrSWsi","colab_type":"code","colab":{}},"cell_type":"code","source":["# state_dict() returns OrderedDict with layer names mapped to weights\n","model.state_dict()"],"execution_count":0,"outputs":[]},{"metadata":{"id":"YhjNYgnEF3Ls","colab_type":"code","outputId":"3d3a4263-6f09-4d2e-baae-fa2a3f94ff4f","executionInfo":{"status":"ok","timestamp":1538807957257,"user_tz":420,"elapsed":1664,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# output from a single data instance can be generated\n","x = torch.randn(5, dtype = torch.float) # generate a single data instance\n","print(x)\n","print(model(x)) # output result"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor([-0.1225, -2.0576, 0.9150, -1.9567, -0.8272])\n"],"name":"stdout"}]},{"metadata":{"id":"geWgKOAfGWEE","colab_type":"code","outputId":"7132c247-a8eb-4b37-d16c-7d97cc83b8d8","executionInfo":{"status":"ok","timestamp":1538808043016,"user_tz":420,"elapsed":1464,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":122}},"cell_type":"code","source":["# output from multiple data instances can be generated\n","x = torch.randn(3, 5, dtype = torch.float) # generate 3 random data instance\n","print(x)\n","print(model(x)) # output result"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor([[-0.4397, 1.2041, -0.6803, -0.6452, -1.2140],\n"," [ 0.0292, -0.9331, 0.4583, 0.1276, -0.9395],\n"," [-0.0179, -0.5324, 0.3720, 0.5575, -1.5623]])\n","tensor([[0.0139],\n"," [0.4732],\n"," [0.6025]], grad_fn=)\n"],"name":"stdout"}]},{"metadata":{"id":"YtjF_56l3d__","colab_type":"text"},"cell_type":"markdown","source":["## 2. Data Types of Models\n","- In Pytorch, models have data types and they should be aligned with input data types\n","- Also, a model can be changed into GPU model. In this case, data type of input should be GPU-compatible as well.\n","```python\n","model.to(device)```"]},{"metadata":{"id":"xz642SBhDGMb","colab_type":"code","colab":{}},"cell_type":"code","source":["model = model.double() # model has data type of double\n","print(x.dtype) # x has data type of float\n","print(model(x)) # error"],"execution_count":0,"outputs":[]},{"metadata":{"id":"bLPvJC3DHtio","colab_type":"code","outputId":"f02c69cf-0981-46a3-f996-014a3b6e7a6c","executionInfo":{"status":"ok","timestamp":1538808443014,"user_tz":420,"elapsed":1301,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":87}},"cell_type":"code","source":["model = model.float() # now change model to float \n","print(x.dtype) # x has data type of float\n","print(model(x)) # no error"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.float32\n","tensor([[0.0139],\n"," [0.4732],\n"," [0.6025]], grad_fn=)\n"],"name":"stdout"}]},{"metadata":{"id":"kc-s_Jb44bjB","colab_type":"code","outputId":"75938f72-2d88-4e31-fd79-fcfa590b5c6a","executionInfo":{"status":"ok","timestamp":1538808495667,"user_tz":420,"elapsed":1453,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":70}},"cell_type":"code","source":["device = torch.device(\"cuda\")\n","model = model.to(device)\n","x = x.to(device)\n","print(model(x))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor([[0.0139],\n"," [0.4732],\n"," [0.6025]], device='cuda:0', grad_fn=)\n"],"name":"stdout"}]},{"metadata":{"id":"uleJxsvvJWkw","colab_type":"text"},"cell_type":"markdown","source":["## 3. nn.Sequential\n","- Sequential container in which modules can be added **in order**\n"," - Also, ```OrderedDict``` can be used as an input"]},{"metadata":{"id":"_EZNzUYfJ3UI","colab_type":"code","colab":{}},"cell_type":"code","source":["# create a simple multi-layer perceptron\n","model = nn.Sequential(\n"," nn.Linear(5, 5),\n"," nn.ReLU(),\n"," nn.Linear(5, 1),\n"," nn.Sigmoid()\n"," )"],"execution_count":0,"outputs":[]},{"metadata":{"id":"B1G16yeBKAYm","colab_type":"code","outputId":"3eb54ff8-b790-4060-e8ff-860e6b7956fd","executionInfo":{"status":"ok","timestamp":1538808910744,"user_tz":420,"elapsed":1627,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["x = torch.randn(5, dtype = torch.float)\n","model(x)"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([0.6740], grad_fn=)"]},"metadata":{"tags":[]},"execution_count":34}]},{"metadata":{"id":"tk_7oBdvKDP-","colab_type":"code","colab":{}},"cell_type":"code","source":["# create MLP using ordereddict - same result as above\n","from collections import OrderedDict\n","\n","model = nn.Sequential(OrderedDict([\n"," (\"dense1\", nn.Linear(5,5)),\n"," (\"relu1\", nn.ReLU()),\n"," (\"dense2\", nn.Linear(5,1)),\n"," (\"relu2\", nn.ReLU())\n","]))"],"execution_count":0,"outputs":[]}]} -------------------------------------------------------------------------------- /Deep Learning with PyTorch/DL-with-pytorch - 3 [MLP].ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"DL-with-pytorch - 3 [MLP].ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Multi Layer Perceptrons\n","- CIFAR-10 image classification with deeper MLP\n"," - Last time, we have attempted to classify images in the CIFAR-10 dataset with MLP. However, we were able to obtain accuracy of only 0.369.\n"," - Here, we attempt to gain higher accuracy by changing the MLP architecture and hyperparameters."]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":336},"outputId":"54e7c80a-d066-4716-bfcc-dd50dab9977a","executionInfo":{"status":"ok","timestamp":1545850085005,"user_tz":420,"elapsed":64720,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":1,"outputs":[{"output_type":"stream","text":["Collecting torch\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/7e/60/66415660aa46b23b5e1b72bc762e816736ce8d7260213e22365af51e8f9c/torch-1.0.0-cp36-cp36m-manylinux1_x86_64.whl (591.8MB)\n","\u001b[K 100% |████████████████████████████████| 591.8MB 26kB/s \n","tcmalloc: large alloc 1073750016 bytes == 0x60d7e000 @ 0x7f87578fc2a4 0x591a07 0x5b5d56 0x502e9a 0x506859 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x504c28 0x502540 0x502f3d 0x507641\n","\u001b[?25hCollecting torchvision\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl (54kB)\n","\u001b[K 100% |████████████████████████████████| 61kB 23.5MB/s \n","\u001b[?25hCollecting pillow>=4.1.1 (from torchvision)\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/62/94/5430ebaa83f91cc7a9f687ff5238e26164a779cca2ef9903232268b0a318/Pillow-5.3.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n","\u001b[K 100% |████████████████████████████████| 2.0MB 4.2MB/s \n","\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n","Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n","Installing collected packages: torch, pillow, torchvision\n"," Found existing installation: Pillow 4.0.0\n"," Uninstalling Pillow-4.0.0:\n"," Successfully uninstalled Pillow-4.0.0\n","Successfully installed pillow-5.3.0 torch-1.0.0 torchvision-0.2.1\n"],"name":"stdout"}]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","outputId":"1b368262-43ed-4ed4-af57-992d8f166a8e","executionInfo":{"status":"ok","timestamp":1545795541495,"user_tz":420,"elapsed":757,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","import torch.nn as nn\n","import torch.nn.functional as F\n","torch.__version__"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'1.0.0'"]},"metadata":{"tags":[]},"execution_count":1}]},{"metadata":{"id":"ewrw93tt2BfV","colab_type":"text"},"cell_type":"markdown","source":["## 1. Import & process dataset\n","- CIFAR10 dataset can be downloaded by ```torchvision```\n"," - [torchvision.datasets](https://pytorch.org/docs/stable/torchvision/datasets.html)"]},{"metadata":{"id":"W5anlYa01w3w","colab_type":"code","outputId":"ca1e77b2-1a79-4e0e-a2fa-d12a8267ecd3","executionInfo":{"status":"ok","timestamp":1545795548014,"user_tz":420,"elapsed":2564,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":52}},"cell_type":"code","source":["from torchvision import datasets\n","import torchvision.transforms as transforms\n","\n","train_dataset = datasets.CIFAR10(root = \"/\", train = True, download = True, transform = transforms.ToTensor())\n","test_dataset = datasets.CIFAR10(root = \"/\", train = False, download = True, transform = transforms.ToTensor())"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Files already downloaded and verified\n","Files already downloaded and verified\n"],"name":"stdout"}]},{"metadata":{"id":"9nznVMdo5edZ","colab_type":"text"},"cell_type":"markdown","source":["## 2. Creating MLP model and training\n","\n","- Create and train MLP with more hidden layers and advanced settings\n","\n","![](http://cs231n.github.io/assets/nn1/neural_net.jpeg)"]},{"metadata":{"id":"YEoNQr10kGUW","colab_type":"code","colab":{}},"cell_type":"code","source":["# create data loaders \n","train_loader = torch.utils.data.DataLoader(train_dataset, batch_size = 128, shuffle = True)\n","test_loader = torch.utils.data.DataLoader(test_dataset, batch_size = 128, shuffle = False)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"AQawpMRPI7jm","colab_type":"code","colab":{}},"cell_type":"code","source":["# create MLP with one hidden layer\n","class net(nn.Module):\n"," def __init__(self, input_dim, hidden_dim, output_dim):\n"," super(net, self).__init__()\n"," self.input_dim = input_dim\n"," self.dense1 = nn.Linear(input_dim, hidden_dim[0]) # first hidden layer\n"," self.dense2 = nn.Linear(hidden_dim[0], hidden_dim[1]) # second hidden layer\n"," self.dense3 = nn.Linear(hidden_dim[1], hidden_dim[2]) # third hidden layer\n"," self.dense4 = nn.Linear(hidden_dim[2], output_dim) \n"," self.relu = nn.ReLU()\n"," \n"," def forward(self, x):\n"," x = x.view(x.size(0), self.input_dim)\n"," x = self.dense1(x)\n"," x = self.relu(x)\n"," x = self.dense2(x)\n"," x = self.relu(x)\n"," x = self.dense3(x)\n"," x = self.relu(x)\n"," x = self.dense4(x)\n"," return x"],"execution_count":0,"outputs":[]},{"metadata":{"id":"rP0Gt5E9ajmd","colab_type":"code","colab":{}},"cell_type":"code","source":["# hyperparameters\n","DEVICE = torch.device('cuda')\n","INPUT_DIM = 32 * 32 * 3\n","HIDDEN_DIM = (100, 100, 100) # 3 hidden dimensions for 3 layers\n","OUTPUT_DIM = 10\n","LEARNING_RATE = 1e-3\n","NUM_EPOCHS = 30 # increase number of epochs"],"execution_count":0,"outputs":[]},{"metadata":{"id":"cPBm8qDrSWsi","colab_type":"code","colab":{}},"cell_type":"code","source":["model = net(INPUT_DIM, HIDDEN_DIM, OUTPUT_DIM).to(DEVICE)\n","criterion = nn.CrossEntropyLoss() # do not need softmax layer when using CEloss criterion\n","optimizer = torch.optim.Adam(model.parameters(), lr = LEARNING_RATE)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"SEBtAPYCFeic","colab_type":"code","outputId":"ff60d729-d864-4e56-a2fa-ae44100f747e","executionInfo":{"status":"ok","timestamp":1545796827881,"user_tz":420,"elapsed":292007,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":545}},"cell_type":"code","source":["# training for NUM_EPOCHS\n","for i in range(NUM_EPOCHS):\n"," temp_loss = []\n"," for (x, y) in train_loader:\n"," x, y = x.float().to(DEVICE), y.to(DEVICE)\n"," outputs = model(x)\n"," loss = criterion(outputs, y)\n"," temp_loss.append(loss.item())\n"," \n"," optimizer.zero_grad()\n"," loss.backward()\n"," optimizer.step()\n"," \n"," print(\"Loss at {}th epoch: {}\".format(i, np.mean(temp_loss)))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Loss at 0th epoch: 1.89627995271512\n","Loss at 1th epoch: 1.693795374592247\n","Loss at 2th epoch: 1.6212971856831895\n","Loss at 3th epoch: 1.5668071813290687\n","Loss at 4th epoch: 1.5262251299665408\n","Loss at 5th epoch: 1.4933200495322343\n","Loss at 6th epoch: 1.4687112606394932\n","Loss at 7th epoch: 1.4454451488411946\n","Loss at 8th epoch: 1.4289868929806877\n","Loss at 9th epoch: 1.408607015829257\n","Loss at 10th epoch: 1.394239951582516\n","Loss at 11th epoch: 1.3733605451291175\n","Loss at 12th epoch: 1.3631285389366052\n","Loss at 13th epoch: 1.343611014011266\n","Loss at 14th epoch: 1.3399841968360764\n","Loss at 15th epoch: 1.3238185805737819\n","Loss at 16th epoch: 1.3146519923149167\n","Loss at 17th epoch: 1.3017306763802647\n","Loss at 18th epoch: 1.2912272934413627\n","Loss at 19th epoch: 1.2836608034570505\n","Loss at 20th epoch: 1.2702278880512012\n","Loss at 21th epoch: 1.2586160532348907\n","Loss at 22th epoch: 1.2533685801279209\n","Loss at 23th epoch: 1.2392126495575966\n","Loss at 24th epoch: 1.2397530463040638\n","Loss at 25th epoch: 1.2295762690741692\n","Loss at 26th epoch: 1.2183618973892973\n","Loss at 27th epoch: 1.2042760227037512\n","Loss at 28th epoch: 1.2044923659175863\n","Loss at 29th epoch: 1.2012101366086994\n"],"name":"stdout"}]},{"metadata":{"id":"qpAJUiHm529m","colab_type":"text"},"cell_type":"markdown","source":["## 3. Evaluation\n","- Evaluate the trained MLP model with ROC AUC score \n"," - Store probability of each instance to a list and compare it with true y label"]},{"metadata":{"id":"txXH3dknFpSx","colab_type":"code","outputId":"0ceabe35-2158-40ac-82e8-428937e4169a","executionInfo":{"status":"ok","timestamp":1545798230878,"user_tz":420,"elapsed":2322,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":72}},"cell_type":"code","source":["y_pred, y_true = [], []\n","with torch.no_grad():\n"," for x, y in test_loader:\n"," x, y = x.float().to(DEVICE), y.to(DEVICE)\n"," outputs = F.softmax(model(x)).max(1)[-1] # predicted label\n"," y_true += list(y.cpu().numpy()) # true label\n"," y_pred += list(outputs.cpu().numpy()) "],"execution_count":0,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n"," \"\"\"\n"],"name":"stderr"}]},{"metadata":{"id":"HV1s3xf5Frkl","colab_type":"code","outputId":"f5ca3a29-ffba-4b9a-993f-5f9c4106e0ae","executionInfo":{"status":"ok","timestamp":1545798232470,"user_tz":420,"elapsed":732,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# evaluation result\n","from sklearn.metrics import accuracy_score\n","accuracy_score(y_true, y_pred)"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.504"]},"metadata":{"tags":[]},"execution_count":25}]}]} -------------------------------------------------------------------------------- /Deep Learning with PyTorch/DL-with-pytorch - 4 [CNN].ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"DL-with-pytorch - 4 [CNN].ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Convolutional Neural Networks\n","- CIFAR-10 image classification with CNN\n"," - Last time, we have attempted to classify images in the CIFAR-10 dataset with MLP and deeper MLP. The final results were accuracy scores of 0.369 and 0.504, respectively.\n"," - Let's try to improve the classification performance by implementing simple CNN"]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","outputId":"0d78cd9a-4aa1-4b43-e240-f16e706115c7","executionInfo":{"status":"ok","timestamp":1545889909282,"user_tz":420,"elapsed":64924,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":336}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":1,"outputs":[{"output_type":"stream","text":["Collecting torch\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/7e/60/66415660aa46b23b5e1b72bc762e816736ce8d7260213e22365af51e8f9c/torch-1.0.0-cp36-cp36m-manylinux1_x86_64.whl (591.8MB)\n","\u001b[K 100% |████████████████████████████████| 591.8MB 29kB/s \n","tcmalloc: large alloc 1073750016 bytes == 0x618be000 @ 0x7f36e05d62a4 0x591a07 0x5b5d56 0x502e9a 0x506859 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x504c28 0x502540 0x502f3d 0x507641\n","\u001b[?25hCollecting torchvision\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl (54kB)\n","\u001b[K 100% |████████████████████████████████| 61kB 18.0MB/s \n","\u001b[?25hCollecting pillow>=4.1.1 (from torchvision)\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/62/94/5430ebaa83f91cc7a9f687ff5238e26164a779cca2ef9903232268b0a318/Pillow-5.3.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n","\u001b[K 100% |████████████████████████████████| 2.0MB 3.8MB/s \n","\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n","Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n","Installing collected packages: torch, pillow, torchvision\n"," Found existing installation: Pillow 4.0.0\n"," Uninstalling Pillow-4.0.0:\n"," Successfully uninstalled Pillow-4.0.0\n","Successfully installed pillow-5.3.0 torch-1.0.0 torchvision-0.2.1\n"],"name":"stdout"}]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","outputId":"59fd3412-a818-47e9-b910-d23da6d76900","executionInfo":{"status":"ok","timestamp":1545889919108,"user_tz":420,"elapsed":1028,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","import torch.nn as nn\n","import torch.nn.functional as F\n","torch.__version__"],"execution_count":2,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'1.0.0'"]},"metadata":{"tags":[]},"execution_count":2}]},{"metadata":{"id":"ewrw93tt2BfV","colab_type":"text"},"cell_type":"markdown","source":["## 1. Import & process dataset\n","- CIFAR10 dataset can be downloaded by ```torchvision```\n"," - [torchvision.datasets](https://pytorch.org/docs/stable/torchvision/datasets.html)"]},{"metadata":{"id":"W5anlYa01w3w","colab_type":"code","outputId":"e48610fb-2c14-46a5-c1e5-1053e125fbe9","executionInfo":{"status":"ok","timestamp":1545889971425,"user_tz":420,"elapsed":50510,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":52}},"cell_type":"code","source":["from torchvision import datasets\n","import torchvision.transforms as transforms\n","\n","train_dataset = datasets.CIFAR10(root = \"/\", train = True, download = True, transform = transforms.ToTensor())\n","test_dataset = datasets.CIFAR10(root = \"/\", train = False, download = True, transform = transforms.ToTensor())"],"execution_count":3,"outputs":[{"output_type":"stream","text":["Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to /cifar-10-python.tar.gz\n","Files already downloaded and verified\n"],"name":"stdout"}]},{"metadata":{"id":"9nznVMdo5edZ","colab_type":"text"},"cell_type":"markdown","source":["## 2. Creating CNN model and training\n","\n","- Create and train simple CNN model\n","\n","![](https://www.researchgate.net/profile/Qianzhou_Du2/publication/322477802/figure/fig3/AS:582461356511232@1515881017676/Illustration-of-Convolutional-Neural-Network-CNN-Architecture.png)"]},{"metadata":{"id":"YEoNQr10kGUW","colab_type":"code","colab":{}},"cell_type":"code","source":["# create data loaders \n","train_loader = torch.utils.data.DataLoader(train_dataset, batch_size = 128, shuffle = True)\n","test_loader = torch.utils.data.DataLoader(test_dataset, batch_size = 128, shuffle = False)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"AQawpMRPI7jm","colab_type":"code","colab":{}},"cell_type":"code","source":["# create CNN with one convolution/pooling layer\n","class net(nn.Module):\n"," def __init__(self, input_dim, num_filters, kernel_size, stride, padding, num_classes):\n"," super(net, self).__init__()\n"," self.input_dim = input_dim\n"," conv_output_size = int((input_dim - kernel_size + 2 * padding)/stride) + 1 # conv layer output size\n"," pool_output_size = int((conv_output_size - kernel_size)/stride) + 1 # pooling layer output size\n"," \n"," self.conv = nn.Conv2d(3, num_filters, kernel_size = kernel_size, stride = stride, padding = padding) \n"," self.pool = nn.MaxPool2d(kernel_size = kernel_size, stride = stride)\n"," self.relu = nn.ReLU()\n"," self.dense = nn.Linear(pool_output_size * pool_output_size * num_filters, num_classes) \n"," \n"," def forward(self, x):\n"," x = self.conv(x)\n"," x = self.relu(x)\n"," x = self.pool(x)\n"," x = x.view(x.size(0), -1) # resize to fit into final dense layer\n"," x = self.dense(x)\n"," return x"],"execution_count":0,"outputs":[]},{"metadata":{"id":"rP0Gt5E9ajmd","colab_type":"code","colab":{}},"cell_type":"code","source":["# hyperparameters\n","DEVICE = torch.device('cuda')\n","INPUT_DIM = 32\n","NUM_FILTERS = 32\n","KERNEL_SIZE = 3\n","STRIDE = 1\n","PADDING = 1\n","NUM_CLASSES = 10\n","LEARNING_RATE = 1e-3\n","NUM_EPOCHS = 30 "],"execution_count":0,"outputs":[]},{"metadata":{"id":"cPBm8qDrSWsi","colab_type":"code","colab":{}},"cell_type":"code","source":["model = net(INPUT_DIM, NUM_FILTERS, KERNEL_SIZE, STRIDE, PADDING, NUM_CLASSES).to(DEVICE)\n","criterion = nn.CrossEntropyLoss() # do not need softmax layer when using CEloss criterion\n","optimizer = torch.optim.Adam(model.parameters(), lr = LEARNING_RATE)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"SEBtAPYCFeic","colab_type":"code","outputId":"3a8c705c-c46e-443d-82ba-af9fa858f9ec","executionInfo":{"status":"ok","timestamp":1545890996474,"user_tz":420,"elapsed":295268,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":545}},"cell_type":"code","source":["# training for NUM_EPOCHS\n","for i in range(NUM_EPOCHS):\n"," temp_loss = []\n"," for (x, y) in train_loader:\n"," x, y = x.float().to(DEVICE), y.to(DEVICE)\n"," outputs = model(x)\n"," loss = criterion(outputs, y)\n"," temp_loss.append(loss.item())\n"," \n"," optimizer.zero_grad()\n"," loss.backward()\n"," optimizer.step()\n"," \n"," print(\"Loss at {}th epoch: {}\".format(i, np.mean(temp_loss)))"],"execution_count":19,"outputs":[{"output_type":"stream","text":["Loss at 0th epoch: 1.5467755730499697\n","Loss at 1th epoch: 1.1874774668527686\n","Loss at 2th epoch: 1.0879874247724137\n","Loss at 3th epoch: 1.0216767588234923\n","Loss at 4th epoch: 0.9694403696547994\n","Loss at 5th epoch: 0.9249361718402189\n","Loss at 6th epoch: 0.8892614440539913\n","Loss at 7th epoch: 0.8543041116746185\n","Loss at 8th epoch: 0.8303252603391857\n","Loss at 9th epoch: 0.8046949812213479\n","Loss at 10th epoch: 0.7781740239514109\n","Loss at 11th epoch: 0.7531528044539644\n","Loss at 12th epoch: 0.7326688110218633\n","Loss at 13th epoch: 0.7106734506614373\n","Loss at 14th epoch: 0.680794730210853\n","Loss at 15th epoch: 0.6707794649521713\n","Loss at 16th epoch: 0.6504438420390839\n","Loss at 17th epoch: 0.6231620132618243\n","Loss at 18th epoch: 0.6014589527836236\n","Loss at 19th epoch: 0.5822262440038763\n","Loss at 20th epoch: 0.5729258664886056\n","Loss at 21th epoch: 0.5556111552983599\n","Loss at 22th epoch: 0.5367484169695383\n","Loss at 23th epoch: 0.5206924786653055\n","Loss at 24th epoch: 0.5027579848876085\n","Loss at 25th epoch: 0.4926028959738934\n","Loss at 26th epoch: 0.47675407138626896\n","Loss at 27th epoch: 0.4579864634424829\n","Loss at 28th epoch: 0.44510160489460393\n","Loss at 29th epoch: 0.4307325435873797\n"],"name":"stdout"}]},{"metadata":{"id":"qpAJUiHm529m","colab_type":"text"},"cell_type":"markdown","source":["## 3. Evaluation\n","- Evaluate the trained CNN model with accuracy score \n"," - Store probability of each instance to a list and compare it with true y label"]},{"metadata":{"id":"txXH3dknFpSx","colab_type":"code","outputId":"d1db27e2-f9ca-4642-971e-98ab60a352f3","executionInfo":{"status":"ok","timestamp":1545891014673,"user_tz":420,"elapsed":3350,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":72}},"cell_type":"code","source":["y_pred, y_true = [], []\n","with torch.no_grad():\n"," for x, y in test_loader:\n"," x, y = x.float().to(DEVICE), y.to(DEVICE)\n"," outputs = F.softmax(model(x)).max(1)[-1] # predicted label\n"," y_true += list(y.cpu().numpy()) # true label\n"," y_pred += list(outputs.cpu().numpy()) "],"execution_count":20,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n"," \"\"\"\n"],"name":"stderr"}]},{"metadata":{"id":"HV1s3xf5Frkl","colab_type":"code","outputId":"8c2be308-ea59-46d7-879f-dee1bde8f264","executionInfo":{"status":"ok","timestamp":1545891017606,"user_tz":420,"elapsed":1795,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# evaluation result\n","from sklearn.metrics import accuracy_score\n","accuracy_score(y_true, y_pred)"],"execution_count":21,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.6454"]},"metadata":{"tags":[]},"execution_count":21}]}]} -------------------------------------------------------------------------------- /Deep Learning with PyTorch/DL-with-pytorch - 10 [CNN-RNN].ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"DL-with-pytorch - 10 [CNN-RNN].ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# CNN-RNN network\n","- CNNs and RNNs both have their own strengths and drawbacks. Hence, it is sometimes recommended to combine the two to model highly complicated data\n","- Here, we combine the two to classify fashion image dataset (Fashion-MNIST)"]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","outputId":"2a3e5076-f593-41ba-ada7-35277c683a89","executionInfo":{"status":"ok","timestamp":1545950044665,"user_tz":420,"elapsed":62689,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":336}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Collecting torch\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/7e/60/66415660aa46b23b5e1b72bc762e816736ce8d7260213e22365af51e8f9c/torch-1.0.0-cp36-cp36m-manylinux1_x86_64.whl (591.8MB)\n","\u001b[K 100% |████████████████████████████████| 591.8MB 26kB/s \n","tcmalloc: large alloc 1073750016 bytes == 0x6140c000 @ 0x7efc5d2cd2a4 0x591a07 0x5b5d56 0x502e9a 0x506859 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x504c28 0x502540 0x502f3d 0x507641\n","\u001b[?25hCollecting torchvision\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl (54kB)\n","\u001b[K 100% |████████████████████████████████| 61kB 20.0MB/s \n","\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n","Collecting pillow>=4.1.1 (from torchvision)\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/62/94/5430ebaa83f91cc7a9f687ff5238e26164a779cca2ef9903232268b0a318/Pillow-5.3.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n","\u001b[K 100% |████████████████████████████████| 2.0MB 3.3MB/s \n","\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n","Installing collected packages: torch, pillow, torchvision\n"," Found existing installation: Pillow 4.0.0\n"," Uninstalling Pillow-4.0.0:\n"," Successfully uninstalled Pillow-4.0.0\n","Successfully installed pillow-5.3.0 torch-1.0.0 torchvision-0.2.1\n"],"name":"stdout"}]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","outputId":"c35a8fa0-4744-468b-dbaf-ea28fabffe75","executionInfo":{"status":"ok","timestamp":1545968024763,"user_tz":420,"elapsed":1126,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","import torch.nn as nn\n","import torch.nn.functional as F\n","torch.__version__"],"execution_count":1,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'1.0.0'"]},"metadata":{"tags":[]},"execution_count":1}]},{"metadata":{"id":"ewrw93tt2BfV","colab_type":"text"},"cell_type":"markdown","source":["## 1. Import & process dataset\n","- Fashion MNIST dataset from torchvision \n","- [Original dataset source](https://github.com/zalandoresearch/fashion-mnist), [paper](https://arxiv.org/abs/1708.07747)\n","\n","![](https://github.com/zalandoresearch/fashion-mnist/raw/master/doc/img/fashion-mnist-sprite.png)"]},{"metadata":{"id":"SxDHXFEsf5em","colab_type":"code","outputId":"2b71fa41-bc2e-4a89-a0a2-e9bbceca50d7","executionInfo":{"status":"ok","timestamp":1545968138956,"user_tz":420,"elapsed":5242,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":122}},"cell_type":"code","source":["from torchvision import datasets\n","import torchvision.transforms as transforms\n","\n","train_dataset = datasets.FashionMNIST(root = \"/\", train = True, download = True, transform = transforms.ToTensor())\n","test_dataset = datasets.FashionMNIST(root = \"/\", train = False, download = True, transform = transforms.ToTensor())"],"execution_count":2,"outputs":[{"output_type":"stream","text":["Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz\n","Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz\n","Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz\n","Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz\n","Processing...\n","Done!\n"],"name":"stdout"}]},{"metadata":{"id":"L5b7U5zL4fQb","colab_type":"code","colab":{}},"cell_type":"code","source":["# create data loaders \n","train_loader = torch.utils.data.DataLoader(train_dataset, batch_size = 128, shuffle = True)\n","test_loader = torch.utils.data.DataLoader(test_dataset, batch_size = 128, shuffle = False)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"9nznVMdo5edZ","colab_type":"text"},"cell_type":"markdown","source":["## 2. Creating CNN-RNN model and training\n","\n","- Create and train CNN-RNN model for fashion MNIST image classification.\n","\n","\n","![](https://www.researchgate.net/profile/Maulik_Kamdar/publication/322167103/figure/fig5/AS:631611880124465@1527599415555/CRNN-Architecture-Overview-Combining-CNN-and-RNN-to-predict-the-methylation-state-from.png)"]},{"metadata":{"id":"AQawpMRPI7jm","colab_type":"code","colab":{}},"cell_type":"code","source":["# create CNN with one convolution/pooling layer\n","class net(nn.Module):\n"," def __init__(self, input_dim, num_filters, conv_kernel_size, pool_kernel_size, stride, padding, hidden_size, num_classes, device):\n"," super(net, self).__init__()\n"," self.input_dim = input_dim\n"," self.device = device\n"," self.num_filters = num_filters\n"," self.hidden_size = hidden_size\n"," conv_output_size = int((input_dim - conv_kernel_size + 2 * padding)/stride) + 1 # conv layer output size\n"," self.pool_output_size = int((conv_output_size - pool_kernel_size)/stride) + 1 # pooling layer output size\n"," \n"," self.conv = nn.Conv2d(1, num_filters, kernel_size = conv_kernel_size, stride = stride, padding = padding) \n"," self.pool = nn.MaxPool2d(kernel_size = pool_kernel_size, stride = stride)\n"," self.rnn = nn.GRU(input_size = self.pool_output_size * self.pool_output_size, hidden_size = hidden_size) # GRU layer that takes into CNN output\n"," self.relu = nn.ReLU()\n"," self.dense = nn.Linear(hidden_size, num_classes) \n"," \n"," def forward(self, x):\n"," x = self.conv(x)\n"," x = self.relu(x)\n"," x = self.pool(x)\n","\n"," x = x.view(self.num_filters, x.size(0), self.pool_output_size * self.pool_output_size) # resize to fit into GRU layer\n"," \n"," h0 = torch.from_numpy(np.zeros((1, x.size(1), self.hidden_size))).float().to(self.device)\n"," x, _ = self.rnn(x, h0)\n"," x = x[-1, :, :] # take only the last sequence output\n"," x = self.dense(x)\n"," return x"],"execution_count":0,"outputs":[]},{"metadata":{"id":"rP0Gt5E9ajmd","colab_type":"code","colab":{}},"cell_type":"code","source":["# hyperparameters\n","DEVICE = torch.device('cuda')\n","INPUT_DIM = 28\n","NUM_FILTERS = 64\n","HIDDEN_SIZE = 30\n","CONV_KERNEL_SIZE = 3\n","POOL_KERNEL_SIZE = 2\n","STRIDE = 1\n","PADDING = 1\n","HIDDEN_SIZE = 10\n","NUM_CLASSES = 10\n","LEARNING_RATE = 1e-1\n","NUM_EPOCHS = 10"],"execution_count":0,"outputs":[]},{"metadata":{"id":"cPBm8qDrSWsi","colab_type":"code","colab":{}},"cell_type":"code","source":["model = net(INPUT_DIM, NUM_FILTERS, CONV_KERNEL_SIZE, POOL_KERNEL_SIZE, STRIDE, PADDING, HIDDEN_SIZE, NUM_CLASSES, DEVICE).to(DEVICE)\n","criterion = nn.CrossEntropyLoss() # do not need softmax layer when using CEloss criterion\n","optimizer = torch.optim.Adam(model.parameters(), lr = LEARNING_RATE)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"SEBtAPYCFeic","colab_type":"code","outputId":"69a901da-5440-441a-81cd-a71ad0993e06","executionInfo":{"status":"ok","timestamp":1545970237133,"user_tz":420,"elapsed":137066,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":193}},"cell_type":"code","source":["# training for NUM_EPOCHS\n","for i in range(NUM_EPOCHS):\n"," temp_loss = []\n"," for (x, y) in train_loader:\n"," x, y = x.float().to(DEVICE), y.to(DEVICE) # beware that input to embedding should be type 'long'\n"," outputs = model(x)\n"," loss = criterion(outputs, y)\n"," temp_loss.append(loss.item())\n"," \n"," optimizer.zero_grad()\n"," loss.backward()\n"," optimizer.step()\n"," \n"," print(\"Loss at {}th epoch: {}\".format(i, np.mean(temp_loss)))"],"execution_count":118,"outputs":[{"output_type":"stream","text":["Loss at 0th epoch: 2.342791672200282\n","Loss at 1th epoch: 2.3466376383929872\n","Loss at 2th epoch: 2.344623132555215\n","Loss at 3th epoch: 2.3397951639537364\n","Loss at 4th epoch: 2.343589466009567\n","Loss at 5th epoch: 2.3421539873965007\n","Loss at 6th epoch: 2.3439814973233353\n","Loss at 7th epoch: 2.3448602634706477\n","Loss at 8th epoch: 2.3438526247101805\n","Loss at 9th epoch: 2.3455153802818836\n"],"name":"stdout"}]},{"metadata":{"id":"qpAJUiHm529m","colab_type":"text"},"cell_type":"markdown","source":["## 3. Evaluation\n","- Evaluate the trained CNN-RNN model with accuracy score \n"," - Store probability of each instance to a list and compare it with true y label"]},{"metadata":{"id":"txXH3dknFpSx","colab_type":"code","outputId":"ab59b91f-acab-4fc3-c7d4-c4f52a9fef51","executionInfo":{"status":"ok","timestamp":1545970889474,"user_tz":420,"elapsed":2443,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":72}},"cell_type":"code","source":["y_pred, y_true = [], []\n","with torch.no_grad():\n"," for x, y in test_loader:\n"," x, y = x.float().to(DEVICE), y.to(DEVICE) # beware that input to embedding should be type 'long'\n"," outputs = F.softmax(model(x)).max(1)[-1] # predicted label\n"," y_true += list(y.cpu().numpy()) # true label\n"," y_pred += list(outputs.cpu().numpy()) "],"execution_count":120,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n"," \"\"\"\n"],"name":"stderr"}]},{"metadata":{"id":"HV1s3xf5Frkl","colab_type":"code","outputId":"0dc389ad-cc65-4d7e-b85a-8a8e1f49f1ac","executionInfo":{"status":"ok","timestamp":1545970893107,"user_tz":420,"elapsed":748,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# evaluation result\n","from sklearn.metrics import accuracy_score\n","accuracy_score(y_true, y_pred)"],"execution_count":121,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.0989"]},"metadata":{"tags":[]},"execution_count":121}]}]} -------------------------------------------------------------------------------- /Deep Learning with PyTorch/DL-with-pytorch - 5 [CNN].ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"DL-with-pytorch - 5 [CNN].ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Convolutional Neural Networks\n","- CIFAR-10 image classification with CNN\n"," - Last time, we have attempted to classify images in the CIFAR-10 dataset with simple CNN having one convolutional & pooling layers. The final result was accuracy score of 0.645.\n"," - Again, let's try to improve the classification performance by implementing deeper CNN with more layers"]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","outputId":"0d78cd9a-4aa1-4b43-e240-f16e706115c7","executionInfo":{"status":"ok","timestamp":1545889909282,"user_tz":420,"elapsed":64924,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":336}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Collecting torch\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/7e/60/66415660aa46b23b5e1b72bc762e816736ce8d7260213e22365af51e8f9c/torch-1.0.0-cp36-cp36m-manylinux1_x86_64.whl (591.8MB)\n","\u001b[K 100% |████████████████████████████████| 591.8MB 29kB/s \n","tcmalloc: large alloc 1073750016 bytes == 0x618be000 @ 0x7f36e05d62a4 0x591a07 0x5b5d56 0x502e9a 0x506859 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x504c28 0x502540 0x502f3d 0x507641\n","\u001b[?25hCollecting torchvision\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl (54kB)\n","\u001b[K 100% |████████████████████████████████| 61kB 18.0MB/s \n","\u001b[?25hCollecting pillow>=4.1.1 (from torchvision)\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/62/94/5430ebaa83f91cc7a9f687ff5238e26164a779cca2ef9903232268b0a318/Pillow-5.3.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n","\u001b[K 100% |████████████████████████████████| 2.0MB 3.8MB/s \n","\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n","Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n","Installing collected packages: torch, pillow, torchvision\n"," Found existing installation: Pillow 4.0.0\n"," Uninstalling Pillow-4.0.0:\n"," Successfully uninstalled Pillow-4.0.0\n","Successfully installed pillow-5.3.0 torch-1.0.0 torchvision-0.2.1\n"],"name":"stdout"}]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","outputId":"1d031f6a-911d-4dc5-89f8-3fe9b92cf142","executionInfo":{"status":"ok","timestamp":1545893714687,"user_tz":420,"elapsed":1539,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","import torch.nn as nn\n","import torch.nn.functional as F\n","torch.__version__"],"execution_count":4,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'1.0.0'"]},"metadata":{"tags":[]},"execution_count":4}]},{"metadata":{"id":"ewrw93tt2BfV","colab_type":"text"},"cell_type":"markdown","source":["## 1. Import & process dataset\n","- CIFAR10 dataset can be downloaded by ```torchvision```\n"," - [torchvision.datasets](https://pytorch.org/docs/stable/torchvision/datasets.html)"]},{"metadata":{"id":"W5anlYa01w3w","colab_type":"code","outputId":"65b19caa-1d78-43d7-ea27-fbf238ddfb47","executionInfo":{"status":"ok","timestamp":1545893719000,"user_tz":420,"elapsed":2878,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":52}},"cell_type":"code","source":["from torchvision import datasets\n","import torchvision.transforms as transforms\n","\n","train_dataset = datasets.CIFAR10(root = \"/\", train = True, download = True, transform = transforms.ToTensor())\n","test_dataset = datasets.CIFAR10(root = \"/\", train = False, download = True, transform = transforms.ToTensor())"],"execution_count":5,"outputs":[{"output_type":"stream","text":["Files already downloaded and verified\n","Files already downloaded and verified\n"],"name":"stdout"}]},{"metadata":{"id":"9nznVMdo5edZ","colab_type":"text"},"cell_type":"markdown","source":["## 2. Creating CNN model and training\n","\n","- Create and train deeper CNN model, having two convolutional & pooling layers\n","\n","![](https://www.researchgate.net/profile/Qianzhou_Du2/publication/322477802/figure/fig3/AS:582461356511232@1515881017676/Illustration-of-Convolutional-Neural-Network-CNN-Architecture.png)"]},{"metadata":{"id":"YEoNQr10kGUW","colab_type":"code","colab":{}},"cell_type":"code","source":["# create data loaders \n","train_loader = torch.utils.data.DataLoader(train_dataset, batch_size = 128, shuffle = True)\n","test_loader = torch.utils.data.DataLoader(test_dataset, batch_size = 128, shuffle = False)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"AQawpMRPI7jm","colab_type":"code","colab":{}},"cell_type":"code","source":["# create CNN with one convolution/pooling layer\n","class net(nn.Module):\n"," def __init__(self, input_dim, num_filters, conv_kernel_size, pool_kernel_size, stride, padding, num_classes):\n"," super(net, self).__init__()\n"," self.input_dim = input_dim\n"," conv_output_size = int((input_dim - conv_kernel_size + 2 * padding)/stride) + 1 # first conv layer output size\n"," pool_output_size = int((conv_output_size - pool_kernel_size)/stride) + 1 # first pooling layer output size\n"," conv_output_size = int((pool_output_size - conv_kernel_size + 2 * padding)/stride) + 1 # second conv layer output size\n"," pool_output_size = int((conv_output_size - pool_kernel_size)/stride) + 1 # second pooling layer output size\n"," \n"," self.conv1 = nn.Conv2d(3, num_filters[0], kernel_size = conv_kernel_size, stride = stride, padding = padding) \n"," self.pool1 = nn.MaxPool2d(kernel_size = pool_kernel_size, stride = stride)\n"," self.conv2 = nn.Conv2d(num_filters[0], num_filters[1], kernel_size = conv_kernel_size, stride = stride, padding = padding) \n"," self.pool2 = nn.MaxPool2d(kernel_size = pool_kernel_size, stride = stride)\n"," self.relu = nn.ReLU()\n"," self.dense = nn.Linear(pool_output_size * pool_output_size * num_filters[-1], num_classes) \n"," \n"," def forward(self, x):\n"," x = self.conv1(x)\n"," x = self.relu(x)\n"," x = self.pool1(x)\n"," x = self.conv2(x)\n"," x = self.relu(x)\n"," x = self.pool2(x)\n"," x = x.view(x.size(0), -1) # resize to fit into final dense layer\n"," x = self.dense(x)\n"," return x"],"execution_count":0,"outputs":[]},{"metadata":{"id":"rP0Gt5E9ajmd","colab_type":"code","colab":{}},"cell_type":"code","source":["# hyperparameters\n","DEVICE = torch.device('cuda')\n","INPUT_DIM = 32\n","NUM_FILTERS = (32, 32)\n","CONV_KERNEL_SIZE = 3\n","POOL_KERNEL_SIZE = 3\n","STRIDE = 1\n","PADDING = 1\n","NUM_CLASSES = 10\n","LEARNING_RATE = 1e-3\n","NUM_EPOCHS = 30 "],"execution_count":0,"outputs":[]},{"metadata":{"id":"cPBm8qDrSWsi","colab_type":"code","colab":{}},"cell_type":"code","source":["model = net(INPUT_DIM, NUM_FILTERS, CONV_KERNEL_SIZE, POOL_KERNEL_SIZE, STRIDE, PADDING, NUM_CLASSES).to(DEVICE)\n","criterion = nn.CrossEntropyLoss() # do not need softmax layer when using CEloss criterion\n","optimizer = torch.optim.Adam(model.parameters(), lr = LEARNING_RATE)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"SEBtAPYCFeic","colab_type":"code","outputId":"fa40e74c-75fb-4821-a991-1a900284ab96","executionInfo":{"status":"ok","timestamp":1545894705557,"user_tz":420,"elapsed":350059,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":545}},"cell_type":"code","source":["# training for NUM_EPOCHS\n","for i in range(NUM_EPOCHS):\n"," temp_loss = []\n"," for (x, y) in train_loader:\n"," x, y = x.float().to(DEVICE), y.to(DEVICE)\n"," outputs = model(x)\n"," loss = criterion(outputs, y)\n"," temp_loss.append(loss.item())\n"," \n"," optimizer.zero_grad()\n"," loss.backward()\n"," optimizer.step()\n"," \n"," print(\"Loss at {}th epoch: {}\".format(i, np.mean(temp_loss)))"],"execution_count":26,"outputs":[{"output_type":"stream","text":["Loss at 0th epoch: 1.488609354209412\n","Loss at 1th epoch: 1.1592735616142487\n","Loss at 2th epoch: 1.0515797013212043\n","Loss at 3th epoch: 0.9784907306856512\n","Loss at 4th epoch: 0.9343700555279432\n","Loss at 5th epoch: 0.8945669358038841\n","Loss at 6th epoch: 0.8600707198957653\n","Loss at 7th epoch: 0.8309158998377183\n","Loss at 8th epoch: 0.8015030218512201\n","Loss at 9th epoch: 0.7767865087675012\n","Loss at 10th epoch: 0.7458175573202656\n","Loss at 11th epoch: 0.720476924153545\n","Loss at 12th epoch: 0.7013726171172793\n","Loss at 13th epoch: 0.6770398076385489\n","Loss at 14th epoch: 0.6584783453313287\n","Loss at 15th epoch: 0.6332657866923096\n","Loss at 16th epoch: 0.6176666906270225\n","Loss at 17th epoch: 0.5958096003898269\n","Loss at 18th epoch: 0.5843624716524578\n","Loss at 19th epoch: 0.5720692123750897\n","Loss at 20th epoch: 0.545465971350365\n","Loss at 21th epoch: 0.5345218773845517\n","Loss at 22th epoch: 0.517139298989035\n","Loss at 23th epoch: 0.5017482235151178\n","Loss at 24th epoch: 0.48472921325422613\n","Loss at 25th epoch: 0.46795453310317703\n","Loss at 26th epoch: 0.45054197570551996\n","Loss at 27th epoch: 0.44360664589783116\n","Loss at 28th epoch: 0.4359766659910417\n","Loss at 29th epoch: 0.41148768883684406\n"],"name":"stdout"}]},{"metadata":{"id":"qpAJUiHm529m","colab_type":"text"},"cell_type":"markdown","source":["## 3. Evaluation\n","- Evaluate the trained CNN model with accuracy score \n"," - Store probability of each instance to a list and compare it with true y label"]},{"metadata":{"id":"txXH3dknFpSx","colab_type":"code","outputId":"7f0cb542-3a02-46ab-ab79-b1b3e18c3b04","executionInfo":{"status":"ok","timestamp":1545894809696,"user_tz":420,"elapsed":3180,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":72}},"cell_type":"code","source":["y_pred, y_true = [], []\n","with torch.no_grad():\n"," for x, y in test_loader:\n"," x, y = x.float().to(DEVICE), y.to(DEVICE)\n"," outputs = F.softmax(model(x)).max(1)[-1] # predicted label\n"," y_true += list(y.cpu().numpy()) # true label\n"," y_pred += list(outputs.cpu().numpy()) "],"execution_count":27,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n"," \"\"\"\n"],"name":"stderr"}]},{"metadata":{"id":"HV1s3xf5Frkl","colab_type":"code","outputId":"11e94351-4eed-4c45-8f59-9ad4f62adceb","executionInfo":{"status":"ok","timestamp":1545894811244,"user_tz":420,"elapsed":1147,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# evaluation result\n","from sklearn.metrics import accuracy_score\n","accuracy_score(y_true, y_pred)"],"execution_count":28,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.6643"]},"metadata":{"tags":[]},"execution_count":28}]}]} -------------------------------------------------------------------------------- /Deep Learning with PyTorch/DL-with-pytorch - 8 [RNN].ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"DL-with-pytorch - 8 [RNN].ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Recurrent Neural Networks\n","- IMDB review sentiment classification with RNN\n"," - Last time, we have started sentence classification with CNN and have achieved accuracy over 0.80. \n"," - This time, we try training with RNN to model text data"]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","outputId":"9e427acb-6e7c-46f8-ccbc-61a5b071edd2","executionInfo":{"status":"ok","timestamp":1545932184206,"user_tz":420,"elapsed":66823,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":336}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Collecting torch\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/7e/60/66415660aa46b23b5e1b72bc762e816736ce8d7260213e22365af51e8f9c/torch-1.0.0-cp36-cp36m-manylinux1_x86_64.whl (591.8MB)\n","\u001b[K 100% |████████████████████████████████| 591.8MB 28kB/s \n","tcmalloc: large alloc 1073750016 bytes == 0x61d2e000 @ 0x7f29476992a4 0x591a07 0x5b5d56 0x502e9a 0x506859 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x504c28 0x502540 0x502f3d 0x507641\n","\u001b[?25hCollecting torchvision\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl (54kB)\n","\u001b[K 100% |████████████████████████████████| 61kB 18.0MB/s \n","\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n","Collecting pillow>=4.1.1 (from torchvision)\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/62/94/5430ebaa83f91cc7a9f687ff5238e26164a779cca2ef9903232268b0a318/Pillow-5.3.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n","\u001b[K 100% |████████████████████████████████| 2.0MB 4.5MB/s \n","\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n","Installing collected packages: torch, pillow, torchvision\n"," Found existing installation: Pillow 4.0.0\n"," Uninstalling Pillow-4.0.0:\n"," Successfully uninstalled Pillow-4.0.0\n","Successfully installed pillow-5.3.0 torch-1.0.0 torchvision-0.2.1\n"],"name":"stdout"}]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","outputId":"076899a6-0d7a-4dc7-c934-34527bb14bd9","executionInfo":{"status":"ok","timestamp":1545941966434,"user_tz":420,"elapsed":1729,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","import torch.nn as nn\n","import torch.nn.functional as F\n","torch.__version__"],"execution_count":1,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'1.0.0'"]},"metadata":{"tags":[]},"execution_count":1}]},{"metadata":{"id":"ewrw93tt2BfV","colab_type":"text"},"cell_type":"markdown","source":["## 1. Import & process dataset\n","- IMDB review dataset for sentiment analysis\n"," - [source](http://ai.stanford.edu/~amaas/data/sentiment/)\n"," - Let's cheat a while and use dataset provided by Keras"]},{"metadata":{"id":"SxDHXFEsf5em","colab_type":"code","outputId":"2a1622aa-c6ba-46bc-d04f-a559347c37c1","executionInfo":{"status":"ok","timestamp":1545941991487,"user_tz":420,"elapsed":9393,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":52}},"cell_type":"code","source":["from keras.datasets import imdb\n","from keras.preprocessing import sequence\n","\n","num_words = 10000\n","maxlen = 50\n","\n","(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words = num_words)\n","\n","X_train = sequence.pad_sequences(X_train, maxlen = maxlen, padding = 'pre')\n","X_test = sequence.pad_sequences(X_test, maxlen = maxlen, padding = 'pre')\n"," \n","print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)"],"execution_count":2,"outputs":[{"output_type":"stream","text":["Using TensorFlow backend.\n"],"name":"stderr"},{"output_type":"stream","text":["(25000, 50) (25000, 50) (25000,) (25000,)\n"],"name":"stdout"}]},{"metadata":{"id":"9nznVMdo5edZ","colab_type":"text"},"cell_type":"markdown","source":["## 2. Creating RNN model and training\n","\n","- Create and train RNN model for sentence classification, with one GRU layer\n","\n","\n","![](http://upload.wikimedia.org/wikipedia/commons/thumb/3/37/Gated_Recurrent_Unit%2C_base_type.svg/440px-Gated_Recurrent_Unit%2C_base_type.svg.png)"]},{"metadata":{"id":"tdmsy2B3weeK","colab_type":"code","colab":{}},"cell_type":"code","source":["class imdbTrainDataset(torch.utils.data.Dataset):\n"," def __init__(self):\n"," self.X = X_train\n"," self.y = y_train\n"," \n"," def __getitem__(self, idx):\n"," return self.X[idx], self.y[idx]\n"," \n"," def __len__(self):\n"," return len(self.X)\n"," \n","class imdbTestDataset(torch.utils.data.Dataset):\n"," def __init__(self):\n"," self.X = X_test\n"," self.y = y_test\n"," \n"," def __getitem__(self, idx):\n"," return self.X[idx], self.y[idx]\n"," \n"," def __len__(self):\n"," return len(self.X)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"xFBmmDxKw1t2","colab_type":"code","colab":{}},"cell_type":"code","source":["# create dataset & dataloader instances\n","train_dataset = imdbTrainDataset()\n","test_dataset = imdbTestDataset()\n","\n","train_loader = torch.utils.data.DataLoader(train_dataset, batch_size = 128, shuffle = True)\n","test_loader = torch.utils.data.DataLoader(test_dataset, batch_size = 128, shuffle = False)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"AQawpMRPI7jm","colab_type":"code","colab":{}},"cell_type":"code","source":["# create RNN with one GRU layer\n","class net(nn.Module):\n"," def __init__(self, input_dim, num_words, embedding_dim, hidden_size, device):\n"," super(net, self).__init__()\n"," self.input_dim = input_dim\n"," self.embedding_dim = embedding_dim\n"," self.hidden_size = hidden_size\n"," self.device = device\n"," \n"," self.embedding = nn.Embedding(num_words, self.embedding_dim)\n"," \n"," # recurrent layer (GRU)\n"," self.rnn = nn.GRU(input_size = self.embedding_dim, hidden_size = hidden_size)\n"," self.dense = nn.Linear(hidden_size, 2) \n"," \n"," def forward(self, x):\n","\n"," x = self.embedding(x) # project to word embedding space\n"," \n"," h0 = torch.from_numpy(np.zeros((1, x.size(1), self.hidden_size))).float().to(self.device)\n"," x, _ = self.rnn(x, h0)\n"," x = x[:, -1, :]\n"," x = self.dense(x)\n"," return x"],"execution_count":0,"outputs":[]},{"metadata":{"id":"rP0Gt5E9ajmd","colab_type":"code","colab":{}},"cell_type":"code","source":["# hyperparameters\n","DEVICE = torch.device('cuda')\n","INPUT_DIM = maxlen\n","EMBEDDING_DIM = 50\n","HIDDEN_SIZE = 10\n","NUM_WORDS = num_words\n","LEARNING_RATE = 1e-3\n","NUM_EPOCHS = 30 "],"execution_count":0,"outputs":[]},{"metadata":{"id":"cPBm8qDrSWsi","colab_type":"code","colab":{}},"cell_type":"code","source":["model = net(INPUT_DIM, NUM_WORDS, EMBEDDING_DIM, HIDDEN_SIZE, DEVICE).to(DEVICE)\n","criterion = nn.CrossEntropyLoss() # do not need softmax layer when using CEloss criterion\n","optimizer = torch.optim.Adam(model.parameters(), lr = LEARNING_RATE)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"SEBtAPYCFeic","colab_type":"code","outputId":"41554f7e-ebdd-44bf-dc22-63162fc583ee","executionInfo":{"status":"ok","timestamp":1545942807966,"user_tz":420,"elapsed":114248,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":545}},"cell_type":"code","source":["# training for NUM_EPOCHS\n","for i in range(NUM_EPOCHS):\n"," temp_loss = []\n"," for (x, y) in train_loader:\n"," x, y = x.long().to(DEVICE), y.to(DEVICE) # beware that input to embedding should be type 'long'\n"," outputs = model(x)\n"," loss = criterion(outputs, y)\n"," temp_loss.append(loss.item())\n"," \n"," optimizer.zero_grad()\n"," loss.backward()\n"," optimizer.step()\n"," \n"," print(\"Loss at {}th epoch: {}\".format(i, np.mean(temp_loss)))"],"execution_count":31,"outputs":[{"output_type":"stream","text":["Loss at 0th epoch: 0.6974587248904365\n","Loss at 1th epoch: 0.6849879643746785\n","Loss at 2th epoch: 0.6775674278638801\n","Loss at 3th epoch: 0.6681843965637441\n","Loss at 4th epoch: 0.656453858224713\n","Loss at 5th epoch: 0.6463168439816456\n","Loss at 6th epoch: 0.6354162738639482\n","Loss at 7th epoch: 0.6256442434933721\n","Loss at 8th epoch: 0.6180111759779404\n","Loss at 9th epoch: 0.6114620164949067\n","Loss at 10th epoch: 0.6037514221911527\n","Loss at 11th epoch: 0.5976227883781705\n","Loss at 12th epoch: 0.5947248400474081\n","Loss at 13th epoch: 0.5908836469966539\n","Loss at 14th epoch: 0.5851703447346784\n","Loss at 15th epoch: 0.5835301590209104\n","Loss at 16th epoch: 0.5775724901836745\n","Loss at 17th epoch: 0.5748130908730079\n","Loss at 18th epoch: 0.5724356636709097\n","Loss at 19th epoch: 0.5707177081886603\n","Loss at 20th epoch: 0.5690213797651992\n","Loss at 21th epoch: 0.5663490290848576\n","Loss at 22th epoch: 0.5647801385850323\n","Loss at 23th epoch: 0.5625148958393505\n","Loss at 24th epoch: 0.5615952776706948\n","Loss at 25th epoch: 0.5612863715814085\n","Loss at 26th epoch: 0.5593391068431796\n","Loss at 27th epoch: 0.557831632695636\n","Loss at 28th epoch: 0.5568314390827198\n","Loss at 29th epoch: 0.556679327116937\n"],"name":"stdout"}]},{"metadata":{"id":"qpAJUiHm529m","colab_type":"text"},"cell_type":"markdown","source":["## 3. Evaluation\n","- Evaluate the trained RNN model with accuracy score \n"," - Store probability of each instance to a list and compare it with true y label"]},{"metadata":{"id":"txXH3dknFpSx","colab_type":"code","outputId":"5a4128c5-75cd-4274-bf5c-34fad648a86b","executionInfo":{"status":"ok","timestamp":1545942833837,"user_tz":420,"elapsed":3065,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":72}},"cell_type":"code","source":["y_pred, y_true = [], []\n","with torch.no_grad():\n"," for x, y in test_loader:\n"," x, y = x.long().to(DEVICE), y.to(DEVICE) # beware that input to embedding should be type 'long'\n"," outputs = F.softmax(model(x)).max(1)[-1] # predicted label\n"," y_true += list(y.cpu().numpy()) # true label\n"," y_pred += list(outputs.cpu().numpy()) "],"execution_count":32,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n"," \"\"\"\n"],"name":"stderr"}]},{"metadata":{"id":"HV1s3xf5Frkl","colab_type":"code","outputId":"41592584-9c51-4c4a-e972-2b6e29efe0f5","executionInfo":{"status":"ok","timestamp":1545942833838,"user_tz":420,"elapsed":1616,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# evaluation result\n","from sklearn.metrics import accuracy_score\n","accuracy_score(y_true, y_pred)"],"execution_count":33,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.59316"]},"metadata":{"tags":[]},"execution_count":33}]}]} -------------------------------------------------------------------------------- /Deep Learning with PyTorch/DL-with-pytorch - 6 [CNN].ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"DL-with-pytorch - 6 [CNN].ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Convolutional Neural Networks\n","- IMDB review sentiment classification with CNN\n"," - Up to last time, we have tried to classify images in the CIFAR-10 dataset with CNN. However, CNNs are not only useful in classifying and recognizing images, but also processing data with temporal dependencies, such as text data.\n"," - Here, let's try classifying movie review data with CNN"]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","outputId":"9e427acb-6e7c-46f8-ccbc-61a5b071edd2","executionInfo":{"status":"ok","timestamp":1545932184206,"user_tz":420,"elapsed":66823,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":336}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":1,"outputs":[{"output_type":"stream","text":["Collecting torch\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/7e/60/66415660aa46b23b5e1b72bc762e816736ce8d7260213e22365af51e8f9c/torch-1.0.0-cp36-cp36m-manylinux1_x86_64.whl (591.8MB)\n","\u001b[K 100% |████████████████████████████████| 591.8MB 28kB/s \n","tcmalloc: large alloc 1073750016 bytes == 0x61d2e000 @ 0x7f29476992a4 0x591a07 0x5b5d56 0x502e9a 0x506859 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x504c28 0x502540 0x502f3d 0x507641\n","\u001b[?25hCollecting torchvision\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl (54kB)\n","\u001b[K 100% |████████████████████████████████| 61kB 18.0MB/s \n","\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n","Collecting pillow>=4.1.1 (from torchvision)\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/62/94/5430ebaa83f91cc7a9f687ff5238e26164a779cca2ef9903232268b0a318/Pillow-5.3.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n","\u001b[K 100% |████████████████████████████████| 2.0MB 4.5MB/s \n","\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n","Installing collected packages: torch, pillow, torchvision\n"," Found existing installation: Pillow 4.0.0\n"," Uninstalling Pillow-4.0.0:\n"," Successfully uninstalled Pillow-4.0.0\n","Successfully installed pillow-5.3.0 torch-1.0.0 torchvision-0.2.1\n"],"name":"stdout"}]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","outputId":"d695e86a-a98a-4951-d9cf-185ef8130dd9","executionInfo":{"status":"ok","timestamp":1545933208234,"user_tz":420,"elapsed":1304,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","import torch.nn as nn\n","import torch.nn.functional as F\n","torch.__version__"],"execution_count":1,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'1.0.0'"]},"metadata":{"tags":[]},"execution_count":1}]},{"metadata":{"id":"ewrw93tt2BfV","colab_type":"text"},"cell_type":"markdown","source":["## 1. Import & process dataset\n","- IMDB review dataset for sentiment analysis\n"," - [source](http://ai.stanford.edu/~amaas/data/sentiment/)\n"," - Let's cheat a while and use dataset provided by Keras"]},{"metadata":{"id":"SxDHXFEsf5em","colab_type":"code","outputId":"6995c6a3-d576-4267-840a-f5362e03ef28","executionInfo":{"status":"ok","timestamp":1545933216936,"user_tz":420,"elapsed":8829,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":52}},"cell_type":"code","source":["from keras.datasets import imdb\n","from keras.preprocessing import sequence\n","\n","num_words = 10000\n","maxlen = 50\n","\n","(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words = num_words)\n","\n","X_train = sequence.pad_sequences(X_train, maxlen = maxlen, padding = 'pre')\n","X_test = sequence.pad_sequences(X_test, maxlen = maxlen, padding = 'pre')\n"," \n","print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)"],"execution_count":2,"outputs":[{"output_type":"stream","text":["Using TensorFlow backend.\n"],"name":"stderr"},{"output_type":"stream","text":["(25000, 50) (25000, 50) (25000,) (25000,)\n"],"name":"stdout"}]},{"metadata":{"id":"9nznVMdo5edZ","colab_type":"text"},"cell_type":"markdown","source":["## 2. Creating CNN model and training\n","\n","- Create and train CNN model for sentence classification, with one convolutional & average pooling layer\n","- Model architecture is adopted from [Kim 2015](https://www.aclweb.org/anthology/D14-1181)\n","\n","![](https://3qeqpr26caki16dnhd19sv6by6v-wpengine.netdna-ssl.com/wp-content/uploads/2017/08/Example-of-a-CNN-Filter-and-Polling-Architecture-for-Natural-Language-Processing.png)"]},{"metadata":{"id":"tdmsy2B3weeK","colab_type":"code","colab":{}},"cell_type":"code","source":["class imdbTrainDataset(torch.utils.data.Dataset):\n"," def __init__(self):\n"," self.X = X_train\n"," self.y = y_train\n"," \n"," def __getitem__(self, idx):\n"," return self.X[idx], self.y[idx]\n"," \n"," def __len__(self):\n"," return len(self.X)\n"," \n","class imdbTestDataset(torch.utils.data.Dataset):\n"," def __init__(self):\n"," self.X = X_test\n"," self.y = y_test\n"," \n"," def __getitem__(self, idx):\n"," return self.X[idx], self.y[idx]\n"," \n"," def __len__(self):\n"," return len(self.X)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"xFBmmDxKw1t2","colab_type":"code","colab":{}},"cell_type":"code","source":["# create dataset & dataloader instances\n","train_dataset = imdbTrainDataset()\n","test_dataset = imdbTestDataset()\n","\n","train_loader = torch.utils.data.DataLoader(train_dataset, batch_size = 128, shuffle = True)\n","test_loader = torch.utils.data.DataLoader(test_dataset, batch_size = 128, shuffle = False)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"AQawpMRPI7jm","colab_type":"code","colab":{}},"cell_type":"code","source":["# create CNN with one convolution/pooling layer\n","class net(nn.Module):\n"," def __init__(self, input_dim, num_words, embedding_dim, num_filters, kernel_size, stride):\n"," super(net, self).__init__()\n"," self.input_dim = input_dim\n"," self.embedding_dim = embedding_dim\n"," \n"," conv_output_size = int((input_dim - kernel_size)/stride) + 1 # first conv layer output size\n"," \n"," self.embedding = nn.Embedding(num_words, self.embedding_dim)\n"," self.conv = nn.Conv2d(1, num_filters, kernel_size = (kernel_size, self.embedding_dim), stride = stride) \n"," self.pool = nn.MaxPool2d((conv_output_size, 1)) # Max-over-time pooling (FYI: avg pooling also works)\n"," self.relu = nn.ReLU()\n"," self.dense = nn.Linear(num_filters, 2) \n"," \n"," def forward(self, x):\n"," x = self.embedding(x) # project to word embedding space\n"," x = x.view(-1, 1, self.input_dim, self.embedding_dim) # resize to fit into convolutional layer\n"," x = self.conv(x)\n"," x = self.relu(x)\n"," x = self.pool(x) \n"," x = x.view(x.size(0), -1) # resize to fit into final dense layer\n"," x = self.dense(x)\n"," return x"],"execution_count":0,"outputs":[]},{"metadata":{"id":"rP0Gt5E9ajmd","colab_type":"code","colab":{}},"cell_type":"code","source":["# hyperparameters\n","DEVICE = torch.device('cuda')\n","INPUT_DIM = maxlen\n","NUM_FILTERS = 64\n","KERNEL_SIZE = 3\n","STRIDE = 1\n","EMBEDDING_DIM = 50\n","NUM_WORDS = num_words\n","LEARNING_RATE = 1e-3\n","NUM_EPOCHS = 30 "],"execution_count":0,"outputs":[]},{"metadata":{"id":"cPBm8qDrSWsi","colab_type":"code","colab":{}},"cell_type":"code","source":["model = net(INPUT_DIM, NUM_WORDS, EMBEDDING_DIM, NUM_FILTERS, KERNEL_SIZE, STRIDE).to(DEVICE)\n","criterion = nn.CrossEntropyLoss() # do not need softmax layer when using CEloss criterion\n","optimizer = torch.optim.Adam(model.parameters(), lr = LEARNING_RATE)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"SEBtAPYCFeic","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":545},"outputId":"d94310e4-194c-455d-9830-a13d3c638039","executionInfo":{"status":"ok","timestamp":1545933874196,"user_tz":420,"elapsed":210646,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# training for NUM_EPOCHS\n","for i in range(NUM_EPOCHS):\n"," temp_loss = []\n"," for (x, y) in train_loader:\n"," x, y = x.long().to(DEVICE), y.to(DEVICE) # beware that input to embedding should be type 'long'\n"," outputs = model(x)\n"," loss = criterion(outputs, y)\n"," temp_loss.append(loss.item())\n"," \n"," optimizer.zero_grad()\n"," loss.backward()\n"," optimizer.step()\n"," \n"," print(\"Loss at {}th epoch: {}\".format(i, np.mean(temp_loss)))"],"execution_count":14,"outputs":[{"output_type":"stream","text":["Loss at 0th epoch: 0.6467448935216787\n","Loss at 1th epoch: 0.5299239850469998\n","Loss at 2th epoch: 0.4440507980025544\n","Loss at 3th epoch: 0.38025763858946005\n","Loss at 4th epoch: 0.3228620941541633\n","Loss at 5th epoch: 0.2733502811467161\n","Loss at 6th epoch: 0.22932577779402538\n","Loss at 7th epoch: 0.18939032276370088\n","Loss at 8th epoch: 0.1521880773029157\n","Loss at 9th epoch: 0.12011083876904176\n","Loss at 10th epoch: 0.09256724133251273\n","Loss at 11th epoch: 0.07036281743904158\n","Loss at 12th epoch: 0.0526158744469285\n","Loss at 13th epoch: 0.03983275468784327\n","Loss at 14th epoch: 0.029970756969509686\n","Loss at 15th epoch: 0.022237090422410746\n","Loss at 16th epoch: 0.016928160087946727\n","Loss at 17th epoch: 0.013268464244902134\n","Loss at 18th epoch: 0.01047601109627178\n","Loss at 19th epoch: 0.008334757876582444\n","Loss at 20th epoch: 0.00675514917606868\n","Loss at 21th epoch: 0.005520799474039932\n","Loss at 22th epoch: 0.004574792754209163\n","Loss at 23th epoch: 0.003809949959276662\n","Loss at 24th epoch: 0.0031995022270296302\n","Loss at 25th epoch: 0.00271612963501402\n","Loss at 26th epoch: 0.0023028627079578917\n","Loss at 27th epoch: 0.0019556615081121574\n","Loss at 28th epoch: 0.00167927734271091\n","Loss at 29th epoch: 0.0014462826259158627\n"],"name":"stdout"}]},{"metadata":{"id":"qpAJUiHm529m","colab_type":"text"},"cell_type":"markdown","source":["## 3. Evaluation\n","- Evaluate the trained CNN model with accuracy score \n"," - Store probability of each instance to a list and compare it with true y label"]},{"metadata":{"id":"txXH3dknFpSx","colab_type":"code","outputId":"35322a5b-a28c-463f-85c9-7b01b6cb66d8","executionInfo":{"status":"ok","timestamp":1545933445208,"user_tz":420,"elapsed":964,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":72}},"cell_type":"code","source":["y_pred, y_true = [], []\n","with torch.no_grad():\n"," for x, y in test_loader:\n"," x, y = x.long().to(DEVICE), y.to(DEVICE) # beware that input to embedding should be type 'long'\n"," outputs = F.softmax(model(x)).max(1)[-1] # predicted label\n"," y_true += list(y.cpu().numpy()) # true label\n"," y_pred += list(outputs.cpu().numpy()) "],"execution_count":9,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n"," \"\"\"\n"],"name":"stderr"}]},{"metadata":{"id":"HV1s3xf5Frkl","colab_type":"code","outputId":"90d07e53-a813-40e9-8227-09cbccd2fa7a","executionInfo":{"status":"ok","timestamp":1545933447202,"user_tz":420,"elapsed":986,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# evaluation result\n","from sklearn.metrics import accuracy_score\n","accuracy_score(y_true, y_pred)"],"execution_count":10,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.77464"]},"metadata":{"tags":[]},"execution_count":10}]}]} -------------------------------------------------------------------------------- /Deep Learning with PyTorch/DL-with-pytorch - 9 [RNN].ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"DL-with-pytorch - 9 [RNN].ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Recurrent Neural Networks\n","- IMDB review sentiment classification with RNN\n"," - Last time, we have started sentence classification with CNN and have achieved accuracy over 0.80. \n"," - This time, we try training with RNN to model text data"]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","outputId":"2a3e5076-f593-41ba-ada7-35277c683a89","executionInfo":{"status":"ok","timestamp":1545950044665,"user_tz":420,"elapsed":62689,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":336}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":2,"outputs":[{"output_type":"stream","text":["Collecting torch\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/7e/60/66415660aa46b23b5e1b72bc762e816736ce8d7260213e22365af51e8f9c/torch-1.0.0-cp36-cp36m-manylinux1_x86_64.whl (591.8MB)\n","\u001b[K 100% |████████████████████████████████| 591.8MB 26kB/s \n","tcmalloc: large alloc 1073750016 bytes == 0x6140c000 @ 0x7efc5d2cd2a4 0x591a07 0x5b5d56 0x502e9a 0x506859 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x504c28 0x502540 0x502f3d 0x507641\n","\u001b[?25hCollecting torchvision\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl (54kB)\n","\u001b[K 100% |████████████████████████████████| 61kB 20.0MB/s \n","\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n","Collecting pillow>=4.1.1 (from torchvision)\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/62/94/5430ebaa83f91cc7a9f687ff5238e26164a779cca2ef9903232268b0a318/Pillow-5.3.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n","\u001b[K 100% |████████████████████████████████| 2.0MB 3.3MB/s \n","\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n","Installing collected packages: torch, pillow, torchvision\n"," Found existing installation: Pillow 4.0.0\n"," Uninstalling Pillow-4.0.0:\n"," Successfully uninstalled Pillow-4.0.0\n","Successfully installed pillow-5.3.0 torch-1.0.0 torchvision-0.2.1\n"],"name":"stdout"}]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","outputId":"eaf4c480-5417-4c4a-92a9-ac51e6484baf","executionInfo":{"status":"ok","timestamp":1545962635136,"user_tz":420,"elapsed":884,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","import torch.nn as nn\n","import torch.nn.functional as F\n","torch.__version__"],"execution_count":19,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'1.0.0'"]},"metadata":{"tags":[]},"execution_count":19}]},{"metadata":{"id":"ewrw93tt2BfV","colab_type":"text"},"cell_type":"markdown","source":["## 1. Import & process dataset\n","- IMDB review dataset for sentiment analysis\n"," - [source](http://ai.stanford.edu/~amaas/data/sentiment/)\n"," - Let's cheat a while and use dataset provided by Keras"]},{"metadata":{"id":"SxDHXFEsf5em","colab_type":"code","outputId":"adf8d22d-c35d-422f-9b82-477796e21232","executionInfo":{"status":"ok","timestamp":1545962643802,"user_tz":420,"elapsed":7679,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["from keras.datasets import imdb\n","from keras.preprocessing import sequence\n","\n","num_words = 10000\n","maxlen = 50\n","\n","(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words = num_words)\n","\n","X_train = sequence.pad_sequences(X_train, maxlen = maxlen, padding = 'pre')\n","X_test = sequence.pad_sequences(X_test, maxlen = maxlen, padding = 'pre')\n"," \n","print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)"],"execution_count":20,"outputs":[{"output_type":"stream","text":["(25000, 50) (25000, 50) (25000,) (25000,)\n"],"name":"stdout"}]},{"metadata":{"id":"9nznVMdo5edZ","colab_type":"text"},"cell_type":"markdown","source":["## 2. Creating RNN model and training\n","\n","- Create and train RNN model for sentence classification, with one GRU layer\n","\n","\n","![](http://upload.wikimedia.org/wikipedia/commons/thumb/3/37/Gated_Recurrent_Unit%2C_base_type.svg/440px-Gated_Recurrent_Unit%2C_base_type.svg.png)"]},{"metadata":{"id":"tdmsy2B3weeK","colab_type":"code","colab":{}},"cell_type":"code","source":["class imdbTrainDataset(torch.utils.data.Dataset):\n"," def __init__(self):\n"," self.X = X_train\n"," self.y = y_train\n"," \n"," def __getitem__(self, idx):\n"," return self.X[idx], self.y[idx]\n"," \n"," def __len__(self):\n"," return len(self.X)\n"," \n","class imdbTestDataset(torch.utils.data.Dataset):\n"," def __init__(self):\n"," self.X = X_test\n"," self.y = y_test\n"," \n"," def __getitem__(self, idx):\n"," return self.X[idx], self.y[idx]\n"," \n"," def __len__(self):\n"," return len(self.X)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"xFBmmDxKw1t2","colab_type":"code","colab":{}},"cell_type":"code","source":["# create dataset & dataloader instances\n","train_dataset = imdbTrainDataset()\n","test_dataset = imdbTestDataset()\n","\n","train_loader = torch.utils.data.DataLoader(train_dataset, batch_size = 128, shuffle = True)\n","test_loader = torch.utils.data.DataLoader(test_dataset, batch_size = 128, shuffle = False)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"AQawpMRPI7jm","colab_type":"code","colab":{}},"cell_type":"code","source":["# create RNN with one GRU layer\n","class net(nn.Module):\n"," def __init__(self, input_dim, num_words, embedding_dim, hidden_size, device):\n"," super(net, self).__init__()\n"," self.input_dim = input_dim\n"," self.embedding_dim = embedding_dim\n"," self.hidden_size = hidden_size\n"," self.device = device\n"," \n"," self.embedding = nn.Embedding(num_words, self.embedding_dim)\n"," \n"," # recurrent layer (GRU)\n"," self.rnn = nn.GRU(input_size = self.embedding_dim, hidden_size = hidden_size, num_layers = 2, bidirectional = True)\n"," self.dense = nn.Linear(hidden_size * 2, 2) \n"," \n"," def forward(self, x):\n","\n"," x = self.embedding(x) # project to word embedding space\n"," \n"," h0 = torch.from_numpy(np.zeros((4, x.size(1), self.hidden_size))).float().to(self.device)\n"," x, _ = self.rnn(x, h0)\n"," x = x[:, -1, :]\n"," x = self.dense(x)\n"," return x"],"execution_count":0,"outputs":[]},{"metadata":{"id":"rP0Gt5E9ajmd","colab_type":"code","colab":{}},"cell_type":"code","source":["# hyperparameters\n","DEVICE = torch.device('cuda')\n","INPUT_DIM = maxlen\n","EMBEDDING_DIM = 50\n","HIDDEN_SIZE = 30\n","NUM_WORDS = num_words\n","LEARNING_RATE = 1e-2\n","NUM_EPOCHS = 50 "],"execution_count":0,"outputs":[]},{"metadata":{"id":"cPBm8qDrSWsi","colab_type":"code","colab":{}},"cell_type":"code","source":["model = net(INPUT_DIM, NUM_WORDS, EMBEDDING_DIM, HIDDEN_SIZE, DEVICE).to(DEVICE)\n","criterion = nn.CrossEntropyLoss() # do not need softmax layer when using CEloss criterion\n","optimizer = torch.optim.Adam(model.parameters(), lr = LEARNING_RATE)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"SEBtAPYCFeic","colab_type":"code","outputId":"db578f95-e97d-45ba-cf80-42f2d3578ce0","executionInfo":{"status":"ok","timestamp":1545968636915,"user_tz":420,"elapsed":561170,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":897}},"cell_type":"code","source":["# training for NUM_EPOCHS\n","for i in range(NUM_EPOCHS):\n"," temp_loss = []\n"," for (x, y) in train_loader:\n"," x, y = x.long().to(DEVICE), y.to(DEVICE) # beware that input to embedding should be type 'long'\n"," outputs = model(x)\n"," loss = criterion(outputs, y)\n"," temp_loss.append(loss.item())\n"," \n"," optimizer.zero_grad()\n"," loss.backward()\n"," optimizer.step()\n"," \n"," print(\"Loss at {}th epoch: {}\".format(i, np.mean(temp_loss)))"],"execution_count":44,"outputs":[{"output_type":"stream","text":["Loss at 0th epoch: 0.6862862052357926\n","Loss at 1th epoch: 0.6511446015567196\n","Loss at 2th epoch: 0.6249510837452752\n","Loss at 3th epoch: 0.6097113420160449\n","Loss at 4th epoch: 0.6029126274646545\n","Loss at 5th epoch: 0.592951243629261\n","Loss at 6th epoch: 0.5927904642054013\n","Loss at 7th epoch: 0.5877855566083169\n","Loss at 8th epoch: 0.5805982343700468\n","Loss at 9th epoch: 0.5823732127948683\n","Loss at 10th epoch: 0.5791789341946038\n","Loss at 11th epoch: 0.5721504489073947\n","Loss at 12th epoch: 0.567636356365924\n","Loss at 13th epoch: 0.5686291468386747\n","Loss at 14th epoch: 0.5680928979905284\n","Loss at 15th epoch: 0.5664665400981903\n","Loss at 16th epoch: 0.5596896381098397\n","Loss at 17th epoch: 0.5586986120562164\n","Loss at 18th epoch: 0.559884287416935\n","Loss at 19th epoch: 0.5565901298303993\n","Loss at 20th epoch: 0.5537556493464781\n","Loss at 21th epoch: 0.5537246392697704\n","Loss at 22th epoch: 0.5574867477526471\n","Loss at 23th epoch: 0.5543662172799207\n","Loss at 24th epoch: 0.5542050087634398\n","Loss at 25th epoch: 0.5516191876056243\n","Loss at 26th epoch: 0.551948319892494\n","Loss at 27th epoch: 0.5519226010964842\n","Loss at 28th epoch: 0.5533486927042202\n","Loss at 29th epoch: 0.5516620681298022\n","Loss at 30th epoch: 0.5526148445752203\n","Loss at 31th epoch: 0.5525032734992553\n","Loss at 32th epoch: 0.5505632563513152\n","Loss at 33th epoch: 0.5500600316694805\n","Loss at 34th epoch: 0.5512854948031659\n","Loss at 35th epoch: 0.5485446401396576\n","Loss at 36th epoch: 0.5485566307087334\n","Loss at 37th epoch: 0.54866475718362\n","Loss at 38th epoch: 0.5484578428523881\n","Loss at 39th epoch: 0.5509587942945714\n","Loss at 40th epoch: 0.5505209299070495\n","Loss at 41th epoch: 0.550063682910131\n","Loss at 42th epoch: 0.5487506185867348\n","Loss at 43th epoch: 0.5487663972438598\n","Loss at 44th epoch: 0.5497341061733207\n","Loss at 45th epoch: 0.5474839957088841\n","Loss at 46th epoch: 0.5488186151701577\n","Loss at 47th epoch: 0.5494940557650158\n","Loss at 48th epoch: 0.5462039275741091\n","Loss at 49th epoch: 0.5493884527561615\n"],"name":"stdout"}]},{"metadata":{"id":"qpAJUiHm529m","colab_type":"text"},"cell_type":"markdown","source":["## 3. Evaluation\n","- Evaluate the trained RNN model with accuracy score \n"," - Store probability of each instance to a list and compare it with true y label"]},{"metadata":{"id":"txXH3dknFpSx","colab_type":"code","outputId":"1333224a-acb7-4f8c-df07-f7d0e10cf0f1","executionInfo":{"status":"ok","timestamp":1545968664991,"user_tz":420,"elapsed":6426,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":72}},"cell_type":"code","source":["y_pred, y_true = [], []\n","with torch.no_grad():\n"," for x, y in test_loader:\n"," x, y = x.long().to(DEVICE), y.to(DEVICE) # beware that input to embedding should be type 'long'\n"," outputs = F.softmax(model(x)).max(1)[-1] # predicted label\n"," y_true += list(y.cpu().numpy()) # true label\n"," y_pred += list(outputs.cpu().numpy()) "],"execution_count":45,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n"," \"\"\"\n"],"name":"stderr"}]},{"metadata":{"id":"HV1s3xf5Frkl","colab_type":"code","outputId":"f35fe987-a7bb-4b6c-cfbd-686c7cde745d","executionInfo":{"status":"ok","timestamp":1545968664992,"user_tz":420,"elapsed":4120,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# evaluation result\n","from sklearn.metrics import accuracy_score\n","accuracy_score(y_true, y_pred)"],"execution_count":46,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.59312"]},"metadata":{"tags":[]},"execution_count":46}]}]} -------------------------------------------------------------------------------- /Deep Learning with PyTorch/DL-with-pytorch - 7 [CNN].ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"DL-with-pytorch - 7 [CNN].ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Convolutional Neural Networks\n","- IMDB review sentiment classification with CNN\n"," - Last time, we have started sentence classification with CNN having only one filter. Here, let's try more complicated CNN architecture with different filter sizes to ameliorate the performance."]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","outputId":"9e427acb-6e7c-46f8-ccbc-61a5b071edd2","executionInfo":{"status":"ok","timestamp":1545932184206,"user_tz":420,"elapsed":66823,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":336}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Collecting torch\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/7e/60/66415660aa46b23b5e1b72bc762e816736ce8d7260213e22365af51e8f9c/torch-1.0.0-cp36-cp36m-manylinux1_x86_64.whl (591.8MB)\n","\u001b[K 100% |████████████████████████████████| 591.8MB 28kB/s \n","tcmalloc: large alloc 1073750016 bytes == 0x61d2e000 @ 0x7f29476992a4 0x591a07 0x5b5d56 0x502e9a 0x506859 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x504c28 0x502540 0x502f3d 0x507641\n","\u001b[?25hCollecting torchvision\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl (54kB)\n","\u001b[K 100% |████████████████████████████████| 61kB 18.0MB/s \n","\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n","Collecting pillow>=4.1.1 (from torchvision)\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/62/94/5430ebaa83f91cc7a9f687ff5238e26164a779cca2ef9903232268b0a318/Pillow-5.3.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n","\u001b[K 100% |████████████████████████████████| 2.0MB 4.5MB/s \n","\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n","Installing collected packages: torch, pillow, torchvision\n"," Found existing installation: Pillow 4.0.0\n"," Uninstalling Pillow-4.0.0:\n"," Successfully uninstalled Pillow-4.0.0\n","Successfully installed pillow-5.3.0 torch-1.0.0 torchvision-0.2.1\n"],"name":"stdout"}]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","outputId":"3ba43d71-c184-412a-ac18-f18c1b5ccf78","executionInfo":{"status":"ok","timestamp":1545933745335,"user_tz":420,"elapsed":1595,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","import torch.nn as nn\n","import torch.nn.functional as F\n","torch.__version__"],"execution_count":1,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'1.0.0'"]},"metadata":{"tags":[]},"execution_count":1}]},{"metadata":{"id":"ewrw93tt2BfV","colab_type":"text"},"cell_type":"markdown","source":["## 1. Import & process dataset\n","- IMDB review dataset for sentiment analysis\n"," - [source](http://ai.stanford.edu/~amaas/data/sentiment/)\n"," - Let's cheat a while and use dataset provided by Keras"]},{"metadata":{"id":"SxDHXFEsf5em","colab_type":"code","outputId":"e48ae898-fb20-421d-abf4-088ff78e5c6c","executionInfo":{"status":"ok","timestamp":1545933757212,"user_tz":420,"elapsed":13458,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":52}},"cell_type":"code","source":["from keras.datasets import imdb\n","from keras.preprocessing import sequence\n","\n","num_words = 10000\n","maxlen = 50\n","\n","(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words = num_words)\n","\n","X_train = sequence.pad_sequences(X_train, maxlen = maxlen, padding = 'pre')\n","X_test = sequence.pad_sequences(X_test, maxlen = maxlen, padding = 'pre')\n"," \n","print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)"],"execution_count":2,"outputs":[{"output_type":"stream","text":["Using TensorFlow backend.\n"],"name":"stderr"},{"output_type":"stream","text":["(25000, 50) (25000, 50) (25000,) (25000,)\n"],"name":"stdout"}]},{"metadata":{"id":"9nznVMdo5edZ","colab_type":"text"},"cell_type":"markdown","source":["## 2. Creating CNN model and training\n","\n","- Create and train CNN model for sentence classification, with three convolutional & max pooling layers concatenated in the end.\n","- Model architecture is adopted from [Kim 2015](https://www.aclweb.org/anthology/D14-1181)\n","\n","![](https://3qeqpr26caki16dnhd19sv6by6v-wpengine.netdna-ssl.com/wp-content/uploads/2017/08/Example-of-a-CNN-Filter-and-Polling-Architecture-for-Natural-Language-Processing.png)"]},{"metadata":{"id":"tdmsy2B3weeK","colab_type":"code","colab":{}},"cell_type":"code","source":["class imdbTrainDataset(torch.utils.data.Dataset):\n"," def __init__(self):\n"," self.X = X_train\n"," self.y = y_train\n"," \n"," def __getitem__(self, idx):\n"," return self.X[idx], self.y[idx]\n"," \n"," def __len__(self):\n"," return len(self.X)\n"," \n","class imdbTestDataset(torch.utils.data.Dataset):\n"," def __init__(self):\n"," self.X = X_test\n"," self.y = y_test\n"," \n"," def __getitem__(self, idx):\n"," return self.X[idx], self.y[idx]\n"," \n"," def __len__(self):\n"," return len(self.X)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"xFBmmDxKw1t2","colab_type":"code","colab":{}},"cell_type":"code","source":["# create dataset & dataloader instances\n","train_dataset = imdbTrainDataset()\n","test_dataset = imdbTestDataset()\n","\n","train_loader = torch.utils.data.DataLoader(train_dataset, batch_size = 128, shuffle = True)\n","test_loader = torch.utils.data.DataLoader(test_dataset, batch_size = 128, shuffle = False)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"AQawpMRPI7jm","colab_type":"code","colab":{}},"cell_type":"code","source":["# create CNN with one convolution/pooling layer\n","class net(nn.Module):\n"," def __init__(self, input_dim, num_words, embedding_dim, num_filters, kernel_size, stride):\n"," super(net, self).__init__()\n"," self.input_dim = input_dim\n"," self.embedding_dim = embedding_dim\n"," \n"," conv_output_size1 = int((input_dim - kernel_size[0])/stride) + 1 # first conv layer output size\n"," conv_output_size2 = int((input_dim - kernel_size[1])/stride) + 1 # first conv layer output size\n"," conv_output_size3 = int((input_dim - kernel_size[2])/stride) + 1 # first conv layer output size\n"," \n"," self.embedding = nn.Embedding(num_words, self.embedding_dim)\n"," \n"," # three convolution & pooling layers\n"," self.conv1 = nn.Conv2d(1, num_filters[0], kernel_size = (kernel_size[0], self.embedding_dim), stride = stride) \n"," self.pool1 = nn.MaxPool2d((conv_output_size1, 1)) # Max-over-time pooling\n"," self.conv2 = nn.Conv2d(1, num_filters[1], kernel_size = (kernel_size[1], self.embedding_dim), stride = stride) \n"," self.pool2 = nn.MaxPool2d((conv_output_size2, 1)) # Max-over-time pooling\n"," self.conv3 = nn.Conv2d(1, num_filters[2], kernel_size = (kernel_size[2], self.embedding_dim), stride = stride) \n"," self.pool3 = nn.MaxPool2d((conv_output_size3, 1)) # Max-over-time pooling\n"," \n"," self.relu = nn.ReLU()\n"," self.dense = nn.Linear(num_filters[0] + num_filters[1] + num_filters[2], 2) \n"," \n"," def forward(self, x):\n"," x = self.embedding(x) # project to word embedding space\n"," x = x.view(-1, 1, self.input_dim, self.embedding_dim) # resize to fit into convolutional layer\n"," x1 = self.pool1(self.relu(self.conv1(x)))\n"," x2 = self.pool2(self.relu(self.conv2(x)))\n"," x3 = self.pool3(self.relu(self.conv3(x)))\n","\n"," x = torch.cat((x1, x2, x3), dim = 1) # concatenate three convolutional outputs\n"," x = x.view(x.size(0), -1) # resize to fit into final dense layer\n"," x = self.dense(x)\n"," return x"],"execution_count":0,"outputs":[]},{"metadata":{"id":"rP0Gt5E9ajmd","colab_type":"code","colab":{}},"cell_type":"code","source":["# hyperparameters\n","DEVICE = torch.device('cuda')\n","INPUT_DIM = maxlen\n","NUM_FILTERS = (16, 32, 64) \n","KERNEL_SIZE = (1, 2, 3)\n","STRIDE = 1\n","EMBEDDING_DIM = 50\n","NUM_WORDS = num_words\n","LEARNING_RATE = 1e-3\n","NUM_EPOCHS = 30 "],"execution_count":0,"outputs":[]},{"metadata":{"id":"cPBm8qDrSWsi","colab_type":"code","colab":{}},"cell_type":"code","source":["model = net(INPUT_DIM, NUM_WORDS, EMBEDDING_DIM, NUM_FILTERS, KERNEL_SIZE, STRIDE).to(DEVICE)\n","criterion = nn.CrossEntropyLoss() # do not need softmax layer when using CEloss criterion\n","optimizer = torch.optim.Adam(model.parameters(), lr = LEARNING_RATE)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"SEBtAPYCFeic","colab_type":"code","outputId":"ed0ebe28-62f1-4e8d-e464-d66cdb3d49cb","colab":{"base_uri":"https://localhost:8080/","height":545},"executionInfo":{"status":"ok","timestamp":1545934527865,"user_tz":420,"elapsed":224951,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# training for NUM_EPOCHS\n","for i in range(NUM_EPOCHS):\n"," temp_loss = []\n"," for (x, y) in train_loader:\n"," x, y = x.long().to(DEVICE), y.to(DEVICE) # beware that input to embedding should be type 'long'\n"," outputs = model(x)\n"," loss = criterion(outputs, y)\n"," temp_loss.append(loss.item())\n"," \n"," optimizer.zero_grad()\n"," loss.backward()\n"," optimizer.step()\n"," \n"," print(\"Loss at {}th epoch: {}\".format(i, np.mean(temp_loss)))"],"execution_count":27,"outputs":[{"output_type":"stream","text":["Loss at 0th epoch: 0.6226605290965158\n","Loss at 1th epoch: 0.4895881914666721\n","Loss at 2th epoch: 0.40360755549401656\n","Loss at 3th epoch: 0.3413624836474049\n","Loss at 4th epoch: 0.2855290376714298\n","Loss at 5th epoch: 0.23665862393622494\n","Loss at 6th epoch: 0.19332292257827155\n","Loss at 7th epoch: 0.1526499463587391\n","Loss at 8th epoch: 0.11854718325241488\n","Loss at 9th epoch: 0.09007945675783012\n","Loss at 10th epoch: 0.06637739711346066\n","Loss at 11th epoch: 0.04888947682493195\n","Loss at 12th epoch: 0.03602837147761365\n","Loss at 13th epoch: 0.026454947622759\n","Loss at 14th epoch: 0.019765267642785092\n","Loss at 15th epoch: 0.015103718741055655\n","Loss at 16th epoch: 0.011833845148319188\n","Loss at 17th epoch: 0.00931674547075313\n","Loss at 18th epoch: 0.007469219165112899\n","Loss at 19th epoch: 0.006082932326980695\n","Loss at 20th epoch: 0.005003999198824927\n","Loss at 21th epoch: 0.004151264236460687\n","Loss at 22th epoch: 0.0034760919789194452\n","Loss at 23th epoch: 0.0029148718803094662\n","Loss at 24th epoch: 0.002471668743148294\n","Loss at 25th epoch: 0.0021142353920256527\n","Loss at 26th epoch: 0.0018004027156073752\n","Loss at 27th epoch: 0.0015497613752888022\n","Loss at 28th epoch: 0.001329448939495891\n","Loss at 29th epoch: 0.001152327883518206\n"],"name":"stdout"}]},{"metadata":{"id":"qpAJUiHm529m","colab_type":"text"},"cell_type":"markdown","source":["## 3. Evaluation\n","- Evaluate the trained CNN model with accuracy score \n"," - Store probability of each instance to a list and compare it with true y label"]},{"metadata":{"id":"txXH3dknFpSx","colab_type":"code","outputId":"eb7f5d0b-a845-49b4-f7b8-f93b536c30a3","executionInfo":{"status":"ok","timestamp":1545935074996,"user_tz":420,"elapsed":1243,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":52}},"cell_type":"code","source":["y_pred, y_true = [], []\n","with torch.no_grad():\n"," for x, y in test_loader:\n"," x, y = x.long().to(DEVICE), y.to(DEVICE) # beware that input to embedding should be type 'long'\n"," outputs = F.softmax(model(x)).max(1)[-1] # predicted label\n"," y_true += list(y.cpu().numpy()) # true label\n"," y_pred += list(outputs.cpu().numpy()) "],"execution_count":28,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n"," \"\"\"\n"],"name":"stderr"}]},{"metadata":{"id":"HV1s3xf5Frkl","colab_type":"code","outputId":"73dc11cb-5863-4e41-f9f8-29ee06fc7c02","executionInfo":{"status":"ok","timestamp":1545935076195,"user_tz":420,"elapsed":1299,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# evaluation result\n","from sklearn.metrics import accuracy_score\n","accuracy_score(y_true, y_pred)"],"execution_count":29,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.80096"]},"metadata":{"tags":[]},"execution_count":29}]}]} -------------------------------------------------------------------------------- /PyTorch Basics/pytorch-tensors.ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"pytorch-tensors.ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Pytorch Tensors\n","- Tensors\n","- Creating Tensors\n","- Tensor Data Types\n","- Size (shape) of Tensors"]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":336},"outputId":"77b5be33-7c05-439c-b124-0903f64a41f7","executionInfo":{"status":"ok","timestamp":1538803854042,"user_tz":420,"elapsed":170845,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":1,"outputs":[{"output_type":"stream","text":["Collecting torch\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/49/0e/e382bcf1a6ae8225f50b99cc26effa2d4cc6d66975ccf3fa9590efcbedce/torch-0.4.1-cp36-cp36m-manylinux1_x86_64.whl (519.5MB)\n","\u001b[K 100% |████████████████████████████████| 519.5MB 29kB/s \n","tcmalloc: large alloc 1073750016 bytes == 0x586b2000 @ 0x7f0833a5f1c4 0x46d6a4 0x5fcbcc 0x4c494d 0x54f3c4 0x553aaf 0x54e4c8 0x54f4f6 0x553aaf 0x54efc1 0x54f24d 0x553aaf 0x54efc1 0x54f24d 0x553aaf 0x54efc1 0x54f24d 0x551ee0 0x54e4c8 0x54f4f6 0x553aaf 0x54efc1 0x54f24d 0x551ee0 0x54efc1 0x54f24d 0x551ee0 0x54e4c8 0x54f4f6 0x553aaf 0x54e4c8\n","\u001b[?25hCollecting torchvision\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl (54kB)\n","\u001b[K 100% |████████████████████████████████| 61kB 21.9MB/s \n","\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n","Collecting pillow>=4.1.1 (from torchvision)\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/62/94/5430ebaa83f91cc7a9f687ff5238e26164a779cca2ef9903232268b0a318/Pillow-5.3.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n","\u001b[K 100% |████████████████████████████████| 2.0MB 4.9MB/s \n","\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n","Installing collected packages: torch, pillow, torchvision\n"," Found existing installation: Pillow 4.0.0\n"," Uninstalling Pillow-4.0.0:\n"," Successfully uninstalled Pillow-4.0.0\n","Successfully installed pillow-5.3.0 torch-0.4.1 torchvision-0.2.1\n"],"name":"stdout"}]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"16bf052d-753a-47a7-8b83-c21d2876b322","executionInfo":{"status":"ok","timestamp":1538803875999,"user_tz":420,"elapsed":7976,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","torch.__version__"],"execution_count":2,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'0.4.1'"]},"metadata":{"tags":[]},"execution_count":2}]},{"metadata":{"id":"ewrw93tt2BfV","colab_type":"text"},"cell_type":"markdown","source":["## 1. Tensors\n","- Pytorch Tensors are similar to Tensors in TensorFlow. Both are based on NumPy arrays and function similar to how the NumPy array does.\n","- In a nutshell, Pytorch tensors are NumPy arrays with GPU computing.\n","- Note: from v.0.4.0, ```Variable``` is deprecated. Use only Tensors\n","- Documentation: [```torch.Tensor```](https://pytorch.org/docs/stable/tensors.html)"]},{"metadata":{"id":"YtjF_56l3d__","colab_type":"text"},"cell_type":"markdown","source":["## 2. Creating tensors\n","Tensors can be created in a number of ways:\n","\n","- From NumPy arrays\n","- From Python list/tuples\n","- Using Pytorch functions"]},{"metadata":{"id":"xz642SBhDGMb","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":87},"outputId":"eeb00578-858d-416a-8423-53b085d3533d","executionInfo":{"status":"ok","timestamp":1538804268528,"user_tz":420,"elapsed":1233,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# tensors can be created from arrays\n","# note that data type is conserved even though array is converted into tensor\n","x = np.array([1, 2, 3])\n","print(x)\n","print(x.dtype)\n","x = torch.from_numpy(x)\n","print(x)\n","print(x.dtype)"],"execution_count":8,"outputs":[{"output_type":"stream","text":["[1 2 3]\n","int64\n","tensor([1, 2, 3])\n","torch.int64\n"],"name":"stdout"}]},{"metadata":{"id":"kc-s_Jb44bjB","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":87},"outputId":"6830c0e1-626d-4e23-a754-f0d61e155a77","executionInfo":{"status":"ok","timestamp":1538804328844,"user_tz":420,"elapsed":1639,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["y = np.array([1., 2., 3.], dtype = np.float32)\n","print(y)\n","print(y.dtype)\n","y = torch.from_numpy(y)\n","print(y)\n","print(y.dtype)"],"execution_count":9,"outputs":[{"output_type":"stream","text":["[1. 2. 3.]\n","float32\n","tensor([1., 2., 3.])\n","torch.float32\n"],"name":"stdout"}]},{"metadata":{"id":"mRnx4sZv3xWJ","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":87},"outputId":"1d5cd232-dd7e-40e2-9708-d549ba84a063","executionInfo":{"status":"ok","timestamp":1538804627869,"user_tz":420,"elapsed":1644,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# when creating tensor from list/tuple, data type can be designated explicitly\n","x = torch.tensor([1, 2, 3])\n","print(x)\n","print(x.dtype)\n","\n","y = torch.IntTensor([1, 2, 3])\n","print(y)\n","print(y.dtype)"],"execution_count":13,"outputs":[{"output_type":"stream","text":["tensor([1, 2, 3])\n","torch.int64\n","tensor([1, 2, 3], dtype=torch.int32)\n","torch.int32\n"],"name":"stdout"}]},{"metadata":{"id":"_6LlSZ3U3VlK","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":175},"outputId":"7f52c40f-ad08-403a-e4d1-54928456d20c","executionInfo":{"status":"ok","timestamp":1538804667410,"user_tz":420,"elapsed":1648,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# there are some pytorch functions that are similar to those in numpy\n","x = torch.empty(5) # 1d tensor, uninitialized\n","print(x)\n","x = torch.randn(4, 2) # 2d random tensor\n","print(x)\n","x = torch.zeros(1, 2, 4) # 3d tensor \n","print(x)\n","x = torch.ones(1, 2, 4) # 3d tensor \n","print(x)"],"execution_count":16,"outputs":[{"output_type":"stream","text":["tensor([[0.0000, 0.0000],\n"," [0.0000, 0.0000],\n"," [0.0000, 0.0000],\n"," [0.0000, 0.0000]])\n","tensor([ 1.1233, 2.0569, 0.6844, -1.1102, -0.4936])\n","tensor([[[0., 0., 0., 0.],\n"," [0., 0., 0., 0.]]])\n","tensor([[[1., 1., 1., 1.],\n"," [1., 1., 1., 1.]]])\n"],"name":"stdout"}]},{"metadata":{"id":"WVZTmJ4CY8fM","colab_type":"text"},"cell_type":"markdown","source":["## 3. Tensor Data Types\n","- There are 8 data types in Pytorch and each has corresponding NumPy data type and CPU/GPU tensor instances\n"," - ```torch.uint8```\n"," - ```torch.float16``` (```torch.half```)\n"," - ```torch.float32``` (```torch.float```)\n"," - ```torch.float64``` (```torch.double```)\n"," - ```torch.int8``` \n"," - ```torch.int16``` (```torch.short```)\n"," - ```torch.int32``` (```torch.int```)\n"," - ```torch.int64``` (```torch.long```)\n","- Each tensor has its own data type and it can be changed\n","- CPU tensor can be changed into GPU tensor and vice versa"]},{"metadata":{"id":"Y9FOkKfy8xW2","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"4a0a3842-33df-4dfd-caab-6516c234a970","executionInfo":{"status":"ok","timestamp":1538805429034,"user_tz":420,"elapsed":1237,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# explicitly setting and printing out data type \n","x = torch.zeros(3, 2, dtype = torch.double)\n","print(x.dtype)"],"execution_count":27,"outputs":[{"output_type":"stream","text":["torch.float64\n"],"name":"stdout"}]},{"metadata":{"id":"wPt1Yip68_oB","colab_type":"code","colab":{}},"cell_type":"code","source":["# operation between tensors with different data types is not allowed\n","x = torch.zeros(2, 2, dtype = torch.int)\n","y = torch.zeros(2, 2, dtype = torch.float)\n","print(x + y) # error"],"execution_count":0,"outputs":[]},{"metadata":{"id":"gjOw2qVC68I0","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":52},"outputId":"53253fe7-dc80-4313-9a8b-a95f2bee0900","executionInfo":{"status":"ok","timestamp":1538805382740,"user_tz":420,"elapsed":1245,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# changing data type of tensor\n","x = torch.FloatTensor([1, 2, 3])\n","print(x.dtype)\n","x = x.long()\n","print(x.dtype)"],"execution_count":26,"outputs":[{"output_type":"stream","text":["torch.float32\n","torch.int64\n"],"name":"stdout"}]},{"metadata":{"id":"MScsQXTX91Ar","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":52},"outputId":"2e498ede-24e7-4856-d053-31b5f1c2a9d9","executionInfo":{"status":"ok","timestamp":1538805717937,"user_tz":420,"elapsed":1615,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# creating a GPU tensor\n","device = torch.device(\"cuda\")\n","x = torch.ones(3, device = device)\n","y = torch.ones(3)\n","print(x)\n","print(y)"],"execution_count":39,"outputs":[{"output_type":"stream","text":["tensor([1., 1., 1.], device='cuda:0')\n","tensor([1., 1., 1.])\n"],"name":"stdout"}]},{"metadata":{"id":"oD_sNKRf8f1N","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":52},"outputId":"d97f6fb7-a73f-41b7-8163-1b1ca2f385b7","executionInfo":{"status":"ok","timestamp":1538805618364,"user_tz":420,"elapsed":1627,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# converting CPU tensor to GPU tensor\n","device = torch.device(\"cuda\")\n","x = torch.FloatTensor([1, 2, 3])\n","print(x)\n","x = x.to(device)\n","print(x)"],"execution_count":36,"outputs":[{"output_type":"stream","text":["tensor([1., 2., 3.])\n","tensor([1., 2., 3.], device='cuda:0')\n"],"name":"stdout"}]},{"metadata":{"id":"nh5GGuln9g_x","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":52},"outputId":"99c11d0b-e0f4-4409-e3d2-6a9a03edca4e","executionInfo":{"status":"ok","timestamp":1538805652086,"user_tz":420,"elapsed":1229,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# converting GPU tensor to CPU tensor\n","print(x)\n","x = x.cpu()\n","print(x)"],"execution_count":37,"outputs":[{"output_type":"stream","text":["tensor([1., 2., 3.], device='cuda:0')\n","tensor([1., 2., 3.])\n"],"name":"stdout"}]},{"metadata":{"id":"7K9iC17g-N2E","colab_type":"text"},"cell_type":"markdown","source":["## 4. Size (shape) of Tensors\n","- Size of a tensor is equivalent to shape of NumPy array\n"," - Size of a tensor can be displayed using ```size()```\n"," - A tensor can be reshaped using ```view()```"]},{"metadata":{"id":"chXz3L5N-GpA","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":87},"outputId":"a8a3e04b-f5a8-410b-e224-6bdd2c0c0684","executionInfo":{"status":"ok","timestamp":1538805900819,"user_tz":420,"elapsed":1607,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# size of tensor is equivalent to shape of NumPy array\n","x = np.asarray([2, 4, 5])\n","print(x.shape)\n","x = torch.tensor([2, 4, 5])\n","print(x.size())\n","\n","x = np.asarray([[1,2], [3,4]])\n","print(x.shape)\n","x = torch.tensor([[1,2], [3,4]])\n","print(x.size())"],"execution_count":42,"outputs":[{"output_type":"stream","text":["(3,)\n","torch.Size([3])\n","(2, 2)\n","torch.Size([2, 2])\n"],"name":"stdout"}]},{"metadata":{"id":"1hgZ8h3B-fFd","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":70},"outputId":"8e175211-fb1f-479d-f1a5-d2c114ea3619","executionInfo":{"status":"ok","timestamp":1538805982602,"user_tz":420,"elapsed":1166,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# reshaping tensors\n","x = torch.ones(2, 8, dtype = torch.float)\n","print(x.size())\n","x = x.view(4, 4)\n","print(x.size())\n","x = x.view(8, -1)\n","print(x.size())"],"execution_count":43,"outputs":[{"output_type":"stream","text":["torch.Size([2, 8])\n","torch.Size([4, 4])\n","torch.Size([8, 2])\n"],"name":"stdout"}]},{"metadata":{"id":"zyAVn_Ng--Cl","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":52},"outputId":"c77901f6-f79d-45bc-ca54-130764504bca","executionInfo":{"status":"ok","timestamp":1538806024196,"user_tz":420,"elapsed":1510,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# to get scalar value of tensor with size 1, use item()\n","x = torch.tensor([7])\n","x /= 7\n","print(x)\n","print(x.item())"],"execution_count":44,"outputs":[{"output_type":"stream","text":["tensor([1])\n","1\n"],"name":"stdout"}]}]} -------------------------------------------------------------------------------- /PyTorch Basics/pytorch-model-basics-3 [CNN].ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"pytorch-model-basics-3 [CNN].ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Building Blocks of Models\n","- Convolution & Pooling\n","- Padding"]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","outputId":"3d971b90-4a62-4564-d3a2-9259fa52b587","executionInfo":{"status":"ok","timestamp":1543983074896,"user_tz":420,"elapsed":4151,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":125}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (0.4.1)\n","Requirement already satisfied: torchvision in /usr/local/lib/python3.6/dist-packages (0.2.1)\n","Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n","Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n","Requirement already satisfied: pillow>=4.1.1 in /usr/local/lib/python3.6/dist-packages (from torchvision) (5.3.0)\n"],"name":"stdout"}]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","outputId":"94a42058-3871-4e5f-8d1b-6a0554a3a75a","executionInfo":{"status":"ok","timestamp":1543981967794,"user_tz":420,"elapsed":1566,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","torch.__version__"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'0.4.1'"]},"metadata":{"tags":[]},"execution_count":2}]},{"metadata":{"id":"gyv2Sy5WO8lK","colab_type":"code","colab":{}},"cell_type":"code","source":["import torch.nn as nn"],"execution_count":0,"outputs":[]},{"metadata":{"id":"NAqYXyzcT-OJ","colab_type":"text"},"cell_type":"markdown","source":["## 1. Convolution & Pooling \n","Convolution and pooling are fundamental operations for building CNN models. There are a number of parameters and if their definitions are not clear, it could lead to great confusion.\n","- Parameters for convolution (pooling) layers\n"," - **stride:** how many \"steps\" that the filter makes for each advance\n"," - **kernel size**: how large is the kernel (filter) is\n"," - **number of filters (channels):** designates the \"depth\" of the data. Most image inputs have three filters (RGB)\n"," - **padding:** how to pad the input sample with zero in the border\n","- How to calculate output size of convolution/pooling operation\n","
\n","*(W - F + 2P)/S + 1*
\n"," - *W*: input size\n"," - *F*: kernel size\n"," - *P*: padding \n"," - *S*: stride\n"," \n","![alt text](https://user-images.githubusercontent.com/22738317/34081046-c3a97518-e347-11e7-98fe-929f602ee857.png)"]},{"metadata":{"id":"XoCXfOh1RQun","colab_type":"text"},"cell_type":"markdown","source":["### 1. Convolution & Pooling 1D\n","\n","- ```torch.nn.Conv1d()```: 1D convolution\n"," - Input: (N, Fi, Li): basically, each input is ```Fi``` vectors of length ```Li```\n"," - N: batch size\n"," - Fi: number of input filters (or channels)\n"," - Li: length of input sequence\n"," - Output: (N, Fo, Lo): each output is ```Fo``` vectors of length ```Lo```\n"," - N: batch size\n"," - Fo: number of output filters (or channels)\n"," - Lo: length of output sequence"]},{"metadata":{"id":"kp_YDg0aQQ0e","colab_type":"code","outputId":"115d3bbd-055e-416b-c550-b1c9ffa45bd8","executionInfo":{"status":"ok","timestamp":1543981975275,"user_tz":420,"elapsed":1323,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# case 1 - kernel size = 1\n","conv1d = nn.Conv1d(16, 32, kernel_size = 1)\n","\n","x = torch.ones(128, 16, 10) # input: batch_size = 128, num_filters = 16, seq_length = 10\n","print(conv1d(x).size()) # input and output size are equal when kernel_size = 1 (assuming no padding)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([128, 32, 10])\n"],"name":"stdout"}]},{"metadata":{"id":"LMcGMyJNSzSA","colab_type":"code","outputId":"5b0b68e1-6b30-4715-8693-96fc6bd724dc","executionInfo":{"status":"ok","timestamp":1538897442176,"user_tz":420,"elapsed":873,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# case 2 - kernel size = 2, stride = 1\n","conv1d = nn.Conv1d(16, 32, kernel_size = 2, padding = 2)\n","\n","x = torch.ones(128, 16, 10) # input: batch_size = 128, num_filters = 16, seq_length = 10\n","print(conv1d(x).size())"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([128, 32, 13])\n"],"name":"stdout"}]},{"metadata":{"id":"Irn1yF5lb3HO","colab_type":"code","outputId":"8a5e5025-bee4-4949-d115-63fdf9e1e70d","executionInfo":{"status":"ok","timestamp":1538897465647,"user_tz":420,"elapsed":916,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# case 2 - kernel size = 2, stride = 2\n","conv1d = nn.Conv1d(16, 64, kernel_size = 2, stride = 2, padding = 2)\n","\n","x = torch.ones(128, 16, 10) # input: batch_size = 128, num_filters = 16, seq_length = 10\n","print(conv1d(x).size())"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([128, 64, 7])\n"],"name":"stdout"}]},{"metadata":{"id":"6BF4TJ0Of1G1","colab_type":"text"},"cell_type":"markdown","source":["### Convolution & Pooling 2D\n","- ```torch.nn.Conv2d()```: 2D convolution\n"," - Largely siimilar to 1D-convolution, but input and outputs are 2-dimensional, rather than 1-dimensional\n","- Image data in Pytorch\n"," - ```Conv2d()``` is the basic building block of modern CNNs to process image, such as GoogleNet or ResNet\n"," - In Pytorch, images usually have shape of **\\[depth, height, width\\]**\n"," - depth corresponds to the number of filters (kernels)\n"," - width and height represent the size of an image. When the image is square (e.g., in CIFAR10), width = height\n"," \n"," ![alt text](http://cs231n.github.io/assets/cnn/cnn.jpeg)"]},{"metadata":{"id":"ddO_D_L2b81J","colab_type":"code","outputId":"923e57f9-8ced-4e2d-eba0-3074893cffcb","executionInfo":{"status":"ok","timestamp":1543983461231,"user_tz":420,"elapsed":1980,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":52}},"cell_type":"code","source":["# case 1 - kernel size = 1\n","conv2d = nn.Conv2d(16, 32, kernel_size = 1) # if kernel size is integer, width and height are equal (i.e., square kernel) \n","\n","x = torch.ones(128, 16, 10, 10) # input: batch_size = 128, num_filters = 16, height = 10, width = 10\n","print(conv2d(x).size()) # input and output size are equal when kernel_size = 1 (assuming no padding)\n","\n","conv2d = nn.Conv2d(16, 32, kernel_size = (1, 1)) # same as kernel size = 1\n","\n","x = torch.ones(128, 16, 10, 10) # input: batch_size = 128, num_filters = 16, height = 10, width = 10\n","print(conv2d(x).size()) # input and output size are equal when kernel_size = 1 (assuming no padding)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([128, 32, 10, 10])\n","torch.Size([128, 32, 10, 10])\n"],"name":"stdout"}]},{"metadata":{"id":"kvg4w7Q7lJBT","colab_type":"code","outputId":"1e6019f5-6756-4ad0-aa83-648f1fbd10f0","executionInfo":{"status":"ok","timestamp":1543983497981,"user_tz":420,"elapsed":1034,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# case 2 - kernel size = 2\n","conv2d = nn.Conv2d(16, 32, kernel_size = 2, padding = 1) # if kernel size is integer, width and height are equal (i.e., square kernel) \n","\n","x = torch.ones(128, 16, 10, 10) # input: batch_size = 128, num_filters = 16, height = 10, width = 10\n","print(conv2d(x).size()) "],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([128, 32, 11, 11])\n"],"name":"stdout"}]},{"metadata":{"id":"WMZooMlVljmy","colab_type":"code","outputId":"4f35276d-58de-4671-c6e2-4a6087216385","executionInfo":{"status":"ok","timestamp":1543983527824,"user_tz":420,"elapsed":1487,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# case 3 - differing kernel size\n","conv2d = nn.Conv2d(16, 32, kernel_size = (2, 1)) # if kernel size is integer, width and height are equal (i.e., square kernel) \n","\n","x = torch.ones(128, 16, 10, 10) # input: batch_size = 128, num_filters = 16, height = 10, width = 10\n","print(conv2d(x).size()) # non-square output"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([128, 32, 9, 10])\n"],"name":"stdout"}]},{"metadata":{"id":"9A514b51lsxS","colab_type":"code","outputId":"f3ae5c31-ba42-41d4-b8b4-456ce43f1f81","executionInfo":{"status":"ok","timestamp":1543983641407,"user_tz":420,"elapsed":1430,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# case 4 - kernel size = 3\n","conv2d = nn.Conv2d(16, 32, kernel_size = (3, 3), padding = 1) # if kernel size is integer, width and height are equal (i.e., square kernel) \n","\n","x = torch.ones(128, 16, 10, 10) # input: batch_size = 128, num_filters = 16, height = 10, width = 10\n","print(conv2d(x).size()) # input and output size are equal"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([128, 32, 10, 10])\n"],"name":"stdout"}]},{"metadata":{"id":"91PuKukfmK07","colab_type":"code","outputId":"1e0560c0-49d5-4580-d6d5-0a5cde6abd2e","executionInfo":{"status":"ok","timestamp":1543983657522,"user_tz":420,"elapsed":1550,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# case 4 - kernel size = 3, stride = 2\n","conv2d = nn.Conv2d(16, 32, kernel_size = (3, 3), stride = 2) # if kernel size is integer, width and height are equal (i.e., square kernel) \n","\n","x = torch.ones(128, 16, 10, 10) # input: batch_size = 128, num_filters = 16, height = 10, width = 10\n","print(conv2d(x).size()) # input and output size are equal"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([128, 32, 4, 4])\n"],"name":"stdout"}]},{"metadata":{"id":"uxSuAXXRmjSJ","colab_type":"text"},"cell_type":"markdown","source":["## 2. Padding\n","- Zero padding can be applied in convolution/pooling as we have seen above. But custom padding can be applied as well\n"," - ```nn.ConstandPad1d(padding, value)```: apply constant padding on 1D data\n"," - ```padding```: the shape of padding (if tuple, (```padingLeft```, ```padingRight```))\n"," - ```value```: the value of padding\n"," - ```nn.ConstantPad2d(padding, value)```: apply constant padding on 2D data\n"," - ```padding```: the shape of padding (if tuple, (```padingLeft```, ```padingRight```, ```paddingTop```, ```padingBottom```))\n"," - ```value```: the value of padding\n"," - ```nn.ZeroPad2d(padding)```: apply zero padding on 2D data \n"," - ```padding```: the shape of padding (if tuple, (```padingLeft```, ```padingRight```, ```paddingTop```, ```padingBottom```))"]},{"metadata":{"id":"l8v3fQXooGyk","colab_type":"code","outputId":"f9380ca5-60ba-421c-c284-51b432558386","executionInfo":{"status":"ok","timestamp":1543984263422,"user_tz":420,"elapsed":1379,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["p = nn.ConstantPad1d(1, 0.75) # 1d padding with constant 0.75\n","x = torch.ones(1, 1, 3)\n","print(p(x))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor([[[0.7500, 1.0000, 1.0000, 1.0000, 0.7500]]])\n"],"name":"stdout"}]},{"metadata":{"id":"QCt72l7LpNoc","colab_type":"code","outputId":"bce9e603-ef1b-4bef-9f03-70b4f3c7e2e5","executionInfo":{"status":"ok","timestamp":1543984484897,"user_tz":420,"elapsed":1077,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":70}},"cell_type":"code","source":["p = nn.ConstantPad2d((2, 2, 0, 0), -1) # 2d padding with -1 (on right and left)\n","x = torch.ones(1, 1, 3, 3)\n","print(p(x))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor([[[[-1., -1., 1., 1., 1., -1., -1.],\n"," [-1., -1., 1., 1., 1., -1., -1.],\n"," [-1., -1., 1., 1., 1., -1., -1.]]]])\n"],"name":"stdout"}]},{"metadata":{"id":"k7ucMutomgGh","colab_type":"code","outputId":"e0f5ce1a-51c2-4e53-c331-4a4172ac665d","executionInfo":{"status":"ok","timestamp":1543984158701,"user_tz":420,"elapsed":1516,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":70}},"cell_type":"code","source":["p = nn.ZeroPad2d((1,0,0,0)) # apply zero padding only on the left of first column\n","x = torch.ones(1, 1, 3, 3)\n","print(p(x))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor([[[[0., 1., 1., 1.],\n"," [0., 1., 1., 1.],\n"," [0., 1., 1., 1.]]]])\n"],"name":"stdout"}]}]} -------------------------------------------------------------------------------- /PyTorch Basics/pytorch-datasets-1.ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"pytorch-datasets-1.ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[],"toc_visible":true},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Generating Data in PyTorch\n","- Generating data from NumPy array\n","- Generating data using custom DataSet and DataLoaders"]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":105},"outputId":"88348d80-9804-4c3b-d18e-951fecb2de4a","executionInfo":{"status":"ok","timestamp":1538721493611,"user_tz":420,"elapsed":3858,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":39,"outputs":[{"output_type":"stream","text":["Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (0.4.1)\n","Requirement already satisfied: torchvision in /usr/local/lib/python3.6/dist-packages (0.2.1)\n","Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n","Requirement already satisfied: pillow>=4.1.1 in /usr/local/lib/python3.6/dist-packages (from torchvision) (5.3.0)\n","Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n"],"name":"stdout"}]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"7c452566-27df-4050-b6ac-51b1b59eb608","executionInfo":{"status":"ok","timestamp":1538720530647,"user_tz":420,"elapsed":7366,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","torch.__version__"],"execution_count":2,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'0.4.1'"]},"metadata":{"tags":[]},"execution_count":2}]},{"metadata":{"id":"WVZTmJ4CY8fM","colab_type":"text"},"cell_type":"markdown","source":["## 1. Generating data from NumPy array\n","- Import data using Pandas or NumPy and convert into Torch tensors"]},{"metadata":{"id":"7K9iC17g-N2E","colab_type":"text"},"cell_type":"markdown","source":["### Import data"]},{"metadata":{"id":"wJX5td0owoB-","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":198},"outputId":"1392e1fd-0640-4153-a3cf-f5e3759b7bdf","executionInfo":{"status":"ok","timestamp":1538679078875,"user_tz":420,"elapsed":362,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# retrive iris dataset from UCI repository using read_table() in Pandas\n","url = \"https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data\"\n","data = pd.read_table(url, header = None, sep = \",\")\n","data.columns = [\"sepal_len\", \"sepal_wid\", \"petal_len\", \"petal_wid\", \"class\"]\n","data.head()"],"execution_count":16,"outputs":[{"output_type":"execute_result","data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
sepal_lensepal_widpetal_lenpetal_widclass
05.13.51.40.2Iris-setosa
14.93.01.40.2Iris-setosa
24.73.21.30.2Iris-setosa
34.63.11.50.2Iris-setosa
45.03.61.40.2Iris-setosa
\n","
"],"text/plain":[" sepal_len sepal_wid petal_len petal_wid class\n","0 5.1 3.5 1.4 0.2 Iris-setosa\n","1 4.9 3.0 1.4 0.2 Iris-setosa\n","2 4.7 3.2 1.3 0.2 Iris-setosa\n","3 4.6 3.1 1.5 0.2 Iris-setosa\n","4 5.0 3.6 1.4 0.2 Iris-setosa"]},"metadata":{"tags":[]},"execution_count":16}]},{"metadata":{"id":"9Yb_HdI1xGst","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":198},"outputId":"3cb7e2a7-1631-41d4-a075-2cb8414b6c88","executionInfo":{"status":"ok","timestamp":1538679125648,"user_tz":420,"elapsed":346,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# first convert class into categorical values ({0, 1, 2})\n","data[\"class\"] = data[\"class\"].astype(\"category\").cat.codes\n","data.head()"],"execution_count":17,"outputs":[{"output_type":"execute_result","data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
sepal_lensepal_widpetal_lenpetal_widclass
05.13.51.40.20
14.93.01.40.20
24.73.21.30.20
34.63.11.50.20
45.03.61.40.20
\n","
"],"text/plain":[" sepal_len sepal_wid petal_len petal_wid class\n","0 5.1 3.5 1.4 0.2 0\n","1 4.9 3.0 1.4 0.2 0\n","2 4.7 3.2 1.3 0.2 0\n","3 4.6 3.1 1.5 0.2 0\n","4 5.0 3.6 1.4 0.2 0"]},"metadata":{"tags":[]},"execution_count":17}]},{"metadata":{"id":"peGq6XBDyg2S","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":36},"outputId":"f607c936-d549-45a1-e61c-c5d840e9d896","executionInfo":{"status":"ok","timestamp":1538679562868,"user_tz":420,"elapsed":248,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["X_data = data[[\"sepal_len\", \"sepal_wid\", \"petal_len\", \"petal_wid\"]].values.astype(\"float32\")\n","y_data = data[\"class\"].values.astype(\"int32\")\n","\n","print(X_data.shape, y_data.shape)"],"execution_count":34,"outputs":[{"output_type":"stream","text":["(150, 4) (150,)\n"],"name":"stdout"}]},{"metadata":{"id":"IFD0FeRccdgJ","colab_type":"text"},"cell_type":"markdown","source":["### Generating tensors\n","- Generating tensors directly"]},{"metadata":{"id":"jvRy7IJzbVMB","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":54},"outputId":"b1e92b7e-bf1e-4672-8f8c-cd89ecb8fdfa","executionInfo":{"status":"ok","timestamp":1538679564113,"user_tz":420,"elapsed":307,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# using from_numpy(): infers data type implicitly from numpy array\n","X_tensor = torch.from_numpy(X_data)\n","y_tensor = torch.from_numpy(y_data)\n","\n","print(X_tensor.type(), y_tensor.type())\n","print(X_tensor.size(), y_tensor.size())"],"execution_count":35,"outputs":[{"output_type":"stream","text":["torch.FloatTensor torch.IntTensor\n","torch.Size([150, 4]) torch.Size([150])\n"],"name":"stdout"}]},{"metadata":{"id":"X7GNYw4fbn9H","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":54},"outputId":"51d779e3-45e7-40aa-c10a-0fff1256fd3e","executionInfo":{"status":"ok","timestamp":1538679966434,"user_tz":420,"elapsed":259,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# assigning data type explicitly\n","X_tensor = torch.FloatTensor(X_data)\n","y_tensor = torch.LongTensor(y_data)\n","\n","print(X_tensor.type(), y_tensor.type())\n","print(X_tensor.size(), y_tensor.size())"],"execution_count":43,"outputs":[{"output_type":"stream","text":["torch.FloatTensor torch.LongTensor\n","torch.Size([150, 4]) torch.Size([150])\n"],"name":"stdout"}]},{"metadata":{"id":"yVWh8UoNhUWe","colab_type":"code","colab":{}},"cell_type":"code","source":["# logistic regression model\n","model = torch.nn.Linear(X_data.shape[-1], len(set(y_data)))\n","criterion = torch.nn.CrossEntropyLoss() \n","optimizer = torch.optim.Adam(model.parameters(), lr=1e-3) \n","model"],"execution_count":0,"outputs":[]},{"metadata":{"id":"pYo3RuXjdKT6","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":203},"outputId":"34b2296e-8a4d-45e9-b572-ec7652a7db83","executionInfo":{"status":"ok","timestamp":1538680046559,"user_tz":420,"elapsed":320,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# model training: whole dataset at a time\n","for epoch in range(100):\n"," outputs = model(X_tensor)\n"," loss = criterion(outputs, y_tensor)\n"," optimizer.zero_grad()\n"," loss.backward()\n"," optimizer.step()\n"," \n"," if (epoch + 1) % 10 == 0:\n"," print(\"Epoch: {}, Loss: {:.5f}\".format(epoch + 1, loss.item()))"],"execution_count":52,"outputs":[{"output_type":"stream","text":["Epoch: 10, Loss: 0.67772\n","Epoch: 20, Loss: 0.67183\n","Epoch: 30, Loss: 0.66608\n","Epoch: 40, Loss: 0.66047\n","Epoch: 50, Loss: 0.65499\n","Epoch: 60, Loss: 0.64963\n","Epoch: 70, Loss: 0.64440\n","Epoch: 80, Loss: 0.63928\n","Epoch: 90, Loss: 0.63428\n","Epoch: 100, Loss: 0.62938\n"],"name":"stdout"}]},{"metadata":{"id":"Tgf96It4fhr9","colab_type":"text"},"cell_type":"markdown","source":["## 2. Generating data using custom DataSet and DataLoaders\n","- Using customized dataset and dataloaders makes it easier to manage training process (e.g., implementing mini-batch SGD)"]},{"metadata":{"id":"SFXadkSTeN4r","colab_type":"code","colab":{}},"cell_type":"code","source":["class IrisDataset(torch.utils.data.Dataset):\n"," def __init__(self):\n"," # import and initialize dataset\n"," url = \"https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data\"\n"," data = pd.read_table(url, header = None, sep = \",\")\n"," data.columns = [\"sepal_len\", \"sepal_wid\", \"petal_len\", \"petal_wid\", \"class\"]\n"," data[\"class\"] = data[\"class\"].astype(\"category\").cat.codes\n"," \n"," self.X = data[[\"sepal_len\", \"sepal_wid\", \"petal_len\", \"petal_wid\"]].values\n"," self.Y = data[\"class\"].values[:, np.newaxis].astype(np.int32)\n"," \n"," def __getitem__(self, idx):\n"," # get item by index\n"," return self.X[idx], self.Y[idx]\n"," \n"," def __len__(self):\n"," # returns length of data\n"," return len(self.X)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"Iv30GdmFg1om","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":52},"outputId":"6bca4e99-ac9e-42a8-8aab-09f9debbb6bd","executionInfo":{"status":"ok","timestamp":1538721091572,"user_tz":420,"elapsed":1479,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# create dataset instance\n","irisdataset = IrisDataset()\n","\n","print(type(irisdataset))\n","print(len(irisdataset))"],"execution_count":25,"outputs":[{"output_type":"stream","text":["\n","150\n"],"name":"stdout"}]},{"metadata":{"id":"m56oNfIDg37o","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"f8607ce6-5921-410d-ec40-42ee044c157f","executionInfo":{"status":"ok","timestamp":1538721092739,"user_tz":420,"elapsed":1079,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# create dataloader instance\n","# set batch size to 32 (mini-batch SGD) and shuffle before training \n","# if batch_size is set to 1, stochastic gradient descent is implemented \n","dataloader = torch.utils.data.DataLoader(irisdataset, batch_size = 32, shuffle = True)\n","dataloader"],"execution_count":26,"outputs":[{"output_type":"execute_result","data":{"text/plain":[""]},"metadata":{"tags":[]},"execution_count":26}]},{"metadata":{"id":"VIfIswG4o6Mt","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"1c3e9268-1ba6-4f8a-e663-5a54c2f61b39","executionInfo":{"status":"ok","timestamp":1538721627737,"user_tz":420,"elapsed":988,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# logistic regression model\n","model = torch.nn.Linear(4, 3).double()\n","criterion = torch.nn.CrossEntropyLoss() \n","optimizer = torch.optim.Adam(model.parameters(), lr=1e-3) \n","model"],"execution_count":44,"outputs":[{"output_type":"execute_result","data":{"text/plain":["Linear(in_features=4, out_features=3, bias=True)"]},"metadata":{"tags":[]},"execution_count":44}]},{"metadata":{"id":"H-chn-XA4tvv","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":193},"outputId":"5ff1caf5-a705-47df-817c-c26dc7330c61","executionInfo":{"status":"ok","timestamp":1538721632640,"user_tz":420,"elapsed":1412,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["steps = len(dataloader)\n","\n","for epoch in range(100):\n"," for i, (x, y) in enumerate(dataloader):\n"," outputs = model(x)\n"," # to match outputs, y should be converted into LongTensor and size should be reduced to 1-d\n"," # this is because CrossEntropyLoss requires y_target to be 1-d\n"," loss = criterion(outputs, y.type(torch.LongTensor).view(-1))\n"," optimizer.zero_grad()\n"," loss.backward()\n"," optimizer.step()\n","\n"," if (epoch + 1) % 10 == 0:\n"," print(\"Epoch: {}, Loss: {:.5f}\".format(epoch + 1, loss.item()))"],"execution_count":45,"outputs":[{"output_type":"stream","text":["Epoch: 10, Loss: 1.28169\n","Epoch: 20, Loss: 1.14514\n","Epoch: 30, Loss: 1.11257\n","Epoch: 40, Loss: 1.04069\n","Epoch: 50, Loss: 0.98725\n","Epoch: 60, Loss: 0.93442\n","Epoch: 70, Loss: 0.88709\n","Epoch: 80, Loss: 0.79886\n","Epoch: 90, Loss: 0.82748\n","Epoch: 100, Loss: 0.80539\n"],"name":"stdout"}]}]} -------------------------------------------------------------------------------- /PyTorch Basics/pytorch-model-basics-4 [RNN].ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"pytorch-model-basics-4 [RNN].ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Recurrent Neural Networks\n","- Vanilla RNN\n","- Gated Recurrent Units\n","- Long Short Term Memory"]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","outputId":"cf6476e7-6200-49b3-988b-2d06b766f76f","executionInfo":{"status":"ok","timestamp":1544639936834,"user_tz":420,"elapsed":62712,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":336}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Collecting torch\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/7e/60/66415660aa46b23b5e1b72bc762e816736ce8d7260213e22365af51e8f9c/torch-1.0.0-cp36-cp36m-manylinux1_x86_64.whl (591.8MB)\n","\u001b[K 100% |████████████████████████████████| 591.8MB 24kB/s \n","tcmalloc: large alloc 1073750016 bytes == 0x60d92000 @ 0x7f44731162a4 0x591a07 0x5b5d56 0x502e9a 0x506859 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x504c28 0x502540 0x502f3d 0x507641\n","\u001b[?25hCollecting torchvision\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl (54kB)\n","\u001b[K 100% |████████████████████████████████| 61kB 21.9MB/s \n","\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n","Collecting pillow>=4.1.1 (from torchvision)\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/62/94/5430ebaa83f91cc7a9f687ff5238e26164a779cca2ef9903232268b0a318/Pillow-5.3.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n","\u001b[K 100% |████████████████████████████████| 2.0MB 4.8MB/s \n","\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n","Installing collected packages: torch, pillow, torchvision\n"," Found existing installation: Pillow 4.0.0\n"," Uninstalling Pillow-4.0.0:\n"," Successfully uninstalled Pillow-4.0.0\n","Successfully installed pillow-5.3.0 torch-1.0.0 torchvision-0.2.1\n"],"name":"stdout"}]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","outputId":"2d1d0d0a-874a-4e70-bc06-02e695726b41","executionInfo":{"status":"ok","timestamp":1544646625339,"user_tz":420,"elapsed":1221,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","torch.__version__"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'1.0.0'"]},"metadata":{"tags":[]},"execution_count":37}]},{"metadata":{"id":"gyv2Sy5WO8lK","colab_type":"code","colab":{}},"cell_type":"code","source":["import torch.nn as nn"],"execution_count":0,"outputs":[]},{"metadata":{"id":"XoCXfOh1RQun","colab_type":"text"},"cell_type":"markdown","source":["## 1. Vanilla RNN\n","![vanilla_RNN](http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-SimpleRNN.png)\n","\n","\n","- Vanilla RNN can be implemented with ```torch.nn.RNN()``` \n","\n","- Key Parameters\n"," - ```input_size```: number of expected features in the input (i.e., dimensionality of feature space)\n"," - ```hidden_size```: number of features in hidden state (i.e., dimensionality of output space)\n"," - ```num_layers```: number of recurrent layers (to create stacked RNN)\n"," - ```batch_first```: If ```True```, input and output tensor shapes are ```(batch_size, seq_length, dim_feature)```. If ```False```, ```(sequence_length, batch_size, dim_feature)```\n"," - ```bidirectional```: If ```True```, bidirectional RNN\n"," \n","- One thing to note is that unlike fully-connected layers or convolutional layers, RNNs take multi inputs/outputs\n"," - In addition to (sequential) inputs, RNN has another called hidden state, which makes RNN special\n"," - This hidden state sends information regarding current step to the next\\\n"," \n","- Inputs to RNN: ```(x0, h0)```\n"," - ```x0```: tensor that contains features of the input sequence\n"," - shape\n"," - ```(seq_len, batch_size, input_size)``` if ```batch_first == False``` (default)\n"," - ```(batch_size, seq_len, input_size)``` if ```batch_fist == True``` \n"," - ```h0```: tensor that contains hidden state for each instance\n"," - shape\n"," - ```(num_layers * num_directions, batch_size, hidden_size)```"]},{"metadata":{"id":"kp_YDg0aQQ0e","colab_type":"code","colab":{}},"cell_type":"code","source":["rnn = nn.RNN(input_size = 10, \n"," hidden_size = 5, \n"," num_layers = 1)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"rtVnd-SJomFA","colab_type":"code","outputId":"4e2aaebc-fe5a-4946-b082-a84390d38312","executionInfo":{"status":"ok","timestamp":1544646557492,"user_tz":420,"elapsed":1030,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["## inputs to RNN\n","# input data (seq_len, batch_size, input_size)\n","x0 = torch.from_numpy(np.random.randn(12, 64, 10)).float() \n","# hidden state (num_layers * num_directions, batch_size, hidden_size)\n","h0 = torch.from_numpy(np.zeros((1, 64, 5))).float() \n","\n","print(x0.shape, h0.shape)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([12, 64, 10]) torch.Size([1, 64, 5])\n"],"name":"stdout"}]},{"metadata":{"id":"fDyuMNs9oyQ-","colab_type":"code","outputId":"398e326a-c805-4c69-bd13-2d894224100d","executionInfo":{"status":"ok","timestamp":1544647163096,"user_tz":420,"elapsed":1503,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["## outputs from RNN\n","# output (seq_len, batch_size, num_directions * hidden_size)\n","# hidden state (num_layers * num_directions, batch_size, hidden_size)\n","out, h1 = rnn(x0, h0)\n","\n","print(out.shape, h1.shape)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([12, 64, 5]) torch.Size([1, 64, 5])\n"],"name":"stdout"}]},{"metadata":{"id":"LMcGMyJNSzSA","colab_type":"code","colab":{}},"cell_type":"code","source":["# when batch_first = True\n","rnn = nn.RNN(input_size = 10, \n"," hidden_size = 5, \n"," num_layers = 2, # stacked RNN (2 layers)\n"," batch_first = True)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"1wKPW7qiWZJt","colab_type":"code","outputId":"a84dc940-c6ee-4e42-a195-a781f6c1971b","executionInfo":{"status":"ok","timestamp":1544647225067,"user_tz":420,"elapsed":1061,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["## inputs to RNN\n","x0 = torch.from_numpy(np.random.randn(64, 12, 10)).float() \n","# note that even batch_first == True, hidden state shape order does not change\n","h0 = torch.from_numpy(np.zeros((2, 64, 5))).float() \n","\n","print(x0.shape, h0.shape)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([64, 12, 10]) torch.Size([2, 64, 5])\n"],"name":"stdout"}]},{"metadata":{"id":"aqUXKwvIXUH-","colab_type":"code","outputId":"81931e92-f228-4de7-ea64-50eb37f16ad4","executionInfo":{"status":"ok","timestamp":1544647249382,"user_tz":420,"elapsed":1058,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["## outputs from RNN\n","out, h1 = rnn(x0, h0)\n","\n","print(out.shape, h1.shape)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([64, 12, 5]) torch.Size([2, 64, 5])\n"],"name":"stdout"}]},{"metadata":{"id":"E4AJ3I9JWnyZ","colab_type":"code","outputId":"952ac553-5608-4888-eafd-ccef2d44dfe6","executionInfo":{"status":"ok","timestamp":1544647257247,"user_tz":420,"elapsed":1064,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# bidirectional, stacked RNN\n","rnn = nn.RNN(input_size = 20, \n"," hidden_size = 10, \n"," num_layers = 4, \n"," bidirectional = True)\n","\n","x0 = torch.from_numpy(np.random.randn(5, 64, 20)).float()\n","h0 = torch.from_numpy(np.zeros((4 * 2, 64, 10))).float() # notice the dimensionality of hidden state\n","out, h1 = rnn(x0, h0)\n","\n","print(out.shape, h1.shape)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([5, 64, 20]) torch.Size([8, 64, 10])\n"],"name":"stdout"}]},{"metadata":{"id":"7jJnvcgAXpSq","colab_type":"text"},"cell_type":"markdown","source":["## 2. Gated Recurrent Units (GRU)\n","\n","![](https://upload.wikimedia.org/wikipedia/commons/thumb/3/37/Gated_Recurrent_Unit%2C_base_type.svg/780px-Gated_Recurrent_Unit%2C_base_type.svg.png)\n","\n","- GRU has rather complicated structure compared to vanilla RNN (see below figure), but in terms of implementing it with Pytorch, largely similar to RNN, using ```torch.nn.GRU```"]},{"metadata":{"id":"Irn1yF5lb3HO","colab_type":"code","colab":{}},"cell_type":"code","source":["gru = nn.GRU(input_size = 10, \n"," hidden_size = 5, \n"," num_layers = 1)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"ddO_D_L2b81J","colab_type":"code","outputId":"ec9c2336-aace-441d-9acb-42506c482c65","executionInfo":{"status":"ok","timestamp":1544640247042,"user_tz":420,"elapsed":1663,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["## inputs to GRU\n","# input data (seq_len, batch_size, input_size)\n","x0 = torch.from_numpy(np.random.randn(12, 64, 10)).float() \n","# hidden state (num_layers * num_directions, batch_size, hidden_size)\n","h0 = torch.from_numpy(np.zeros((1, 64, 5))).float() \n","\n","print(x0.shape, h0.shape)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([12, 64, 10]) torch.Size([1, 64, 5])\n"],"name":"stdout"}]},{"metadata":{"id":"ukQSVR0yY2VC","colab_type":"code","outputId":"b98de175-9c9a-4297-ab94-e7808a25f1da","executionInfo":{"status":"ok","timestamp":1544640247498,"user_tz":420,"elapsed":1021,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["## outputs from GRU\n","# output (seq_len, batch_size, num_directions * hidden_size)\n","# hidden state (num_layers * num_directions, batch_size, hidden_size)\n","out, h1 = gru(x0, h0)\n","\n","print(out.shape, h1.shape)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([12, 64, 5]) torch.Size([1, 64, 5])\n"],"name":"stdout"}]},{"metadata":{"id":"MaM0ruFEtlQA","colab_type":"text"},"cell_type":"markdown","source":["## 3. Long Short Term Memory (LSTM)\n","\n","![](https://upload.wikimedia.org/wikipedia/commons/thumb/3/3b/The_LSTM_cell.png/1200px-The_LSTM_cell.png)\n","\n","- LSTM is another variant of vanilla RNN that is widely used. Though there exist some differences in structure, when implementing one just need to attend to the cell state (c_t)\n"," - Inputs to LSTM: (x0, (h0, c0)\n"," - ```x0```: tensor that contains features of the input sequence\n"," - shape: ```(seq_len, batch_size, input_size)```\n"," - ```h0```: tensor that contains initial hidden state\n"," - shape: ```(num_layers * num_directions, batch_size, hidden_size)```\n"," - ```c0```: tensor that contains initial cell state\n"," - shape: ```(num_layers * num_directions, batch_size, hidden_size)``` (same as h0)\n"," - Outputs to LSTM: (xn, (hn, cn))\n"," - ```xn```: tensor that contains output features from the last layer\n"," - shape: ```(seq_len, batch_size, num_directions * hidden_size)```\n"," - ```hn```: tensor containing the hidden state\n"," - shape: ```(num_layers * num_directions, batch_size, hidden_size)```\n"," - ```cn```: tensor containing the cell state\n"," - shape: ```(num_layers * num_directions, batch_size, hidden_size)```"]},{"metadata":{"id":"LSUeaLXatsK-","colab_type":"code","colab":{}},"cell_type":"code","source":["lstm = nn.LSTM(input_size = 10, \n"," hidden_size = 5, \n"," num_layers = 1)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"uOcAQRqSt826","colab_type":"code","outputId":"3bd8e411-0d5b-4ad1-9da5-daa7ba78a139","executionInfo":{"status":"ok","timestamp":1544647697084,"user_tz":420,"elapsed":588,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["## inputs to LSTM\n","# input data (seq_len, batch_size, input_size)\n","x0 = torch.from_numpy(np.random.randn(1, 64, 10)).float() \n","\n","print(x0.shape)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([1, 64, 10])\n"],"name":"stdout"}]},{"metadata":{"id":"iVEKLB3xLWAR","colab_type":"code","outputId":"00d14088-7077-4262-a068-832ad811493b","executionInfo":{"status":"ok","timestamp":1544647738377,"user_tz":420,"elapsed":923,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":52}},"cell_type":"code","source":["# outputs from LSTM\n","# when initial hidden & cell state are not given, they are regarded as zero\n","xn, (hn, cn) = lstm(x0)\n","\n","print(xn.shape) # (seq_len, batch_size, hidden_size)\n","print(hn.shape, cn.shape) # (num_layers, batch_size, hidden_size)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([1, 64, 5])\n","torch.Size([1, 64, 5]) torch.Size([1, 64, 5])\n"],"name":"stdout"}]},{"metadata":{"id":"uNiSC9sPLf_w","colab_type":"code","outputId":"1c8c0dc3-0892-44fe-ab7d-f625ac51d393","executionInfo":{"status":"ok","timestamp":1544647909318,"user_tz":420,"elapsed":1511,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":52}},"cell_type":"code","source":["# when initial hidden & cell states are given\n","x0 = torch.from_numpy(np.random.randn(1, 64, 10)).float() \n","h0, c0 = torch.from_numpy(np.zeros((1, 64, 5))).float(), torch.from_numpy(np.zeros((1, 64, 5))).float()\n","\n","xn, (hn, cn) = lstm(x0, (h0, c0))\n","\n","print(xn.shape) # (seq_len, batch_size, hidden_size)\n","print(hn.shape, cn.shape) # (num_layers, batch_size, hidden_size)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([1, 64, 5])\n","torch.Size([1, 64, 5]) torch.Size([1, 64, 5])\n"],"name":"stdout"}]},{"metadata":{"id":"7uiNzfCkuhHu","colab_type":"code","colab":{}},"cell_type":"code","source":["# stacked, bidirectional LSTM\n","lstm = nn.LSTM(input_size = 10, \n"," hidden_size = 5, \n"," num_layers = 2,\n"," bidirectional = True)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"3SLQllzOMhEN","colab_type":"code","outputId":"20d4d277-e0d0-42aa-c9bb-d3b1fdc6c9e6","executionInfo":{"status":"ok","timestamp":1544648053540,"user_tz":420,"elapsed":1543,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":52}},"cell_type":"code","source":["# inputs to LSTM\n","x0 = torch.from_numpy(np.random.randn(5, 64, 10)).float()\n","h0, c0 = torch.from_numpy(np.zeros((4, 64, 5))).float(), torch.from_numpy(np.zeros((4, 64, 5))).float()\n","\n","xn, (hn, cn) = lstm(x0, (h0, c0))\n","\n","print(xn.shape)\n","print(hn.shape, cn.shape)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([5, 64, 10])\n","torch.Size([4, 64, 5]) torch.Size([4, 64, 5])\n"],"name":"stdout"}]}]} -------------------------------------------------------------------------------- /PyTorch Basics/pytorch-datasets-2.ipynb: -------------------------------------------------------------------------------- 1 | {"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"pytorch-datasets-2.ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[],"toc_visible":true},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Training and evaluating machine learning models\n","- Train-test split\n","- k-fold Cross-Validation"]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":105},"outputId":"88348d80-9804-4c3b-d18e-951fecb2de4a","executionInfo":{"status":"ok","timestamp":1538721493611,"user_tz":420,"elapsed":3858,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (0.4.1)\n","Requirement already satisfied: torchvision in /usr/local/lib/python3.6/dist-packages (0.2.1)\n","Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n","Requirement already satisfied: pillow>=4.1.1 in /usr/local/lib/python3.6/dist-packages (from torchvision) (5.3.0)\n","Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n"],"name":"stdout"}]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"11f2cf42-843d-4af1-e16c-bc93f6fa91a0","executionInfo":{"status":"ok","timestamp":1538725184915,"user_tz":420,"elapsed":1016,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","torch.__version__"],"execution_count":2,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'0.4.1'"]},"metadata":{"tags":[]},"execution_count":2}]},{"metadata":{"id":"xz642SBhDGMb","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"0be980dc-ef58-47e2-bda2-90e1d151e8a6","executionInfo":{"status":"ok","timestamp":1538725211625,"user_tz":420,"elapsed":988,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# to use GPU\n","device = torch.device(\"cuda\")\n","device"],"execution_count":7,"outputs":[{"output_type":"execute_result","data":{"text/plain":["device(type='cuda')"]},"metadata":{"tags":[]},"execution_count":7}]},{"metadata":{"id":"WVZTmJ4CY8fM","colab_type":"text"},"cell_type":"markdown","source":["## 1. Train-test split\n","- Splitting train and test data in Pytorch"]},{"metadata":{"id":"7K9iC17g-N2E","colab_type":"text"},"cell_type":"markdown","source":["### Import data\n","- Import [epileptic seizure data](https://archive.ics.uci.edu/ml/datasets/Epileptic+Seizure+Recognition) from UCI ML repository\n","- Split train and test data using ```random_split()```\n","- Train logistic regression model with training data and evaluate results with test data"]},{"metadata":{"id":"fiu8hP6P_19B","colab_type":"code","colab":{}},"cell_type":"code","source":["class SeizureDataset(torch.utils.data.Dataset):\n"," def __init__(self):\n"," # import and initialize dataset\n"," df = pd.read_csv(\"https://archive.ics.uci.edu/ml/machine-learning-databases/00388/data.csv\")\n"," df = df[df.columns[1:]]\n"," \n"," self.X = df[df.columns[:-1]].values\n"," self.Y = df[\"y\"].astype(\"category\").cat.codes.values.astype(np.int32)\n"," \n"," def __getitem__(self, idx):\n"," # get item by index\n"," return self.X[idx], self.Y[idx]\n"," \n"," def __len__(self):\n"," # returns length of data\n"," return len(self.X)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"o7TZBFXm_seR","colab_type":"code","colab":{}},"cell_type":"code","source":["seizuredataset = SeizureDataset()"],"execution_count":0,"outputs":[]},{"metadata":{"id":"qTJSFTAkBO-u","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"7fd4f527-f5d7-4ea6-eb47-865f0cea676d","executionInfo":{"status":"ok","timestamp":1538723895957,"user_tz":420,"elapsed":811,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["NUM_INSTANCES = len(seizuredataset)\n","TEST_RATIO = 0.3\n","TEST_SIZE = int(NUM_INSTANCES * 0.3)\n","TRAIN_SIZE = NUM_INSTANCES - TEST_SIZE\n","\n","print(NUM_INSTANCES, TRAIN_SIZE, TEST_SIZE)"],"execution_count":7,"outputs":[{"output_type":"stream","text":["11500 8050 3450\n"],"name":"stdout"}]},{"metadata":{"id":"wJX5td0owoB-","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"b8e5c90d-b4d2-4c13-e1d9-4586272d80c1","executionInfo":{"status":"ok","timestamp":1538723898137,"user_tz":420,"elapsed":917,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["train_data, test_data = torch.utils.data.random_split(seizuredataset, (TRAIN_SIZE, TEST_SIZE))\n","\n","print(len(train_data), len(test_data))"],"execution_count":8,"outputs":[{"output_type":"stream","text":["8050 3450\n"],"name":"stdout"}]},{"metadata":{"id":"mh4YPbf2Cm41","colab_type":"code","colab":{}},"cell_type":"code","source":["# when splitting train and test sets, data loader for each dataset should be made separately\n","train_loader = torch.utils.data.DataLoader(train_data, batch_size = 64, shuffle = True)\n","test_loader = torch.utils.data.DataLoader(test_data, batch_size = 64, shuffle = False)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"x-ICPT4BB6Sd","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"6ccf642b-6605-4fc5-d03f-895c9e6cba1b","executionInfo":{"status":"ok","timestamp":1538725216613,"user_tz":420,"elapsed":3557,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# logistic regression model\n","model = torch.nn.Linear(178, 5).to(device)\n","criterion = torch.nn.CrossEntropyLoss() \n","optimizer = torch.optim.Adam(model.parameters(), lr=1e-2) \n","model"],"execution_count":8,"outputs":[{"output_type":"execute_result","data":{"text/plain":["Linear(in_features=178, out_features=5, bias=True)"]},"metadata":{"tags":[]},"execution_count":8}]},{"metadata":{"id":"bpgjzV4HCX-F","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":193},"outputId":"bf5f96fd-c447-4c76-87c3-92aa17dec2f1","executionInfo":{"status":"ok","timestamp":1538724003097,"user_tz":420,"elapsed":30707,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["num_step = len(train_loader)\n","\n","for epoch in range(100):\n"," for i, (x, y) in enumerate(train_loader):\n"," x, y = x.float().to(device), y.long().to(device)\n"," outputs = model(x)\n"," \n"," loss = criterion(outputs, y)\n"," optimizer.zero_grad()\n"," loss.backward()\n"," optimizer.step()\n","\n"," if (epoch + 1) % 10 == 0:\n"," print(\"Epoch: {}, Loss: {:.5f}\".format(epoch + 1, loss.item()))"],"execution_count":15,"outputs":[{"output_type":"stream","text":["Epoch: 10, Loss: 40.67493\n","Epoch: 20, Loss: 65.14543\n","Epoch: 30, Loss: 49.11377\n","Epoch: 40, Loss: 34.16365\n","Epoch: 50, Loss: 46.56900\n","Epoch: 60, Loss: 33.15329\n","Epoch: 70, Loss: 32.82365\n","Epoch: 80, Loss: 39.61089\n","Epoch: 90, Loss: 30.58812\n","Epoch: 100, Loss: 46.22044\n"],"name":"stdout"}]},{"metadata":{"id":"XC1vyxwLHE2y","colab_type":"code","colab":{}},"cell_type":"code","source":["y_true, y_pred, y_prob = [], [], []\n","with torch.no_grad():\n"," for x, y in test_loader:\n"," # ground truth\n"," y = list(y.numpy())\n"," y_true += y\n"," \n"," x = x.float().to(device)\n"," outputs = model(x)\n","\n"," # predicted label\n"," _, predicted = torch.max(outputs.data, 1)\n"," predicted = list(predicted.cpu().numpy())\n"," y_pred += predicted\n"," \n"," # probability for each label\n"," prob = list(outputs.cpu().numpy())\n"," y_prob += prob"],"execution_count":0,"outputs":[]},{"metadata":{"id":"4rvewoaPIfd2","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"7e1731b1-b7ef-4d4c-a76e-d3b32c6630c4","executionInfo":{"status":"ok","timestamp":1538724699196,"user_tz":420,"elapsed":1039,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# calculating overall accuracy\n","num_correct = 0\n","\n","for i in range(len(y_true)):\n"," if y_true[i] == y_pred[i]:\n"," num_correct += 1\n","\n","print(\"Accuracy: \", num_correct/len(y_true))"],"execution_count":28,"outputs":[{"output_type":"stream","text":["Accuracy: 0.22\n"],"name":"stdout"}]},{"metadata":{"id":"Tgf96It4fhr9","colab_type":"text"},"cell_type":"markdown","source":["## 2. k-fold Cross-Validation\n","- Perform k-fold cross validation in Pytorch\n","- Cross validation can be implemented using NumPy, but we rely on ```skorch``` and ```sklearn``` here for the facility of implementation"]},{"metadata":{"id":"SFXadkSTeN4r","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":318},"outputId":"3a825712-e5cc-4d25-8703-abcb02640d93","executionInfo":{"status":"ok","timestamp":1538724154847,"user_tz":420,"elapsed":5891,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["!pip install -U skorch"],"execution_count":16,"outputs":[{"output_type":"stream","text":["Collecting skorch\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/49/61/d0949b994b8e1faa7c0218c45e94034d6ebf1c4fd87e99663eddfe761e95/skorch-0.3.0-py3-none-any.whl (89kB)\n","\u001b[K 100% |████████████████████████████████| 92kB 4.5MB/s \n","\u001b[?25hRequirement already satisfied, skipping upgrade: scipy in /usr/local/lib/python3.6/dist-packages (from skorch) (0.19.1)\n","Collecting tabulate (from skorch)\n","\u001b[?25l Downloading https://files.pythonhosted.org/packages/12/c2/11d6845db5edf1295bc08b2f488cf5937806586afe42936c3f34c097ebdc/tabulate-0.8.2.tar.gz (45kB)\n","\u001b[K 100% |████████████████████████████████| 51kB 5.3MB/s \n","\u001b[?25hRequirement already satisfied, skipping upgrade: scikit-learn>=0.18 in /usr/local/lib/python3.6/dist-packages (from skorch) (0.19.2)\n","Requirement already satisfied, skipping upgrade: tqdm in /usr/local/lib/python3.6/dist-packages (from skorch) (4.26.0)\n","Requirement already satisfied, skipping upgrade: numpy in /usr/local/lib/python3.6/dist-packages (from skorch) (1.14.6)\n","Building wheels for collected packages: tabulate\n"," Running setup.py bdist_wheel for tabulate ... \u001b[?25l-\b \b\\\b \bdone\n","\u001b[?25h Stored in directory: /root/.cache/pip/wheels/2a/85/33/2f6da85d5f10614cbe5a625eab3b3aebfdf43e7b857f25f829\n","Successfully built tabulate\n","Installing collected packages: tabulate, skorch\n","Successfully installed skorch-0.3.0 tabulate-0.8.2\n"],"name":"stdout"}]},{"metadata":{"id":"Iv30GdmFg1om","colab_type":"code","colab":{}},"cell_type":"code","source":["from skorch import NeuralNetClassifier\n","from sklearn.model_selection import cross_val_score"],"execution_count":0,"outputs":[]},{"metadata":{"id":"znOfjXfFJKr7","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"42adbfc4-fec6-4787-8245-6b5a9f892f94","executionInfo":{"status":"ok","timestamp":1538725195030,"user_tz":420,"elapsed":2425,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# import data\n","df = pd.read_csv(\"https://archive.ics.uci.edu/ml/machine-learning-databases/00388/data.csv\")\n","df = df[df.columns[1:]]\n","\n","X_data = df[df.columns[:-1]].values.astype(np.float32)\n","y_data = df[\"y\"].astype(\"category\").cat.codes.values.astype(np.int64)\n","\n","print(X_data.shape, y_data.shape)"],"execution_count":4,"outputs":[{"output_type":"stream","text":["(11500, 178) (11500,)\n"],"name":"stdout"}]},{"metadata":{"id":"m56oNfIDg37o","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":1073},"outputId":"fe05ce9a-de49-47a3-9022-86d7884b6be3","executionInfo":{"status":"ok","timestamp":1538725227609,"user_tz":420,"elapsed":7665,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# generate skorch high-level classifier and perform 5-fold cross validation using cross_val_score()\n","logistic = NeuralNetClassifier(model, max_epochs = 10, lr = 1e-2)\n","scores = cross_val_score(logistic, X_data, y_data, cv = 5, scoring = \"accuracy\")"],"execution_count":9,"outputs":[{"output_type":"stream","text":[" epoch train_loss valid_acc valid_loss dur\n","------- ------------ ----------- ------------ ------\n"," 1 nan \u001b[32m0.1933\u001b[0m nan 0.1225\n"," 2 nan 0.1846 nan 0.1125\n"," 3 nan 0.1857 nan 0.1121\n"," 4 nan 0.1906 nan 0.1101\n"," 5 nan 0.1851 nan 0.1093\n"," 6 nan 0.1873 nan 0.1124\n"," 7 nan 0.1851 nan 0.1141\n"," 8 nan 0.1884 nan 0.1102\n"," 9 nan 0.1878 nan 0.1104\n"," 10 nan 0.1878 nan 0.1118\n"," epoch train_loss valid_acc valid_loss dur\n","------- ------------ ----------- ------------ ------\n"," 1 nan \u001b[32m0.2205\u001b[0m nan 0.1072\n"," 2 nan 0.2184 nan 0.1111\n"," 3 nan 0.2048 nan 0.1048\n"," 4 nan 0.2108 nan 0.1094\n"," 5 nan 0.2113 nan 0.1133\n"," 6 nan 0.2086 nan 0.1113\n"," 7 nan 0.1912 nan 0.1124\n"," 8 nan 0.1890 nan 0.1108\n"," 9 nan 0.1912 nan 0.1105\n"," 10 nan 0.1912 nan 0.1141\n"," epoch train_loss valid_acc valid_loss dur\n","------- ------------ ----------- ------------ ------\n"," 1 nan \u001b[32m0.2029\u001b[0m nan 0.1093\n"," 2 nan \u001b[32m0.2138\u001b[0m nan 0.1077\n"," 3 nan \u001b[32m0.2149\u001b[0m nan 0.1109\n"," 4 nan 0.2122 nan 0.1150\n"," 5 nan 0.2105 nan 0.1127\n"," 6 nan 0.2127 nan 0.1097\n"," 7 nan \u001b[32m0.2160\u001b[0m nan 0.1093\n"," 8 nan 0.2122 nan 0.1128\n"," 9 nan 0.2094 nan 0.1140\n"," 10 nan 0.2084 nan 0.1049\n"," epoch train_loss valid_acc valid_loss dur\n","------- ------------ ----------- ------------ ------\n"," 1 nan \u001b[32m0.2108\u001b[0m nan 0.1086\n"," 2 nan \u001b[32m0.2113\u001b[0m nan 0.1136\n"," 3 nan 0.2091 nan 0.1124\n"," 4 nan 0.2113 nan 0.1117\n"," 5 nan 0.2053 nan 0.1115\n"," 6 nan \u001b[32m0.2118\u001b[0m nan 0.1111\n"," 7 nan 0.2108 nan 0.1128\n"," 8 nan 0.2113 nan 0.1105\n"," 9 nan 0.2097 nan 0.1177\n"," 10 nan 0.2118 nan 0.1110\n"," epoch train_loss valid_acc valid_loss dur\n","------- ------------ ----------- ------------ ------\n"," 1 nan \u001b[32m0.2090\u001b[0m nan 0.1078\n"," 2 nan \u001b[32m0.2161\u001b[0m nan 0.1111\n"," 3 nan 0.2079 nan 0.1118\n"," 4 nan \u001b[32m0.2172\u001b[0m nan 0.1144\n"," 5 nan \u001b[32m0.2242\u001b[0m nan 0.1120\n"," 6 nan 0.2128 nan 0.1104\n"," 7 nan 0.2172 nan 0.1126\n"," 8 nan \u001b[32m0.2280\u001b[0m nan 0.1237\n"," 9 nan \u001b[32m0.2291\u001b[0m nan 0.1109\n"," 10 nan \u001b[32m0.2329\u001b[0m nan 0.1133\n"],"name":"stdout"}]},{"metadata":{"id":"VIfIswG4o6Mt","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":52},"outputId":"f5eea279-06b3-45be-99fd-2ca6ea2253b1","executionInfo":{"status":"ok","timestamp":1538725338008,"user_tz":420,"elapsed":981,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# print out results\n","print(scores)\n","print(scores.mean(), scores.std())"],"execution_count":12,"outputs":[{"output_type":"stream","text":["[0.19173913 0.1973913 0.21347826 0.20869565 0.23521739]\n","0.20930434782608698 0.01509791607948889\n"],"name":"stdout"}]}]} -------------------------------------------------------------------------------- /Deep Learning with PyTorch/DL_with_pytorch_11_[Multi_Inputs].ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "DL-with-pytorch - 11 [Multi-Inputs].ipynb", 7 | "provenance": [], 8 | "collapsed_sections": [] 9 | }, 10 | "kernelspec": { 11 | "name": "python3", 12 | "display_name": "Python 3" 13 | }, 14 | "accelerator": "GPU" 15 | }, 16 | "cells": [ 17 | { 18 | "cell_type": "markdown", 19 | "metadata": { 20 | "id": "bIAYbYajk1w9", 21 | "colab_type": "text" 22 | }, 23 | "source": [ 24 | "# Multi Input networks\n", 25 | "- In many cases, the problem and dataset of interest requires multiple inputs to a prediction model\n", 26 | "- In this tutorial, we learn how to implement and train multi-input networks with the Quora question retrieval dataset" 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "metadata": { 32 | "id": "8yy37hEYOEiQ", 33 | "colab_type": "code", 34 | "outputId": "92b84ac0-e171-4c4b-a7e8-c072e7a7dc9e", 35 | "colab": { 36 | "base_uri": "https://localhost:8080/", 37 | "height": 35 38 | } 39 | }, 40 | "source": [ 41 | "import numpy as np\n", 42 | "import pandas as pd\n", 43 | "import torch, torchvision\n", 44 | "import torch.nn as nn\n", 45 | "import torch.nn.functional as F\n", 46 | "torch.__version__" 47 | ], 48 | "execution_count": 3, 49 | "outputs": [ 50 | { 51 | "output_type": "execute_result", 52 | "data": { 53 | "text/plain": [ 54 | "'1.3.1'" 55 | ] 56 | }, 57 | "metadata": { 58 | "tags": [] 59 | }, 60 | "execution_count": 3 61 | } 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": { 67 | "id": "ewrw93tt2BfV", 68 | "colab_type": "text" 69 | }, 70 | "source": [ 71 | "## 1. Import & process dataset\n", 72 | "- Quora question retrieval dataset\n", 73 | "- Find out whether the two questions are equivalent or not (i.e., duplicate)\n", 74 | "- Source: http://qim.fs.quoracdn.net/quora_duplicate_questions.tsv" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "metadata": { 80 | "id": "SPjXQWMHZyov", 81 | "colab_type": "code", 82 | "colab": { 83 | "base_uri": "https://localhost:8080/", 84 | "height": 251 85 | }, 86 | "outputId": "68ae7380-cb08-425e-86b7-75075afae152" 87 | }, 88 | "source": [ 89 | "data = pd.read_csv(\"http://qim.fs.quoracdn.net/quora_duplicate_questions.tsv\", sep = \"\\t\")\n", 90 | "print(data.shape)\n", 91 | "data.head()" 92 | ], 93 | "execution_count": 6, 94 | "outputs": [ 95 | { 96 | "output_type": "stream", 97 | "text": [ 98 | "(404290, 6)\n" 99 | ], 100 | "name": "stdout" 101 | }, 102 | { 103 | "output_type": "execute_result", 104 | "data": { 105 | "text/html": [ 106 | "
\n", 107 | "\n", 120 | "\n", 121 | " \n", 122 | " \n", 123 | " \n", 124 | " \n", 125 | " \n", 126 | " \n", 127 | " \n", 128 | " \n", 129 | " \n", 130 | " \n", 131 | " \n", 132 | " \n", 133 | " \n", 134 | " \n", 135 | " \n", 136 | " \n", 137 | " \n", 138 | " \n", 139 | " \n", 140 | " \n", 141 | " \n", 142 | " \n", 143 | " \n", 144 | " \n", 145 | " \n", 146 | " \n", 147 | " \n", 148 | " \n", 149 | " \n", 150 | " \n", 151 | " \n", 152 | " \n", 153 | " \n", 154 | " \n", 155 | " \n", 156 | " \n", 157 | " \n", 158 | " \n", 159 | " \n", 160 | " \n", 161 | " \n", 162 | " \n", 163 | " \n", 164 | " \n", 165 | " \n", 166 | " \n", 167 | " \n", 168 | " \n", 169 | " \n", 170 | " \n", 171 | " \n", 172 | " \n", 173 | " \n", 174 | " \n", 175 | " \n", 176 | " \n", 177 | " \n", 178 | " \n", 179 | "
idqid1qid2question1question2is_duplicate
0012What is the step by step guide to invest in sh...What is the step by step guide to invest in sh...0
1134What is the story of Kohinoor (Koh-i-Noor) Dia...What would happen if the Indian government sto...0
2256How can I increase the speed of my internet co...How can Internet speed be increased by hacking...0
3378Why am I mentally very lonely? How can I solve...Find the remainder when [math]23^{24}[/math] i...0
44910Which one dissolve in water quikly sugar, salt...Which fish would survive in salt water?0
\n", 180 | "
" 181 | ], 182 | "text/plain": [ 183 | " id qid1 ... question2 is_duplicate\n", 184 | "0 0 1 ... What is the step by step guide to invest in sh... 0\n", 185 | "1 1 3 ... What would happen if the Indian government sto... 0\n", 186 | "2 2 5 ... How can Internet speed be increased by hacking... 0\n", 187 | "3 3 7 ... Find the remainder when [math]23^{24}[/math] i... 0\n", 188 | "4 4 9 ... Which fish would survive in salt water? 0\n", 189 | "\n", 190 | "[5 rows x 6 columns]" 191 | ] 192 | }, 193 | "metadata": { 194 | "tags": [] 195 | }, 196 | "execution_count": 6 197 | } 198 | ] 199 | }, 200 | { 201 | "cell_type": "code", 202 | "metadata": { 203 | "id": "SxDHXFEsf5em", 204 | "colab_type": "code", 205 | "colab": {} 206 | }, 207 | "source": [ 208 | "# we use only 10000 first instances and 10 tokens for each question for faster training in this tutorial\n", 209 | "# you can try learning with all instances yourself and compare the results!\n", 210 | "num_instances, max_length = 10000, 10\n", 211 | "\n", 212 | "data = data.iloc[:num_instances]\n", 213 | "q1 = [str(x).split() for x in data[\"question1\"]]\n", 214 | "q2 = [str(x).split() for x in data[\"question2\"]]" 215 | ], 216 | "execution_count": 0, 217 | "outputs": [] 218 | }, 219 | { 220 | "cell_type": "code", 221 | "metadata": { 222 | "id": "fJ8M3h4bb6AT", 223 | "colab_type": "code", 224 | "colab": {} 225 | }, 226 | "source": [ 227 | "# gather unique tokens and convert into list for indexing\n", 228 | "unique_tokens = set()\n", 229 | "for i in range(len(q1)):\n", 230 | " unique_tokens.update(q1[i])\n", 231 | " unique_tokens.update(q2[i])\n", 232 | "unique_tokens = list(unique_tokens)" 233 | ], 234 | "execution_count": 0, 235 | "outputs": [] 236 | }, 237 | { 238 | "cell_type": "code", 239 | "metadata": { 240 | "id": "GGbC_Hmib7BE", 241 | "colab_type": "code", 242 | "colab": {} 243 | }, 244 | "source": [ 245 | "# create X, y data\n", 246 | "# note that there are two X data (X_data_1, X_data_2)!\n", 247 | "y_data = data[\"is_duplicate\"].values\n", 248 | "X_data_1, X_data_2 = np.zeros((num_instances, max_length)), np.zeros((num_instances, max_length))\n", 249 | "\n", 250 | "for i in range(len(q1)):\n", 251 | " if len(q1[i]) < max_length:\n", 252 | " for j in range(len(q1[i])):\n", 253 | " X_data_1[i][j] = unique_tokens.index(q1[i][j])\n", 254 | " else:\n", 255 | " for j in range(max_length):\n", 256 | " X_data_1[i][j] = unique_tokens.index(q1[i][j])\n", 257 | "\n", 258 | "for i in range(len(q2)):\n", 259 | " if len(q2[i]) < max_length:\n", 260 | " for j in range(len(q2[i])):\n", 261 | " X_data_2[i][j] = unique_tokens.index(q2[i][j])\n", 262 | " else:\n", 263 | " for j in range(max_length):\n", 264 | " X_data_2[i][j] = unique_tokens.index(q2[i][j])" 265 | ], 266 | "execution_count": 0, 267 | "outputs": [] 268 | }, 269 | { 270 | "cell_type": "code", 271 | "metadata": { 272 | "id": "6KV4rDQ0c1mm", 273 | "colab_type": "code", 274 | "colab": {} 275 | }, 276 | "source": [ 277 | "# generate dataset and data loader instances\n", 278 | "# we use SubsetRandomSampler to sample test instances here\n", 279 | "from torch.utils.data.sampler import SubsetRandomSampler\n", 280 | "\n", 281 | "class QuoraDataset(torch.utils.data.Dataset):\n", 282 | " def __init__(self):\n", 283 | " self.x1 = X_data_1\n", 284 | " self.x2 = X_data_2\n", 285 | " self.y = y_data.astype(\"int64\")\n", 286 | " \n", 287 | " def __getitem__(self, idx):\n", 288 | " return self.x1[idx], self.x2[idx], self.y[idx]\n", 289 | " \n", 290 | " def __len__(self):\n", 291 | " return len(self.x1)\n", 292 | "\n", 293 | "dataset = QuoraDataset()\n", 294 | "NUM_INSTANCES = len(dataset)\n", 295 | "TEST_RATIO = 0.3\n", 296 | "TEST_SIZE = int(NUM_INSTANCES * 0.3)\n", 297 | "\n", 298 | "indices = list(range(NUM_INSTANCES))\n", 299 | "\n", 300 | "test_idx = np.random.choice(indices, size = TEST_SIZE, replace = False)\n", 301 | "train_idx = list(set(indices) - set(test_idx))\n", 302 | "train_sampler, test_sampler = SubsetRandomSampler(train_idx), SubsetRandomSampler(test_idx)\n", 303 | "\n", 304 | "train_loader = torch.utils.data.DataLoader(dataset, batch_size = 128, sampler = train_sampler)\n", 305 | "test_loader = torch.utils.data.DataLoader(dataset, batch_size = 128, sampler = test_sampler)" 306 | ], 307 | "execution_count": 0, 308 | "outputs": [] 309 | }, 310 | { 311 | "cell_type": "markdown", 312 | "metadata": { 313 | "id": "9nznVMdo5edZ", 314 | "colab_type": "text" 315 | }, 316 | "source": [ 317 | "## 2. Creating Multi Input Network\n", 318 | "\n", 319 | "- Create and train Multi Input Network for question retrieval" 320 | ] 321 | }, 322 | { 323 | "cell_type": "code", 324 | "metadata": { 325 | "id": "AQawpMRPI7jm", 326 | "colab_type": "code", 327 | "colab": {} 328 | }, 329 | "source": [ 330 | "# create CNN with one convolution/pooling layer\n", 331 | "class net(nn.Module):\n", 332 | " def __init__(self, input_dim, embedding_dim, num_words, num_hidden_cells):\n", 333 | " super(net, self).__init__()\n", 334 | " self.embedding = nn.Embedding(num_words, embedding_dim)\n", 335 | " self.dense_1 = nn.Linear(embedding_dim * input_dim, num_hidden_cells)\n", 336 | " self.dense_2 = nn.Linear(embedding_dim * input_dim, num_hidden_cells)\n", 337 | " self.final_dense = nn.Linear(num_hidden_cells * 2, 2) \n", 338 | " \n", 339 | " def forward(self, x1, x2):\n", 340 | " x1, x2 = self.embedding(x1), self.embedding(x2)\n", 341 | " x1, x2 = x1.view(x1.size(0), -1), x2.view(x2.size(0), -1)\n", 342 | " x1, x2 = self.dense_1(x1), self.dense_2(x2)\n", 343 | " x = torch.cat((x1, x2), dim = 1)\n", 344 | " x = self.final_dense(x)\n", 345 | " return x" 346 | ], 347 | "execution_count": 0, 348 | "outputs": [] 349 | }, 350 | { 351 | "cell_type": "code", 352 | "metadata": { 353 | "id": "rP0Gt5E9ajmd", 354 | "colab_type": "code", 355 | "colab": {} 356 | }, 357 | "source": [ 358 | "# hyperparameters\n", 359 | "INPUT_DIM = max_length\n", 360 | "EMBEDDING_DIM = 50\n", 361 | "NUM_WORDS = len(unique_tokens)\n", 362 | "HIDDEN_SIZE = 30\n", 363 | "LEARNING_RATE = 1e-2\n", 364 | "NUM_EPOCHS = 10" 365 | ], 366 | "execution_count": 0, 367 | "outputs": [] 368 | }, 369 | { 370 | "cell_type": "code", 371 | "metadata": { 372 | "id": "cPBm8qDrSWsi", 373 | "colab_type": "code", 374 | "colab": {} 375 | }, 376 | "source": [ 377 | "model = net(INPUT_DIM, EMBEDDING_DIM, NUM_WORDS, HIDDEN_SIZE)\n", 378 | "criterion = nn.CrossEntropyLoss() # do not need softmax layer when using CEloss criterion\n", 379 | "optimizer = torch.optim.Adam(model.parameters(), lr = LEARNING_RATE)" 380 | ], 381 | "execution_count": 0, 382 | "outputs": [] 383 | }, 384 | { 385 | "cell_type": "code", 386 | "metadata": { 387 | "id": "SEBtAPYCFeic", 388 | "colab_type": "code", 389 | "outputId": "6509c82d-c8a0-4f91-913c-d02c543d59b5", 390 | "colab": { 391 | "base_uri": "https://localhost:8080/", 392 | "height": 199 393 | } 394 | }, 395 | "source": [ 396 | "# training for NUM_EPOCHS\n", 397 | "for i in range(NUM_EPOCHS):\n", 398 | " temp_loss = []\n", 399 | " for x1, x2, y in train_loader:\n", 400 | " x1, x2 = x1.long(), x2.long()\n", 401 | " outputs = model(x1, x2)\n", 402 | " loss = criterion(outputs, y)\n", 403 | " temp_loss.append(loss.item())\n", 404 | " \n", 405 | " optimizer.zero_grad()\n", 406 | " loss.backward()\n", 407 | " optimizer.step()\n", 408 | " \n", 409 | " print(\"Loss at {}th epoch: {}\".format(i, np.mean(temp_loss)))" 410 | ], 411 | "execution_count": 39, 412 | "outputs": [ 413 | { 414 | "output_type": "stream", 415 | "text": [ 416 | "Loss at 0th epoch: 8.358021567057056e-05\n", 417 | "Loss at 1th epoch: 6.927385127875658e-05\n", 418 | "Loss at 2th epoch: 5.8683096689161506e-05\n", 419 | "Loss at 3th epoch: 5.074793185816485e-05\n", 420 | "Loss at 4th epoch: 4.445249596756185e-05\n", 421 | "Loss at 5th epoch: 3.919280668262879e-05\n", 422 | "Loss at 6th epoch: 3.4757104962905446e-05\n", 423 | "Loss at 7th epoch: 3.1304369787738486e-05\n", 424 | "Loss at 8th epoch: 2.82746555802243e-05\n", 425 | "Loss at 9th epoch: 2.5681897552038372e-05\n" 426 | ], 427 | "name": "stdout" 428 | } 429 | ] 430 | }, 431 | { 432 | "cell_type": "markdown", 433 | "metadata": { 434 | "id": "qpAJUiHm529m", 435 | "colab_type": "text" 436 | }, 437 | "source": [ 438 | "## 3. Evaluation\n", 439 | "- Evaluate the trained multi input model with accuracy score \n", 440 | " - Store probability of each instance to a list and compare it with true y label" 441 | ] 442 | }, 443 | { 444 | "cell_type": "code", 445 | "metadata": { 446 | "id": "txXH3dknFpSx", 447 | "colab_type": "code", 448 | "outputId": "2eb7e09f-cc17-405e-c95a-88357c3809fb", 449 | "colab": { 450 | "base_uri": "https://localhost:8080/", 451 | "height": 74 452 | } 453 | }, 454 | "source": [ 455 | "y_pred, y_true = [], []\n", 456 | "with torch.no_grad():\n", 457 | " for x1, x2, y in test_loader:\n", 458 | " x1, x2 = x1.long(), x2.long()\n", 459 | " outputs = F.softmax(model(x1, x2)).max(1)[-1] # predicted label\n", 460 | " y_true += list(y.numpy()) # true label\n", 461 | " y_pred += list(outputs.numpy()) " 462 | ], 463 | "execution_count": 40, 464 | "outputs": [ 465 | { 466 | "output_type": "stream", 467 | "text": [ 468 | "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n", 469 | " \"\"\"\n" 470 | ], 471 | "name": "stderr" 472 | } 473 | ] 474 | }, 475 | { 476 | "cell_type": "code", 477 | "metadata": { 478 | "id": "HV1s3xf5Frkl", 479 | "colab_type": "code", 480 | "outputId": "f335b304-4c50-4447-9483-476be56f784d", 481 | "colab": { 482 | "base_uri": "https://localhost:8080/", 483 | "height": 35 484 | } 485 | }, 486 | "source": [ 487 | "# evaluation result\n", 488 | "from sklearn.metrics import accuracy_score\n", 489 | "accuracy_score(y_true, y_pred)" 490 | ], 491 | "execution_count": 41, 492 | "outputs": [ 493 | { 494 | "output_type": "execute_result", 495 | "data": { 496 | "text/plain": [ 497 | "0.6016666666666667" 498 | ] 499 | }, 500 | "metadata": { 501 | "tags": [] 502 | }, 503 | "execution_count": 41 504 | } 505 | ] 506 | } 507 | ] 508 | } --------------------------------------------------------------------------------