├── AAE.ipynb ├── Basic and Linear Regression.ipynb ├── CNN and ResNet.ipynb ├── Custom dataset and Transfer learning.ipynb ├── GANS.ipynb ├── Initialize weight of parameter of model by yourself.ipynb ├── Logistic regression and Neural Network.ipynb ├── Practise Notebook_basic.ipynb └── README.md /Basic and Linear Regression.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 2, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import torch \n", 10 | "import torchvision\n", 11 | "import torch.nn as nn\n", 12 | "import numpy as np\n", 13 | "import torchvision.transforms as transforms\n", 14 | "import matplotlib.pyplot as plt" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": 3, 20 | "metadata": {}, 21 | "outputs": [ 22 | { 23 | "name": "stdout", 24 | "output_type": "stream", 25 | "text": [ 26 | "dx: tensor(2.)\n", 27 | "dw: tensor(1.)\n", 28 | "b: tensor(3.)\n", 29 | "x: tensor(1., requires_grad=True)\n", 30 | "None\n" 31 | ] 32 | } 33 | ], 34 | "source": [ 35 | "# Create tensors. As for learning we need to compute the gradient also, so we use tensors \n", 36 | "\n", 37 | "x = torch.tensor(1., requires_grad=True)\n", 38 | "w = torch.tensor(2., requires_grad=True)\n", 39 | "b = torch.tensor(3., requires_grad=False)\n", 40 | "\n", 41 | "# Build a computational graph.\n", 42 | "a = w*x\n", 43 | "y = a + b # y = 2 * x + 3\n", 44 | "\n", 45 | "# Compute gradients.\n", 46 | "y.backward()\n", 47 | "\n", 48 | "# Print out the gradients.\n", 49 | "print(\"dx: \",x.grad) # x.grad = 2 \n", 50 | "print(\"dw: \",w.grad) # w.grad = 1 \n", 51 | "print(\"b: \",b) # b.grad = 1 \n", 52 | "print(\"x: \", x)\n", 53 | "print(b.grad)" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": 3, 59 | "metadata": {}, 60 | "outputs": [ 61 | { 62 | "name": "stdout", 63 | "output_type": "stream", 64 | "text": [ 65 | "W: Parameter containing:\n", 66 | "tensor([[ 0.3043, -0.5016, -0.2386],\n", 67 | " [ 0.0296, 0.1085, -0.1317]], requires_grad=True)\n", 68 | "b: Parameter containing:\n", 69 | "tensor([0.5423, 0.0027], requires_grad=True)\n" 70 | ] 71 | } 72 | ], 73 | "source": [ 74 | "input = torch.randn(10, 3)#Note:10 is the number of examples and 3 is no. of feature \n", 75 | "output = torch.randn(10,2)#no. of output elements is 2(for an example), total examples 10\n", 76 | "\n", 77 | "c = nn.Linear(3,2)#takes 3 no. of input feature and gives 2 output feature\n", 78 | "print(\"W:\",c.weight)\n", 79 | "print(\"b:\", c.bias)" 80 | ] 81 | }, 82 | { 83 | "cell_type": "code", 84 | "execution_count": 4, 85 | "metadata": {}, 86 | "outputs": [ 87 | { 88 | "name": "stdout", 89 | "output_type": "stream", 90 | "text": [ 91 | "The value of a is: tensor([[-0.8575, 1.3380],\n", 92 | " [ 0.0211, -0.2638]])\n", 93 | "The value of c is: [[-0.85748863 1.3379784 ]\n", 94 | " [ 0.02113914 -0.2638441 ]]\n", 95 | "The value of d is: tensor([[-0.8575, 1.3380],\n", 96 | " [ 0.0211, -0.2638]])\n" 97 | ] 98 | } 99 | ], 100 | "source": [ 101 | "#torch to numpy and Vice versa\n", 102 | "a = torch.randn(2,2)\n", 103 | "c = a.numpy()\n", 104 | "d = torch.from_numpy(c)\n", 105 | "\n", 106 | "print(\"The value of a is: \",a)\n", 107 | "print(\"The value of c is: \",c)\n", 108 | "print(\"The value of d is: \",d)" 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": 5, 114 | "metadata": {}, 115 | "outputs": [ 116 | { 117 | "name": "stdout", 118 | "output_type": "stream", 119 | "text": [ 120 | "The shape of X is: (15, 1)\n", 121 | "Epoch [10/50], Loss: 4.9680\n", 122 | "Epoch [20/50], Loss: 0.9578\n", 123 | "Epoch [30/50], Loss: 0.2997\n", 124 | "Epoch [40/50], Loss: 0.1916\n", 125 | "Epoch [50/50], Loss: 0.1739\n" 126 | ] 127 | } 128 | ], 129 | "source": [ 130 | "#Linear Regression \n", 131 | "\n", 132 | "input_size = 1\n", 133 | "output_size = 1\n", 134 | "num_epochs = 50\n", 135 | "learning_rate = 0.001\n", 136 | "\n", 137 | "#sample dataset\n", 138 | "x_train = np.array([[3.3], [4.4], [5.5], [6.71], [6.93], [4.168], \n", 139 | " [9.779], [6.182], [7.59], [2.167], [7.042], \n", 140 | " [10.791], [5.313], [7.997], [3.1]], dtype=np.float32)\n", 141 | "\n", 142 | "y_train = np.array([[1.7], [2.76], [2.09], [3.19], [1.694], [1.573], \n", 143 | " [3.366], [2.596], [2.53], [1.221], [2.827], \n", 144 | " [3.465], [1.65], [2.904], [1.3]], dtype=np.float32)\n", 145 | "print(\"The shape of X is: \", x_train.shape)\n", 146 | "\n", 147 | "model = nn.Linear(input_size, output_size)\n", 148 | "\n", 149 | "criterion = nn.MSELoss()#used to set the Loss. Here we have set Mean Square Error Loss\n", 150 | "optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate)#this is used to set the optimizer you use to update the params\n", 151 | "\n", 152 | "for epoch in range(num_epochs):\n", 153 | " inputs = torch.from_numpy(x_train)\n", 154 | " labels = torch.from_numpy(y_train)\n", 155 | " outputs = model(inputs)#input is passed to the data\n", 156 | " loss = criterion(outputs, labels)#loss is computed here. Above we have just set the type of loss\n", 157 | " optimizer.zero_grad()#It is always important set the gradient as zero before back propagating. \n", 158 | " loss.backward()#now all the gradients are computed with respect to loss. In graph if you see it computes the derivative of the parameter which is used to calculate the loss\n", 159 | " optimizer.step()#To update the parametes\n", 160 | " \n", 161 | " if (epoch+1) % 10 == 0:\n", 162 | " print ('Epoch [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, loss.item()))\n" 163 | ] 164 | }, 165 | { 166 | "cell_type": "code", 167 | "execution_count": 6, 168 | "metadata": {}, 169 | "outputs": [ 170 | { 171 | "data": { 172 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xt4VNXd9vHvSogJISgKalEMExGVc4AgYKpVkDPiEUXz2mJb8VS174MoEIoIhGL11fJULA8eirSpVlAQxbOCcpIakGNQERkQtIr4IIQIJLDePyaGzDAhM8nM7D2T+3NdXpO9sjP75yTc2fntNWsbay0iIpJYkpwuQEREIk/hLiKSgBTuIiIJSOEuIpKAFO4iIglI4S4ikoAU7iIiCUjhLiKSgBTuIiIJqIFTB27WrJn1eDxOHV5EJC6tWrXqO2vtqTXt51i4ezweioqKnDq8iEhcMsZsC2U/tWVERBKQwl1EJAEp3EVEEpBjPfdgysrK2LFjBwcOHHC6FAHS0tJo0aIFKSkpTpciImFyVbjv2LGDxo0b4/F4MMY4XU69Zq1l9+7d7Nixg6ysLKfLEZEwuaotc+DAAZo2bapgdwFjDE2bNtVfUSJxylXhDijYXUTfC5H45bpwFxFJVPsPlvPIm5/y1Z4fo34shXuAHTt2cMUVV9C6dWtatWrFPffcw6FDh4Lu+9VXX3HttdfW+JwDBw5kz549tapnwoQJPPLIIzXul5GRcdzP79mzhyeeeKJWNYhI3f3x9U20e+BNHl/0OUs//y7qx4vvcC8sBI8HkpJ8j4WFdXo6ay1XX301V155JZs3b+azzz6jpKSE/Pz8Y/YtLy/njDPOYO7cuTU+72uvvUaTJk3qVFtdKdxFnLFq2/d4Ri/kf97/AoDhF3q4LuesqB83fsO9sBBGjIBt28Ba3+OIEXUK+Pfee4+0tDRuvvlmAJKTk3nsscd45plnKC0tZdasWQwZMoRevXrRu3dvvF4v7du3B6C0tJTrrruOtm3bctVVV9G9e/fK5RU8Hg/fffcdXq+XNm3acMstt9CuXTv69u3Ljz/6/jx78skn6datG506deKaa66htLT0uLVu3bqVnj170qFDB8aNG1c5XlJSQu/evenSpQsdOnTg5ZdfBmD06NFs2bKF7OxsRo0aVe1+IhIZ+w+W02HCm1zz1xUApDZIYt2EvkwY0i4mx4/fcM/Ph8AALC31jdfSxo0b6dq1q9/YiSeeSGZmJp9//jkAq1evZu7cubz//vt++z3xxBOcfPLJFBcXM2nSJFatWhX0GJs3b+bOO+9k48aNNGnShBdffBGAq6++mo8++oi1a9fSpk0bnn766ePWes8993D77bezfv16mjdvXjmelpbGvHnzWL16NYsWLWLkyJFYa5k6dSqtWrVizZo1PPzww9XuJyJ1N+U1Xwtm34FyAJ4f0YNPJw/gxLTYvWekxnnuxpg04AMgtWL/udbaBwL2GQ48DOysGHrcWvtUZEsNsH17eOMR0qdPH0455ZRjxpcuXco999wDQPv27enYsWPQr8/KyiI7OxuArl274vV6AdiwYQPjxo1jz549lJSU0K9fv+PWsWzZsspfDDfddBP3338/4GstjR07lg8++ICkpCR27tzJN998c8zXV7ffz372s9BeCBE5RpH3e66dsaJy++ZcDw9cHpsz9UChvInpINDLWltijEkBlhpjXrfWfhiw37+stb+LfInVyMz0tWKCjddS27Ztj+mh7927l+3bt3POOeewevVqGjVqVOvnB0hNTa38ODk5ubItM3z4cObPn0+nTp2YNWsWixcvrvG5gk1VLCwsZNeuXaxatYqUlBQ8Hk/Queqh7iciNSs5WE6PKe9SctB3pt4wJZl/5/emcQzP1APV2JaxPiUVmykV/zn/93tBAaSn+4+lp/vGa6l3796UlpYye/ZsAA4fPszIkSMZPnw46YHHCpCbm8sLL7wAQHFxMevXrw/r2Pv27aN58+aUlZVRGMJ1g9zcXJ5//nkAv/1/+OEHTjvtNFJSUli0aBHbKn4BNm7cmH379tW4n4iEZ/KrxbR/4M3KYP/XiB5smtTf0WCHEHvuxphkY8wa4FvgbWvtyiC7XWOMWWeMmWuMCXop2BgzwhhTZIwp2rVrVx3KBvLyYOZMaNkSjPE9zpzpG68lYwzz5s1jzpw5tG7dmnPPPZe0tDSmTJlS49fecccd7Nq1i7Zt2zJu3DjatWvHSSedFPKxJ02aRPfu3cnNzeX888+vcf9p06Yxffp0OnTowM6dOyvH8/LyKCoqokOHDsyePbvyuZo2bUpubi7t27dn1KhR1e4nIqH5yOubBfPU0q0A/Do3C+/UQXQ/u2n1XxThGX7HY8K5iGaMaQLMA+6y1m6oMt4UKLHWHjTG3Apcb63tdbznysnJsYE369i0aRNt2rQJp37XOHz4MGVlZaSlpbFlyxYuu+wyPv30U0444QSnS6uTeP6eiETDvgNl9JjyLvsPHQYg/YRkVo4NoQXz0wy/qhNB0tPDPik1xqyy1ubUtF9YC4dZa/cYYxYB/YENVcZ3V9ntKeBP4TxvIigtLeXSSy+lrKwMay1PPPFE3Ae7iPib+EoxzyzbWrn9wq09uSDr2AkWQR1vhl8dOg7VCWW2zKlAWUWwNwT6AA8F7NPcWvt1xeYQYFPEK3W5xo0b67aBIglq5Re7uX7m0Tkkt1yURf6gtuE9SYxn+IVy5t4ceNYYk4yvR/+CtfZVY8xEoMhauwC42xgzBCgHvgeGR6VaEZEY2negjG4F73Cg7AgAjVMbsGJsbzJSa7FaehRm+B1PjRVaa9cBnYOMj6/y8RhgTGRLExFxzoOvbORvy7yV23Nu60k3T4gtmGAKCoL33Osww+94XHWzDhERp334xW6G1bUFE8xPffX8fF8rJjPTF+xR6LeDwl1EBIC9B8rImfwOh8p9LZgT0xqwfEwtWzDVycuLWpgHit+1ZaIkOTmZ7Ozsyv+8Xi9FRUXcfffdACxevJjly5dX7j9//nyKi4vDPk51S/T+NB7qcsIichwhziufsGAjHSe8VRnsL97ek3UT+kU22GMsfiuPkoYNG7JmzRq/MY/HQ06Ob1rp4sWLycjI4MILLwR84T548GDato3An21VhLqcsIhUI3Be+U8rx0Ll2fOKLbu54cmjLZhbLz6bMQMT430dOnMPweLFixk8eDBer5cZM2bw2GOPkZ2dzfvvv8+CBQsYNWoU2dnZbNmyhS1bttC/f3+6du3KRRddxCeffAJUv0RvdaouJzxr1iyuvvpq+vfvT+vWrbnvvvsq93vrrbfo2bMnXbp0YejQoZSUlFT3lCL1y3Hmle89UMa5+a9XBvtJDVPY+GC/hAl2cPGZ+4OvbKT4q70Rfc62Z5xY4wptP/74Y+WqjVlZWcybN6/ycx6Ph9tuu42MjAzuvfdeAIYMGcLgwYMrWyi9e/dmxowZtG7dmpUrV3LHHXfw3nvvVS7R+8tf/pLp06eHXfuaNWv4+OOPSU1N5bzzzuOuu+6iYcOGTJ48mXfeeYdGjRrx0EMP8eijjzJ+/Pian1Ak0VUzf/wP5w7k7xPeqtx+8fYL6dry5FhVFTOuDXenBGvLhKqkpITly5czdOjQyrGDBw8C1S/RG6revXtXrlXTtm1btm3bxp49eyguLiY3NxeAQ4cO0bNnz1rVLpJwAuaVL8/swI03/LFy+7ZftGL0gMRdU8m14e7UGsh1ceTIEZo0aVLtL4dgS/SGKnCp4PLycqy19OnTh+eee67WzyuSsCrmlf9w2ND57n9yJCkZgFOSj7Bk/AAaxfHF0lCo5x6mwKVzq26feOKJZGVlMWfOHMB3Q4y1a9cC1S/RWxc9evRg2bJllXeJ2r9/P5999llEnlsk7uXl4bnrBTr9/l+Vwf5Sq32sLrg84YMdFO5hu/zyy5k3bx7Z2dksWbKEYcOG8fDDD9O5c2e2bNlCYWEhTz/9NJ06daJdu3aV9yatboneujj11FOZNWsWN9xwAx07dqRnz56VF3BF6rNH3vwUz+iFldvNMlLxTh1El1uGOVhVbIW15G8kJdqSv4lK3xOJJ19+X8pFf1rkN1Y07jKaZaRW8xXxJypL/oqIuFXVM3WACZe3ZXhulkPVOE/hLiJx7aE3PuGvi7f4jXmnDnKoGvdwXbhba+s0q0Qix6mWnUgotu8u5eKH/Vswq8ZdRtMEasHUhavCPS0tjd27d9O0aVMFvMOstezevZu0tDSnSxE5RmALZuIV7fhlT48zxbiUq8K9RYsW7NixgzrfPFsiIi0tjRYtWjhdhkilP76+if95/wu/MbVggnNVuKekpJCVVX8vgIhIcNt27+cXDy/2G1v9hz6c0kj3Ka6O5rmLhCPEJWQlMqy1eEYv9Av2yVe2xzt1kIK9Bq46cxdxtRCWkJXIKVhYzJNLtvqNqQUTOle9iUnE1Tye4Dc4btkSvN5YV5Owtn63n0sfWew39vEf+nCyztQBvYlJJPKqWUK22nEJi7WWrDGv+Y1NuaoDN3bPdKii+KZwFwlVwBKyfuNSJxNfKeaZZUdbMMbA1j+qBVMXCneRUFUsIet3d5/0dN+41MoXu0ro9f/e9xtbM74PTdLVgqkrhbtIqH66aJqf72vFZGb6gl0XU8MWrAUz9eoODLtAfwVFisJdJBx5eQrzOpqwYCOzlnsrt1OSDZsLBjpXUIJSuItITGzZVULvgBbM2vF9OSk9xaGKEpvCXUSiKlgL5k/XdOS6bmc5VFH9oHAXkagZ//IGZq84OsMoLSWJTyYNcLCi+kPhLiIR9/m3+7js0Q/8xtY+0JeTGqoFEysKdxGJmGAtmEeGduLarlpdNNYU7iISEePmr+cfHx59t26jE5LZOLG/gxXVbwp3EamTzd/so89jasG4TY3hboxJAz4AUiv2n2utfSBgn1RgNtAV2A1cb631RrxaEXGNYC2YR6/rxNVd1IJxg1DO3A8Cvay1JcaYFGCpMeZ1a+2HVfb5DfC/1tpzjDHDgIeA66NQr4i4wJiX1vPcv4+2YE5Ma8C6Cf0crEgC1Rju1rcmcEnFZkrFf4HrBF8BTKj4eC7wuDHGWN1hWSShfPbNPvoGtGDWT+hL4zS1YNwmpJ67MSYZWAWcA0y31q4M2OVM4EsAa225MeYHoCnwXQRrFRGHBGvB/Pn6bK7sfKZDFUlNQgp3a+1hINsY0wSYZ4xpb63dEO7BjDEjgBEAmVomVSQu3D93Hf8q+rJy++T0FD4e39fBiiQUYc2WsdbuMcYsAvoDVcN9J3AWsMMY0wA4Cd+F1cCvnwnMBN+dmGpbtIhE3yf/2Uv/Py/xG9vwYD8yUjXJLh6EMlvmVKCsItgbAn3wXTCtagHwK2AFcC3wnvrtIvEpWAtm2rBsrshWCyaehPIruDnwbEXfPQl4wVr7qjFmIlBkrV0APA383RjzOfA9MCxqFYtI1Nw7Zy1zV+2o3G6WkUrRuMscrEhqK5TZMuuAzkHGx1f5+AAwNLKliUisbPp6LwOmqQWTSJKcLkBEoqywEDweSEryPRYWVn7KWotn9EK/YP/vGzrjnTpIwR7n9N0TSWSFhf73fd22zbcN/FdKO15avbNy19NPTGXlWLVgEoXCXSSR5ef739Ab2JhxOoPWN8E3ya1i7MF+NNKZekLRd1MkkW0/ukTAEQxn3/+K36en39iFQR2bx7oqiQGFu0giy8yEbdvofsezfNO4aeXwGfu/Z/lfbnKwMIk2hbtIAlt0/0PcvC3Db2zjjJtoNP0vDlUksaJwF0lAh49YWo19DTga7COX/J27dn4I0/8CeXnOFScxoXAXiZbCQt8Fze3bfe2RgoKYhGrr/NcoO+z/BnHv1EHAoKgfW9xD4S4SDceZghitgH930zf85tkivzHdEan+Mk4tAZOTk2OLiopq3lEkHnk8vkAP1LIleL0RPdTRFsxRo/qdx52XnhPR44g7GGNWWWtzatpPZ+4i0VBlCmJI47WUNWYhgednvhaM1HcKd5FoqJiCGHQ8At4u/oZbZvv/5as7IklVWlumvjjO+iISBQUFkJ7uP5ae7huvg8NHfGvBVA320QPOxzt1kIJd/OjMvT5w4OJevffT6xrB2TKe0QuPGVMLRqqjC6r1QQwv7knkPbvcywMLNvqNqQVTf+mCqhwVo4t7Elllh4/QOv91v7Fh3c5i6jUdHapI4onCvT6I8sU9iTy1YKSudEG1PojSxT2JvGeWbj0m2NeO76tgl7DpzL0+iMLFPYmsQ+VHOHecfwsmr3smBVd1cKgiiXcK9/oiL09h7lJqwUg0KNxFHPLUki+YvHCT35jWgpFIUbiLxFiwFszwCz1MGNLOoYokESncRWJILRiJFYW7SAzMeH8LU1//xG9Mb0SSaFK4i0TRwfLDnDfuDb+xX+dmMf7ytg5VJPWFwl0kStSCESfpTUyS+GK8Iub0RZ8fE+wbHuynYJeY0pm7JLYYroj546HDtBnv34IZcfHZjB3YJqLHEQmFVoWUxBajFTHVgpFY0aqQIhD1FTFHv7iO5z/60m9s3YS+nKhZMOIwhbsktiitiFl6qJy249/0GxvUsTnTb+xSp+cViRRdUJXEFoUVMT2jFx4T7N6pg6If7LpVooRBZ+6S2CK4IubIF9by4uodfmMxeyOSbpUoYarxgqox5ixgNnA6YIGZ1tppAftcArwMbK0YeslaO/F4z6sLqhIv9h8sp90D/mfqV2SfwbRhnWNXhG6VKBUieUG1HBhprV1tjGkMrDLGvG2tLQ7Yb4m1dnBtihVxK9fMgtGtEiVMNYa7tfZr4OuKj/cZYzYBZwKB4S6SMH7//MfMX/OV39iGB/uRkepQJ1O3SpQwhXVB1RjjAToDK4N8uqcxZq0x5nVjjNYulbi070AZntEL/YL9mi4t8E4d5Fywg26VKGEL+afVGJMBvAj83lq7N+DTq4GW1toSY8xAYD7QOshzjABGAGTqjENcxjUtmGB0q0QJU0jvUDXGpACvAm9aax8NYX8vkGOt/a66fXRBVdzid/9czavrvvYbK57Yj/QTNJlM3CdiF1SNMQZ4GthUXbAbY34GfGOttcaYC/C1e3aHWbNITO09UEbHCW/5jV2fcxYPXdvRoYpEIieUU5Nc4CZgvTFmTcXYWCATwFo7A7gWuN0YUw78CAyzTi1aIxICV7dgRCIglNkySwFTwz6PA49HqiiRaLnt76t4Y+N//MY2TexPwxOSHapIJDrUVJR64Ycfy+j0oH8L5sbumUy5qoNDFYlEl8JdEp5aMFIfKdwlYU18pZhnlm31G/tkUn/SUtSCkcSncJeEs+9AGR0CZsGMHnA+t/2ilUMVicSewl0SilowIj4Kd0kIExZsZNZyr9/Yp5P7k9pALRipnxTuEteCvRFp3KA2/Paisx2qSMQdFO4St9SCEamewl3izrj56/nHh/7rmH82eQAnNNBdI0V+onCXuPFDaRmdJvq3YMYPbsuvf57lUEUi7qVwl7igFoxIeBTu4mpjXlrHc//+0m9sc8EAUpLVghE5HoW7uNKe0kNkT3zbb+zBIe341YUeZwoSiTMKd3EdtWBE6k7hLq4xas5a5qza4TemFoxI7SjcxXH/u/8QnSf5t2AmXdmem3q0dKgikfincBdHqQUjEh0Kd3HEf72whpdW7/Qb+7xgAA3UghGJCIW7xNTukoN0nfyO39iUqzpwY/dMhyoSSUwKd4kZtWBEYkfhLlH33+9u5tG3P/Mb2zJlIMlJx73vuojUgcJdoibYWjB/G96NS88/zaGKROoPhbtERWALJrVBEp9OHuBQNSL1j8JdIuqxtz9j2rub/ca+mDKQJLVgRGJK4S4REWwtmFk3d+OS89SCEXGCwl3qLLAFk5HagA0P9nOoGhEB0DtGpNYeffuzY4L9iykDww/2wkLweCApyfdYWBixGkXqK525S9iCtWD+/psLuKj1qeE/WWEhjBgBpaW+7W3bfNsAeXl1rFSk/jLWWkcOnJOTY4uKihw5ttRe4Jn6SQ1TWPtA3zo8occX6IFatgSvt/bPK5KgjDGrrLU5Ne2nM3cJyZ/e+IQnFm/xG4vILJjt28MbF5GQKNzluL7ff4guAcvxFv62O7nnNIvMATIzg5+5Z2qtGZG6ULhLtQJbME0bncCqP/SJ7EEKCvx77gDp6b5xEak1hbscY+rrnzDjff8WzNY/DsSYKLwR6aeLpvn5vlZMZqYv2HUxVaROagx3Y8xZwGzgdMACM6210wL2McA0YCBQCgy31q6OfLkSTd+VHCQnYDnef/62OxdGqgVTnbw8hblIhIVy5l4OjLTWrjbGNAZWGWPettYWV9lnANC64r/uwF8rHiVOBLZgTmucyr/zL3OoGhGpqxrD3Vr7NfB1xcf7jDGbgDOBquF+BTDb+uZVfmiMaWKMaV7xteJiU17bxMwPvvAbi1oLRkRiJqyeuzHGA3QGVgZ86kzgyyrbOyrGFO4uFawF8/yIHvQ4u6lDFYlIJIUc7saYDOBF4PfW2r21OZgxZgQwAiBTU90cE9iCOeOkNJaP6e1QNSISDSGFuzEmBV+wF1prXwqyy07grCrbLSrG/FhrZwIzwfcO1bCrlTqZ+Eoxzyzb6jemFoxIYgpltowBngY2WWsfrWa3BcDvjDHP47uQ+oP67e7x7b4DXFDwrt/YC7f25IKsUxyqSESiLZQz91zgJmC9MWZNxdhYIBPAWjsDeA3fNMjP8U2FvDnypUptBLZgzjqlIUvu6+VQNSISK6HMllkKHPfv9opZMndGqiipuwkLNjJruddvTC0YkfpD71BNMN/uPcAFU/xbMHNu60k3j1owIvWJwj2BBLZgPE3TWTzqUoeqEREnKdwTwPP/3s7ol9b7jakFI1K/KdzjWLA7Ir038hecfWqGQxWJiFso3OPU2WMWcqTKOwV+nZvF+MvbOleQiLiKwj3O/HPldsbOUwtGRI5P4R4n/nf/IToH3BFp0b2XkNWskUMViYibKdzjQOAsmN/+PItxg9WCEZHqKdxd7B8fbmPc/A1+Y2rBiEgoFO4uFOym1IvvvQSPWjAiEiKFu8sEtmBuvfhsxgxs41A1IhKvFO4uMXuFl/Evb/Qb804d5EwxIhL3FO4O211ykK4Bd0T6YNSlZDZNd6giEUkECncHBbZgbr+kFff3P9+hakQkkSjcHTBr2VYmvFLsN6YWjIhEksI9hoLdlHrJfZdy1ilqwYhIZCncYySwBXPnpa0Y1U8tGBGJjiSnC0gohYXg8UBSku+xsJBnlm49Jti9Uwcp2BNVkJ8BESfozD1SCgthxAgoLQVg164f6La+Caw/2ltXCybBBfwMsG2bbxsgL8+5uqReMr7bn8ZeTk6OLSoqcuTYUeHx+P4xA577X/X71F29zmFk3/McKEpiqsrPgJ+WLcHrjXU1kqCMMaustTk17acz90jZvp1Xz/85v7titN+w90+Xw9QjDhUlMbV9e3jjIlGknnsE7Ck9hOe+V/yCfelfb8b70GDIzHSwMhepD73o6r7X+hkQB+jMvY4GTltC8dd7K7cfXvgYQze869tIT4eCAocqc5H60osuKPD//wT9DIhjdOZeS6+s/QrP6IWVwe5pmo63wx6G7vscjPH1WWfOTKzwqq38fP/AA992fr4z9URLXp7ve96ypX4GxHG6oBqmkoPltH/gTb+xonGX0Swj1aGK4kBSEgT7OTMGjuh6hEg4dEE1Cia/WsxTS7dWbj8ytBPXdm3hYEVxIjMz+CwS9aJFokbhHoKPvN8zdMaKyu3hF3qYMKSdgxXFGfWiRWJOPffj2HegjHbj36gM9oYpyayb0Nedwe7m2SjqRYvEnM7cqxHYgvnXiB50P7upgxUdRzzMRsnLc08tIvWALqgGWPnFbq6f+WHl9q9zsxh/eVsHKwqB3hkpUm/ogmqY9h0oo1vBOxwo883eSD8hmZVje9M4LcXhykKgd0aKSACFO/DgKxv52zJv5fYLt/bkgqxTnCsoXJqNIiIB6nW4f/jFboZVacH85udZ/GGwy1swwWg2iogEqDHcjTHPAIOBb6217YN8/hLgZeCnq48vWWsnRrLISNt7oIxuk9/hYLmvBdM4tQErxvYmIzVOf9f9dKEyP9/XisnM9AW7LmCK1FuhpNks4HFg9nH2WWKtHRyRiqJswoKNzFrurdyec1tPunniqAVTHc1GEZEqagx3a+0HxhhP9EuJrhVbdnPDk0dbMLdclEX+oDhswYiIhCBSfYiexpi1wFfAvdbajRF63jrbe6CMnEnvcOhwRQsmrQErxsRxC0ZEJASRSLjVQEtrbYkxZiAwH2gdbEdjzAhgBEBmDGZyPPDyBp5dcXQWydzbepKTCC0YEZEa1DncrbV7q3z8mjHmCWNMM2vtd0H2nQnMBN+bmOp67OoEtmBuvfhsxgxsE63DiYi4Tp3D3RjzM+Aba601xlyAb72a3XWurBZ++LGMrpPepvyI7/fGSQ1TWD66F43UghGReiaUqZDPAZcAzYwxO4AHgBQAa+0M4FrgdmNMOfAjMMw6sKbBuPnr+ceHR9+R+eLtF9K15cmxLkNExBVCmS1zQw2ffxzfVElHLP/8O258amXl9m2/aMXoAec7VY6IiCvEbb/ih9IyOk96i4oODKc0OoEl912qFoyICHEa7vnz1lO48mgL5qU7LqRLplowIiI/ibtwL/5qb2Ww33FJK+7rrxaMiEiguAv3c0/P4PEbO9Pr/NNIPyHuyhcRiYm4S8cGyUkM7niG02WIiLia7qEqIpKAFO4iIglI4S4ikoAU7uEqLPTdkDopyfdYWOh0RSIix4i7C6qOKiz0v53dtm2+bdCNMkTEVXTmHo78fP/7lIJvOz/fmXpERKqhcA/H9u3hjYuIOEThHo7qbjASgxuPiIiEQ+EejoICSE/3H0tP942LiLiIwj0ceXkwcya0bAnG+B5nztTFVBFxnfgKdzdMQ8zLA68XjhzxPSrYRcSF4mcqpKYhioiELH7O3DUNUUQkZPET7pqGKCISsvgJd01DFBEJWfyEu6YhioiELH7CXdMQRURCFj+zZcAX5ApzEZEaxc+Zu4iIhEzhLiKSgBTuIiIJSOEuIpKAFO4iIgm/WtsRAAADI0lEQVTIWGudObAxu4BtIezaDPguyuXEI70u1dNrE5xel+rF02vT0lp7ak07ORbuoTLGFFlrc5yuw230ulRPr01wel2ql4ivjdoyIiIJSOEuIpKA4iHcZzpdgEvpdameXpvg9LpUL+FeG9f33EVEJHzxcOYuIiJhcmW4G2POMsYsMsYUG2M2GmPucbomNzHGJBtjPjbGvOp0LW5ijGlijJlrjPnEGLPJGNPT6Zrcwhjzfyv+LW0wxjxnjElzuianGGOeMcZ8a4zZUGXsFGPM28aYzRWPJztZYyS4MtyBcmCktbYt0AO40xjT1uGa3OQeYJPTRbjQNOANa+35QCf0GgFgjDkTuBvIsda2B5KBYc5W5ahZQP+AsdHAu9ba1sC7FdtxzZXhbq392lq7uuLjffj+kZ7pbFXuYIxpAQwCnnK6FjcxxpwEXAw8DWCtPWSt3eNsVa7SAGhojGkApANfOVyPY6y1HwDfBwxfATxb8fGzwJUxLSoKXBnuVRljPEBnYKWzlbjGn4H7gCNOF+IyWcAu4G8VLaunjDGNnC7KDay1O4FHgO3A18AP1tq3nK3KdU631n5d8fF/gNOdLCYSXB3uxpgM4EXg99bavU7X4zRjzGDgW2vtKqdrcaEGQBfgr9bazsB+EuBP60io6B9fge8X4BlAI2PM/3G2KveyvimEcT+N0LXhboxJwRfshdbal5yuxyVygSHGGC/wPNDLGPMPZ0tyjR3ADmvtT3/hzcUX9gKXAVuttbustWXAS8CFDtfkNt8YY5oDVDx+63A9debKcDfGGHy9003W2kedrsctrLVjrLUtrLUefBfE3rPW6gwMsNb+B/jSGHNexVBvoNjBktxkO9DDGJNe8W+rN7rYHGgB8KuKj38FvOxgLRHhynDHd4Z6E74z0zUV/w10uihxvbuAQmPMOiAbmOJwPa5Q8dfMXGA1sB7fv/uEe0dmqIwxzwErgPOMMTuMMb8BpgJ9jDGb8f2lM9XJGiNB71AVEUlAbj1zFxGROlC4i4gkIIW7iEgCUriLiCQghbuISAJSuIuIJCCFu4hIAlK4i4gkoP8PsvMlLWvuPNAAAAAASUVORK5CYII=\n", 173 | "text/plain": [ 174 | "
" 175 | ] 176 | }, 177 | "metadata": {}, 178 | "output_type": "display_data" 179 | } 180 | ], 181 | "source": [ 182 | "#continuation of above, plotting the graph\n", 183 | "predicted = model(torch.from_numpy(x_train)).detach().numpy()\n", 184 | "plt.plot(x_train, y_train, 'ro', label='Original data')\n", 185 | "plt.plot(x_train, predicted, label='Fitted line')\n", 186 | "plt.legend()\n", 187 | "plt.show()" 188 | ] 189 | }, 190 | { 191 | "cell_type": "code", 192 | "execution_count": null, 193 | "metadata": {}, 194 | "outputs": [], 195 | "source": [] 196 | } 197 | ], 198 | "metadata": { 199 | "kernelspec": { 200 | "display_name": "Python 3", 201 | "language": "python", 202 | "name": "python3" 203 | }, 204 | "language_info": { 205 | "codemirror_mode": { 206 | "name": "ipython", 207 | "version": 3 208 | }, 209 | "file_extension": ".py", 210 | "mimetype": "text/x-python", 211 | "name": "python", 212 | "nbconvert_exporter": "python", 213 | "pygments_lexer": "ipython3", 214 | "version": "3.5.2" 215 | } 216 | }, 217 | "nbformat": 4, 218 | "nbformat_minor": 2 219 | } 220 | -------------------------------------------------------------------------------- /CNN and ResNet.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import torch\n", 10 | "import torch.nn as nn\n", 11 | "import torchvision\n", 12 | "import numpy as np\n", 13 | "import torchvision.transforms as transforms\n", 14 | "import matplotlib.pyplot as plt" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": 4, 20 | "metadata": {}, 21 | "outputs": [ 22 | { 23 | "name": "stdout", 24 | "output_type": "stream", 25 | "text": [ 26 | "Epoch [1/5], Step [100/600], Loss: 0.1840\n", 27 | "Epoch [1/5], Step [200/600], Loss: 0.1319\n", 28 | "Epoch [1/5], Step [300/600], Loss: 0.0985\n", 29 | "Epoch [1/5], Step [400/600], Loss: 0.0702\n", 30 | "Epoch [1/5], Step [500/600], Loss: 0.0764\n", 31 | "Epoch [1/5], Step [600/600], Loss: 0.0713\n", 32 | "Epoch [2/5], Step [100/600], Loss: 0.0245\n", 33 | "Epoch [2/5], Step [200/600], Loss: 0.0640\n", 34 | "Epoch [2/5], Step [300/600], Loss: 0.0906\n", 35 | "Epoch [2/5], Step [400/600], Loss: 0.0248\n", 36 | "Epoch [2/5], Step [500/600], Loss: 0.0114\n", 37 | "Epoch [2/5], Step [600/600], Loss: 0.1116\n", 38 | "Epoch [3/5], Step [100/600], Loss: 0.0559\n", 39 | "Epoch [3/5], Step [200/600], Loss: 0.0767\n", 40 | "Epoch [3/5], Step [300/600], Loss: 0.1913\n", 41 | "Epoch [3/5], Step [400/600], Loss: 0.0453\n", 42 | "Epoch [3/5], Step [500/600], Loss: 0.0302\n", 43 | "Epoch [3/5], Step [600/600], Loss: 0.0527\n", 44 | "Epoch [4/5], Step [100/600], Loss: 0.0102\n", 45 | "Epoch [4/5], Step [200/600], Loss: 0.0629\n", 46 | "Epoch [4/5], Step [300/600], Loss: 0.0215\n", 47 | "Epoch [4/5], Step [400/600], Loss: 0.0314\n", 48 | "Epoch [4/5], Step [500/600], Loss: 0.0040\n", 49 | "Epoch [4/5], Step [600/600], Loss: 0.0225\n", 50 | "Epoch [5/5], Step [100/600], Loss: 0.0394\n", 51 | "Epoch [5/5], Step [200/600], Loss: 0.0253\n", 52 | "Epoch [5/5], Step [300/600], Loss: 0.0266\n", 53 | "Epoch [5/5], Step [400/600], Loss: 0.0113\n", 54 | "Epoch [5/5], Step [500/600], Loss: 0.0361\n", 55 | "Epoch [5/5], Step [600/600], Loss: 0.0081\n", 56 | "Test Accuracy of the model on the 10000 test images: 99 %\n" 57 | ] 58 | } 59 | ], 60 | "source": [ 61 | "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n", 62 | "num_epochs = 5\n", 63 | "num_classes = 10\n", 64 | "batch_size = 100 \n", 65 | "learning_rate = 0.001\n", 66 | "\n", 67 | "class Conv(nn.Module):\n", 68 | " def __init__(self, num_classes):\n", 69 | " super(Conv, self).__init__()\n", 70 | " self.layer1 = nn.Sequential(\n", 71 | " nn.Conv2d(1,16, kernel_size=5, stride = 1, padding = 2),\n", 72 | " nn.BatchNorm2d(16),\n", 73 | " nn.ReLU(),\n", 74 | " nn.MaxPool2d(kernel_size = 2, stride=2))\n", 75 | " self.layer2 = nn.Sequential(\n", 76 | " nn.Conv2d(16,32,kernel_size = 5, padding =2, stride =1),\n", 77 | " nn.BatchNorm2d(32),\n", 78 | " nn.ReLU(),\n", 79 | " nn.MaxPool2d(kernel_size = 2, stride =2))\n", 80 | " self.fc = nn.Linear(7*7*32, num_classes)\n", 81 | " def forward(self, x):\n", 82 | " out = self.layer1(x)\n", 83 | " out = self.layer2(out)\n", 84 | "# print(out.size(0)) #this is to check your implementation\n", 85 | " out = out.reshape(out.size(0), -1)#here we provide a batch of image so out.size(0) is the batch_size\n", 86 | " out = self.fc(out)\n", 87 | " return out\n", 88 | "\n", 89 | "train_dataset = torchvision.datasets.MNIST(root='Data', train=True, transform = transforms.ToTensor(), download = True)\n", 90 | "test_dataset = torchvision.datasets.MNIST(root='Data', train=False, transform = transforms.ToTensor(), download = False)\n", 91 | "train_loader = torch.utils.data.DataLoader(dataset = train_dataset, batch_size = batch_size, shuffle = True)\n", 92 | "test_loader = torch.utils.data.DataLoader(dataset = test_dataset, batch_size = batch_size, shuffle = True)\n", 93 | "\n", 94 | "model = Conv(num_classes).to(device)\n", 95 | "criterion = nn.CrossEntropyLoss()\n", 96 | "optimizer = torch.optim.Adam(model.parameters(), lr= learning_rate)\n", 97 | "# out = model(train_dataset[0][0].reshape(1,1,28,28))#1-no. of images, 1-depth of the image, height, width\n", 98 | "\n", 99 | "total_step = len(train_loader)\n", 100 | "for epoch in range(num_epochs):\n", 101 | " for i, (images, labels) in enumerate(train_loader):\n", 102 | " images = images.to(device)\n", 103 | " labels = labels.to(device)\n", 104 | "# print(images.shape)\n", 105 | " outputs = model(images)\n", 106 | " loss = criterion(outputs, labels)\n", 107 | " \n", 108 | " optimizer.zero_grad()\n", 109 | " loss.backward()\n", 110 | " optimizer.step()\n", 111 | " if (i+1) % 100 == 0:\n", 112 | " print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}' \n", 113 | " .format(epoch+1, num_epochs, i+1, total_step, loss.item()))\n", 114 | "\n", 115 | "model.eval() #it is typically used for batchnorm so that it uses moving variance and mean instead of the whole batch\n", 116 | "with torch.no_grad():\n", 117 | " correct = 0\n", 118 | " total = 0\n", 119 | " for images, labels in test_loader:\n", 120 | " images = images.to(device)\n", 121 | " labels = labels.to(device)\n", 122 | " outputs = model(images)\n", 123 | " _, predicted = torch.max(outputs.data, 1)\n", 124 | " total += labels.size(0)\n", 125 | " correct += (predicted == labels).sum()\n", 126 | "\n", 127 | " print('Test Accuracy of the model on the 10000 test images: {} %'.format(100 * correct / total))" 128 | ] 129 | }, 130 | { 131 | "cell_type": "code", 132 | "execution_count": null, 133 | "metadata": {}, 134 | "outputs": [], 135 | "source": [ 136 | "torch.save(model.state_dict(), 'params.ckpt') #we are saving only the parameters\n", 137 | "torch.save(model, 'model.ckpt') #if memory is not an issue then better save the whole model. \n", 138 | "c = torch.load('model.ckpt')#one more important thing is that the device which you have used to train will be only used for testing also(cuda or CPU)\n", 139 | "d = torch.load('params.ckpt')\n", 140 | "#print both" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": null, 146 | "metadata": {}, 147 | "outputs": [], 148 | "source": [ 149 | "#Do Practise after transfer learning and loading a custom dataset. \n", 150 | "\n", 151 | "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n", 152 | "batch_size = 100\n", 153 | "num_epochs = 30\n", 154 | "learning_rate = 0.001\n", 155 | "transform = transforms.Compose([\n", 156 | " transforms.Pad(4),\n", 157 | " transforms.RandomHorizontalFlip(), \n", 158 | " transforms.RandomCrop(32), \n", 159 | " transforms.ToTensor()])\n", 160 | "\n", 161 | "train_dataset = torchvision.datasets.CIFAR10(root='Data', train=True, transform = transform, download = True)\n", 162 | "test_dataset = torchvision.datasets.CIFAR10(root='Data', train=False, transform = transforms.ToTensor(), download = False)\n", 163 | "train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size = batch_size, shuffle= True)\n", 164 | "test_loader = torch.utils.data.DataLoader(dataset=test_dataset, batch_size = batch_size, shuffle= False)\n", 165 | "\n", 166 | "class ResidualBlock(nn.Module):\n", 167 | " def __init__(self, in_channel, out_channel,stride=1, downsample=None):\n", 168 | " super(ResidualBlock, self).__init__()\n", 169 | " self.conv1 = nn.Conv2d(in_channel, out_channel, kernel_size = 3, stride = stride, padding=1)\n", 170 | " self.bn1 = nn.BatchNorm2d(out_channel)\n", 171 | " self.relu = nn.ReLU()\n", 172 | " self.conv2 = nn.Conv2d(out_channel, out_channel, kernel_size = 3, stride = 1, padding=1) #we change the size only once\n", 173 | " self.bn2 = nn.BatchNorm2d(out_channel)\n", 174 | " self.downsample = downsample\n", 175 | " def forward(self,x):\n", 176 | " residual = x\n", 177 | " out = self.conv1(x)\n", 178 | " out = self.bn1(out)\n", 179 | " out = self.relu(out)\n", 180 | " out = self.conv2(out)\n", 181 | " out = self.bn2(out)\n", 182 | " if self.downsample:#to be used when input size does not match output size\n", 183 | " residual = self.downsample(x)\n", 184 | " out += residual\n", 185 | " out = self.relu(out)\n", 186 | " return(out)\n", 187 | "\n", 188 | "class ResNet(nn.Module):\n", 189 | " def __init__(self, block, num_classes=10):\n", 190 | " super(ResNet, self).__init__()\n", 191 | " self.in_channel = 16\n", 192 | " self.conv1 = nn.Conv2d(3,16, stride =1, kernel_size = 3, padding = 1)\n", 193 | " self.bn = nn.BatchNorm2d(16)\n", 194 | " self.relu = nn.ReLU()\n", 195 | " self.block1 = self.make_layer(block, 16, 1)\n", 196 | " self.block2 = self.make_layer(block, 16, 1)\n", 197 | " self.block3 = self.make_layer(block, 32, 2)\n", 198 | " self.block4 = self.make_layer(block, 32, 1)\n", 199 | " self.block5 = self.make_layer(block, 64, 2)\n", 200 | " self.block6 = self.make_layer(block, 64, 1)\n", 201 | " self.avg_pool = nn.AvgPool2d(8) #8 is the kernel size so it is taking average of 8x8\n", 202 | " self.fc = nn.Linear(64, num_classes)\n", 203 | " def make_layer(self, block, out_channel, stride=1):\n", 204 | " downsample = None\n", 205 | " if(stride!=1) or (self.in_channel != out_channel):#input size not equal to output size only when stride not 1 or input channel and output channel are not same \n", 206 | " downsample = nn.Sequential(\n", 207 | " nn.Conv2d(self.in_channel, out_channel, kernel_size = 3, stride = stride, padding = 1),\n", 208 | " nn.BatchNorm2d(out_channel))\n", 209 | " out_layer = block(self.in_channel, out_channel, stride, downsample)\n", 210 | " self.in_channel = out_channel\n", 211 | " return(out_layer)\n", 212 | " def forward(self,x):\n", 213 | " out = self.conv1(x)\n", 214 | " out = self.bn(out)\n", 215 | " out = self.relu(out)\n", 216 | " out = self.block1(out)\n", 217 | " out = self.block2(out)\n", 218 | " out = self.block3(out)\n", 219 | " out = self.block4(out)\n", 220 | " out = self.block5(out)\n", 221 | " out = self.block6(out)\n", 222 | " out = self.avg_pool(out)\n", 223 | " out = out.view(out.size(0), -1)\n", 224 | " out = self.fc(out)\n", 225 | " return out\n", 226 | "model = ResNet(ResidualBlock).to(device)\n", 227 | "# print(train_dataset[0][0].shape)\n", 228 | "# out = model(train_dataset[0][0].reshape(1,3,32,32))\n", 229 | "\n", 230 | "# Loss and optimizer\n", 231 | "criterion = nn.CrossEntropyLoss()\n", 232 | "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n", 233 | "\n", 234 | "# For updating learning rate\n", 235 | "def update_lr(optimizer, lr): \n", 236 | " for param_group in optimizer.param_groups:\n", 237 | " param_group['lr'] = lr\n", 238 | "\n", 239 | "# Train the model\n", 240 | "total_step = len(train_loader)\n", 241 | "curr_lr = learning_rate\n", 242 | "for epoch in range(num_epochs):\n", 243 | " for i, (images, labels) in enumerate(train_loader):\n", 244 | " images = images.to(device)\n", 245 | " labels = labels.to(device)\n", 246 | " \n", 247 | " # Forward pass\n", 248 | " outputs = model(images)\n", 249 | " loss = criterion(outputs, labels)\n", 250 | " \n", 251 | " # Backward and optimize\n", 252 | " optimizer.zero_grad()\n", 253 | " loss.backward()\n", 254 | " optimizer.step()\n", 255 | " \n", 256 | " if (i+1) % 100 == 0:\n", 257 | " print (\"Epoch [{}/{}], Step [{}/{}] Loss: {:.4f}\"\n", 258 | " .format(epoch+1, num_epochs, i+1, total_step, loss.item()))\n", 259 | "\n", 260 | " # Decay learning rate\n", 261 | " if (epoch+1) % 20 == 0:\n", 262 | " curr_lr /= 3\n", 263 | " update_lr(optimizer, curr_lr)" 264 | ] 265 | } 266 | ], 267 | "metadata": { 268 | "kernelspec": { 269 | "display_name": "Python 3", 270 | "language": "python", 271 | "name": "python3" 272 | }, 273 | "language_info": { 274 | "codemirror_mode": { 275 | "name": "ipython", 276 | "version": 3 277 | }, 278 | "file_extension": ".py", 279 | "mimetype": "text/x-python", 280 | "name": "python", 281 | "nbconvert_exporter": "python", 282 | "pygments_lexer": "ipython3", 283 | "version": "3.5.2" 284 | } 285 | }, 286 | "nbformat": 4, 287 | "nbformat_minor": 2 288 | } 289 | -------------------------------------------------------------------------------- /Custom dataset and Transfer learning.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import os\n", 10 | "import torch\n", 11 | "import torchvision\n", 12 | "import torch.nn as nn\n", 13 | "from torchvision import transforms\n", 14 | "from torchvision.utils import save_image" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": 2, 20 | "metadata": {}, 21 | "outputs": [ 22 | { 23 | "name": "stdout", 24 | "output_type": "stream", 25 | "text": [ 26 | "The child is - Conv2d(3, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 27 | "The value of child number is 0\n", 28 | "The child is - BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", 29 | "The value of child number is 1\n", 30 | "The child is - ReLU()\n", 31 | "The value of child number is 2\n", 32 | "The child is - ResidualBlock(\n", 33 | " (conv1): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 34 | " (bn1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", 35 | " (relu): ReLU()\n", 36 | " (conv2): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 37 | " (bn2): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", 38 | ")\n", 39 | "The value of child number is 3\n", 40 | "The child is - ResidualBlock(\n", 41 | " (conv1): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 42 | " (bn1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", 43 | " (relu): ReLU()\n", 44 | " (conv2): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 45 | " (bn2): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", 46 | ")\n", 47 | "The value of child number is 4\n", 48 | "The child is - ResidualBlock(\n", 49 | " (conv1): Conv2d(16, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n", 50 | " (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", 51 | " (relu): ReLU()\n", 52 | " (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 53 | " (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", 54 | " (downsample): Sequential(\n", 55 | " (0): Conv2d(16, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n", 56 | " (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", 57 | " )\n", 58 | ")\n", 59 | "The value of child number is 5\n", 60 | "The child is - ResidualBlock(\n", 61 | " (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 62 | " (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", 63 | " (relu): ReLU()\n", 64 | " (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 65 | " (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", 66 | ")\n", 67 | "The value of child number is 6\n", 68 | "The child is - ResidualBlock(\n", 69 | " (conv1): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n", 70 | " (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", 71 | " (relu): ReLU()\n", 72 | " (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 73 | " (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", 74 | " (downsample): Sequential(\n", 75 | " (0): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n", 76 | " (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", 77 | " )\n", 78 | ")\n", 79 | "The value of child number is 7\n", 80 | "The child is - ResidualBlock(\n", 81 | " (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 82 | " (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", 83 | " (relu): ReLU()\n", 84 | " (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", 85 | " (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", 86 | ")\n", 87 | "The value of child number is 8\n", 88 | "The child is - AvgPool2d(kernel_size=8, stride=8, padding=0)\n", 89 | "The value of child number is 9\n", 90 | "The child is - Linear(in_features=64, out_features=10, bias=True)\n", 91 | "The value of child number is 10\n", 92 | "Linear(in_features=64, out_features=10, bias=True)\n", 93 | "Linear(in_features=64, out_features=5, bias=True)\n" 94 | ] 95 | } 96 | ], 97 | "source": [ 98 | "class ResidualBlock(nn.Module):\n", 99 | " def __init__(self, in_channel, out_channel,stride=1, downsample=None):\n", 100 | " super(ResidualBlock, self).__init__()\n", 101 | " self.conv1 = nn.Conv2d(in_channel, out_channel, kernel_size = 3, stride = stride, padding=1)\n", 102 | " self.bn1 = nn.BatchNorm2d(out_channel)\n", 103 | " self.relu = nn.ReLU()\n", 104 | " self.conv2 = nn.Conv2d(out_channel, out_channel, kernel_size = 3, stride = 1, padding=1) #we change the size only once\n", 105 | " self.bn2 = nn.BatchNorm2d(out_channel)\n", 106 | " self.downsample = downsample\n", 107 | " def forward(self,x):\n", 108 | " residual = x\n", 109 | " out = self.conv1(x)\n", 110 | " out = self.bn1(out)\n", 111 | " out = self.relu(out)\n", 112 | " out = self.conv2(out)\n", 113 | " out = self.bn2(out)\n", 114 | " if self.downsample:#to be used when input size does not match output size\n", 115 | " residual = self.downsample(x)\n", 116 | " out += residual\n", 117 | " out = self.relu(out)\n", 118 | " return(out)\n", 119 | "\n", 120 | "class ResNet(nn.Module):\n", 121 | " def __init__(self, block, num_classes=10):\n", 122 | " super(ResNet, self).__init__()\n", 123 | " self.in_channel = 16\n", 124 | " self.conv1 = nn.Conv2d(3,16, stride =1, kernel_size = 3, padding = 1)\n", 125 | " self.bn = nn.BatchNorm2d(16)\n", 126 | " self.relu = nn.ReLU()\n", 127 | " self.block1 = self.make_layer(block, 16, 1)\n", 128 | " self.block2 = self.make_layer(block, 16, 1)\n", 129 | " self.block3 = self.make_layer(block, 32, 2)\n", 130 | " self.block4 = self.make_layer(block, 32, 1)\n", 131 | " self.block5 = self.make_layer(block, 64, 2)\n", 132 | " self.block6 = self.make_layer(block, 64, 1)\n", 133 | " self.avg_pool = nn.AvgPool2d(8) #8 is the kernel size so it is taking average of 8x8\n", 134 | " self.fc = nn.Linear(64, num_classes)\n", 135 | " def make_layer(self, block, out_channel, stride=1):\n", 136 | " downsample = None\n", 137 | " if(stride!=1) or (self.in_channel != out_channel):#input size not equal to output size only when stride not 1 or input channel and output channel are not same \n", 138 | " downsample = nn.Sequential(\n", 139 | " nn.Conv2d(self.in_channel, out_channel, kernel_size = 3, stride = stride, padding = 1),\n", 140 | " nn.BatchNorm2d(out_channel))\n", 141 | " out_layer = block(self.in_channel, out_channel, stride, downsample)\n", 142 | " self.in_channel = out_channel\n", 143 | " return(out_layer)\n", 144 | " def forward(self,x):\n", 145 | " out = self.conv1(x)\n", 146 | " out = self.bn(out)\n", 147 | " out = self.relu(out)\n", 148 | " out = self.block1(out)\n", 149 | " out = self.block2(out)\n", 150 | " out = self.block3(out)\n", 151 | " out = self.block4(out)\n", 152 | " out = self.block5(out)\n", 153 | " out = self.block6(out)\n", 154 | " out = self.avg_pool(out)\n", 155 | " out = out.view(out.size(0), -1)\n", 156 | " out = self.fc(out)\n", 157 | " return out\n", 158 | "model = ResNet(ResidualBlock)\n", 159 | "i=0\n", 160 | "# this is very important. To freeze the weights we use iteration and freeze accordingly.\n", 161 | "# Here I have freezed all layers except the last fc.\n", 162 | "#try to play with this by yourself. \n", 163 | "for c in model.children():\n", 164 | " print(\"The child is -\",c)\n", 165 | " print(\"The value of child number is \",i)\n", 166 | " i+=1\n", 167 | " if(i<=9):\n", 168 | " for param in c.parameters():\n", 169 | " param.reqires_grad = False\n", 170 | " \n", 171 | "print(model.fc)\n", 172 | "model.fc = nn.Linear(model.fc.in_features, 5)\n", 173 | "print(model.fc)\n" 174 | ] 175 | }, 176 | { 177 | "cell_type": "code", 178 | "execution_count": 33, 179 | "metadata": {}, 180 | "outputs": [ 181 | { 182 | "name": "stdout", 183 | "output_type": "stream", 184 | "text": [ 185 | "Linear(in_features=512, out_features=1000, bias=True)\n", 186 | "torch.Size([64, 1000])\n", 187 | "torch.Size([64, 100])\n" 188 | ] 189 | } 190 | ], 191 | "source": [ 192 | "# the below is to download a pretrained model and then playing with that\n", 193 | "resnet = torchvision.models.resnet18(pretrained=True)\n", 194 | "print(resnet.fc)\n", 195 | "for param in resnet.parameters():\n", 196 | " param.requires_grad = False\n", 197 | "\n", 198 | "images = torch.randn(64,3,224,224)\n", 199 | "output = resnet(images)\n", 200 | "print(output.shape)\n", 201 | "resnet.fc = nn.Linear(resnet.fc.in_features, 100)\n", 202 | "# print(resnet.fc)\n", 203 | "images = torch.randn(64,3,224,224)\n", 204 | "output = resnet(images)\n", 205 | "print(output.shape)\n", 206 | "\n" 207 | ] 208 | }, 209 | { 210 | "cell_type": "code", 211 | "execution_count": 8, 212 | "metadata": {}, 213 | "outputs": [], 214 | "source": [ 215 | "#It's a good practice to first convert your data into the numpy file and then load. It makes computation faster as well as writing code for that is realtively easy. \n", 216 | "#For fun, check the dataLoader of COCO for image captioning. \n", 217 | "class CustomDatasetFromCSV(torch.utils.data.Dataset):\n", 218 | " def __init__(self, csv_path, height, width):#this is used to initialize the datapath and define data transform\n", 219 | " \"\"\"\n", 220 | " Args:\n", 221 | " csv_path (string): path to csv file\n", 222 | " height (int): image height\n", 223 | " width (int): image width\n", 224 | " transform: pytorch transforms for transforms and tensor conversion\n", 225 | " \"\"\"\n", 226 | " self.data = pd.read_csv(csv_path)\n", 227 | " self.labels = np.asarray(self.data.iloc[:, 0])\n", 228 | " self.height = height\n", 229 | " self.width = width\n", 230 | " \n", 231 | " def transform(self): # this \n", 232 | " return (transforms.Compose([\n", 233 | " transforms.Pad(4),\n", 234 | " transforms.RandomHorizontalFlip(),\n", 235 | " transforms.RandomCrop(32),\n", 236 | " transforms.ToTensor()]))\n", 237 | " \n", 238 | " def __getitem__(self, index):#this is used to get single image. As this will be iteratively used.\n", 239 | " single_image_label = self.labels[index]\n", 240 | " # Read each 784 pixels and reshape the 1D array ([784]) to 2D array ([28,28]) \n", 241 | " img_as_np = np.asarray(self.data.iloc[index][1:]).reshape(28,28).astype('uint8')\n", 242 | " # Convert image from numpy array to PIL image, mode 'L' is for grayscale\n", 243 | " img_as_img = Image.fromarray(img_as_np)\n", 244 | " img_as_img = img_as_img.convert('L')\n", 245 | " # Transform image to tensor\n", 246 | " img_as_tensor = self.transform(img_as_img)\n", 247 | " # Return image and the label\n", 248 | " return (img_as_tensor, single_image_label)\n", 249 | "\n", 250 | " def __len__(self):\n", 251 | " return len(self.data.index)\n", 252 | " \n", 253 | "#another approach is to define the transforms below \n", 254 | "class CustomDatasetFromCSV(Dataset):\n", 255 | " def __init__(self, csv_path, height, width, transforms=None):\n", 256 | " \"\"\"\n", 257 | " Args:\n", 258 | " csv_path (string): path to csv file\n", 259 | " height (int): image height\n", 260 | " width (int): image width\n", 261 | " transform: pytorch transforms for transforms and tensor conversion\n", 262 | " \"\"\"\n", 263 | " self.data = pd.read_csv(csv_path)\n", 264 | " self.labels = np.asarray(self.data.iloc[:, 0])\n", 265 | " self.height = height\n", 266 | " self.width = width\n", 267 | " self.transforms = transform\n", 268 | "\n", 269 | " def __getitem__(self, index):\n", 270 | " single_image_label = self.labels[index]\n", 271 | " # Read each 784 pixels and reshape the 1D array ([784]) to 2D array ([28,28]) \n", 272 | " img_as_np = np.asarray(self.data.iloc[index][1:]).reshape(28,28).astype('uint8')\n", 273 | "\t# Convert image from numpy array to PIL image, mode 'L' is for grayscale\n", 274 | " img_as_img = Image.fromarray(img_as_np)\n", 275 | " img_as_img = img_as_img.convert('L')\n", 276 | " # Transform image to tensor\n", 277 | " if self.transforms is not None:\n", 278 | " img_as_tensor = self.transforms(img_as_img)\n", 279 | " # Return image and the label\n", 280 | " return (img_as_tensor, single_image_label)\n", 281 | "\n", 282 | " def __len__(self):\n", 283 | " return len(self.data.index)\n", 284 | "\n", 285 | "#Above was the case when you have a normal classification problem, when you have other kind of problem like object detection of so \n", 286 | "#then you can make a csv file which has annotations and filename(with path) to parse and load the data. \n", 287 | "\n", 288 | "\n", 289 | "# transformations = transforms.Compose([transforms.ToTensor()])\n", 290 | "# custom_mnist_from_csv = CustomDatasetFromCSV('../data/mnist_in_csv.csv', 28, 28, transformations)" 291 | ] 292 | }, 293 | { 294 | "cell_type": "code", 295 | "execution_count": null, 296 | "metadata": {}, 297 | "outputs": [], 298 | "source": [] 299 | } 300 | ], 301 | "metadata": { 302 | "kernelspec": { 303 | "display_name": "Python 3", 304 | "language": "python", 305 | "name": "python3" 306 | }, 307 | "language_info": { 308 | "codemirror_mode": { 309 | "name": "ipython", 310 | "version": 3 311 | }, 312 | "file_extension": ".py", 313 | "mimetype": "text/x-python", 314 | "name": "python", 315 | "nbconvert_exporter": "python", 316 | "pygments_lexer": "ipython3", 317 | "version": "3.5.2" 318 | } 319 | }, 320 | "nbformat": 4, 321 | "nbformat_minor": 2 322 | } 323 | -------------------------------------------------------------------------------- /GANS.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import os\n", 12 | "import torch\n", 13 | "import torchvision\n", 14 | "import torch.nn as nn\n", 15 | "from torchvision import transforms\n", 16 | "from torchvision.utils import save_image\n", 17 | "\n", 18 | "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 13, 24 | "metadata": {}, 25 | "outputs": [ 26 | { 27 | "name": "stdout", 28 | "output_type": "stream", 29 | "text": [ 30 | "Epoch [0/10], Step [200/600], d_loss: 0.0452, g_loss: 4.3386 \n", 31 | "Epoch [0/10], Step [400/600], d_loss: 0.1360, g_loss: 4.9729 \n", 32 | "Epoch [0/10], Step [600/600], d_loss: 0.0328, g_loss: 5.3808 \n", 33 | "Epoch [1/10], Step [200/600], d_loss: 0.0616, g_loss: 5.6563 \n", 34 | "Epoch [1/10], Step [400/600], d_loss: 0.2106, g_loss: 3.5196 \n", 35 | "Epoch [1/10], Step [600/600], d_loss: 0.3836, g_loss: 4.4486 \n", 36 | "Epoch [2/10], Step [200/600], d_loss: 0.1566, g_loss: 3.4256 \n", 37 | "Epoch [2/10], Step [400/600], d_loss: 1.1676, g_loss: 2.7942 \n", 38 | "Epoch [2/10], Step [600/600], d_loss: 0.1919, g_loss: 4.2086 \n", 39 | "Epoch [3/10], Step [200/600], d_loss: 0.2330, g_loss: 3.6334 \n", 40 | "Epoch [3/10], Step [400/600], d_loss: 0.8132, g_loss: 3.4572 \n", 41 | "Epoch [3/10], Step [600/600], d_loss: 0.1976, g_loss: 3.9961 \n", 42 | "Epoch [4/10], Step [200/600], d_loss: 0.6469, g_loss: 3.4096 \n", 43 | "Epoch [4/10], Step [400/600], d_loss: 0.3115, g_loss: 3.8466 \n", 44 | "Epoch [4/10], Step [600/600], d_loss: 0.1797, g_loss: 3.2961 \n", 45 | "Epoch [5/10], Step [200/600], d_loss: 0.4727, g_loss: 5.3163 \n", 46 | "Epoch [5/10], Step [400/600], d_loss: 0.2535, g_loss: 3.6409 \n", 47 | "Epoch [5/10], Step [600/600], d_loss: 0.5925, g_loss: 3.4327 \n", 48 | "Epoch [6/10], Step [200/600], d_loss: 0.5229, g_loss: 3.4864 \n", 49 | "Epoch [6/10], Step [400/600], d_loss: 0.3635, g_loss: 4.3978 \n", 50 | "Epoch [6/10], Step [600/600], d_loss: 0.3305, g_loss: 3.5535 \n", 51 | "Epoch [7/10], Step [200/600], d_loss: 0.1542, g_loss: 3.9531 \n", 52 | "Epoch [7/10], Step [400/600], d_loss: 0.1438, g_loss: 3.3679 \n", 53 | "Epoch [7/10], Step [600/600], d_loss: 0.2528, g_loss: 5.8108 \n", 54 | "Epoch [8/10], Step [200/600], d_loss: 0.1854, g_loss: 5.6493 \n", 55 | "Epoch [8/10], Step [400/600], d_loss: 0.3392, g_loss: 5.4587 \n", 56 | "Epoch [8/10], Step [600/600], d_loss: 0.0764, g_loss: 4.8581 \n", 57 | "Epoch [9/10], Step [200/600], d_loss: 0.1503, g_loss: 4.1301 \n", 58 | "Epoch [9/10], Step [400/600], d_loss: 0.1419, g_loss: 5.1282 \n", 59 | "Epoch [9/10], Step [600/600], d_loss: 0.1980, g_loss: 5.8345 \n" 60 | ] 61 | } 62 | ], 63 | "source": [ 64 | "random_size = 64 \n", 65 | "hidden_size = 256 \n", 66 | "image_size = 784\n", 67 | "num_epoch = 10\n", 68 | "batch_size = 100\n", 69 | "learning_rate = 0.0002\n", 70 | "sample_dir = 'samples'\n", 71 | "\n", 72 | "\n", 73 | "if not os.path.exists(sample_dir):\n", 74 | " os.makedirs(sample_dir)\n", 75 | " \n", 76 | "transform = transforms.Compose([\n", 77 | " transforms.ToTensor(),\n", 78 | " transforms.Normalize(mean =(0.5,0.5,0.5),# this is for RGB image(3 channel)\n", 79 | " std = (0.5,0.5,0.5))])\n", 80 | "\n", 81 | "data = torchvision.datasets.MNIST(root = 'Data', transform = transform, train=True, download = True)\n", 82 | "data_loader = torch.utils.data.DataLoader(dataset = data, shuffle = True, batch_size = batch_size)\n", 83 | "\n", 84 | "def denorm(x):\n", 85 | " out = (x + 1) / 2\n", 86 | " return out.clamp(0, 1)\n", 87 | "\n", 88 | "\n", 89 | "class Discriminator(nn.Module):\n", 90 | " def __init__(self, image_size, hidden_size):\n", 91 | " super(Discriminator, self).__init__()\n", 92 | " self.fc1 = nn.Linear(image_size, hidden_size)\n", 93 | " self.relu = nn.LeakyReLU(0.2)#this is the negative slope. \n", 94 | " self.fc2 = nn.Linear(hidden_size, hidden_size)\n", 95 | " self.fc3 = nn.Linear(hidden_size, 1)\n", 96 | " self.sigmoid = nn.Sigmoid()\n", 97 | " \n", 98 | " def forward(self, x):\n", 99 | " out = self.fc1(x)\n", 100 | " out = self.relu(out)\n", 101 | " out = self.fc2(out)\n", 102 | " out = self.relu(out)\n", 103 | " out = self.fc3(out)\n", 104 | " out = self.sigmoid(out)\n", 105 | " return (out)\n", 106 | "class Generator(nn.Module):\n", 107 | " def __init__(self, random_size, hidden_size, image_size):\n", 108 | " super(Generator, self).__init__()\n", 109 | " self.fc1 = nn.Linear(random_size, hidden_size)\n", 110 | " self.relu = nn.ReLU()\n", 111 | " self.fc2 = nn.Linear(hidden_size, hidden_size)\n", 112 | " self.fc3 = nn.Linear(hidden_size, image_size)\n", 113 | " self.tanh = nn.Tanh()\n", 114 | " def forward(self, x):\n", 115 | " out = self.fc1(x)\n", 116 | " out = self.relu(out)\n", 117 | " out = self.fc2(out)\n", 118 | " out = self.relu(out)\n", 119 | " out = self.fc3(out)\n", 120 | " out = self.tanh(out)\n", 121 | " return(out)\n", 122 | " \n", 123 | "model_d = Discriminator(image_size, hidden_size).to(device)\n", 124 | "model_g = Generator(random_size, hidden_size, image_size).to(device)\n", 125 | "\n", 126 | "criterion = nn.BCELoss() #using cross entropy loss\n", 127 | "optimizer_d = torch.optim.Adam(model_d.parameters(), lr = learning_rate)\n", 128 | "optimizer_g = torch.optim.Adam(model_g.parameters(), lr = learning_rate)\n", 129 | "\n", 130 | "total_step = len(data_loader)\n", 131 | "\n", 132 | "for epoch in range(num_epoch):\n", 133 | " for i, (images, labels) in enumerate(data_loader):\n", 134 | " images = images.reshape(-1, image_size).to(device)\n", 135 | " \n", 136 | " real_labels = torch.ones(batch_size, 1).to(device)\n", 137 | " fake_labels = torch.zeros(batch_size, 1).to(device)\n", 138 | " \n", 139 | " \n", 140 | " #To train the Discriminator\n", 141 | " output_d_real = model_d(images)\n", 142 | " d_real_loss = criterion(output_d_real, real_labels)\n", 143 | " \n", 144 | " z = torch.randn(batch_size, random_size).to(device)\n", 145 | " fake_images = model_g(z)\n", 146 | " output_d_fake = model_d(fake_images)\n", 147 | " d_fake_loss = criterion(output_d_fake, fake_labels)\n", 148 | " d_loss = d_real_loss + d_fake_loss\n", 149 | " \n", 150 | " optimizer_d.zero_grad()\n", 151 | " d_loss.backward()\n", 152 | " optimizer_d.step()#this is going to update only parameters of discriminator\n", 153 | " \n", 154 | " #to train the generator\n", 155 | " z = torch.randn(batch_size, random_size).to(device)\n", 156 | " fake_images = model_g(z)\n", 157 | " outputs = model_d(fake_images)\n", 158 | " \n", 159 | " #to train the generator the output of this should be compared with real_labels. \n", 160 | " #so we compare the output by real label. \n", 161 | " \n", 162 | " g_loss = criterion(outputs, real_labels)\n", 163 | " \n", 164 | " optimizer_g.zero_grad()\n", 165 | " g_loss.backward()\n", 166 | " optimizer_g.step()\n", 167 | " \n", 168 | " if (i+1) % 200 == 0:\n", 169 | " print('Epoch [{}/{}], Step [{}/{}], d_loss: {:.4f}, g_loss: {:.4f} ' \n", 170 | " .format(epoch, num_epoch, i+1, total_step, d_loss.item(), g_loss.item()))\n", 171 | " if (epoch == num_epoch-1):\n", 172 | " fake_image = fake_images.reshape(fake_images.size(0),1,28,28)\n", 173 | " save_image(denorm(fake_images), os.path.join(sample_dir, 'fake_images-{}.png'.format(epoch+1)))\n", 174 | " \n", 175 | " " 176 | ] 177 | }, 178 | { 179 | "cell_type": "code", 180 | "execution_count": 17, 181 | "metadata": {}, 182 | "outputs": [ 183 | { 184 | "name": "stderr", 185 | "output_type": "stream", 186 | "text": [ 187 | "/usr/local/lib/python3.5/dist-packages/torch/serialization.py:241: UserWarning: Couldn't retrieve source code for container of type Generator. It won't be checked for correctness upon loading.\n", 188 | " \"type \" + obj.__name__ + \". It won't be checked \"\n" 189 | ] 190 | } 191 | ], 192 | "source": [ 193 | "torch.save(model_g.state_dict(), 'G.ckpt') #we are saving only the parameters\n", 194 | "torch.save(model_d.state_dict(), 'D.ckpt') #if memory is not an issue then better save the whole model. \n", 195 | "torch.save(model_g, 'G_model.ckpt')#one more important thing is that the device which you have used to train will be only used for testing also(cuda or CPU)" 196 | ] 197 | }, 198 | { 199 | "cell_type": "code", 200 | "execution_count": 16, 201 | "metadata": {}, 202 | "outputs": [ 203 | { 204 | "name": "stdout", 205 | "output_type": "stream", 206 | "text": [ 207 | "OrderedDict([('fc1.weight', tensor([[ 0.0204, -0.0074, 0.0106, ..., -0.0078, 0.0017, 0.0054],\n", 208 | " [ 0.0491, -0.0023, 0.0294, ..., 0.0021, -0.0113, 0.0027],\n", 209 | " [ 0.0573, -0.0045, 0.0484, ..., -0.0216, -0.0126, -0.0022],\n", 210 | " ...,\n", 211 | " [-0.0088, -0.0236, -0.0427, ..., 0.0254, 0.0002, -0.0157],\n", 212 | " [-0.0049, 0.0072, 0.0023, ..., -0.0027, 0.0008, 0.0084],\n", 213 | " [ 0.0249, -0.0165, 0.0199, ..., 0.0613, -0.0519, -0.0095]],\n", 214 | " device='cuda:0')), ('fc1.bias', tensor([-0.0935, -0.1042, -0.0458, -0.1942, -0.1662, -0.1799, -0.1838, 0.0796,\n", 215 | " -0.1442, -0.2187, -0.1250, -0.1063, -0.1021, -0.1250, -0.0782, -0.2127,\n", 216 | " -0.1127, -0.1682, -0.0990, -0.0644, -0.0494, -0.1956, -0.0590, -0.0468,\n", 217 | " -0.1630, -0.0769, -0.1232, -0.0913, -0.0860, -0.1368, -0.0371, -0.0885,\n", 218 | " -0.0736, -0.1410, -0.1843, -0.0407, -0.2049, -0.1075, -0.0249, -0.0482,\n", 219 | " -0.0548, -0.0398, -0.0879, -0.0353, -0.2352, -0.1684, -0.1707, -0.1164,\n", 220 | " -0.0800, -0.0319, -0.0514, -0.1349, -0.1283, -0.1729, -0.1164, -0.0999,\n", 221 | " -0.1893, -0.1635, -0.2028, -0.0941, -0.1761, -0.0672, -0.1793, -0.1666,\n", 222 | " -0.1821, -0.2158, -0.0844, -0.1840, -0.0303, -0.2316, -0.1125, -0.0538,\n", 223 | " -0.2100, -0.1143, -0.1565, -0.2016, -0.1728, -0.2313, -0.0472, -0.1172,\n", 224 | " -0.0825, -0.0648, -0.0684, -0.1542, -0.0715, -0.0310, -0.0850, -0.1648,\n", 225 | " -0.0563, -0.0818, -0.0682, -0.1277, -0.1691, -0.1561, -0.0501, 0.0446,\n", 226 | " -0.1529, -0.0630, -0.1402, -0.1311, -0.1183, -0.1901, -0.1478, -0.0948,\n", 227 | " -0.2099, -0.0782, -0.0953, -0.1590, -0.2191, -0.1385, -0.0625, -0.2217,\n", 228 | " -0.1918, -0.1955, -0.2050, -0.0477, -0.0392, -0.0238, -0.2103, -0.1548,\n", 229 | " -0.1002, -0.0451, -0.1933, -0.1141, -0.0313, -0.1269, -0.0959, -0.1358,\n", 230 | " -0.0873, -0.1374, -0.1114, -0.1889, -0.0724, -0.1071, -0.0923, -0.1055,\n", 231 | " -0.0649, -0.0380, -0.1781, -0.1872, -0.0498, -0.0100, -0.0705, -0.1470,\n", 232 | " -0.1411, -0.1161, -0.1436, -0.0814, -0.1162, -0.0795, -0.0896, -0.1049,\n", 233 | " 0.0579, -0.0763, -0.1580, -0.0318, -0.2293, -0.0401, -0.0820, -0.1420,\n", 234 | " -0.0569, -0.0433, -0.0806, -0.0797, -0.1364, -0.0448, -0.2215, -0.1989,\n", 235 | " -0.0537, -0.2223, -0.1914, -0.0522, -0.0538, -0.1702, -0.1559, -0.0756,\n", 236 | " -0.1500, -0.0536, -0.0593, -0.0975, -0.0493, -0.2211, -0.0917, -0.0361,\n", 237 | " -0.0585, -0.2104, -0.1718, -0.0667, -0.1911, -0.0942, -0.0463, -0.0556,\n", 238 | " -0.0421, -0.0948, -0.0459, -0.1007, -0.1941, 0.0123, -0.1455, -0.1318,\n", 239 | " -0.1376, -0.1491, -0.1021, -0.1023, -0.0572, -0.1994, -0.1569, -0.1394,\n", 240 | " -0.1620, -0.0806, -0.0007, -0.0681, -0.2161, -0.0992, -0.0911, -0.0345,\n", 241 | " -0.1358, -0.1533, -0.0931, -0.0004, -0.0559, -0.0642, -0.1594, -0.0670,\n", 242 | " -0.0226, -0.0763, -0.0963, -0.1268, -0.0691, -0.0805, -0.0334, -0.0300,\n", 243 | " -0.1667, -0.2015, -0.1083, -0.1967, -0.0505, 0.0412, -0.0348, -0.0584,\n", 244 | " -0.1462, -0.1407, -0.1125, -0.0682, -0.2115, -0.0895, -0.0277, -0.0548,\n", 245 | " -0.1531, -0.0463, -0.1328, -0.1178, -0.1609, -0.0373, -0.1322, -0.1329],\n", 246 | " device='cuda:0')), ('fc2.weight', tensor([[ 0.0054, 0.0027, 0.0045, ..., 0.0270, 0.0058, 0.0159],\n", 247 | " [ 0.0504, 0.0559, 0.0623, ..., 0.0716, 0.0389, 0.0516],\n", 248 | " [ 0.1473, 0.0806, 0.2018, ..., -0.1838, -0.0065, 0.0061],\n", 249 | " ...,\n", 250 | " [-0.0606, -0.0019, -0.0537, ..., 0.0489, -0.0370, 0.0819],\n", 251 | " [ 0.0015, -0.0233, 0.0342, ..., 0.0753, -0.0566, -0.0523],\n", 252 | " [ 0.0592, -0.0628, -0.0447, ..., 0.1201, -0.0358, -0.0364]],\n", 253 | " device='cuda:0')), ('fc2.bias', tensor([-0.0613, 0.0255, 0.0234, 0.0403, 0.0448, -0.0202, -0.0043, -0.0022,\n", 254 | " 0.0154, 0.1299, 0.0915, -0.0365, -0.0253, 0.0210, 0.0232, -0.0541,\n", 255 | " 0.0299, -0.0507, 0.1946, 0.0283, 0.1019, 0.0235, -0.0135, 0.2176,\n", 256 | " 0.1342, 0.0415, -0.0676, -0.0143, 0.0753, -0.0497, -0.0361, 0.0234,\n", 257 | " 0.0038, -0.0334, -0.0020, 0.0613, 0.0747, 0.0216, 0.0299, 0.0003,\n", 258 | " 0.0227, -0.0159, 0.1137, 0.0172, -0.0623, 0.0321, 0.0143, 0.0488,\n", 259 | " 0.0201, 0.2275, -0.0076, 0.0022, -0.0235, 0.0214, 0.0854, 0.1174,\n", 260 | " -0.0663, 0.0458, 0.1916, 0.0743, 0.0154, 0.0006, 0.0107, -0.0308,\n", 261 | " 0.0436, -0.0395, -0.0699, -0.0316, 0.0657, 0.0155, -0.0417, 0.1151,\n", 262 | " 0.0069, 0.0391, -0.0326, 0.0447, 0.0177, 0.0277, 0.0108, 0.0779,\n", 263 | " 0.1404, 0.0823, -0.0117, -0.0610, -0.0597, 0.0538, 0.0866, 0.0464,\n", 264 | " 0.0404, -0.0091, -0.0101, 0.0424, -0.0205, -0.0005, 0.0183, 0.0564,\n", 265 | " 0.0060, -0.0331, 0.0583, -0.0688, 0.0320, 0.0379, -0.0457, 0.1410,\n", 266 | " 0.0142, -0.0057, -0.0636, 0.0823, 0.0742, 0.1572, -0.0028, -0.0360,\n", 267 | " -0.0541, 0.0942, 0.0561, -0.0129, 0.0030, 0.0319, 0.0396, 0.0526,\n", 268 | " 0.0110, 0.0197, -0.0340, -0.0593, 0.0911, 0.0028, 0.1351, 0.0855,\n", 269 | " -0.0691, 0.0818, 0.0417, 0.0495, -0.0045, -0.0222, -0.0429, -0.0858,\n", 270 | " 0.0124, 0.0328, 0.0094, 0.0196, 0.0065, 0.1674, -0.0095, 0.0469,\n", 271 | " -0.0609, 0.0353, -0.0831, 0.0322, 0.0403, -0.0651, 0.0871, 0.0174,\n", 272 | " -0.0338, 0.0344, 0.1531, -0.0535, 0.0475, 0.0497, 0.0676, -0.0058,\n", 273 | " -0.0100, 0.0161, 0.0471, -0.0052, -0.0117, -0.0256, 0.1153, -0.0003,\n", 274 | " -0.0327, -0.0346, -0.0033, 0.0849, 0.1284, -0.0697, 0.0091, 0.1899,\n", 275 | " -0.0095, -0.0148, 0.0436, -0.0328, -0.0182, 0.1387, 0.0127, -0.0460,\n", 276 | " 0.0199, 0.0710, 0.2011, 0.0194, 0.0514, 0.0401, 0.0085, 0.0744,\n", 277 | " 0.0295, 0.0261, 0.0152, 0.0006, 0.0537, -0.0319, -0.0401, 0.1119,\n", 278 | " -0.0353, 0.0911, 0.1575, 0.0163, 0.0441, 0.0067, 0.1541, -0.0705,\n", 279 | " 0.0314, -0.0472, -0.0228, 0.0579, 0.0073, 0.0103, -0.0048, 0.0111,\n", 280 | " -0.0904, 0.0963, -0.0436, -0.0320, -0.0027, -0.0321, -0.0306, 0.0791,\n", 281 | " -0.0038, -0.0367, 0.0333, 0.0738, 0.0295, -0.0086, -0.0035, 0.0154,\n", 282 | " 0.0158, 0.0198, -0.0069, 0.0206, -0.0131, -0.0267, 0.0191, 0.0416,\n", 283 | " 0.0464, 0.1962, -0.0451, -0.0161, 0.0952, 0.0243, 0.0390, -0.0318,\n", 284 | " -0.0007, 0.0075, 0.0509, 0.1550, -0.0577, 0.0101, 0.0446, 0.0380],\n", 285 | " device='cuda:0')), ('fc3.weight', tensor([[-0.1492, -0.0610, 0.0273, ..., 0.0452, 0.0148, -0.0877],\n", 286 | " [-0.0954, 0.0590, -0.0632, ..., 0.0868, -0.0370, -0.0224],\n", 287 | " [-0.0201, -0.1648, -0.0643, ..., -0.0797, -0.0438, -0.1373],\n", 288 | " ...,\n", 289 | " [-0.0300, 0.0006, -0.0890, ..., -0.0369, -0.0312, -0.1059],\n", 290 | " [-0.0237, -0.0533, -0.1081, ..., -0.0454, -0.0993, -0.0991],\n", 291 | " [ 0.0216, -0.0690, -0.0440, ..., -0.0580, -0.0363, 0.0306]],\n", 292 | " device='cuda:0')), ('fc3.bias', tensor([-0.0495, -0.0753, -0.1018, -0.1297, -0.0808, -0.0899, -0.0717, -0.0261,\n", 293 | " -0.1950, -0.0722, -0.0018, -0.1645, -0.0691, -0.1630, -0.1245, -0.0792,\n", 294 | " -0.1737, -0.1028, -0.0984, -0.0683, -0.0858, -0.1461, -0.0690, -0.0786,\n", 295 | " -0.1512, -0.0947, -0.1457, -0.1076, -0.1455, -0.1183, -0.1169, -0.0483,\n", 296 | " -0.0628, -0.0445, -0.1277, -0.0741, -0.0473, -0.1675, -0.0188, -0.0737,\n", 297 | " -0.1186, -0.1169, -0.0791, -0.0919, -0.0399, -0.1711, -0.1444, -0.0842,\n", 298 | " -0.1263, -0.1608, -0.1647, -0.1731, -0.1459, -0.0845, -0.1094, -0.0945,\n", 299 | " -0.0969, -0.0305, -0.1084, -0.1125, -0.1405, -0.0451, -0.1467, -0.1103,\n", 300 | " -0.0954, -0.1582, -0.1340, -0.0864, -0.1032, -0.0927, -0.0533, -0.0837,\n", 301 | " -0.1102, -0.0625, -0.1556, -0.1360, -0.0923, -0.0932, -0.1506, -0.1065,\n", 302 | " -0.0941, -0.0671, -0.0627, -0.0782, -0.1808, -0.0357, -0.0719, -0.1152,\n", 303 | " -0.0477, -0.0872, -0.0718, -0.0808, -0.1135, -0.0599, -0.0797, -0.0683,\n", 304 | " -0.1084, -0.0943, -0.0658, -0.0828, -0.0633, -0.0377, -0.0336, -0.0021,\n", 305 | " -0.0502, -0.0409, -0.0723, -0.0773, -0.1398, -0.1096, -0.0805, -0.1116,\n", 306 | " -0.0577, -0.0628, -0.0725, -0.0844, -0.1160, -0.1612, -0.1329, -0.1452,\n", 307 | " -0.0627, -0.0901, -0.0860, -0.0576, -0.0645, -0.0425, -0.1020, -0.0693,\n", 308 | " 0.0170, -0.0802, -0.0459, -0.0321, -0.0536, -0.0853, -0.1067, -0.0649,\n", 309 | " -0.0519, -0.1066, -0.0482, -0.1414, -0.0933, -0.0935, -0.0364, -0.1624,\n", 310 | " -0.1391, -0.0901, -0.1109, -0.0875, -0.0292, -0.1122, -0.0158, -0.0611,\n", 311 | " -0.1119, -0.0657, -0.0275, -0.0426, -0.0350, -0.0455, 0.0029, -0.0134,\n", 312 | " -0.0481, -0.1307, -0.0874, -0.0784, -0.1012, -0.0738, -0.1786, -0.0606,\n", 313 | " -0.1576, -0.1542, -0.0877, -0.0811, -0.1092, -0.0614, -0.0668, -0.0637,\n", 314 | " -0.1534, -0.1101, -0.1283, -0.0294, 0.0270, -0.0448, -0.0605, -0.0341,\n", 315 | " -0.0144, 0.0656, 0.0029, -0.0418, -0.0950, -0.0049, -0.1508, -0.1189,\n", 316 | " -0.1199, -0.0539, -0.0754, -0.1613, -0.1241, -0.0837, -0.1362, -0.0862,\n", 317 | " -0.0681, -0.1520, -0.1018, -0.0371, -0.1295, -0.0869, 0.0084, -0.0595,\n", 318 | " 0.0434, -0.0300, 0.0393, -0.0405, 0.0257, -0.0580, -0.0044, 0.0395,\n", 319 | " -0.0179, -0.0410, -0.0828, -0.0539, -0.0777, -0.1119, -0.0629, -0.1637,\n", 320 | " -0.1290, -0.0681, -0.0824, -0.1001, -0.1623, -0.0716, -0.0170, -0.0377,\n", 321 | " -0.0347, -0.0031, -0.0602, -0.0649, -0.0473, -0.0073, -0.0377, 0.0406,\n", 322 | " -0.0378, -0.0315, -0.0563, -0.0243, -0.0165, -0.0483, -0.0028, -0.1015,\n", 323 | " -0.1135, -0.1458, -0.0735, -0.0621, -0.0443, -0.1047, -0.0520, -0.0725,\n", 324 | " -0.0252, -0.0597, -0.1237, -0.0774, -0.1050, -0.0707, -0.0348, 0.0258,\n", 325 | " -0.0021, -0.0772, -0.0384, 0.0000, -0.0455, -0.0285, 0.0125, 0.0076,\n", 326 | " -0.0552, -0.0023, -0.1359, -0.1005, -0.1599, -0.1048, -0.0777, -0.1158,\n", 327 | " -0.1428, -0.0697, -0.1021, -0.0762, -0.0656, -0.1568, -0.1429, -0.1098,\n", 328 | " -0.0479, -0.0567, -0.0390, -0.0003, -0.0059, -0.0715, -0.0610, 0.0076,\n", 329 | " -0.0587, -0.0451, 0.0532, -0.0642, -0.0571, -0.0686, -0.0952, -0.0619,\n", 330 | " -0.0370, -0.0661, -0.1557, -0.1391, -0.0372, -0.1006, -0.1214, -0.1439,\n", 331 | " -0.0581, -0.1089, -0.1090, -0.0763, -0.1275, -0.0132, 0.0262, -0.0095,\n", 332 | " -0.0640, -0.0581, -0.0469, 0.0134, -0.0519, 0.0254, 0.0194, -0.0601,\n", 333 | " 0.0174, 0.0082, -0.0277, -0.1532, -0.0884, -0.0780, -0.0705, -0.1560,\n", 334 | " -0.1525, -0.1213, -0.0884, -0.0983, -0.0565, -0.0926, -0.0833, -0.0992,\n", 335 | " -0.0435, -0.0060, -0.0066, 0.0290, -0.0313, -0.0073, -0.0626, -0.0683,\n", 336 | " 0.0115, -0.0551, -0.0417, -0.0461, -0.0457, -0.0761, -0.0442, -0.0850,\n", 337 | " -0.1374, -0.0561, -0.0616, -0.1494, -0.0673, -0.1393, -0.1165, -0.1097,\n", 338 | " -0.0826, -0.1180, -0.1240, -0.0231, -0.1113, -0.0951, -0.0814, -0.0250,\n", 339 | " 0.0568, -0.0764, 0.0439, -0.0377, -0.0147, -0.0587, -0.0361, -0.0409,\n", 340 | " -0.0533, -0.0623, -0.0574, -0.0680, -0.1577, -0.0608, -0.1227, -0.0664,\n", 341 | " -0.0492, -0.0253, -0.0981, -0.1566, -0.1305, -0.0976, -0.0666, -0.0709,\n", 342 | " -0.0690, 0.0168, 0.0155, -0.0132, -0.0557, -0.0590, 0.0133, 0.0172,\n", 343 | " -0.0057, 0.0142, -0.0616, 0.0313, -0.0614, -0.0614, -0.0868, -0.1222,\n", 344 | " -0.0607, -0.0658, -0.0679, -0.0939, -0.0675, -0.0819, -0.0957, -0.0196,\n", 345 | " -0.0359, 0.0003, -0.0582, 0.0188, -0.0447, -0.0672, -0.0812, -0.0691,\n", 346 | " -0.0294, -0.0126, 0.0224, -0.0445, -0.0440, -0.0526, -0.0238, -0.0778,\n", 347 | " -0.0596, -0.0177, -0.1127, -0.0411, -0.0364, -0.1130, -0.0190, -0.1635,\n", 348 | " -0.0529, -0.1386, -0.0529, -0.0634, -0.0687, -0.1303, -0.0743, -0.0993,\n", 349 | " -0.0521, 0.0080, -0.0363, -0.1049, -0.0082, -0.0069, -0.0540, -0.0780,\n", 350 | " -0.0732, -0.0711, -0.0123, -0.0704, -0.0065, -0.0536, -0.0651, -0.0350,\n", 351 | " -0.0312, -0.1396, -0.1007, -0.1945, -0.1277, -0.0723, -0.0737, -0.1137,\n", 352 | " -0.1214, -0.0527, -0.0987, -0.0337, 0.0242, -0.0234, 0.0155, -0.0550,\n", 353 | " -0.0018, -0.0844, -0.0503, -0.0457, 0.0232, 0.0094, -0.0377, -0.0655,\n", 354 | " 0.0121, -0.0056, -0.0936, -0.0600, -0.0784, -0.0625, -0.0906, -0.1779,\n", 355 | " -0.0392, -0.0480, -0.1765, -0.1239, -0.1365, -0.0597, -0.1157, -0.0397,\n", 356 | " -0.0230, 0.0067, -0.0151, -0.0020, -0.0259, -0.0407, 0.0180, -0.0759,\n", 357 | " -0.0022, 0.0182, -0.0514, 0.0146, -0.0851, -0.0291, -0.1127, -0.1504,\n", 358 | " -0.0181, -0.0633, -0.1111, -0.0511, -0.0927, -0.0585, -0.1733, -0.1119,\n", 359 | " -0.1038, -0.0060, -0.0104, -0.0111, -0.0656, -0.0311, -0.0659, 0.0057,\n", 360 | " -0.0897, -0.0044, -0.0406, -0.0276, -0.0335, 0.0637, 0.0086, -0.0104,\n", 361 | " -0.0926, -0.1075, -0.0815, -0.0704, -0.0574, -0.1384, -0.0421, -0.0347,\n", 362 | " -0.0546, -0.0930, -0.0284, 0.0121, -0.1460, -0.0566, -0.0413, -0.0405,\n", 363 | " -0.0011, 0.0398, 0.0613, 0.0261, 0.0400, -0.0488, 0.0306, -0.0308,\n", 364 | " -0.0033, 0.0175, -0.0691, -0.0233, -0.1091, -0.0437, -0.0323, -0.1149,\n", 365 | " -0.1263, -0.1460, -0.1255, -0.0784, -0.1691, -0.1634, -0.0452, -0.0840,\n", 366 | " -0.1441, -0.0699, -0.0682, -0.0846, -0.0508, -0.0078, -0.0112, -0.0291,\n", 367 | " 0.0001, -0.0060, 0.0028, 0.0148, -0.0226, 0.0050, 0.0116, -0.0729,\n", 368 | " -0.1043, -0.1221, -0.1068, -0.0677, -0.0686, -0.1528, -0.0924, -0.0917,\n", 369 | " -0.1638, -0.0720, -0.1306, -0.0677, -0.0410, -0.0805, -0.0517, -0.0264,\n", 370 | " 0.0253, 0.0176, 0.0508, -0.0238, -0.0163, -0.0494, 0.0370, -0.0054,\n", 371 | " -0.0294, -0.0670, -0.0519, -0.0266, -0.1198, -0.1045, -0.0536, -0.1169,\n", 372 | " -0.0656, -0.1450, -0.0355, -0.1734, -0.1174, -0.1359, -0.0878, -0.0910,\n", 373 | " -0.0674, -0.1485, -0.1274, -0.1138, -0.0268, -0.0785, -0.0718, 0.0313,\n", 374 | " 0.0096, -0.0599, 0.0167, -0.0691, -0.0530, -0.0660, -0.0199, -0.0642,\n", 375 | " -0.0445, -0.1175, -0.1266, -0.0514, -0.1006, -0.0691, -0.0791, -0.1602,\n", 376 | " -0.1240, -0.0632, -0.0994, -0.0819, -0.1630, -0.1339, -0.0880, -0.1443,\n", 377 | " -0.1135, -0.0919, 0.0042, -0.0326, -0.0736, -0.0177, -0.0586, -0.1043,\n", 378 | " -0.1317, -0.1194, -0.0921, -0.1348, -0.1052, -0.0792, -0.0624, -0.1434,\n", 379 | " -0.0535, -0.1026, 0.0093, -0.1074, -0.1268, -0.1260, -0.1306, -0.1010,\n", 380 | " -0.0998, -0.0889, -0.1103, -0.0083, -0.0359, -0.1584, -0.0740, -0.0945,\n", 381 | " -0.1038, -0.1650, -0.1406, -0.1849, -0.1200, -0.0258, -0.1006, -0.0610,\n", 382 | " -0.1058, -0.1442, -0.1418, -0.0416, -0.1072, -0.1030, -0.1174, -0.0761,\n", 383 | " -0.1297, -0.1312, -0.0925, -0.0691, -0.1533, -0.1151, -0.0365, -0.1737,\n", 384 | " -0.1309, -0.0977, -0.0754, -0.1685, -0.0951, -0.1467, -0.0965, -0.0891,\n", 385 | " -0.0529, -0.1641, -0.1348, -0.1569, -0.0762, -0.1315, -0.0816, -0.0994,\n", 386 | " -0.0551, -0.0692, -0.0588, -0.0627, -0.0755, -0.0168, -0.0666, -0.0842,\n", 387 | " -0.1039, -0.1246, -0.1591, -0.1065, -0.0611, -0.1464, -0.1353, -0.0574,\n", 388 | " -0.1507, -0.1154, -0.0858, -0.1069, -0.0858, -0.0860, -0.0593, -0.0755,\n", 389 | " -0.1327, -0.0936, -0.0921, -0.1021, -0.1229, -0.1307, -0.1694, -0.0877],\n", 390 | " device='cuda:0'))])\n" 391 | ] 392 | } 393 | ], 394 | "source": [ 395 | "g_model = torch.load('G.ckpt') \n", 396 | "d_model = torch.load('D.ckpt')\n", 397 | "\n", 398 | "print(g_model)" 399 | ] 400 | }, 401 | { 402 | "cell_type": "code", 403 | "execution_count": 23, 404 | "metadata": {}, 405 | "outputs": [ 406 | { 407 | "name": "stdout", 408 | "output_type": "stream", 409 | "text": [ 410 | "Generator(\n", 411 | " (fc1): Linear(in_features=64, out_features=256, bias=True)\n", 412 | " (relu): ReLU()\n", 413 | " (fc2): Linear(in_features=256, out_features=256, bias=True)\n", 414 | " (fc3): Linear(in_features=256, out_features=784, bias=True)\n", 415 | " (tanh): Tanh()\n", 416 | ")\n", 417 | "tensor([[-0.8309, -0.7986, -0.9823, ..., -0.9560, -0.9585, -0.8854],\n", 418 | " [-0.9631, -0.9526, -0.9981, ..., -0.9983, -0.9878, -0.9978],\n", 419 | " [-0.8655, -0.7915, -0.9885, ..., -0.9779, -0.9512, -0.9676],\n", 420 | " ...,\n", 421 | " [-0.8522, -0.8073, -0.9909, ..., -0.9791, -0.9622, -0.9327],\n", 422 | " [-0.6143, -0.6682, -0.9381, ..., -0.8887, -0.8573, -0.6515],\n", 423 | " [-0.9929, -0.9685, -1.0000, ..., -0.9999, -0.9998, -0.9992]],\n", 424 | " device='cuda:0', grad_fn=)\n" 425 | ] 426 | } 427 | ], 428 | "source": [ 429 | "model = torch.load('G_model.ckpt')\n", 430 | "print(model)\n", 431 | "z = torch.randn(batch_size, random_size).to(device) #to show that is stores model as well as parameters\n", 432 | "fake_images = model(z)\n", 433 | "print(fake_images)" 434 | ] 435 | }, 436 | { 437 | "cell_type": "code", 438 | "execution_count": null, 439 | "metadata": {}, 440 | "outputs": [], 441 | "source": [] 442 | } 443 | ], 444 | "metadata": { 445 | "kernelspec": { 446 | "display_name": "Python 3", 447 | "language": "python", 448 | "name": "python3" 449 | }, 450 | "language_info": { 451 | "codemirror_mode": { 452 | "name": "ipython", 453 | "version": 3 454 | }, 455 | "file_extension": ".py", 456 | "mimetype": "text/x-python", 457 | "name": "python", 458 | "nbconvert_exporter": "python", 459 | "pygments_lexer": "ipython3", 460 | "version": "3.5.2" 461 | } 462 | }, 463 | "nbformat": 4, 464 | "nbformat_minor": 2 465 | } 466 | -------------------------------------------------------------------------------- /Initialize weight of parameter of model by yourself.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 2, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import torch\n", 10 | "import torch.nn as nn\n", 11 | "import torchvision\n", 12 | "import torchvision.transforms as transforms\n", 13 | "import torch.nn.functional as F\n", 14 | "import torch.optim as optim\n", 15 | "from torch.autograd import Variable, Function" 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": 9, 21 | "metadata": {}, 22 | "outputs": [ 23 | { 24 | "name": "stdout", 25 | "output_type": "stream", 26 | "text": [ 27 | "tensor([[0.5000, 0.5000]], dtype=torch.float64, grad_fn=)\n", 28 | "\n", 29 | " Loss:\n", 30 | "tensor(0.6931, dtype=torch.float64, grad_fn=)\n", 31 | "\n", 32 | " FC gradients :\n", 33 | "tensor([[-0.7000, -0.9750, -0.6000, -0.8250, -1.1250, -0.6750, -0.4000, -0.5250,\n", 34 | " -0.3000],\n", 35 | " [ 0.7000, 0.9750, 0.6000, 0.8250, 1.1250, 0.6750, 0.4000, 0.5250,\n", 36 | " 0.3000]], dtype=torch.float64)\n", 37 | "\n", 38 | "Conv Gradients :\n", 39 | "tensor([[[[0., 0., 0.],\n", 40 | " [0., 0., 0.],\n", 41 | " [0., 0., 0.]]]], dtype=torch.float64)\n", 42 | "\n", 43 | "Updated Conv Weights :\n", 44 | "Parameter containing:\n", 45 | "tensor([[[[0.1000, 0.2000, 0.3000],\n", 46 | " [0.4000, 0.5000, 0.6000],\n", 47 | " [0.7000, 0.8000, 0.9000]]]],\n", 48 | " dtype=torch.float64, requires_grad=True)\n", 49 | "\n", 50 | "Updated FC Weights :\n", 51 | "Parameter containing:\n", 52 | "tensor([[0.1007, 0.2010, 0.3006, 0.4008, 0.5011, 0.6007, 0.7004, 0.8005, 0.9003],\n", 53 | " [0.0993, 0.1990, 0.2994, 0.3992, 0.4989, 0.5993, 0.6996, 0.7995, 0.8997]],\n", 54 | " dtype=torch.float64, requires_grad=True)\n", 55 | "tensor([[0.5095, 0.4905]], dtype=torch.float64, grad_fn=)\n", 56 | "\n", 57 | " Loss:\n", 58 | "tensor(0.6837, dtype=torch.float64, grad_fn=)\n", 59 | "tensor([[0.5273, 0.4727]], dtype=torch.float64, grad_fn=)\n", 60 | "\n", 61 | " Loss:\n", 62 | "tensor(0.6662, dtype=torch.float64, grad_fn=)\n", 63 | "tensor([[0.5524, 0.4476]], dtype=torch.float64, grad_fn=)\n", 64 | "\n", 65 | " Loss:\n", 66 | "tensor(0.6421, dtype=torch.float64, grad_fn=)\n", 67 | "tensor([[0.5834, 0.4166]], dtype=torch.float64, grad_fn=)\n", 68 | "\n", 69 | " Loss:\n", 70 | "tensor(0.6132, dtype=torch.float64, grad_fn=)\n", 71 | "tensor([[0.6188, 0.3812]], dtype=torch.float64, grad_fn=)\n", 72 | "\n", 73 | " Loss:\n", 74 | "tensor(0.5814, dtype=torch.float64, grad_fn=)\n", 75 | "tensor([[0.6567, 0.3433]], dtype=torch.float64, grad_fn=)\n", 76 | "\n", 77 | " Loss:\n", 78 | "tensor(0.5487, dtype=torch.float64, grad_fn=)\n", 79 | "tensor([[0.6954, 0.3046]], dtype=torch.float64, grad_fn=)\n", 80 | "\n", 81 | " Loss:\n", 82 | "tensor(0.5167, dtype=torch.float64, grad_fn=)\n", 83 | "tensor([[0.7331, 0.2669]], dtype=torch.float64, grad_fn=)\n", 84 | "\n", 85 | " Loss:\n", 86 | "tensor(0.4870, dtype=torch.float64, grad_fn=)\n", 87 | "tensor([[0.7685, 0.2315]], dtype=torch.float64, grad_fn=)\n", 88 | "\n", 89 | " Loss:\n", 90 | "tensor(0.4602, dtype=torch.float64, grad_fn=)\n", 91 | "tensor([[0.8007, 0.1993]], dtype=torch.float64, grad_fn=)\n", 92 | "\n", 93 | " Loss:\n", 94 | "tensor(0.4370, dtype=torch.float64, grad_fn=)\n", 95 | "tensor([[0.8292, 0.1708]], dtype=torch.float64, grad_fn=)\n", 96 | "\n", 97 | " Loss:\n", 98 | "tensor(0.4172, dtype=torch.float64, grad_fn=)\n", 99 | "tensor([[0.8539, 0.1461]], dtype=torch.float64, grad_fn=)\n", 100 | "\n", 101 | " Loss:\n", 102 | "tensor(0.4006, dtype=torch.float64, grad_fn=)\n", 103 | "tensor([[0.8749, 0.1251]], dtype=torch.float64, grad_fn=)\n", 104 | "\n", 105 | " Loss:\n", 106 | "tensor(0.3870, dtype=torch.float64, grad_fn=)\n", 107 | "tensor([[0.8926, 0.1074]], dtype=torch.float64, grad_fn=)\n", 108 | "\n", 109 | " Loss:\n", 110 | "tensor(0.3757, dtype=torch.float64, grad_fn=)\n", 111 | "tensor([[0.9074, 0.0926]], dtype=torch.float64, grad_fn=)\n", 112 | "\n", 113 | " Loss:\n", 114 | "tensor(0.3666, dtype=torch.float64, grad_fn=)\n", 115 | "tensor([[0.9197, 0.0803]], dtype=torch.float64, grad_fn=)\n", 116 | "\n", 117 | " Loss:\n", 118 | "tensor(0.3591, dtype=torch.float64, grad_fn=)\n", 119 | "tensor([[0.9300, 0.0700]], dtype=torch.float64, grad_fn=)\n", 120 | "\n", 121 | " Loss:\n", 122 | "tensor(0.3529, dtype=torch.float64, grad_fn=)\n", 123 | "tensor([[0.9385, 0.0615]], dtype=torch.float64, grad_fn=)\n", 124 | "\n", 125 | " Loss:\n", 126 | "tensor(0.3479, dtype=torch.float64, grad_fn=)\n", 127 | "tensor([[0.9456, 0.0544]], dtype=torch.float64, grad_fn=)\n", 128 | "\n", 129 | " Loss:\n", 130 | "tensor(0.3437, dtype=torch.float64, grad_fn=)\n", 131 | "tensor([[0.9516, 0.0484]], dtype=torch.float64, grad_fn=)\n", 132 | "\n", 133 | " Loss:\n", 134 | "tensor(0.3402, dtype=torch.float64, grad_fn=)\n", 135 | "tensor([[0.9566, 0.0434]], dtype=torch.float64, grad_fn=)\n", 136 | "\n", 137 | " Loss:\n", 138 | "tensor(0.3374, dtype=torch.float64, grad_fn=)\n", 139 | "tensor([[0.9608, 0.0392]], dtype=torch.float64, grad_fn=)\n", 140 | "\n", 141 | " Loss:\n", 142 | "tensor(0.3349, dtype=torch.float64, grad_fn=)\n", 143 | "tensor([[0.9644, 0.0356]], dtype=torch.float64, grad_fn=)\n", 144 | "\n", 145 | " Loss:\n", 146 | "tensor(0.3329, dtype=torch.float64, grad_fn=)\n", 147 | "tensor([[0.9675, 0.0325]], dtype=torch.float64, grad_fn=)\n", 148 | "\n", 149 | " Loss:\n", 150 | "tensor(0.3312, dtype=torch.float64, grad_fn=)\n", 151 | "tensor([[0.9701, 0.0299]], dtype=torch.float64, grad_fn=)\n", 152 | "\n", 153 | " Loss:\n", 154 | "tensor(0.3297, dtype=torch.float64, grad_fn=)\n", 155 | "tensor([[0.9724, 0.0276]], dtype=torch.float64, grad_fn=)\n", 156 | "\n", 157 | " Loss:\n", 158 | "tensor(0.3284, dtype=torch.float64, grad_fn=)\n", 159 | "tensor([[0.9743, 0.0257]], dtype=torch.float64, grad_fn=)\n", 160 | "\n", 161 | " Loss:\n", 162 | "tensor(0.3273, dtype=torch.float64, grad_fn=)\n", 163 | "tensor([[0.9760, 0.0240]], dtype=torch.float64, grad_fn=)\n", 164 | "\n", 165 | " Loss:\n", 166 | "tensor(0.3264, dtype=torch.float64, grad_fn=)\n", 167 | "tensor([[0.9775, 0.0225]], dtype=torch.float64, grad_fn=)\n", 168 | "\n", 169 | " Loss:\n", 170 | "tensor(0.3256, dtype=torch.float64, grad_fn=)\n", 171 | "tensor([[0.9788, 0.0212]], dtype=torch.float64, grad_fn=)\n", 172 | "\n", 173 | " Loss:\n", 174 | "tensor(0.3249, dtype=torch.float64, grad_fn=)\n", 175 | "tensor([[0.9799, 0.0201]], dtype=torch.float64, grad_fn=)\n", 176 | "\n", 177 | " Loss:\n", 178 | "tensor(0.3242, dtype=torch.float64, grad_fn=)\n", 179 | "tensor([[0.9809, 0.0191]], dtype=torch.float64, grad_fn=)\n", 180 | "\n", 181 | " Loss:\n", 182 | "tensor(0.3237, dtype=torch.float64, grad_fn=)\n", 183 | "tensor([[0.9818, 0.0182]], dtype=torch.float64, grad_fn=)\n", 184 | "\n", 185 | " Loss:\n", 186 | "tensor(0.3232, dtype=torch.float64, grad_fn=)\n", 187 | "tensor([[0.9826, 0.0174]], dtype=torch.float64, grad_fn=)\n", 188 | "\n", 189 | " Loss:\n", 190 | "tensor(0.3227, dtype=torch.float64, grad_fn=)\n", 191 | "tensor([[0.9833, 0.0167]], dtype=torch.float64, grad_fn=)\n", 192 | "\n", 193 | " Loss:\n", 194 | "tensor(0.3223, dtype=torch.float64, grad_fn=)\n", 195 | "tensor([[0.9840, 0.0160]], dtype=torch.float64, grad_fn=)\n", 196 | "\n", 197 | " Loss:\n", 198 | "tensor(0.3220, dtype=torch.float64, grad_fn=)\n", 199 | "tensor([[0.9846, 0.0154]], dtype=torch.float64, grad_fn=)\n", 200 | "\n", 201 | " Loss:\n", 202 | "tensor(0.3217, dtype=torch.float64, grad_fn=)\n", 203 | "tensor([[0.9851, 0.0149]], dtype=torch.float64, grad_fn=)\n", 204 | "\n", 205 | " Loss:\n", 206 | "tensor(0.3214, dtype=torch.float64, grad_fn=)\n", 207 | "tensor([[0.9856, 0.0144]], dtype=torch.float64, grad_fn=)\n", 208 | "\n", 209 | " Loss:\n", 210 | "tensor(0.3211, dtype=torch.float64, grad_fn=)\n", 211 | "tensor([[0.9860, 0.0140]], dtype=torch.float64, grad_fn=)\n", 212 | "\n", 213 | " Loss:\n", 214 | "tensor(0.3209, dtype=torch.float64, grad_fn=)\n", 215 | "tensor([[0.9864, 0.0136]], dtype=torch.float64, grad_fn=)\n", 216 | "\n", 217 | " Loss:\n", 218 | "tensor(0.3207, dtype=torch.float64, grad_fn=)\n", 219 | "tensor([[0.9867, 0.0133]], dtype=torch.float64, grad_fn=)\n", 220 | "\n", 221 | " Loss:\n", 222 | "tensor(0.3205, dtype=torch.float64, grad_fn=)\n", 223 | "tensor([[0.9871, 0.0129]], dtype=torch.float64, grad_fn=)\n", 224 | "\n", 225 | " Loss:\n", 226 | "tensor(0.3203, dtype=torch.float64, grad_fn=)\n", 227 | "tensor([[0.9874, 0.0126]], dtype=torch.float64, grad_fn=)\n", 228 | "\n", 229 | " Loss:\n", 230 | "tensor(0.3201, dtype=torch.float64, grad_fn=)\n", 231 | "tensor([[0.9877, 0.0123]], dtype=torch.float64, grad_fn=)\n", 232 | "\n", 233 | " Loss:\n", 234 | "tensor(0.3200, dtype=torch.float64, grad_fn=)\n", 235 | "tensor([[0.9879, 0.0121]], dtype=torch.float64, grad_fn=)\n", 236 | "\n", 237 | " Loss:\n", 238 | "tensor(0.3198, dtype=torch.float64, grad_fn=)\n", 239 | "tensor([[0.9882, 0.0118]], dtype=torch.float64, grad_fn=)\n", 240 | "\n", 241 | " Loss:\n", 242 | "tensor(0.3197, dtype=torch.float64, grad_fn=)\n", 243 | "tensor([[0.9884, 0.0116]], dtype=torch.float64, grad_fn=)\n", 244 | "\n", 245 | " Loss:\n", 246 | "tensor(0.3195, dtype=torch.float64, grad_fn=)\n", 247 | "tensor([[0.9886, 0.0114]], dtype=torch.float64, grad_fn=)\n", 248 | "\n", 249 | " Loss:\n", 250 | "tensor(0.3194, dtype=torch.float64, grad_fn=)\n", 251 | "tensor([[0.9888, 0.0112]], dtype=torch.float64, grad_fn=)\n", 252 | "\n", 253 | " Loss:\n", 254 | "tensor(0.3193, dtype=torch.float64, grad_fn=)\n", 255 | "tensor([[0.9890, 0.0110]], dtype=torch.float64, grad_fn=)\n", 256 | "\n", 257 | " Loss:\n", 258 | "tensor(0.3192, dtype=torch.float64, grad_fn=)\n", 259 | "tensor([[0.9892, 0.0108]], dtype=torch.float64, grad_fn=)\n", 260 | "\n", 261 | " Loss:\n", 262 | "tensor(0.3191, dtype=torch.float64, grad_fn=)\n", 263 | "tensor([[0.9894, 0.0106]], dtype=torch.float64, grad_fn=)\n", 264 | "\n", 265 | " Loss:\n", 266 | "tensor(0.3190, dtype=torch.float64, grad_fn=)\n", 267 | "tensor([[0.9895, 0.0105]], dtype=torch.float64, grad_fn=)\n", 268 | "\n", 269 | " Loss:\n", 270 | "tensor(0.3189, dtype=torch.float64, grad_fn=)\n", 271 | "tensor([[0.9897, 0.0103]], dtype=torch.float64, grad_fn=)\n", 272 | "\n", 273 | " Loss:\n", 274 | "tensor(0.3189, dtype=torch.float64, grad_fn=)\n", 275 | "tensor([[0.9898, 0.0102]], dtype=torch.float64, grad_fn=)\n", 276 | "\n", 277 | " Loss:\n", 278 | "tensor(0.3188, dtype=torch.float64, grad_fn=)\n", 279 | "tensor([[0.9900, 0.0100]], dtype=torch.float64, grad_fn=)\n", 280 | "\n", 281 | " Loss:\n", 282 | "tensor(0.3187, dtype=torch.float64, grad_fn=)\n", 283 | "tensor([[0.9901, 0.0099]], dtype=torch.float64, grad_fn=)\n", 284 | "\n", 285 | " Loss:\n", 286 | "tensor(0.3186, dtype=torch.float64, grad_fn=)\n", 287 | "tensor([[0.9902, 0.0098]], dtype=torch.float64, grad_fn=)\n", 288 | "\n", 289 | " Loss:\n", 290 | "tensor(0.3186, dtype=torch.float64, grad_fn=)\n", 291 | "tensor([[0.9903, 0.0097]], dtype=torch.float64, grad_fn=)\n", 292 | "\n", 293 | " Loss:\n", 294 | "tensor(0.3185, dtype=torch.float64, grad_fn=)\n", 295 | "tensor([[0.9904, 0.0096]], dtype=torch.float64, grad_fn=)\n", 296 | "\n", 297 | " Loss:\n", 298 | "tensor(0.3184, dtype=torch.float64, grad_fn=)\n", 299 | "tensor([[0.9906, 0.0094]], dtype=torch.float64, grad_fn=)\n", 300 | "\n", 301 | " Loss:\n", 302 | "tensor(0.3184, dtype=torch.float64, grad_fn=)\n", 303 | "tensor([[0.9907, 0.0093]], dtype=torch.float64, grad_fn=)\n", 304 | "\n", 305 | " Loss:\n", 306 | "tensor(0.3183, dtype=torch.float64, grad_fn=)\n", 307 | "tensor([[0.9908, 0.0092]], dtype=torch.float64, grad_fn=)\n", 308 | "\n", 309 | " Loss:\n", 310 | "tensor(0.3183, dtype=torch.float64, grad_fn=)\n", 311 | "tensor([[0.9909, 0.0091]], dtype=torch.float64, grad_fn=)\n", 312 | "\n", 313 | " Loss:\n", 314 | "tensor(0.3182, dtype=torch.float64, grad_fn=)\n", 315 | "tensor([[0.9910, 0.0090]], dtype=torch.float64, grad_fn=)\n", 316 | "\n", 317 | " Loss:\n", 318 | "tensor(0.3182, dtype=torch.float64, grad_fn=)\n", 319 | "tensor([[0.9910, 0.0090]], dtype=torch.float64, grad_fn=)\n", 320 | "\n", 321 | " Loss:\n", 322 | "tensor(0.3181, dtype=torch.float64, grad_fn=)\n", 323 | "tensor([[0.9911, 0.0089]], dtype=torch.float64, grad_fn=)\n", 324 | "\n", 325 | " Loss:\n", 326 | "tensor(0.3181, dtype=torch.float64, grad_fn=)\n", 327 | "tensor([[0.9912, 0.0088]], dtype=torch.float64, grad_fn=)\n", 328 | "\n", 329 | " Loss:\n", 330 | "tensor(0.3180, dtype=torch.float64, grad_fn=)\n", 331 | "tensor([[0.9913, 0.0087]], dtype=torch.float64, grad_fn=)\n", 332 | "\n", 333 | " Loss:\n", 334 | "tensor(0.3180, dtype=torch.float64, grad_fn=)\n", 335 | "tensor([[0.9914, 0.0086]], dtype=torch.float64, grad_fn=)\n", 336 | "\n", 337 | " Loss:\n", 338 | "tensor(0.3179, dtype=torch.float64, grad_fn=)\n", 339 | "tensor([[0.9915, 0.0085]], dtype=torch.float64, grad_fn=)\n", 340 | "\n", 341 | " Loss:\n", 342 | "tensor(0.3179, dtype=torch.float64, grad_fn=)\n", 343 | "tensor([[0.9915, 0.0085]], dtype=torch.float64, grad_fn=)\n", 344 | "\n", 345 | " Loss:\n", 346 | "tensor(0.3178, dtype=torch.float64, grad_fn=)\n", 347 | "tensor([[0.9916, 0.0084]], dtype=torch.float64, grad_fn=)\n", 348 | "\n", 349 | " Loss:\n", 350 | "tensor(0.3178, dtype=torch.float64, grad_fn=)\n", 351 | "tensor([[0.9917, 0.0083]], dtype=torch.float64, grad_fn=)\n", 352 | "\n", 353 | " Loss:\n", 354 | "tensor(0.3178, dtype=torch.float64, grad_fn=)\n", 355 | "tensor([[0.9918, 0.0082]], dtype=torch.float64, grad_fn=)\n", 356 | "\n", 357 | " Loss:\n", 358 | "tensor(0.3177, dtype=torch.float64, grad_fn=)\n", 359 | "tensor([[0.9918, 0.0082]], dtype=torch.float64, grad_fn=)\n", 360 | "\n", 361 | " Loss:\n", 362 | "tensor(0.3177, dtype=torch.float64, grad_fn=)\n", 363 | "tensor([[0.9919, 0.0081]], dtype=torch.float64, grad_fn=)\n", 364 | "\n", 365 | " Loss:\n", 366 | "tensor(0.3176, dtype=torch.float64, grad_fn=)\n", 367 | "tensor([[0.9920, 0.0080]], dtype=torch.float64, grad_fn=)\n", 368 | "\n", 369 | " Loss:\n", 370 | "tensor(0.3176, dtype=torch.float64, grad_fn=)\n", 371 | "tensor([[0.9920, 0.0080]], dtype=torch.float64, grad_fn=)\n", 372 | "\n", 373 | " Loss:\n", 374 | "tensor(0.3176, dtype=torch.float64, grad_fn=)\n", 375 | "tensor([[0.9921, 0.0079]], dtype=torch.float64, grad_fn=)\n", 376 | "\n", 377 | " Loss:\n", 378 | "tensor(0.3175, dtype=torch.float64, grad_fn=)\n", 379 | "tensor([[0.9922, 0.0078]], dtype=torch.float64, grad_fn=)\n", 380 | "\n", 381 | " Loss:\n", 382 | "tensor(0.3175, dtype=torch.float64, grad_fn=)\n", 383 | "tensor([[0.9922, 0.0078]], dtype=torch.float64, grad_fn=)\n", 384 | "\n", 385 | " Loss:\n", 386 | "tensor(0.3175, dtype=torch.float64, grad_fn=)\n", 387 | "tensor([[0.9923, 0.0077]], dtype=torch.float64, grad_fn=)\n", 388 | "\n", 389 | " Loss:\n", 390 | "tensor(0.3174, dtype=torch.float64, grad_fn=)\n", 391 | "tensor([[0.9924, 0.0076]], dtype=torch.float64, grad_fn=)\n", 392 | "\n", 393 | " Loss:\n", 394 | "tensor(0.3174, dtype=torch.float64, grad_fn=)\n", 395 | "tensor([[0.9924, 0.0076]], dtype=torch.float64, grad_fn=)\n", 396 | "\n", 397 | " Loss:\n", 398 | "tensor(0.3174, dtype=torch.float64, grad_fn=)\n", 399 | "tensor([[0.9925, 0.0075]], dtype=torch.float64, grad_fn=)\n", 400 | "\n", 401 | " Loss:\n", 402 | "tensor(0.3173, dtype=torch.float64, grad_fn=)\n", 403 | "tensor([[0.9925, 0.0075]], dtype=torch.float64, grad_fn=)\n", 404 | "\n", 405 | " Loss:\n", 406 | "tensor(0.3173, dtype=torch.float64, grad_fn=)\n", 407 | "tensor([[0.9926, 0.0074]], dtype=torch.float64, grad_fn=)\n", 408 | "\n", 409 | " Loss:\n", 410 | "tensor(0.3173, dtype=torch.float64, grad_fn=)\n", 411 | "tensor([[0.9926, 0.0074]], dtype=torch.float64, grad_fn=)\n", 412 | "\n", 413 | " Loss:\n", 414 | "tensor(0.3172, dtype=torch.float64, grad_fn=)\n", 415 | "tensor([[0.9927, 0.0073]], dtype=torch.float64, grad_fn=)\n", 416 | "\n", 417 | " Loss:\n", 418 | "tensor(0.3172, dtype=torch.float64, grad_fn=)\n", 419 | "tensor([[0.9927, 0.0073]], dtype=torch.float64, grad_fn=)\n", 420 | "\n", 421 | " Loss:\n", 422 | "tensor(0.3172, dtype=torch.float64, grad_fn=)\n", 423 | "tensor([[0.9928, 0.0072]], dtype=torch.float64, grad_fn=)\n", 424 | "\n", 425 | " Loss:\n", 426 | "tensor(0.3172, dtype=torch.float64, grad_fn=)\n", 427 | "tensor([[0.9928, 0.0072]], dtype=torch.float64, grad_fn=)\n", 428 | "\n", 429 | " Loss:\n", 430 | "tensor(0.3171, dtype=torch.float64, grad_fn=)\n", 431 | "tensor([[0.9929, 0.0071]], dtype=torch.float64, grad_fn=)\n", 432 | "\n", 433 | " Loss:\n", 434 | "tensor(0.3171, dtype=torch.float64, grad_fn=)\n", 435 | "tensor([[0.9929, 0.0071]], dtype=torch.float64, grad_fn=)\n", 436 | "\n", 437 | " Loss:\n", 438 | "tensor(0.3171, dtype=torch.float64, grad_fn=)\n", 439 | "tensor([[0.9930, 0.0070]], dtype=torch.float64, grad_fn=)\n", 440 | "\n", 441 | " Loss:\n", 442 | "tensor(0.3170, dtype=torch.float64, grad_fn=)\n", 443 | "tensor([[0.9930, 0.0070]], dtype=torch.float64, grad_fn=)\n", 444 | "\n", 445 | " Loss:\n", 446 | "tensor(0.3170, dtype=torch.float64, grad_fn=)\n", 447 | "tensor([[0.9931, 0.0069]], dtype=torch.float64, grad_fn=)\n", 448 | "\n", 449 | " Loss:\n", 450 | "tensor(0.3170, dtype=torch.float64, grad_fn=)\n", 451 | "\n", 452 | " FC gradients :\n", 453 | "tensor([[-0.0106, -0.0148, -0.0091, -0.0126, -0.0171, -0.0103, -0.0062, -0.0081,\n", 454 | " -0.0047],\n", 455 | " [ 0.0106, 0.0148, 0.0091, 0.0126, 0.0171, 0.0103, 0.0062, 0.0081,\n", 456 | " 0.0047]], dtype=torch.float64)\n", 457 | "\n", 458 | "Conv Gradients :\n", 459 | "tensor([[[[-0.0025, -0.0037, -0.0028],\n", 460 | " [-0.0040, -0.0059, -0.0044],\n", 461 | " [-0.0032, -0.0047, -0.0035]]]], dtype=torch.float64)\n", 462 | "\n", 463 | "Updated Conv Weights :\n", 464 | "Parameter containing:\n", 465 | "tensor([[[[0.1059, 0.2086, 0.3064],\n", 466 | " [0.4094, 0.5137, 0.6102],\n", 467 | " [0.7075, 0.8109, 0.9081]]]],\n", 468 | " dtype=torch.float64, requires_grad=True)\n", 469 | "\n", 470 | "Updated FC Weights :\n", 471 | "Parameter containing:\n", 472 | "tensor([[0.1900, 0.3253, 0.3772, 0.5061, 0.6447, 0.6869, 0.7515, 0.8676, 0.9387],\n", 473 | " [0.0100, 0.0747, 0.2228, 0.2939, 0.3553, 0.5131, 0.6485, 0.7324, 0.8613]],\n", 474 | " dtype=torch.float64, requires_grad=True)\n" 475 | ] 476 | }, 477 | { 478 | "name": "stderr", 479 | "output_type": "stream", 480 | "text": [ 481 | "/home/harsh/.local/lib/python3.5/site-packages/ipykernel_launcher.py:25: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n" 482 | ] 483 | } 484 | ], 485 | "source": [ 486 | "import numpy as np\n", 487 | "\n", 488 | "class Net(nn.Module):\n", 489 | " def __init__(self):\n", 490 | " super(Net, self).__init__()\n", 491 | " \n", 492 | " self.conv = nn.Conv2d(1,1,kernel_size=(3,3), stride = 1, padding = 1)\n", 493 | " self.conv.weight = torch.nn.Parameter(data = torch.DoubleTensor([[[[0.1,0.2,0.3],[0.4,0.5,0.6],[0.7,0.8,0.9]]]]), requires_grad =True) #since the input channel and output channel is 1, dimension of filter is (1x1x3x3)\n", 494 | " self.conv.bias = torch.nn.Parameter(data = torch.DoubleTensor([0]), requires_grad=True)\n", 495 | " self.fc = nn.Linear(9,2)\n", 496 | " self.fc.weight = torch.nn.Parameter(data = torch.DoubleTensor([[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9],[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9]]), requires_grad = True)\n", 497 | " self.fc.bias = torch.nn.Parameter(data=torch.DoubleTensor([0.0]), requires_grad=True)\n", 498 | " self.relu = nn.ReLU()\n", 499 | " self.softmax = nn.Softmax()\n", 500 | " def forward(self, x):\n", 501 | " x = self.conv(x)\n", 502 | "# print(\"\\nThe output of conv is\")\n", 503 | "# print(x)\n", 504 | " x = self.relu(x)\n", 505 | " x = torch.reshape(x,(-1,9))\n", 506 | " x = self.fc(x)\n", 507 | "# print(\"\\nThe output of FC is\")\n", 508 | "# print(x)\n", 509 | " x = self.relu(x)\n", 510 | " x = self.softmax(x)\n", 511 | " return x\n", 512 | "img = torch.from_numpy(np.ones((1,1,3,3))) #or torch.ones((1,1,3,3))\n", 513 | "label = torch.LongTensor([0])\n", 514 | "\n", 515 | "net = Net()\n", 516 | "# print(net)\n", 517 | "criterion = nn.CrossEntropyLoss()\n", 518 | "optimizer = torch.optim.SGD(net.parameters(), lr = 0.001, momentum = 0.9)\n", 519 | "\n", 520 | "for i in range(100):\n", 521 | " output = net(img)\n", 522 | " print(output)\n", 523 | " \n", 524 | " loss = criterion(output, label)\n", 525 | " optimizer.zero_grad()\n", 526 | " loss.backward()\n", 527 | " optimizer.step()\n", 528 | " print(\"\\n Loss:\")\n", 529 | " print(loss)\n", 530 | " if(i==0 or i == 99):\n", 531 | "\n", 532 | " print(\"\\n FC gradients :\")\n", 533 | " print(net.fc.weight.grad)\n", 534 | "\n", 535 | " print(\"\\nConv Gradients :\")\n", 536 | " print(net.conv.weight.grad)\n", 537 | " print(\"\\nUpdated Conv Weights :\")\n", 538 | " print(net.conv.weight)\n", 539 | "\n", 540 | " print(\"\\nUpdated FC Weights :\")\n", 541 | " print(net.fc.weight)\n", 542 | " " 543 | ] 544 | } 545 | ], 546 | "metadata": { 547 | "kernelspec": { 548 | "display_name": "Python 3", 549 | "language": "python", 550 | "name": "python3" 551 | }, 552 | "language_info": { 553 | "codemirror_mode": { 554 | "name": "ipython", 555 | "version": 3 556 | }, 557 | "file_extension": ".py", 558 | "mimetype": "text/x-python", 559 | "name": "python", 560 | "nbconvert_exporter": "python", 561 | "pygments_lexer": "ipython3", 562 | "version": "3.5.2" 563 | } 564 | }, 565 | "nbformat": 4, 566 | "nbformat_minor": 2 567 | } 568 | -------------------------------------------------------------------------------- /Logistic regression and Neural Network.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import torch\n", 10 | "import torch.nn as nn\n", 11 | "import torchvision\n", 12 | "import numpy as np\n", 13 | "import torchvision.transforms as transforms\n", 14 | "import matplotlib.pyplot as plt" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": 2, 20 | "metadata": {}, 21 | "outputs": [ 22 | { 23 | "name": "stdout", 24 | "output_type": "stream", 25 | "text": [ 26 | "60000\n", 27 | "600\n", 28 | "Epoch [1/20], Iteration [100/600], Loss: 2.1939\n", 29 | "Epoch [1/20], Iteration [200/600], Loss: 2.0933\n", 30 | "Epoch [1/20], Iteration [300/600], Loss: 1.9931\n", 31 | "Epoch [1/20], Iteration [400/600], Loss: 1.9060\n", 32 | "Epoch [1/20], Iteration [500/600], Loss: 1.8546\n", 33 | "Epoch [1/20], Iteration [600/600], Loss: 1.8569\n", 34 | "Epoch [6/20], Iteration [100/600], Loss: 1.1003\n", 35 | "Epoch [6/20], Iteration [200/600], Loss: 0.9031\n", 36 | "Epoch [6/20], Iteration [300/600], Loss: 0.9449\n", 37 | "Epoch [6/20], Iteration [400/600], Loss: 0.9166\n", 38 | "Epoch [6/20], Iteration [500/600], Loss: 0.8977\n", 39 | "Epoch [6/20], Iteration [600/600], Loss: 0.8527\n", 40 | "Epoch [11/20], Iteration [100/600], Loss: 0.7223\n", 41 | "Epoch [11/20], Iteration [200/600], Loss: 0.6693\n", 42 | "Epoch [11/20], Iteration [300/600], Loss: 0.7644\n", 43 | "Epoch [11/20], Iteration [400/600], Loss: 0.7312\n", 44 | "Epoch [11/20], Iteration [500/600], Loss: 0.8660\n", 45 | "Epoch [11/20], Iteration [600/600], Loss: 0.6407\n", 46 | "Epoch [16/20], Iteration [100/600], Loss: 0.6034\n", 47 | "Epoch [16/20], Iteration [200/600], Loss: 0.6197\n", 48 | "Epoch [16/20], Iteration [300/600], Loss: 0.6956\n", 49 | "Epoch [16/20], Iteration [400/600], Loss: 0.6883\n", 50 | "Epoch [16/20], Iteration [500/600], Loss: 0.5473\n", 51 | "Epoch [16/20], Iteration [600/600], Loss: 0.5219\n", 52 | "Accuracy of the model on the 10000 test images: 87 %\n" 53 | ] 54 | } 55 | ], 56 | "source": [ 57 | "input_size = 784#here we are going to use PyTorch MNIST dataset. Will talk of custom dataset afterwards. Size of image is 28x28, hence if we make them a 1-D vector then it is 784\n", 58 | "output_size = 10#total possible output 0-9\n", 59 | "num_epochs = 20\n", 60 | "batch_size = 100#total number of images is 60000\n", 61 | "learning_rate = 0.001\n", 62 | "#here we are using PyTorch dataset hence torchvision.datasets.MNIST, then the image is normal array to convert them to tensor we use transforms.ToTensor()\n", 63 | "#later we will see that we can include many other transforms here also.\n", 64 | "train_dataset = torchvision.datasets.MNIST(root='Data', train = True, transform = transforms.ToTensor(), download = True) \n", 65 | "test_dataset = torchvision.datasets.MNIST(root='Data', train = False, transform= transforms.ToTensor())\n", 66 | "train_loader = torch.utils.data.DataLoader(dataset = train_dataset, batch_size = batch_size, shuffle = True)\n", 67 | "test_loader = torch.utils.data.DataLoader(dataset = test_dataset, batch_size = batch_size, shuffle = False) \n", 68 | "\n", 69 | "#in DataLoader we use toch.utils.data.DataLoader(), then we define batch_size and shuffle here. \n", 70 | "#Basically it divides the dataset and loads data in batches\n", 71 | "\n", 72 | "class Logistic(nn.Module):\n", 73 | " def __init__(self, input_size, output_size):\n", 74 | " super(Logistic, self).__init__()\n", 75 | " self.linear = nn.Linear(input_size, output_size)\n", 76 | " def forward(self, x):\n", 77 | " out = self.linear(x)\n", 78 | " return(out)\n", 79 | " \n", 80 | " \n", 81 | "\n", 82 | "model = Logistic(input_size, output_size)\n", 83 | "\n", 84 | "criterion = nn.CrossEntropyLoss()#this could be nn.BCELoss() if we were using it for binary classification\n", 85 | "optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate)\n", 86 | "\n", 87 | "print(len(train_dataset))\n", 88 | "print(len(train_loader))\n", 89 | "\n", 90 | "total_step = len(train_loader)#this will give total number of iteration.\n", 91 | "for epoch in range(num_epochs):\n", 92 | " for i, (images, labels) in enumerate(train_loader):#enumerate is used to give the value of i also. \n", 93 | " images = images.reshape(-1, 28*28)#600,784\n", 94 | " outputs = model(images)\n", 95 | " loss = criterion(outputs, labels)\n", 96 | " optimizer.zero_grad()\n", 97 | " loss.backward()\n", 98 | " optimizer.step()\n", 99 | " if(epoch%5 == 0):\n", 100 | " if (i+1) % 100 == 0:\n", 101 | " print ('Epoch [{}/{}], Iteration [{}/{}], Loss: {:.4f}' \n", 102 | " .format(epoch+1, num_epochs, i+1, total_step, loss.item()))\n", 103 | "\n", 104 | "with torch.no_grad():\n", 105 | " correct = 0\n", 106 | " total = 0\n", 107 | " for images, labels in test_loader:\n", 108 | " images = images.reshape(-1, 28*28)\n", 109 | " outputs = model(images)\n", 110 | " _, predicted = torch.max(outputs.data, 1) #return max value and the indices\n", 111 | " total += labels.size(0)\n", 112 | " correct += (predicted==labels).sum()\n", 113 | " print('Accuracy of the model on the 10000 test images: {} %'.format(100 * correct / total)) \n", 114 | " " 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": 4, 120 | "metadata": {}, 121 | "outputs": [ 122 | { 123 | "name": "stdout", 124 | "output_type": "stream", 125 | "text": [ 126 | "Epoch [1/5], Iteration [100/600], Loss: 0.3462\n", 127 | "Epoch [1/5], Iteration [200/600], Loss: 0.3088\n", 128 | "Epoch [1/5], Iteration [300/600], Loss: 0.2861\n", 129 | "Epoch [1/5], Iteration [400/600], Loss: 0.1597\n", 130 | "Epoch [1/5], Iteration [500/600], Loss: 0.3207\n", 131 | "Epoch [1/5], Iteration [600/600], Loss: 0.1445\n", 132 | "Epoch [2/5], Iteration [100/600], Loss: 0.1394\n", 133 | "Epoch [2/5], Iteration [200/600], Loss: 0.1824\n", 134 | "Epoch [2/5], Iteration [300/600], Loss: 0.1940\n", 135 | "Epoch [2/5], Iteration [400/600], Loss: 0.1417\n", 136 | "Epoch [2/5], Iteration [500/600], Loss: 0.0526\n", 137 | "Epoch [2/5], Iteration [600/600], Loss: 0.2374\n", 138 | "Epoch [3/5], Iteration [100/600], Loss: 0.1548\n", 139 | "Epoch [3/5], Iteration [200/600], Loss: 0.1229\n", 140 | "Epoch [3/5], Iteration [300/600], Loss: 0.0985\n", 141 | "Epoch [3/5], Iteration [400/600], Loss: 0.0805\n", 142 | "Epoch [3/5], Iteration [500/600], Loss: 0.1887\n", 143 | "Epoch [3/5], Iteration [600/600], Loss: 0.1625\n", 144 | "Epoch [4/5], Iteration [100/600], Loss: 0.1755\n", 145 | "Epoch [4/5], Iteration [200/600], Loss: 0.0969\n", 146 | "Epoch [4/5], Iteration [300/600], Loss: 0.0539\n", 147 | "Epoch [4/5], Iteration [400/600], Loss: 0.0600\n", 148 | "Epoch [4/5], Iteration [500/600], Loss: 0.0893\n", 149 | "Epoch [4/5], Iteration [600/600], Loss: 0.1310\n", 150 | "Epoch [5/5], Iteration [100/600], Loss: 0.0916\n", 151 | "Epoch [5/5], Iteration [200/600], Loss: 0.1421\n", 152 | "Epoch [5/5], Iteration [300/600], Loss: 0.1182\n", 153 | "Epoch [5/5], Iteration [400/600], Loss: 0.1086\n", 154 | "Epoch [5/5], Iteration [500/600], Loss: 0.0648\n", 155 | "Epoch [5/5], Iteration [600/600], Loss: 0.1162\n", 156 | "Accuracy of the network on the 10000 test images: 97 %\n" 157 | ] 158 | } 159 | ], 160 | "source": [ 161 | "#Write Neural Network By yourself. With HyperParameters\n", 162 | "\n", 163 | "input_size = 784\n", 164 | "hidden_size = 100\n", 165 | "num_classes = 10\n", 166 | "learning_rate = 0.001\n", 167 | "num_epochs = 5\n", 168 | "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n", 169 | "\n", 170 | "class NeuralNet(nn.Module):\n", 171 | " def __init__(self, input_size, hidden_size, num_classes):\n", 172 | " super(NeuralNet, self).__init__()\n", 173 | " self.fc1 = nn.Linear(input_size, hidden_size)\n", 174 | " self.relu = nn.ReLU()\n", 175 | " self.fc2 = nn.Linear(hidden_size, num_classes)\n", 176 | " def forward(self, x):\n", 177 | " out = self.fc1(x)\n", 178 | " out = self.relu(out)\n", 179 | " out = self.fc2(out)\n", 180 | " return out\n", 181 | " \n", 182 | "model = NeuralNet(input_size, hidden_size, num_classes).to(device)\n", 183 | "\n", 184 | "criterion = nn.CrossEntropyLoss()\n", 185 | "optimizer = torch.optim.Adam(model.parameters(),lr = learning_rate)\n", 186 | "\n", 187 | "total_step = len(train_loader)\n", 188 | "for epoch in range(num_epochs):\n", 189 | " for i, (images, labels) in enumerate(train_loader):\n", 190 | " images = images.reshape(-1,28*28).to(device)\n", 191 | " labels = labels.to(device)\n", 192 | " outputs = model(images)\n", 193 | " loss = criterion(outputs, labels)\n", 194 | " optimizer.zero_grad()\n", 195 | " loss.backward()\n", 196 | " optimizer.step()\n", 197 | " if (i+1) % 100 == 0:\n", 198 | " print ('Epoch [{}/{}], Iteration [{}/{}], Loss: {:.4f}' \n", 199 | " .format(epoch+1, num_epochs, i+1, total_step, loss.item()))\n", 200 | "\n", 201 | "with torch.no_grad():\n", 202 | " correct = 0 \n", 203 | " total = 0\n", 204 | " for images, labels in (test_loader):\n", 205 | " images= images.reshape(-1, 28*28).to(device)\n", 206 | " labels = labels.to(device)\n", 207 | " outputs = model(images)\n", 208 | " _, predicted = torch.max(outputs.data, 1)\n", 209 | " total+=predicted.size(0)\n", 210 | " correct += (predicted==labels).sum()\n", 211 | " \n", 212 | " print('Accuracy of the network on the 10000 test images: {} %'.format(100 * correct / total))" 213 | ] 214 | }, 215 | { 216 | "cell_type": "code", 217 | "execution_count": null, 218 | "metadata": {}, 219 | "outputs": [], 220 | "source": [] 221 | } 222 | ], 223 | "metadata": { 224 | "kernelspec": { 225 | "display_name": "Python 3", 226 | "language": "python", 227 | "name": "python3" 228 | }, 229 | "language_info": { 230 | "codemirror_mode": { 231 | "name": "ipython", 232 | "version": 3 233 | }, 234 | "file_extension": ".py", 235 | "mimetype": "text/x-python", 236 | "name": "python", 237 | "nbconvert_exporter": "python", 238 | "pygments_lexer": "ipython3", 239 | "version": "3.5.2" 240 | } 241 | }, 242 | "nbformat": 4, 243 | "nbformat_minor": 2 244 | } 245 | -------------------------------------------------------------------------------- /Practise Notebook_basic.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import torch\n", 10 | "import torch.nn as nn\n", 11 | "import torchvision\n", 12 | "import numpy as np\n", 13 | "import torchvision.transforms as transforms\n", 14 | "import matplotlib.pyplot as plt" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": 2, 20 | "metadata": {}, 21 | "outputs": [ 22 | { 23 | "name": "stdout", 24 | "output_type": "stream", 25 | "text": [ 26 | "60000\n", 27 | "600\n", 28 | "Epoch [1/5], Iteration [100/600], Loss: 2.2360\n", 29 | "Epoch [1/5], Iteration [200/600], Loss: 2.1408\n", 30 | "Epoch [1/5], Iteration [300/600], Loss: 2.0421\n", 31 | "Epoch [1/5], Iteration [400/600], Loss: 1.9860\n", 32 | "Epoch [1/5], Iteration [500/600], Loss: 1.8835\n", 33 | "Epoch [1/5], Iteration [600/600], Loss: 1.7920\n", 34 | "Epoch [2/5], Iteration [100/600], Loss: 1.7255\n", 35 | "Epoch [2/5], Iteration [200/600], Loss: 1.6768\n", 36 | "Epoch [2/5], Iteration [300/600], Loss: 1.6237\n", 37 | "Epoch [2/5], Iteration [400/600], Loss: 1.5295\n", 38 | "Epoch [2/5], Iteration [500/600], Loss: 1.5145\n", 39 | "Epoch [2/5], Iteration [600/600], Loss: 1.4964\n", 40 | "Epoch [3/5], Iteration [100/600], Loss: 1.5087\n", 41 | "Epoch [3/5], Iteration [200/600], Loss: 1.3562\n", 42 | "Epoch [3/5], Iteration [300/600], Loss: 1.3707\n", 43 | "Epoch [3/5], Iteration [400/600], Loss: 1.3737\n", 44 | "Epoch [3/5], Iteration [500/600], Loss: 1.3118\n", 45 | "Epoch [3/5], Iteration [600/600], Loss: 1.3638\n", 46 | "Epoch [4/5], Iteration [100/600], Loss: 1.2265\n", 47 | "Epoch [4/5], Iteration [200/600], Loss: 1.0935\n", 48 | "Epoch [4/5], Iteration [300/600], Loss: 1.2359\n", 49 | "Epoch [4/5], Iteration [400/600], Loss: 1.1087\n", 50 | "Epoch [4/5], Iteration [500/600], Loss: 1.1397\n", 51 | "Epoch [4/5], Iteration [600/600], Loss: 1.0544\n", 52 | "Epoch [5/5], Iteration [100/600], Loss: 1.1278\n", 53 | "Epoch [5/5], Iteration [200/600], Loss: 1.0547\n", 54 | "Epoch [5/5], Iteration [300/600], Loss: 1.0745\n", 55 | "Epoch [5/5], Iteration [400/600], Loss: 1.0204\n", 56 | "Epoch [5/5], Iteration [500/600], Loss: 1.0686\n", 57 | "Epoch [5/5], Iteration [600/600], Loss: 1.1287\n", 58 | "Accuracy of the model on the 10000 test images: 82 %\n" 59 | ] 60 | } 61 | ], 62 | "source": [ 63 | "input_size = 784\n", 64 | "output_size = 10\n", 65 | "num_epochs = 5\n", 66 | "batch_size = 100\n", 67 | "learning_rate = 0.001\n", 68 | "\n", 69 | "train_dataset = torchvision.datasets.MNIST(root='../ai61002s19/MNIST', train = True, transform = transforms.ToTensor(), download = False)\n", 70 | "test_dataset = torchvision.datasets.MNIST(root='../ai61002s19/MNIST', train = False, transform= transforms.ToTensor())\n", 71 | "train_loader = torch.utils.data.DataLoader(dataset = train_dataset, batch_size = batch_size, shuffle = True)\n", 72 | "test_loader = torch.utils.data.DataLoader(dataset = test_dataset, batch_size = batch_size, shuffle = False) \n", 73 | "\n", 74 | "model = nn.Linear(input_size, output_size)\n", 75 | "\n", 76 | "criterion = nn.CrossEntropyLoss()\n", 77 | "optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate)\n", 78 | "\n", 79 | "print(len(train_dataset))\n", 80 | "print(len(train_loader))\n", 81 | "\n", 82 | "total_step = len(train_loader)\n", 83 | "for epoch in range(num_epochs):\n", 84 | " for i, (images, labels) in enumerate(train_loader):\n", 85 | " images = images.reshape(-1, 28*28)\n", 86 | " outputs = model(images)\n", 87 | " loss = criterion(outputs, labels)\n", 88 | " optimizer.zero_grad()\n", 89 | " loss.backward()\n", 90 | " optimizer.step()\n", 91 | " \n", 92 | " if (i+1) % 100 == 0:\n", 93 | " print ('Epoch [{}/{}], Iteration [{}/{}], Loss: {:.4f}' \n", 94 | " .format(epoch+1, num_epochs, i+1, total_step, loss.item()))\n", 95 | "\n", 96 | "with torch.no_grad():\n", 97 | " correct = 0\n", 98 | " total = 0\n", 99 | " for images, labels in test_loader:\n", 100 | " images = images.reshape(-1, 28*28)\n", 101 | " outputs = model(images)\n", 102 | " _, predicted = torch.max(outputs.data, 1) #return max value and the indices\n", 103 | " total += labels.size(0)\n", 104 | " correct += (predicted==labels).sum()\n", 105 | " print('Accuracy of the model on the 10000 test images: {} %'.format(100 * correct / total)) \n", 106 | " " 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": 4, 112 | "metadata": {}, 113 | "outputs": [ 114 | { 115 | "name": "stdout", 116 | "output_type": "stream", 117 | "text": [ 118 | "Epoch [1/5], Iteration [100/600], Loss: 0.5375\n", 119 | "Epoch [1/5], Iteration [200/600], Loss: 0.3712\n", 120 | "Epoch [1/5], Iteration [300/600], Loss: 0.2424\n", 121 | "Epoch [1/5], Iteration [400/600], Loss: 0.4327\n", 122 | "Epoch [1/5], Iteration [500/600], Loss: 0.2944\n", 123 | "Epoch [1/5], Iteration [600/600], Loss: 0.3506\n", 124 | "Epoch [2/5], Iteration [100/600], Loss: 0.1986\n", 125 | "Epoch [2/5], Iteration [200/600], Loss: 0.1941\n", 126 | "Epoch [2/5], Iteration [300/600], Loss: 0.2853\n", 127 | "Epoch [2/5], Iteration [400/600], Loss: 0.1549\n", 128 | "Epoch [2/5], Iteration [500/600], Loss: 0.1121\n", 129 | "Epoch [2/5], Iteration [600/600], Loss: 0.1507\n", 130 | "Epoch [3/5], Iteration [100/600], Loss: 0.0889\n", 131 | "Epoch [3/5], Iteration [200/600], Loss: 0.1409\n", 132 | "Epoch [3/5], Iteration [300/600], Loss: 0.0846\n", 133 | "Epoch [3/5], Iteration [400/600], Loss: 0.0784\n", 134 | "Epoch [3/5], Iteration [500/600], Loss: 0.0881\n", 135 | "Epoch [3/5], Iteration [600/600], Loss: 0.2132\n", 136 | "Epoch [4/5], Iteration [100/600], Loss: 0.1325\n", 137 | "Epoch [4/5], Iteration [200/600], Loss: 0.0613\n", 138 | "Epoch [4/5], Iteration [300/600], Loss: 0.1321\n", 139 | "Epoch [4/5], Iteration [400/600], Loss: 0.1268\n", 140 | "Epoch [4/5], Iteration [500/600], Loss: 0.1063\n", 141 | "Epoch [4/5], Iteration [600/600], Loss: 0.1586\n", 142 | "Epoch [5/5], Iteration [100/600], Loss: 0.1263\n", 143 | "Epoch [5/5], Iteration [200/600], Loss: 0.1355\n", 144 | "Epoch [5/5], Iteration [300/600], Loss: 0.0722\n", 145 | "Epoch [5/5], Iteration [400/600], Loss: 0.0694\n", 146 | "Epoch [5/5], Iteration [500/600], Loss: 0.0414\n", 147 | "Epoch [5/5], Iteration [600/600], Loss: 0.0394\n", 148 | "Accuracy of the network on the 10000 test images: 97 %\n" 149 | ] 150 | } 151 | ], 152 | "source": [ 153 | "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n", 154 | "input_size = 784\n", 155 | "hidden_size = 100\n", 156 | "num_classes = 10\n", 157 | "num_epochs = 5 \n", 158 | "batch_size = 100\n", 159 | "learning_rate = 0.001\n", 160 | "\n", 161 | "train_dataset = torchvision.datasets.MNIST(root = '../ai61002s19/MNIST', train = True, transform = transforms.ToTensor(), download = False)\n", 162 | "test_dataset = torchvision.datasets.MNIST(root = '../ai61002s19/MNIST', train = False, transform = transforms.ToTensor())\n", 163 | "train_loader = torch.utils.data.DataLoader(dataset = train_dataset, batch_size = batch_size, shuffle=True)\n", 164 | "test_loader = torch.utils.data.DataLoader(dataset = test_dataset, batch_size = batch_size, shuffle=False)\n", 165 | "\n", 166 | "class NeuralNet(nn.Module):\n", 167 | " def __init__(self, input_size, hidden_size, num_classes):\n", 168 | " super(NeuralNet, self).__init__()\n", 169 | " self.fc1 = nn.Linear(input_size, hidden_size)\n", 170 | " self.relu = nn.ReLU()\n", 171 | " self.fc2 = nn.Linear(hidden_size, num_classes)\n", 172 | " def forward(self, x):\n", 173 | " out = self.fc1(x)\n", 174 | " out = self.relu(out)\n", 175 | " out = self.fc2(out)\n", 176 | " return out\n", 177 | " \n", 178 | "model = NeuralNet(input_size, hidden_size, num_classes).to(device)\n", 179 | "\n", 180 | "criterion = nn.CrossEntropyLoss()\n", 181 | "optimizer = torch.optim.Adam(model.parameters(),lr = learning_rate)\n", 182 | "\n", 183 | "total_step = len(train_loader)\n", 184 | "for epoch in range(num_epochs):\n", 185 | " for i, (images, labels) in enumerate(train_loader):\n", 186 | " images = images.reshape(-1,28*28).to(device)\n", 187 | " labels = labels.to(device)\n", 188 | " outputs = model(images)\n", 189 | " loss = criterion(outputs, labels)\n", 190 | " optimizer.zero_grad()\n", 191 | " loss.backward()\n", 192 | " optimizer.step()\n", 193 | " if (i+1) % 100 == 0:\n", 194 | " print ('Epoch [{}/{}], Iteration [{}/{}], Loss: {:.4f}' \n", 195 | " .format(epoch+1, num_epochs, i+1, total_step, loss.item()))\n", 196 | "\n", 197 | "with torch.no_grad():\n", 198 | " correct = 0 \n", 199 | " total = 0\n", 200 | " for images, labels in (test_loader):\n", 201 | " images= images.reshape(-1, 28*28).to(device)\n", 202 | " labels = labels.to(device)\n", 203 | " outputs = model(images)\n", 204 | " _, predicted = torch.max(outputs.data, 1)\n", 205 | " total+=predicted.size(0)\n", 206 | " correct += (predicted==labels).sum()\n", 207 | " \n", 208 | " print('Accuracy of the network on the 10000 test images: {} %'.format(100 * correct / total))" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": 1, 214 | "metadata": {}, 215 | "outputs": [ 216 | { 217 | "ename": "NameError", 218 | "evalue": "name 'torch' is not defined", 219 | "output_type": "error", 220 | "traceback": [ 221 | "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", 222 | "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", 223 | "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mdevice\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'cuda:0'\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcuda\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_available\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;34m'cpu'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0mnum_epochs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m5\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mnum_classes\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m10\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0mbatch_size\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m100\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 224 | "\u001b[0;31mNameError\u001b[0m: name 'torch' is not defined" 225 | ] 226 | } 227 | ], 228 | "source": [ 229 | "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n", 230 | "\n", 231 | "num_epochs = 5\n", 232 | "num_classes = 10\n", 233 | "batch_size = 100 \n", 234 | "learning_rate = 0.001\n", 235 | "\n", 236 | "class Conv(nn.Module):\n", 237 | " def __init__(self, num_classes):\n", 238 | " super(Conv, self).__init__()\n", 239 | " self.layer1 = nn.Sequential(\n", 240 | " nn.Conv2d(1,16, kernel_size=5, stride = 1, padding = 2),\n", 241 | " nn.BatchNorm2d(16),\n", 242 | " nn.ReLU(),\n", 243 | " nn.MaxPool2d(kernel_size = 2, stride=2))\n", 244 | " self.layer2 = nn.Sequential(\n", 245 | " nn.Conv2d(16,32,kernel_size = 5, padding =2, stride =1),\n", 246 | " nn.BatchNorm2d(32),\n", 247 | " nn.ReLU(),\n", 248 | " nn.MaxPool2d(kernel_size = 2, stride =2))\n", 249 | " self.fc = nn.Linear(7*7*32, num_classes)\n", 250 | " def forward(self, x):\n", 251 | " out = self.layer1(x)\n", 252 | " out = self.layer2(out)\n", 253 | "# print(out.size(0))\n", 254 | " out = out.reshape(out.size(0), -1)#here we provide a batch of image so out.size(0) is the batch_size\n", 255 | " out = self.fc(out)\n", 256 | " return out\n", 257 | "train_dataset = torchvision.datasets.MNIST(root='../ai61002s19/MNIST', train=True, transform = transforms.ToTensor(), download = False)\n", 258 | "test_dataset = torchvision.datasets.MNIST(root='../ai61002s19/MNIST', train=False, transform = transforms.ToTensor(), download = False)\n", 259 | "train_loader = torch.utils.data.DataLoader(dataset = train_dataset, batch_size = batch_size, shuffle = True)\n", 260 | "test_loader = torch.utils.data.DataLoader(dataset = test_dataset, batch_size = batch_size, shuffle = True)\n", 261 | "\n", 262 | "model = Conv(num_classes)#.to(device)\n", 263 | "criterion = nn.CrossEntropyLoss()\n", 264 | "optimizer = torch.optim.Adam(model.parameters(), lr= learning_rate)\n", 265 | "# out = model(train_dataset[0][0].reshape(1,1,28,28))\n", 266 | "\n", 267 | "train_step = len(train_loader)\n", 268 | "for epoch in range(num_epochs):\n", 269 | " for i, (images, labels) in enumerate(train_loader):\n", 270 | " images = images.to(device)\n", 271 | " labels = labels.to(device)\n", 272 | "# print(images.shape)\n", 273 | " outputs = model(images)\n", 274 | " loss = criterion(outputs, labels)\n", 275 | " \n", 276 | " optimizer.zero_grad()\n", 277 | " loss.backward()\n", 278 | " optimizer.step()\n", 279 | " if (i+1) % 100 == 0:\n", 280 | " print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}' \n", 281 | " .format(epoch+1, num_epochs, i+1, total_step, loss.item()))\n", 282 | "\n", 283 | "model.eval() #it is typically used for batchnorm so that it uses moving variance and mean instead of the whole batch\n", 284 | "with torch.no_grad():\n", 285 | " correct = 0\n", 286 | " total = 0\n", 287 | " for images, labels in test_loader:\n", 288 | " images = images.to(device)\n", 289 | " labels = labels.to(device)\n", 290 | " outputs = model(images)\n", 291 | " _, predicted = torch.max(outputs.data, 1)\n", 292 | " total += labels.size(0)\n", 293 | " correct += (predicted == labels).sum().item()\n", 294 | "\n", 295 | " print('Test Accuracy of the model on the 10000 test images: {} %'.format(100 * correct / total)) " 296 | ] 297 | }, 298 | { 299 | "cell_type": "code", 300 | "execution_count": 14, 301 | "metadata": {}, 302 | "outputs": [ 303 | { 304 | "name": "stdout", 305 | "output_type": "stream", 306 | "text": [ 307 | "Files already downloaded and verified\n", 308 | "Epoch [1/30], Step [100/500] Loss: 1.6097\n", 309 | "Epoch [1/30], Step [200/500] Loss: 1.4205\n", 310 | "Epoch [1/30], Step [300/500] Loss: 1.4091\n", 311 | "Epoch [1/30], Step [400/500] Loss: 1.2042\n", 312 | "Epoch [1/30], Step [500/500] Loss: 1.0310\n" 313 | ] 314 | }, 315 | { 316 | "ename": "KeyboardInterrupt", 317 | "evalue": "", 318 | "output_type": "error", 319 | "traceback": [ 320 | "\u001b[0;31m---------------------------------------\u001b[0m", 321 | "\u001b[0;31mKeyboardInterrupt\u001b[0mTraceback (most recent call last)", 322 | "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 101\u001b[0m \u001b[0;31m# Backward and optimize\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 102\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 103\u001b[0;31m \u001b[0mloss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 104\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 105\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", 323 | "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/torch/tensor.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(self, gradient, retain_graph, create_graph)\u001b[0m\n\u001b[1;32m 91\u001b[0m \u001b[0mproducts\u001b[0m\u001b[0;34m.\u001b[0m \u001b[0mDefaults\u001b[0m \u001b[0mto\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 92\u001b[0m \"\"\"\n\u001b[0;32m---> 93\u001b[0;31m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgradient\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 94\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 95\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mregister_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 324 | "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/torch/autograd/__init__.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(tensors, grad_tensors, retain_graph, create_graph, grad_variables)\u001b[0m\n\u001b[1;32m 88\u001b[0m Variable._execution_engine.run_backward(\n\u001b[1;32m 89\u001b[0m \u001b[0mtensors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad_tensors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 90\u001b[0;31m allow_unreachable=True) # allow_unreachable flag\n\u001b[0m\u001b[1;32m 91\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 92\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", 325 | "\u001b[0;31mKeyboardInterrupt\u001b[0m: " 326 | ] 327 | } 328 | ], 329 | "source": [ 330 | "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n", 331 | "batch_size = 100\n", 332 | "num_epochs = 30\n", 333 | "learning_rate = 0.001\n", 334 | "transform = transforms.Compose([\n", 335 | " transforms.Pad(4),\n", 336 | " transforms.RandomHorizontalFlip(), \n", 337 | " transforms.RandomCrop(32), \n", 338 | " transforms.ToTensor()])\n", 339 | "\n", 340 | "train_dataset = torchvision.datasets.CIFAR10(root='../ai61002s19/', train=True, transform = transform, download = True)\n", 341 | "test_dataset = torchvision.datasets.CIFAR10(root='../ai61002s19/', train=False, transform = transforms.ToTensor(), download = False)\n", 342 | "train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size = batch_size, shuffle= True)\n", 343 | "test_loader = torch.utils.data.DataLoader(dataset=test_dataset, batch_size = batch_size, shuffle= False)\n", 344 | "\n", 345 | "class ResidualBlock(nn.Module):\n", 346 | " def __init__(self, in_channel, out_channel,stride=1, downsample=None):\n", 347 | " super(ResidualBlock, self).__init__()\n", 348 | " self.conv1 = nn.Conv2d(in_channel, out_channel, kernel_size = 3, stride = stride, padding=1)\n", 349 | " self.bn1 = nn.BatchNorm2d(out_channel)\n", 350 | " self.relu = nn.ReLU()\n", 351 | " self.conv2 = nn.Conv2d(out_channel, out_channel, kernel_size = 3, stride = 1, padding=1) #we change the size only once\n", 352 | " self.bn2 = nn.BatchNorm2d(out_channel)\n", 353 | " self.downsample = downsample\n", 354 | " def forward(self,x):\n", 355 | " residual = x\n", 356 | " out = self.conv1(x)\n", 357 | " out = self.bn1(out)\n", 358 | " out = self.relu(out)\n", 359 | " out = self.conv2(out)\n", 360 | " out = self.bn2(out)\n", 361 | " if self.downsample:#to be used when input size does not match output size\n", 362 | " residual = self.downsample(x)\n", 363 | " out += residual\n", 364 | " out = self.relu(out)\n", 365 | " return(out)\n", 366 | "\n", 367 | "class ResNet(nn.Module):\n", 368 | " def __init__(self, block, num_classes=10):\n", 369 | " super(ResNet, self).__init__()\n", 370 | " self.in_channel = 16\n", 371 | " self.conv1 = nn.Conv2d(3,16, stride =1, kernel_size = 3, padding = 1)\n", 372 | " self.bn = nn.BatchNorm2d(16)\n", 373 | " self.relu = nn.ReLU()\n", 374 | " self.block1 = self.make_layer(block, 16, 1)\n", 375 | " self.block2 = self.make_layer(block, 16, 1)\n", 376 | " self.block3 = self.make_layer(block, 32, 2)\n", 377 | " self.block4 = self.make_layer(block, 32, 1)\n", 378 | " self.block5 = self.make_layer(block, 64, 2)\n", 379 | " self.block6 = self.make_layer(block, 64, 1)\n", 380 | " self.avg_pool = nn.AvgPool2d(8) #8 is the kernel size so it is taking average of 8x8\n", 381 | " self.fc = nn.Linear(64, num_classes)\n", 382 | " def make_layer(self, block, out_channel, stride=1):\n", 383 | " downsample = None\n", 384 | " if(stride!=1) or (self.in_channel != out_channel):#input size not equal to output size only when stride not 1 or input channel and output channel are not same \n", 385 | " downsample = nn.Sequential(\n", 386 | " nn.Conv2d(self.in_channel, out_channel, kernel_size = 3, stride = stride, padding = 1),\n", 387 | " nn.BatchNorm2d(out_channel))\n", 388 | " out_layer = block(self.in_channel, out_channel, stride, downsample)\n", 389 | " self.in_channel = out_channel\n", 390 | " return(out_layer)\n", 391 | " def forward(self,x):\n", 392 | " out = self.conv1(x)\n", 393 | " out = self.bn(out)\n", 394 | " out = self.relu(out)\n", 395 | " out = self.block1(out)\n", 396 | " out = self.block2(out)\n", 397 | " out = self.block3(out)\n", 398 | " out = self.block4(out)\n", 399 | " out = self.block5(out)\n", 400 | " out = self.block6(out)\n", 401 | " out = self.avg_pool(out)\n", 402 | " out = out.view(out.size(0), -1)\n", 403 | " out = self.fc(out)\n", 404 | " return out\n", 405 | "model = ResNet(ResidualBlock).to(device)\n", 406 | "# print(train_dataset[0][0].shape)\n", 407 | "# out = model(train_dataset[0][0].reshape(1,3,32,32))\n", 408 | "\n", 409 | "# Loss and optimizer\n", 410 | "criterion = nn.CrossEntropyLoss()\n", 411 | "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n", 412 | "\n", 413 | "# For updating learning rate\n", 414 | "def update_lr(optimizer, lr): \n", 415 | " for param_group in optimizer.param_groups:\n", 416 | " param_group['lr'] = lr\n", 417 | "\n", 418 | "# Train the model\n", 419 | "total_step = len(train_loader)\n", 420 | "curr_lr = learning_rate\n", 421 | "for epoch in range(num_epochs):\n", 422 | " for i, (images, labels) in enumerate(train_loader):\n", 423 | " images = images.to(device)\n", 424 | " labels = labels.to(device)\n", 425 | " \n", 426 | " # Forward pass\n", 427 | " outputs = model(images)\n", 428 | " loss = criterion(outputs, labels)\n", 429 | " \n", 430 | " # Backward and optimize\n", 431 | " optimizer.zero_grad()\n", 432 | " loss.backward()\n", 433 | " optimizer.step()\n", 434 | " \n", 435 | " if (i+1) % 100 == 0:\n", 436 | " print (\"Epoch [{}/{}], Step [{}/{}] Loss: {:.4f}\"\n", 437 | " .format(epoch+1, num_epochs, i+1, total_step, loss.item()))\n", 438 | "\n", 439 | " # Decay learning rate\n", 440 | " if (epoch+1) % 20 == 0:\n", 441 | " curr_lr /= 3\n", 442 | " update_lr(optimizer, curr_lr)" 443 | ] 444 | }, 445 | { 446 | "cell_type": "code", 447 | "execution_count": null, 448 | "metadata": {}, 449 | "outputs": [], 450 | "source": [ 451 | "\n", 452 | "\n", 453 | "\n", 454 | "\n", 455 | "\n", 456 | "\n", 457 | "\n", 458 | "\n", 459 | "\n", 460 | "\n", 461 | "\n", 462 | "\n", 463 | "\n", 464 | "\n", 465 | "\n", 466 | "\n", 467 | "\n", 468 | "\n", 469 | "\n", 470 | "\n", 471 | "\n", 472 | "\n", 473 | "\n", 474 | "\n", 475 | "\n", 476 | "\n", 477 | "\n", 478 | "\n", 479 | "\n", 480 | "\n" 481 | ] 482 | } 483 | ], 484 | "metadata": { 485 | "kernelspec": { 486 | "display_name": "Python 3", 487 | "language": "python", 488 | "name": "python3" 489 | }, 490 | "language_info": { 491 | "codemirror_mode": { 492 | "name": "ipython", 493 | "version": 3 494 | }, 495 | "file_extension": ".py", 496 | "mimetype": "text/x-python", 497 | "name": "python", 498 | "nbconvert_exporter": "python", 499 | "pygments_lexer": "ipython3", 500 | "version": "3.5.2" 501 | } 502 | }, 503 | "nbformat": 4, 504 | "nbformat_minor": 2 505 | } 506 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # PyTorch-Tutorials 2 | This repository contains notebooks to learn Pytorch step by step. All are written in jupyter notebook so that one can tweak the code and check the results for the same. I assume that reader have basic understanding of Machine learning and deep learning. 3 | ### Content 4 | * Basic and Linear Regression - Tensors, tensors to numpy and vice versa, training a normal linear regression model 5 | * Logistic regression and Neural Network - Help you load a dataset, Basic architecture of writing a neural network, build a logistic regression model and neural network. 6 | * CNN and Resnet - This help you to write a CNN network and a complex network like Resnet so that you can then write any other netwrok by your own. Also it tells how to save and load model and what's the difference between saving the parameter and whole model. 7 | * Custom Dataset and Trasfer Learning - It helps you to learn how to load a pretrained network and how to freeze layers, how to tweak a pretrained model and Finally some important tips on how to make a custom dataset loader which is of great essence now. 8 | * Initialize weight of parameter of model by yourself- This notebook help you learn how you can set initial weight of parameter of Network by yourself, without using Pytorch weight initializer. This is very helpful when you have to compare your implementation with a tested library like PyTorch. 9 | * AAE - This helps you learn how to train a multiple networks simultanelously. Here we are going to train three network (Encoder, Decoder and Discriminator). Also plotting the images and curve is shown in the tutorial. 10 | * GANS - Moving to more advanced networks, an application of what you have learned above. 11 | 12 | 13 | ### Dependencies 14 | * Pytorch 15 | * Jupyter Notebook 16 | * Python 3 17 | ### Contributing 18 | Have anything in mind which you think can be helpful, please create an issue and let me know. So that I can incorporate that here. If you wish to contribute feel free to send a pull request. 19 | --------------------------------------------------------------------------------