├── README.md ├── Week 1 ├── 1_D tensors exercise.py ├── 2_D Tensors exercise.py ├── 2_D Tensors.py ├── Dataset Exercise.py ├── Dataset and imgs.py ├── Differentiation.py ├── Prebuilt Dataset Exercise.py ├── Simple_dataset.py ├── Tensors 1 D Overview.py ├── img.tar.gz └── index.csv ├── Week 2 ├── LR with DataLoader and Optimizers Train and Val.py ├── LR with DataLoader and Optimizers.py ├── Linear Regression 1D Stochastic Gradient Descent and DataLoader.py ├── MiniBatch Gradient Descent.py ├── Simple Linear Regression 1D Training One Parameter.py ├── Simple Linear Regression.py └── Simple Linear regression 1D Training Two Parameter.py ├── Week 3 ├── Logistic Regression Training (Cross-Entropy).py ├── Logistic Regression and Bad Initialization Value.py ├── Logistic Regression.py ├── Multiple Linear Regression Training.py ├── Multiple Linear Regression with Multiple Outputs training.py └── Multiple Linear Regression.py ├── Week 4 ├── Activation Functions.py ├── Neural Networks More Hidden Neutrons.py ├── Neural Networks with One Hidden Layer1.py ├── Neural Networks with one Hidden Layer.py ├── Simple One Hidden Layer Neural Network.py ├── Softmax 1D.py ├── Softmax MNIST Classifier.py └── Test Activation Functions on the MNIST Dataset.py ├── Week 5 ├── Batch Normalization with MNIST Dataset.py ├── Creating a Neural Network with nn.Modulelist.py ├── Deep Network Sigmoid, Tanh and Relu on MNIST Dataset.py ├── Dropout in Classification.py ├── Dropout in Regression.py ├── Initialization with same weights experiment.py ├── Neural Networks with Momentum.py └── Weights Experiment_Uniform_He_Xavier_Identical_Default.py ├── Week 6 ├── Convolution Multiple Inputs and Outputs.py ├── Convolution Network Simple.py ├── Convolution.py ├── Convolutional Neural Network with Small Images.py └── MNIST CNN BatchNorm vs Normal.py └── Week 7 ├── FashionMNIST Project Honors.py ├── First 3 Items.png ├── Image1 Validation_set.png ├── Image2 Validation_set.png ├── Image3 Validation_set.png └── Plot Cost and Accuracy.png /README.md: -------------------------------------------------------------------------------- 1 | # Deep-Neural-Networks-with-PyTorch 2 | Developing deep learning models using Pytorch 3 | -------------------------------------------------------------------------------- /Week 1/1_D tensors exercise.py: -------------------------------------------------------------------------------- 1 | 2 | """ 3 | 4 | 1-D Tensors 5 | Exercise 6 | 7 | 8 | 9 | 10 | 32-bit float - Float Tensor 11 | 64-bit float - Double Tensor 12 | 16-bit float - Half Tensor 13 | 8-bit int (unsigned)(8-bit images)- Byte Tensor 14 | 8-bit int (signed)- Char Tensor 15 | 16-bit int (singed)- Short Tensor 16 | 32-bit int (singed)- Int Tensor 17 | 64-bit int (signed)- Long Tensor 18 | 19 | 20 | """ 21 | 22 | import torch 23 | import numpy as np 24 | import pandas as pd 25 | import matplotlib.pyplot as plt 26 | 27 | 28 | #Lets define the function to plot vectors in coorindate system 29 | def plotVec(vectors): 30 | ax = plt.axes() 31 | 32 | # Vectors = [{"vector": vector variable, "name": name of vector, "color": color of the vector on diagram}] 33 | #For loop for drawing vectors 34 | for vec in vectors: 35 | ax.arrow(0, 0, *vec["vector"], head_width = 0.05, color = vec["color"], head_length = 0.1) 36 | plt.text(*(vec["vector"] + 0.1), vec["name"]) 37 | 38 | plt.ylim(-2, 2) #Setting limits for y-axis 39 | plt.xlim(-2, 2) #Setting limits for x-axis 40 | 41 | 42 | 43 | # Convert a integer list with length 5 to a tensor 44 | 45 | ints_to_tensor = torch.tensor([0, 1, 2, 3, 4]) 46 | print("The dtype of tensor object after converting it to tensor: ", ints_to_tensor.dtype) 47 | print("The type of tensor object after converting it to tensor: ", ints_to_tensor.type()) 48 | type(ints_to_tensor) 49 | 50 | 51 | #Convert float list to a float tensor 32bit 52 | list_floats = [0.0, 1.0, 2.0, 3.0, 4.0] 53 | list_floats = torch.tensor(list_floats) 54 | list_floats.type() 55 | 56 | #convert the float tensor to int64 Long Tensor 57 | floats_int_tensor=torch.tensor(list_floats,dtype=torch.int64) 58 | floats_int_tensor.type() 59 | 60 | 61 | # Convert a integer list with length 5 to float tensor 62 | new_float_tensor = torch.FloatTensor([0, 1, 2, 3, 4]) 63 | new_float_tensor.type() 64 | print("The type of the new_float_tensor:", new_float_tensor.type()) 65 | new_float_tensor = torch.FloatTensor([0, 1, 2, 3, 4]) 66 | 67 | # Another method to convert the integer list to float tensor 68 | old_int_tensor = torch.tensor([0, 1, 2, 3, 4]) 69 | new_float_tensor = old_int_tensor.type(torch.FloatTensor) 70 | print("The type of the new_float_tensor:", new_float_tensor.type()) 71 | 72 | 73 | # Introduce the tensor_obj.size() & tensor_ndimension.size() methods 74 | print("The size of the new_float_tensor: ", new_float_tensor.size()) 75 | print("The dimension of the new_float_tensor: ",new_float_tensor.ndimension()) 76 | 77 | 78 | ''' 79 | The tensor_obj.view(row, column) is used for reshaping a tensor object. 80 | 81 | What if you have a tensor object with torch.Size([5]) as a new_float_tensor as shown in the previous example? 82 | After you execute new_float_tensor.view(5, 1), the size of new_float_tensor will be torch.Size([5, 1]). 83 | This means that the tensor object new_float_tensor has been reshaped from 84 | a one-dimensional tensor object with 5 elements to a two-dimensional tensor object with 5 rows and 1 column. 85 | 86 | ''' 87 | 88 | # Introduce the tensor_obj.view(row, column) method 89 | twoD_float_tensor = new_float_tensor.view(5, 1) 90 | print("Original Size: ", new_float_tensor) 91 | print("Size after view method", twoD_float_tensor) 92 | print("Size after view method", twoD_float_tensor.size()) #Tensor of size (5,1) 93 | print("Size after view method", twoD_float_tensor.ndimension()) #Tensor dimension is now 2 94 | 95 | #What if you have a tensor with dynamic size but you want to reshape it? You can use -1 to do just that. 96 | # Introduce the use of -1 in tensor_obj.view(row, column) method 97 | twoD_float_tensor = new_float_tensor.view(-1, 1) 98 | print("Original Size: ", new_float_tensor) 99 | print("Size after view method", twoD_float_tensor) 100 | 101 | # Convert a numpy array to a tensor 102 | numpy_array = np.array([0.0, 1.0, 2.0, 3.0, 4.0]) 103 | new_tensor = torch.from_numpy(numpy_array) 104 | print("The dtype of new tensor: ", new_tensor.dtype) 105 | print("The type of new tensor: ", new_tensor.type()) 106 | 107 | # Convert a tensor to a numpy array 108 | back_to_numpy = new_tensor.numpy() 109 | print("The numpy array from tensor: ", back_to_numpy) 110 | print("The dtype of numpy array: ", back_to_numpy.dtype) 111 | 112 | # Set all elements in numpy array to zero 113 | numpy_array[:] = 0 114 | print("The new tensor points to numpy_array : ", new_tensor) 115 | print("and back to numpy array points to the tensor: ", back_to_numpy) 116 | 117 | 118 | #Converting a tensor to (1,5) tensor 119 | your_tensor = torch.tensor([1, 2, 3, 4, 5]) 120 | your_tensor.size() 121 | your_tensor.ndimesnion() 122 | your_new_tensor = your_tensor.view(1, 5) 123 | your_new_tensor.ndimension() 124 | print("Original Size: ", your_tensor) 125 | print("Size after view method", your_new_tensor) 126 | 127 | 128 | 129 | import matplotlib.pyplot as plt 130 | plt.plot(x.numpy(), y.numpy()) #We need to convert tensors to numpy arrays 131 | 132 | 133 | #Constructing tensor with 25 steps in the range of 0 and pi/2. 134 | x = torch.linspace(0, np.pi/2, steps=25) 135 | y = torch.sin(x) 136 | 137 | plt.plot(x.numpy(), y.numpy()) 138 | 139 | print("Max value from tensor x = ", x.max()) 140 | print("Min value from tensor x = ", x.min()) 141 | 142 | #Convert the list [-1, 1] and [1, 1] to tensors u and v. 143 | #Plot the tensor u and v as a vector by using the function plotVec and find the dot product 144 | 145 | u = torch.tensor([-1, 1]) 146 | v = torch.tensor([1, 1]) 147 | z = torch.dot(u,v) 148 | print("Dot product of tensors u and v = ", np.dot(u,v)) 149 | 150 | 151 | plotVec([ 152 | {"vector": u.numpy(), "name": 'u', "color": 'r'}, 153 | {"vector": v.numpy(), "name": 'v', "color": 'b'} 154 | ]) 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | -------------------------------------------------------------------------------- /Week 1/2_D Tensors exercise.py: -------------------------------------------------------------------------------- 1 | 2 | """ 3 | 4 | 2-D Tensors 5 | Exercise 6 | 7 | 8 | 9 | 10 | 32-bit float - Float Tensor 11 | 64-bit float - Double Tensor 12 | 16-bit float - Half Tensor 13 | 8-bit int (unsigned)(8-bit images)- Byte Tensor 14 | 8-bit int (signed)- Char Tensor 15 | 16-bit int (singed)- Short Tensor 16 | 32-bit int (singed)- Int Tensor 17 | 64-bit int (signed)- Long Tensor 18 | 19 | 20 | """ 21 | 22 | import numpy as np 23 | import matplotlib.pyplot as plt 24 | import torch 25 | import pandas as pd 26 | 27 | #Creating 2D tensor from a list 28 | twoD_list = [[11, 12, 13], [21, 22, 23], [31, 32, 33]] 29 | twoD_tensor = torch.tensor(twoD_list) 30 | print("The New 2D Tensor: ", twoD_tensor) 31 | 32 | #Converting tensor to numpy array and back to tensor 33 | twoD_numpy = twoD_tensor.numpy() 34 | print("Tensor -> Numpy Array:") 35 | print("The numpy array after converting: ", twoD_numpy) 36 | print("Type after converting: ", twoD_numpy.dtype) 37 | 38 | new_twoD_tensor = torch.from_numpy(twoD_numpy) 39 | print("Numpy Array -> Tensor:") 40 | print("The tensor after converting:", new_twoD_tensor) 41 | print("Type after converting: ", new_twoD_tensor.dtype) 42 | 43 | #Lets convert panadas dataframe to a tenosr 44 | df = pd.DataFrame({'a':[11,21,31],'b':[12,22,312]}) 45 | 46 | print("Pandas Dataframe to numpy: ", df.values) 47 | print("Type BEFORE converting: ", df.values.dtype) 48 | 49 | 50 | new_tensor = torch.from_numpy(df.values) 51 | print("Tensor AFTER converting: ", new_tensor) 52 | print("Type AFTER converting: ", new_tensor.dtype) 53 | 54 | #Lets convert a Pandas Series to a tensor 55 | df = pd.DataFrame({'A':[11, 33, 22],'B':[3, 3, 2]}) 56 | pandas_to_numpy = df.values 57 | numpy_to_tensor = torch.tensor(pandas_to_numpy) 58 | numpy_to_tensor 59 | 60 | 61 | 62 | #Slicing rows 63 | tensor_example = torch.tensor([[11, 12, 13], [21, 22, 23], [31, 32, 33]]) 64 | sliced_tensor_example = tensor_example[1:3] #Slicing 2nd and 3rd row 65 | print("Result after tensor_example[1:3]: ", sliced_tensor_example) 66 | print("Dimension after tensor_example[1:3]: ", sliced_tensor_example.ndimension()) 67 | 68 | #Dimension of 2nd row of sliced tensor 69 | print("Result after sliced_tensor_example[1]: ", sliced_tensor_example[1]) 70 | print("Dimension after sliced_tensor_example[1]: ", sliced_tensor_example[1].ndimension()) 71 | 72 | #Lets try to get the values in row 2 and 3 in the second column. Note that the code below will not work. 73 | print("Result: ", tensor_example[1:3][1]) 74 | tensor_example[1:3][1] 75 | print("Dimension: ", tensor_example[1:3][1].ndimension()) #This gives dimension of 1 76 | 77 | #In order to get the values in row 2 and 3 in the second column. we have to separate with a comma. 78 | tensor_example[1:3, 1] 79 | 80 | 81 | #Lets modify values in a 2D tensor. 82 | tensor_ques = torch.tensor([[11, 12, 13], [21, 22, 23], [31, 32, 33]]) 83 | #We will modify the values in the second column of the second and 3rd row. 84 | tensor_ques[1:3, 1] = 0 85 | tensor_ques 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | -------------------------------------------------------------------------------- /Week 1/2_D Tensors.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | 4 | 2-D Tensors 5 | 6 | """ 7 | 8 | """ 9 | A 2d tensor can be viewed as a container the holds numerical values of the same type. 10 | In 2D tensors are essentially a matrix. Each row is a different sample and each column is a feature or attribute. 11 | 12 | We can also represent gray-scale images as 2D tensors. 13 | The image intensity values can be represented as numbers between 0 and 255. 0 corresponds to color black and 255 white. 14 | 15 | Tensors can be extended to any number of dimensions. A 3D tensor is a combination of 3 1D tensors. 16 | 17 | 18 | 19 | """ 20 | 21 | 22 | import torch 23 | import numpy 24 | import pandas 25 | 26 | #Lets create a 2-D tensor 27 | #We first create a list with 3 nested lists. 28 | a = [[11,12,13], [21,22,23], [31,32,33]] 29 | b = torch.tensor([[11,12,13], [21,22,23], [31,32,33]]) 30 | 31 | #We then cast the list to a torch tensor 32 | A = torch.tensor(a) 33 | print(A) 34 | 35 | #Lets check the no. of dimensions or rank 36 | print("Numer of dimensions of A = ", A.ndimension()) 37 | 38 | #The first list [] represents the first dimensions and the second represents the second dimension 39 | #2D Tensors is as follows: [[]] 40 | 41 | #Lets check the number of rows and columns of A. It should be 3,3 --- 3 rows, 3 columns 42 | print("Shape of tensor A: ", A.shape) 43 | #OR 44 | print("Shape of tensor A: ", A.size()) 45 | 46 | #The 3,3 tensor has 2 axes. Axis = 0 (vertical) and Axis = 1 (Horizontal) 47 | 48 | #Number of elements in a tensor -- using numel() method 49 | print("Number of elements in A: ", A.numel()) 50 | 51 | 52 | #Indexing and Slicing 2D Tensors 53 | 54 | #Indexing 55 | print(A) 56 | A[0][1] #Element in 1st row and 2nd column 57 | A[1][2] #Element in 2nd row and 3rd column 58 | A[2][0] #Element in 3rd row and first column 59 | 60 | #Slicing 61 | A[1:3,2] #Slicing elements in rows 2 and 3 from the 3rd column 62 | A[2,0:3] #Slicing all the elements in the 3rd row 63 | 64 | 65 | #Adding 2D tensors only works for tensors of the same type 66 | #Lets add A and B. Elements of the same position will be added 67 | B = torch.tensor([[11,12,13], [21,22,23], [31,32,33]]) 68 | C = A + B 69 | C 70 | 71 | #Multiplication by a scalar is the same as multiplying a matrix by a scalr 72 | #Multiplication of tensors is an elemenet-wise multiplication. Same position elements 73 | D = A*B 74 | print(D) 75 | 76 | 77 | #Matrix multiplication can be done in torch but same rules will apply 78 | #First matrix must have equal columns to the rows of the second matrix 79 | 80 | A = torch.tensor([[0,1,1],[1,0,1]]) 81 | B = torch.tensor([[1,1],[1,1],[-1,1]]) 82 | 83 | #Matrix multiplication is done by using the mm method 84 | C = torch.mm(A,B) 85 | print(C) 86 | -------------------------------------------------------------------------------- /Week 1/Dataset Exercise.py: -------------------------------------------------------------------------------- 1 | ''' 2 | 3 | Dataset Exercise 4 | 5 | ''' 6 | 7 | #Imports needed 8 | import torch 9 | import matplotlib.pylab as plt 10 | import numpy as np 11 | from torch.utils.data import Dataset, DataLoader 12 | torch.manual_seed(0) 13 | from matplotlib.pyplot import imshow 14 | from PIL import Image 15 | import pandas as pd 16 | import os 17 | 18 | #Function needed for this exercise 19 | def show_data(data_sample, shape = (28, 28)): 20 | plt.imshow(data_sample[0].numpy().reshape(shape), cmap='gray') #Coverts to numpy and reshapes 21 | plt.title('y = ' + data_sample[1]) #Plots title of the image 22 | 23 | # Read CSV file from the URL and print out the first five samples 24 | directory = "" 25 | csv_file ='index.csv' 26 | csv_path = os.path.join(directory,csv_file) 27 | 28 | data_name = pd.read_csv(csv_path) 29 | data_name.head() 30 | 31 | #The first column of the dataframe corresponds to the type of clothing. 32 | #The second column is the name of the image file corresponding to the clothing. 33 | data_name.iloc[0, 0] #Obtaining the class of first sample. 34 | data_name.iloc[0, 1] #Obtaining the name of the first sample 35 | 36 | 37 | #Lets load the images and assign their name and path 38 | image_name = data_name.iloc[1, 1] 39 | image_name 40 | 41 | image_path = os.path.join(directory,image_name) 42 | image_path 43 | 44 | #Lets create a loop that loads all 100 images 45 | for i in range (0,101): #Lets plot all the images in the imageset 46 | image_name = data_name.iloc[i,1] #we iterate through every image in the i'th row, 2nd column. 47 | image_path = os.path.join(directory,image_name) #We set the image path 48 | image = Image.open(image_path) #We open the image from its path 49 | #Lets see the image 50 | plt.imshow(image, cmap = 'gray', vmin = 0, vmax = 255) 51 | plt.title(data_name.iloc[i,1]) 52 | plt.show() 53 | 54 | 55 | #Lets create a Dataset class 56 | class Dataset(Dataset): 57 | 58 | # Constructor 59 | def __init__(self, csv_file, data_dir, transform=None): 60 | 61 | # Image directory 62 | self.data_dir = data_dir 63 | 64 | # The transform is goint to be used on image 65 | self.transform = transform 66 | data_dircsv_file = os.path.join(self.data_dir,csv_file) 67 | # Load the CSV file contians image info 68 | self.data_name = pd.read_csv(data_dircsv_file) 69 | 70 | # Number of images in dataset 71 | self.len=self.data_name.shape[0] 72 | 73 | # Get the length 74 | def __len__(self): 75 | return self.len 76 | 77 | # Getter 78 | def __getitem__(self, idx): 79 | 80 | # Image file path 81 | img_name = os.path.join(self.data_dir,self.data_name.iloc[idx, 1]) 82 | # Open image file 83 | image = Image.open(img_name) 84 | 85 | # The class label for the image 86 | y = self.data_name.iloc[idx, 0] 87 | 88 | # If there is any transform method, apply it onto the image 89 | if self.transform: 90 | image = self.transform(image) 91 | 92 | return image, y 93 | 94 | #Creating a dataset object 95 | dataset = Dataset(csv_file = csv_file, data_dir = directory) 96 | 97 | #Lets check the image name and label and plot 98 | image = dataset[0][0] 99 | y = dataset[0][1] 100 | plt.imshow(image,cmap='gray', vmin=0, vmax=255) 101 | plt.title(y) 102 | plt.show() 103 | 104 | 105 | #Lets transform the images 106 | #We need transforms from torchvision 107 | import torchvision.transforms as transforms 108 | 109 | #We can use the Compose method to combine transforms in the order we want 110 | #We will transform 28x28 images to 20x20 tensor 111 | croptensor_data_transform = transforms.Compose([transforms.CenterCrop(20), transforms.ToTensor()]) 112 | dataset = Dataset(csv_file=csv_file , data_dir=directory,transform=croptensor_data_transform ) 113 | print("The shape of the first element tensor: ", dataset[0][0].shape) 114 | 115 | 116 | #Plot the first element in the dataset 117 | #We will use the function shwo_data that was created 118 | show_data(dataset[1], shape = (20, 20)) 119 | show_data(dataset[2], shape = (20, 20)) 120 | 121 | #Lets try another combination of transforms 122 | fliptensor_data_transform = transforms.Compose([transforms.RandomVerticalFlip(p=1),transforms.ToTensor()]) 123 | dataset_flipped = Dataset(csv_file=csv_file , data_dir=directory,transform=fliptensor_data_transform ) 124 | show_data(dataset_flipped[1]) 125 | 126 | #Lets try another transform combination 127 | my_data_transform = transforms.Compose([transforms.RandomVerticalFlip(p = 1), transforms.RandomHorizontalFlip(p = 1), transforms.ToTensor()]) 128 | dataset_transformed = Dataset(csv_file=csv_file , data_dir=directory,transform=fliptensor_data_transform ) 129 | show_data(dataset_transformed[1]) 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | -------------------------------------------------------------------------------- /Week 1/Dataset and imgs.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Dataset for Images 4 | 5 | """ 6 | 7 | import torch 8 | from PIL import Image 9 | import pandas as pd 10 | import os 11 | from matplotlib.pyplot import imshow 12 | import matplotlib.pyplot as plt 13 | from torch.utils.data import Dataset, DataLoader 14 | import torchvision.transforms as transforms 15 | 16 | #We will use MNIST dataset, 28x28x1 images 17 | #The dataset has 10 labels or classes 18 | 19 | directory = r"C:\*******\Deep Neural Networks with PyTorch\Week 1" 20 | csv_file = 'index.csv' 21 | csv_path = os.path.join(directory,csv_file) 22 | 23 | data_name = pd.read_csv((csv_path)) 24 | data_name.head() #Lets view the dataframe 25 | 26 | print('File name:', data_name.iloc[0,1]) 27 | print('class or y:', data_name.iloc[0,0]) 28 | 29 | for i in range(len(data_name)): #Lets plot all the images in the imageset 30 | image_name = data_name.iloc[i,1] #we iterate through every image in the i'th row, 2nd column. 31 | image_path = os.path.join(directory,image_name) #We set the image path 32 | image = Image.open(image_path) #We open the image from its path 33 | #Lets see the image 34 | plt.imshow(image, cmap = 'gray', vmin = 0, vmax = 255) 35 | plt.title(data_name.iloc[i,1]) 36 | plt.show() 37 | 38 | 39 | #Lets create a dataset class 40 | 41 | class Dataset(Dataset): 42 | def __init__(self, csv_file, data_dir, transform = None): 43 | 44 | self.transform = transform 45 | self.data_dir = data_dir 46 | data_dircsv_file = os.path.join(self.data_dir,csv_file) 47 | self.data_name = pd.read_csv(data_dircsv_file) 48 | self.len = self.data_name.shape[0] 49 | 50 | def __lef__(self): 51 | return self.len 52 | 53 | def __getitem__(self, idx): 54 | 55 | img_name = os.path.join(self.data_dir, self.data_name.iloc[idx, 1]) 56 | image = Image.open(img_name) 57 | 58 | y = self.data_name.iloc[idx, 0] 59 | 60 | if self.transform: 61 | image = self.transform(image) 62 | return image, y 63 | 64 | 65 | #Lets use torchvision pre-built transforms for images 66 | transforms.CenterCrop(20) #Lets crop the image to 20x20 67 | transforms.ToTensor() #Converting the image to a tensor 68 | 69 | #Or we can combine the transforms 70 | #Lets compose the transforms 71 | croptensor_data_transform = transforms.Compose([transforms.CenterCrop(20), transforms.ToTensor()]) 72 | 73 | #Now we apply the composed transforms to the dataset constructor 74 | dataset = Dataset(csv_file = csv_file, data_dir = directory, transform = croptensor_data_transform) 75 | dataset[0][0].shape #The dataset is 20x20x1 76 | 77 | 78 | 79 | #Lets try a prebuilt dataset from torchvision 80 | import torchvision.datasets as dsets 81 | 82 | #We creat the dataset object 83 | #Root is root directory of dataset, train parameter indicates if we want to use training or testing sets. 84 | #Download = True: downlaods the dataset into the directory. We set transform parameter to convert image to tensor. 85 | dataset = dsets.MNIST(root = './data', train = False, download = True, transform = transforms.ToTensor) 86 | 87 | 88 | 89 | 90 | 91 | -------------------------------------------------------------------------------- /Week 1/Differentiation.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Differentiation in Pytorch 4 | 5 | """ 6 | 7 | import torch 8 | import numpy as np 9 | import pandas as pd 10 | import matplotlib.pylab as plt 11 | 12 | #Evaluation of derivatives in torch 13 | x = torch.tensor(2, requires_grad=True, dtype=float) #requires_grad tells torch that x value will be used to evaluate functions 14 | y = x**2 15 | #To calculate derivative of y 16 | y.backward() #Backward function Calculates the derivative of y wrt. to x 17 | print(x.grad) 18 | 19 | 20 | #Lets create a new tensor z and evaluate its derivative wrt x 21 | x = torch.tensor(2, requires_grad=True, dtype=float) 22 | z = x**2 + 2*x + 1 23 | z.backward() 24 | print(x.grad) 25 | 26 | 27 | #Partial Derivatives 28 | u = torch.tensor(1, requires_grad=True, dtype=float) 29 | v = torch.tensor(2, requires_grad=True, dtype=float) 30 | f = u*v + u**2 31 | f.backward() 32 | print(u.grad) 33 | print(v.grad) 34 | 35 | 36 | ############################################################################## 37 | 38 | # Create a tensor x 39 | x = torch.tensor(2.0, requires_grad = True) 40 | print("The tensor x: ", x) 41 | 42 | # Create a tensor y according to y = x^2 43 | y = x ** 2 44 | print("The result of y = x^2: ", y) 45 | 46 | # Take the derivative. Try to print out the derivative at the value x = 2 47 | y.backward() 48 | print("The dervative at x = 2: ", x.grad) 49 | 50 | #Below are the attributes of x and y that torch creates. 51 | print('data:',x.data) 52 | print('grad_fn:',x.grad_fn) 53 | print('grad:',x.grad) 54 | print("is_leaf:",x.is_leaf) 55 | print("requires_grad:",x.requires_grad) 56 | 57 | print('data:',y.data) 58 | print('grad_fn:',y.grad_fn) 59 | print('grad:',y.grad) 60 | print("is_leaf:",y.is_leaf) 61 | print("requires_grad:",y.requires_grad) 62 | 63 | 64 | #We can implement our own custom autograd Functions by subclassing torch.autograd.Function 65 | #And implementing the forward and backward passes which operate on Tensors 66 | 67 | class SQ(torch.autograd.Function): 68 | 69 | 70 | @staticmethod 71 | def forward(ctx,i): 72 | """ 73 | In the forward pass we receive a Tensor containing the input and return 74 | a Tensor containing the output. ctx is a context object that can be used 75 | to stash information for backward computation. You can cache arbitrary 76 | objects for use in the backward pass using the ctx.save_for_backward method. 77 | """ 78 | result=i**2 79 | ctx.save_for_backward(i) 80 | return result 81 | 82 | @staticmethod 83 | def backward(ctx, grad_output): 84 | """ 85 | In the backward pass we receive a Tensor containing the gradient of the loss 86 | with respect to the output, and we need to compute the gradient of the loss 87 | with respect to the input. 88 | """ 89 | i, = ctx.saved_tensors 90 | grad_output = 2*i 91 | return grad_output 92 | 93 | 94 | #Lets apply the function 95 | x=torch.tensor(2.0,requires_grad=True ) 96 | sq=SQ.apply 97 | y=sq(x) 98 | y 99 | print(y.grad_fn) 100 | y.backward() 101 | x.grad 102 | 103 | 104 | # Calculate the derivative with multiple values 105 | x = torch.linspace(-10, 10, 10, requires_grad = True) 106 | Y = x ** 2 107 | y = torch.sum(x ** 2) 108 | 109 | # Take the derivative with respect to multiple value. Plot out the function and its derivative 110 | y.backward() 111 | #The method detach()excludes further tracking of operations in the graph, and therefore the subgraph will not record operations 112 | plt.plot(x.detach().numpy(), Y.detach().numpy(), label = 'function') 113 | plt.plot(x.detach().numpy(), x.grad.detach().numpy(), label = 'derivative') 114 | plt.xlabel('x') 115 | plt.legend() 116 | plt.show() 117 | 118 | # Take the derivative of Relu with respect to multiple value. Plot out the function and its derivative 119 | x = torch.linspace(-10, 10, 1000, requires_grad = True) 120 | Y = torch.relu(x) 121 | y = Y.sum() 122 | y.backward() 123 | plt.plot(x.detach().numpy(), Y.detach().numpy(), label = 'function') 124 | plt.plot(x.detach().numpy(), x.grad.detach().numpy(), label = 'derivative') 125 | plt.xlabel('x') 126 | plt.legend() 127 | plt.show() 128 | print(y.grad_fn) 129 | -------------------------------------------------------------------------------- /Week 1/Prebuilt Dataset Exercise.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Prebuilt Dataset 4 | 5 | """ 6 | #Imports needed 7 | import torch 8 | import matplotlib.pylab as plt 9 | import numpy as np 10 | torch.manual_seed(0) 11 | from matplotlib.pyplot import imshow 12 | from PIL import Image 13 | import torchvision.transforms as transforms 14 | import torchvision.datasets as dsets 15 | 16 | #Function needed for this exercise 17 | def show_data(data_sample, shape = (28, 28)): 18 | plt.imshow(data_sample[0].numpy().reshape(shape), cmap='gray') #Coverts to numpy and reshapes 19 | plt.title('y = ' + data_sample[1]) #Plots title of the image 20 | 21 | 22 | #Lets Import the prebuilt dataset into variable dataset 23 | #We creat the dataset object 24 | #Root is root directory of dataset, train parameter indicates if we want to use training or testing sets. 25 | #Download = True: downlaods the dataset into the directory. We set transform parameter to convert image to tensor. 26 | dataset = dsets.MNIST(root = './data', train = False, download = True, transform = transforms.ToTensor()) 27 | 28 | #Each element of the dataset object contains a tuple. 29 | #Lets check the attributes of the elements 30 | print("Type of the first element: ", type(dataset[0])) 31 | print("The length of the tuple: ", len(dataset[0])) 32 | print("The shape of the first element in the tuple: ", dataset[0][0].shape) 33 | print("The type of the first element in the tuple", type(dataset[0][0])) 34 | print("The second element in the tuple: ", dataset[0][1]) 35 | print("The type of the second element in the tuple: ", type(dataset[0][1])) 36 | print("As the result, the structure of the first element in the dataset is (tensor([1, 28, 28]), tensor(7)).") 37 | 38 | show_data(dataset[0]) #The first sample at index 0 is the img of number 7 39 | show_data(dataset[1]) #The second sample at index 1 is the img of number 2 40 | 41 | 42 | #Lets apply some combination of transforms on the dataset using Compose 43 | #We will crop to 20x20 and transform to tensor 44 | croptensor_data_transform = transforms.Compose([transforms.CenterCrop(20), transforms.ToTensor()]) 45 | dataset = dsets.MNIST(root = './data', train = False, download = True, transform = croptensor_data_transform) 46 | print("The shape of the first element in the first tuple: ", dataset[0][0].shape) #The shape is now 1x20x20 from 1x28x28 47 | 48 | # Plot the first element in the dataset (The number 7) 49 | show_data(dataset[0],shape = (20, 20)) #The new cropped img does not have as much black as before 50 | # Plot the second element in the dataset (The number 2) 51 | show_data(dataset[1],shape = (20, 20)) #The new cropped img does not have as much black as before 52 | 53 | #Lets try another kind of transform combination 54 | #Lets flip the image horizontally and convert it to a tensor 55 | #We will have to transform the original source dataset 56 | fliptensor_data_transform = transforms.Compose([transforms.RandomHorizontalFlip(p = 1),transforms.ToTensor()]) 57 | dataset = dsets.MNIST(root = './data', train = False, download = True, transform = fliptensor_data_transform) 58 | show_data(dataset[1]) 59 | show_data(dataset[2]) 60 | 61 | #Lets do a vertical flip then a horizontal flip then transform to tensor 62 | my_data_transform = transforms.Compose([transforms.RandomVerticalFlip(p = 1), transforms.RandomHorizontalFlip(p = 1), transforms.ToTensor()]) 63 | dataset = dsets.MNIST(root = './data', train = False, download = True, transform = my_data_transform) 64 | show_data(dataset[1]) 65 | show_data(dataset[2]) 66 | -------------------------------------------------------------------------------- /Week 1/Simple_dataset.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Simple Dataset 4 | 5 | """ 6 | 7 | #To create a dataset class in torch. 8 | import torch 9 | from torch.utils.data import Dataset #for dataset creation in torch 10 | #The torch.manual_seed() is for forcing the random function to give the same number every time we try to recompile it. 11 | torch.manual_seed(1) 12 | 13 | #We create out own dataset toy_set 14 | class toy_set(Dataset): 15 | 16 | # Constructor with defult values 17 | def __init__(self, length = 100, transform = None): 18 | self.len = length 19 | self.x = 2 * torch.ones(length, 2) 20 | self.y = torch.ones(length, 1) 21 | self.transform = transform 22 | 23 | # Getter 24 | def __getitem__(self, index): 25 | sample = self.x[index], self.y[index] 26 | if self.transform: 27 | sample = self.transform(sample) 28 | return sample 29 | 30 | # Get Length 31 | def __len__(self): 32 | return self.len 33 | 34 | #let us create our toy_set object, find out the value on index 1 and the length of the inital dataset 35 | our_dataset = toy_set() 36 | print("Our toy_set object: ", our_dataset) 37 | print("Value on index 0 of our toy_set object: ", our_dataset[0]) 38 | print("Our toy_set length: ", len(our_dataset)) 39 | 40 | #We are able to customize the indexing and length method by def __getitem__(self, index) and def __len__(self). 41 | 42 | #Print out the first 3 elements and assign them to x and y. We use a for loop for this. 43 | 44 | for i in range(3): #This loop will iterate through indexes 0,1,2 and fetches the x, y value for each index. 45 | x, y = our_dataset[i] 46 | print("index", i, "x:", x, "y", y) 47 | 48 | 49 | for i in range(len(our_dataset)): #This loop prints all the x,y values in the dataset with each index numbered. 50 | print(i) 51 | print(' x:', x, 'y:', y) 52 | 53 | #Another way to do this is as follows: 54 | for x,y in our_dataset: 55 | print(' x:', x, 'y:', y) 56 | 57 | 58 | #Lets create a new dataset with length 50 59 | my_new_dataset = toy_set(length=50) 60 | print("My new dataset length: ", len(my_new_dataset)) 61 | #Lets print all the values. 62 | for i in range(len(my_new_dataset)): #This loop prints all the x,y values in the dataset with each index numbered. 63 | print(i) 64 | print(' x:', x, 'y:', y) 65 | 66 | 67 | 68 | """ 69 | Lets create a class to transform data. 70 | 71 | """ 72 | 73 | class add_mult(object): #We create a transform class 74 | 75 | #Constructor 76 | def __init__(self, addx = 1, muly = 2): 77 | self.addx = addx 78 | self.muly = muly 79 | 80 | #Executor 81 | def __call__(self, sample): # 82 | x = sample[0] 83 | y = sample[1] 84 | x = x + self.addx #Transform will add 1 to x 85 | y = y * self.muly #Transform will multiply y by 2 86 | sample = x, y 87 | return sample 88 | 89 | #Now lets create a transform object 90 | a_m = add_mult() 91 | data_set = toy_set() 92 | 93 | #Lets Assign the outputs of the original dataset to x and y. 94 | #And apply the transform add_mult to the dataset and output the values as x_ and y_, respectively: 95 | #Lets iterate over the first 10 elements and show the original x and y and tranformed x and y 96 | for i in range(10): 97 | x, y = data_set[i] 98 | print('Index : ', i, 'Original x: ', x, 'Original y: ', y) 99 | x_,y_ = a_m(data_set[i]) 100 | print('Index : ', i, 'Transformed x: ', x_, 'Transformed y: ', y_) 101 | 102 | 103 | 104 | """ 105 | 106 | Composing multiple transform on data. 107 | 108 | """ 109 | 110 | #You can compose multiple transforms on the dataset object. 111 | #We need to import transforms from torchvision 112 | 113 | from torchvision import transforms 114 | 115 | #Lets create a new transform class 116 | 117 | class mult(object): 118 | 119 | # Constructor 120 | def __init__(self, mult = 100): 121 | self.mult = mult 122 | 123 | # Executor 124 | def __call__(self, sample): 125 | x = sample[0] 126 | y = sample[1] 127 | x = x * self.mult 128 | y = y * self.mult 129 | sample = x, y 130 | return sample 131 | 132 | 133 | 134 | #Now we have 2 transform classes and we can use Compose to combine them 135 | data_transform = transforms.Compose([add_mult(), mult()]) 136 | print("The combination of transforms (Compose: ", data_transform) 137 | data_transform(data_set[0]) #Lets apply our combined transform to index 0 of data_set and see it 138 | x,y = data_transform(data_set[0]) 139 | 140 | print( 'Original x: ', x, 'Original y: ', y) 141 | print( 'Transformed x_:', x_, 'Transformed y_:', y_) 142 | 143 | #Now we can pass the new Compose object (The combination of methods add_mult() and mult) to the constructor for creating toy_set object. 144 | #Create a new toy_set object with compose object as transform 145 | compose_data_set = toy_set(transform = data_transform) 146 | 147 | #Lets print out the elements in our new transformed dataset 148 | for i in range(len(compose_data_set)): 149 | print('Index: ', i, ' x: ', x, 'y: ', y) 150 | 151 | 152 | #Lets reverse the order of the transforms 153 | my_compose = transforms.Compose([mult(), add_mult()]) 154 | my_transformed_dataset = toy_set(transform = my_compose) 155 | for i in range(len(my_transformed_dataset)): 156 | print('Index: ', i, 'Transformed x:', x_, 'Transformed y:', y_) 157 | 158 | 159 | 160 | 161 | 162 | -------------------------------------------------------------------------------- /Week 1/Tensors 1 D Overview.py: -------------------------------------------------------------------------------- 1 | """ 2 | 1-D Tensors in Pytorch 3 | ------------------------------------------------------------------------------ 4 | 5 | Tenors are arrays that are the building blocks of a neural network. 6 | 7 | - A 0-D tenors is just a number: 1, 2, 0.2 , 10 8 | 9 | 1-D tensor is an array of numbers. It can be a row in a database, a vector, time series etc... 10 | A tensor contains elements of a single data type. The Tensor Type is the type of tensor. 11 | For real numbers: Tensor type is either a float or a double tensor or even half tensor. 12 | 13 | 32-bit float - Float Tensor 14 | 64-bit float - Double Tensor 15 | 16-bit float - Half Tensor 16 | 8-bit int (unsigned)(8-bit images)- Byte Tensor 17 | 8-bit int (signed)- Char Tensor 18 | 16-bit int (singed)- Short Tensor 19 | 32-bit int (singed)- Int Tensor 20 | 64-bit int (signed)- Long Tensor 21 | 22 | 23 | """ 24 | 25 | 26 | #To create a 1-D Tensor: 27 | 28 | import torch 29 | 30 | torch.__version__ 31 | 32 | 33 | a = torch.tensor([7,4,3,2,6]) # create a list and cast to pytorch tensor 34 | 35 | #To find the datatype in the tensor: 36 | print(a.dtype) 37 | 38 | #To find the type of tensor: 39 | print(a.type()) 40 | 41 | 42 | #Lets create a float tensor 43 | b = torch.tensor([0.0, 1.0, 2.0, 3.0, 4.0]) 44 | 45 | #To find the datatype in the tensor: 46 | print(b.dtype) 47 | 48 | #To find the type of tensor: 49 | print(b.type()) 50 | 51 | 52 | #Specifying data type of tensor within constructor also works 53 | c = torch.tensor([0.0, 1.0, 2.0, 3.0, 4.0], dtype=torch.float64) 54 | 55 | print(c.dtype) 56 | 57 | 58 | #Explicitely creating a tensor of a specific type 59 | d = torch.FloatTensor([0, 1, 2, 3, 4]) 60 | print(d.dtype) 61 | 62 | #Converting tensor type 63 | a = a.type(torch.FloatTensor) 64 | print("New type of a: ", a.dtype) 65 | 66 | 67 | #Checking tensor size. How many elements in the tensor? 68 | print(a.size()) 69 | 70 | #Checking no. of dimensions of the tensor 71 | print(c.ndimension()) 72 | 73 | 74 | #Converting a 1-D tensor to 2-D tensor by casting 75 | a_col = a.view(5,1) #5 is no. of rows. 1 is number of columns 76 | #If we dont know the number of elements (5) then we can use this argument 77 | a_col = a.view(-1,1) 78 | 79 | #Lets make sure it worked 80 | print(a_col.ndimension()) 81 | 82 | 83 | 84 | #Creating a column tensor. Using an extra set of squared brackets[] 85 | Col_tensor = torch.tensor([[1,2,3,4,5]]) 86 | Col_tensor.ndimension() 87 | 88 | #We can convert a numpy array to torch tensor this way 89 | import numpy as np 90 | numpy_array = np.array([0.0, 1.0, 2.0, 3.0, 4.0]) 91 | torch_tensor = torch.from_numpy(numpy_array) 92 | torch_tensor 93 | 94 | #Converting torch tensor back to numpy array 95 | back_to_numpy = torch_tensor.numpy() 96 | back_to_numpy 97 | 98 | #Converting Pandas series to tensor 99 | import pandas as pd 100 | pandas_series = pd.Series([0.1,2,0.3,10.1]) 101 | pandas_to_torch = torch.from_numpy(pandas_series.values) 102 | pandas_to_torch 103 | 104 | #Converting tensor to list 105 | this_tensor = torch.tensor([0,1,2,3]) 106 | torch_to_list=this_tensor.tolist() 107 | torch_to_list 108 | 109 | ''' 110 | 111 | Individual values of tensors are also tensors 112 | 113 | ''' 114 | 115 | new_tensor = torch.tensor([5,2,6,1]) 116 | new_tensor[0] 117 | new_tensor[1] 118 | 119 | #We can use .item to return the number in the tensor 120 | new_tensor[0].item() 121 | new_tensor[1].item() 122 | 123 | 124 | 125 | ''' 126 | 127 | Indexing and slicing methods for Tensors 128 | 129 | ''' 130 | 131 | c = torch.tensor([20,1,2,3,4]) 132 | #lets change the first tensor in C tensor to 0 133 | c[0] = 0 134 | 135 | print(c) 136 | 137 | #slice - select elements in c and assign them to d 138 | d = c[1:4] 139 | print(d) 140 | 141 | #Add or replace values in tensor c 142 | c[3:5] = torch.tensor([5,6]) 143 | 144 | 145 | #Vector addition in pytorch 146 | #Addition must be done with vectors of the same type 147 | u = torch.tensor([1.0, 5.0]) 148 | v = torch.tensor([0.0, 3.0]) 149 | z = u + v 150 | z = z.type(torch.FloatTensor) 151 | print(z) 152 | 153 | 154 | #Vector multiplicaiton with a Scalar 155 | y = torch.tensor([1,2]) 156 | z = 2*y 157 | print(z) 158 | 159 | #Product of two tensors (element wise multiplication) 160 | u = torch.tensor([1, 2]) 161 | v = torch.tensor([3, 2]) 162 | z = u*v 163 | print(z) 164 | 165 | #Dot product - element-wise multiplication and summation 166 | y = torch.dot(u,v) 167 | print(y) #y is still a tensor 168 | 169 | #Adding a scalar value to a tensor aka. BROADCASTING 170 | u = torch.tensor([1,2,3,-1]) 171 | z = u+1 172 | print(z) 173 | 174 | #Changing value of many indexes at once 175 | practice_tensor = torch.tensor([2, 7, 3, 4, 6, 2, 3, 1, 2]) 176 | selected_indexes = [3, 4, 7] 177 | practice_tensor[[3,4,7]] = 0 178 | #Or 179 | practice_tensor[selected_indexes] = 0 180 | practice_tensor 181 | ''' 182 | Functions on tensors 183 | ''' 184 | #MEAN 185 | a = torch.tensor([1,-1,1,-1], dtype=float) 186 | print(a.dtype) 187 | mean_a = a.mean() 188 | mean_a 189 | 190 | #STANDARD DEVIATION 191 | Standard_deviation = a.std() 192 | Standard_deviation 193 | 194 | #MAX 195 | b = torch.tensor([1, -1, 3, 4, 100]) 196 | MaxB = b.max() 197 | MaxB 198 | 199 | #MIN 200 | MinB = b.min() 201 | MinB 202 | 203 | #Create a torch tensor in radiance using pi 204 | np.pi 205 | x = torch.tensor([0, np.pi/2, np.pi]) 206 | x 207 | 208 | #Apply sin(x) to tensor x 209 | y = torch.sin(x) 210 | y 211 | 212 | #Using linspace to plot mathematical functions 213 | M = torch.linspace(-2, 2, steps=5) #-2 is starting point, 2 is ending, 5 is number of elements to generate 214 | M 215 | 216 | x = torch.linspace(0, 2*np.pi, 100) #Generating 100 evenly spaced elements from 0 to 2pi 217 | x 218 | 219 | y = torch.sin(x) 220 | y 221 | 222 | 223 | import matplotlib.pyplot as plt 224 | plt.plot(x.numpy(), y.numpy()) #We need to convert tensors to numpy arrays 225 | 226 | 227 | #Constructing tensor with 25 steps in the range of 0 and pi/2. 228 | x = torch.linspace(0, np.pi/2, steps=25) 229 | y = torch.sin(x) 230 | 231 | plt.plot(x.numpy(), y.numpy()) 232 | 233 | print("Max value from tensor x = ", x.max()) 234 | print("Min value from tensor x = ", x.min()) 235 | 236 | #Convert the list [-1, 1] and [1, 1] to tensors u and v. 237 | #Plot the tensor u and v as a vector by using the function plotVec and find the dot product 238 | 239 | u = torch.tensor([-1, 1]) 240 | v = torch.tensor([1, 1]) 241 | z = torch.dot(u,v) 242 | print("Dot product of tensors u and v = ", np.dot(u,v)) 243 | 244 | -------------------------------------------------------------------------------- /Week 1/img.tar.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AyadiGithub/Deep-Neural-Networks-with-PyTorch/824a98999143ede21ba9f0b7fb020fbd5676e603/Week 1/img.tar.gz -------------------------------------------------------------------------------- /Week 2/LR with DataLoader and Optimizers Train and Val.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Linear Regression with DataLoader, Pytorch way 4 | 5 | """ 6 | import torch 7 | from torch import nn 8 | import numpy as np 9 | import matplotlib.pyplot as plt 10 | from mpl_toolkits import mplot3d 11 | from torch.utils.data import Dataset, DataLoader 12 | from torch import optim 13 | 14 | #Creating Data Class 15 | class Data(Dataset): 16 | 17 | #Constructor 18 | def __init__(self, train = True): 19 | self.x = torch.arange(-3, 3, 0.1).view(-1, 1) 20 | self.f = -3 * self.x + 1 21 | self.y = self.f + 0.1 * torch.randn(self.x.size()) 22 | self.len = self.x.shape[0] 23 | 24 | #Creating outliers 25 | if train == True: 26 | self.y[0] = 0 27 | self.y[50:55] = 20 28 | else: 29 | pass 30 | 31 | #Getter 32 | def __getitem__(self, index): 33 | return self.x[index], self.y[index] 34 | 35 | #Get Length 36 | def __len__(self): 37 | return self.len 38 | 39 | #Creating random data 40 | torch.manual_seed(1) 41 | 42 | #Creating train_data object and validation data 43 | train_data = Data() 44 | val_data = Data(train = False) 45 | 46 | #Plot out training points 47 | plt.plot(train_data.x.numpy(), train_data.y.numpy(), 'xr',label="training data ") 48 | plt.plot(train_data.x.numpy(), train_data.f.numpy(),label="true function ") 49 | plt.xlabel('x') 50 | plt.ylabel('y') 51 | plt.legend() 52 | plt.show() 53 | 54 | 55 | class linear_regression(nn.Module): #Creating linear_regression class with attributes from nn.Module 56 | 57 | #Constructor 58 | def __init__(self, input_size, output_size): 59 | super().__init__() #Inheriting methods from parent class nn.module 60 | self.linear = nn.Linear(input_size, output_size) 61 | 62 | #Prediction 63 | def forward(self, x): 64 | yhat = self.linear(x) 65 | return yhat 66 | 67 | 68 | #Using Pytorch built-in functions to create a criterion function 69 | #Using the MSE loss 70 | criterion = nn.MSELoss() 71 | 72 | #Create a DataLoader object 73 | trainloader = DataLoader(dataset = train_data, batch_size = 1) #batch_size 1 74 | 75 | 76 | # Create Learning Rate list, the error lists and the MODELS list 77 | learning_rates=[0.0001, 0.001, 0.01, 0.1] 78 | train_error=torch.zeros(len(learning_rates)) 79 | validation_error=torch.zeros(len(learning_rates)) 80 | MODELS=[] 81 | 82 | 83 | # Define the train model function and train the model 84 | def train_model_with_lr (iter, lr_list): 85 | 86 | #iterate through different learning rates 87 | for i, lr in enumerate(lr_list): 88 | 89 | model = linear_regression(1, 1) #LR with 1 input and 1 output 90 | 91 | optimizer = optim.SGD(model.parameters(), lr = lr) #SGD Optimizer 92 | 93 | for epoch in range(iter): 94 | 95 | for x, y in trainloader: 96 | yhat = model(x) #prediction 97 | 98 | loss = criterion(yhat, y) #loss calculation 99 | 100 | optimizer.zero_grad() #Zeroing gradient 101 | 102 | loss.backward() #backward pass 103 | 104 | optimizer.step() #Re-initialization 105 | 106 | print(model.state_dict()) #Printing the updated parameters for each iteration 107 | 108 | #Training Data 109 | Yhat = model(train_data.x) 110 | 111 | train_loss = criterion(Yhat, train_data.y) 112 | 113 | train_error[i] = train_loss.item() 114 | 115 | #Validation Data 116 | Yhat = model(val_data.x) 117 | 118 | val_loss = criterion(Yhat, val_data.y) 119 | 120 | validation_error[i] = val_loss.item() 121 | 122 | MODELS.append(model) 123 | 124 | train_model_with_lr(10, learning_rates) 125 | 126 | 127 | 128 | #Plot the training loss and validation loss 129 | #Validation error will be smaller because Outliers were added to the train_data for visualization 130 | #Plotting log plot due to learning being in order of 10 131 | plt.semilogx(np.array(learning_rates), train_error.numpy(), label = 'training loss/total Loss') 132 | plt.semilogx(np.array(learning_rates), validation_error.numpy(), label = 'validation cost/total Loss') 133 | plt.ylabel('Cost\ Total Loss') 134 | plt.xlabel('learning rate') 135 | plt.legend() 136 | plt.show() 137 | 138 | 139 | 140 | #Plot the predictions 141 | i = 0 142 | for model, learning_rate in zip(MODELS, learning_rates): 143 | 144 | yhat = model(val_data.x) #Making predictions on Val_data 145 | 146 | plt.plot(val_data.x.numpy(), yhat.detach().numpy(), label = 'lr:' + str(learning_rate)) #Plot yhat for each val_data.x 147 | 148 | print('i', yhat.detach().numpy()[0:3]) 149 | 150 | #Plotting learning rate versus validation data. 151 | plt.plot(val_data.x.numpy(), val_data.f.numpy(), 'or', label = 'validation data') 152 | plt.xlabel('x') 153 | plt.ylabel('y') 154 | plt.legend() 155 | plt.show() 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | -------------------------------------------------------------------------------- /Week 2/MiniBatch Gradient Descent.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Mini-Batch Gradient Descent with DataLoader 4 | 5 | """ 6 | import torch 7 | import numpy as np 8 | import matplotlib.pyplot as plt 9 | from mpl_toolkits import mplot3d 10 | from torch.utils.data import Dataset, DataLoader 11 | 12 | #Add a class plot_error_surfaces to visualize the data space and parameters. 13 | class plot_error_surfaces(object): 14 | 15 | # Constructor 16 | def __init__(self, w_range, b_range, X, Y, n_samples = 30, go = True): 17 | W = np.linspace(-w_range, w_range, n_samples) 18 | B = np.linspace(-b_range, b_range, n_samples) 19 | w, b = np.meshgrid(W, B) 20 | Z = np.zeros((30, 30)) 21 | count1 = 0 22 | self.y = Y.numpy() 23 | self.x = X.numpy() 24 | for w1, b1 in zip(w, b): 25 | count2 = 0 26 | for w2, b2 in zip(w1, b1): 27 | Z[count1, count2] = np.mean((self.y - w2 * self.x + b2) ** 2) 28 | count2 += 1 29 | count1 += 1 30 | self.Z = Z 31 | self.w = w 32 | self.b = b 33 | self.W = [] 34 | self.B = [] 35 | self.LOSS = [] 36 | self.n = 0 37 | if go == True: 38 | plt.figure() 39 | plt.figure(figsize = (7.5, 5)) 40 | plt.axes(projection = '3d').plot_surface(self.w, self.b, self.Z, rstride = 1, cstride = 1,cmap = 'viridis', edgecolor = 'none') 41 | plt.title('Loss Surface') 42 | plt.xlabel('w') 43 | plt.ylabel('b') 44 | plt.show() 45 | plt.figure() 46 | plt.title('Loss Surface Contour') 47 | plt.xlabel('w') 48 | plt.ylabel('b') 49 | plt.contour(self.w, self.b, self.Z) 50 | plt.show() 51 | 52 | # Setter 53 | def set_para_loss(self, W, B, loss): 54 | self.n = self.n + 1 55 | self.W.append(W) 56 | self.B.append(B) 57 | self.LOSS.append(loss) 58 | 59 | # Plot diagram 60 | def final_plot(self): 61 | ax = plt.axes(projection = '3d') 62 | ax.plot_wireframe(self.w, self.b, self.Z) 63 | ax.scatter(self.W, self.B, self.LOSS, c = 'r', marker = 'x', s = 200, alpha = 1) 64 | plt.figure() 65 | plt.contour(self.w, self.b, self.Z) 66 | plt.scatter(self.W, self.B, c = 'r', marker = 'x') 67 | plt.xlabel('w') 68 | plt.ylabel('b') 69 | plt.show() 70 | 71 | # Plot diagram 72 | def plot_ps(self): 73 | plt.subplot(121) 74 | plt.ylim 75 | plt.plot(self.x, self.y, 'ro', label = "training points") 76 | plt.plot(self.x, self.W[-1] * self.x + self.B[-1], label = "estimated line") 77 | plt.xlabel('x') 78 | plt.ylabel('y') 79 | plt.ylim((-10, 15)) 80 | plt.title('Data Space Iteration: ' + str(self.n)) 81 | plt.subplot(122) 82 | plt.contour(self.w, self.b, self.Z) 83 | plt.scatter(self.W, self.B, c = 'r', marker = 'x') 84 | plt.title('Loss Surface Contour Iteration' + str(self.n)) 85 | plt.xlabel('w') 86 | plt.ylabel('b') 87 | plt.show() 88 | 89 | 90 | #Creating random data 91 | torch.manual_seed(1) 92 | 93 | #Generating values 94 | X = torch.arange(-3, 3, 0.1).view(-1, 1) #X -3 to 3 with 0.1 steps 95 | f = 1 * X - 1 96 | Y = f + 0.1 * torch.randn(X.size()) #Adding random noise 97 | 98 | #Lets plot Y, X, f 99 | plt.plot(X.numpy(), Y.numpy(), 'rx', label = 'y') 100 | plt.plot(X.numpy(), f.numpy(), label = 'f') 101 | plt.xlabel('x') 102 | plt.ylabel('y') 103 | plt.legend() 104 | plt.show() 105 | 106 | # Define the forward function 107 | def forward(x): 108 | return w * x + b 109 | 110 | # Define the MSE Loss function 111 | def criterion(yhat, y): 112 | return torch.mean((yhat - y) ** 2) #Mean Squared Error 113 | 114 | #We use plot_error_surfaces to visualize 115 | get_surface = plot_error_surfaces(15, 13, X, Y, 30, go = False) 116 | 117 | #Dataset Creation Class 118 | class Data(Dataset): 119 | #Constructor 120 | def __init__(self): 121 | self.x = torch.arange(-3, 3, 0.1).view(-1, 1) 122 | self.y = 1 * self.x - 1 123 | self.len = self.x.shape[0] 124 | 125 | #Getter 126 | def __getitem__(self,index): 127 | return self.x[index], self.y[index] 128 | 129 | #Return the length 130 | def __len__(self): 131 | return self.len 132 | 133 | """ 134 | 135 | Mini-Batch with batch size = 5 136 | 137 | """ 138 | 139 | #Create a Dataset object 140 | dataset = Data() 141 | trainloader = DataLoader(dataset = dataset, batch_size = 5) #Selecting a batch size of 5. 142 | print("The length of dataset: ", len(dataset)) #Using the len method in the Data class 143 | 144 | # Define the parameters w, b for y = wx + b 145 | w = torch.tensor(-15.0, requires_grad = True) #Requires_grad = true because torch needs to learn it. 146 | b = torch.tensor(-10.0, requires_grad = True) #Requires_grad = true because torch needs to learn it. 147 | 148 | #Define a learning rate and create an empty list LOSS to store the loss for each iteration 149 | lr = 0.1 #Our first choice learning rate 150 | LOSS_MINI5 = [] #Create the MiniBatch Loss empty list 151 | 152 | def train_model_Mini5(epochs): 153 | 154 | for epoch in range(epochs): 155 | Yhat = forward(X) 156 | get_surface.set_para_loss(w.data.tolist(), b.data.tolist(), criterion(Yhat, Y).tolist()) 157 | get_surface.plot_ps() 158 | LOSS_MINI5.append(criterion(forward(X), Y).tolist()) 159 | 160 | for x, y in trainloader: 161 | yhat = forward(x) 162 | loss = criterion(yhat, y) 163 | get_surface.set_para_loss(w.data.tolist(), b.data.tolist(), loss.tolist()) 164 | loss.backward() 165 | w.data = w.data - lr * w.grad.data 166 | b.data = b.data - lr * b.grad.data 167 | w.grad.data.zero_() 168 | b.grad.data.zero_() 169 | 170 | train_model_Mini5(5) 171 | 172 | 173 | """ 174 | 175 | Mini-Batch with batch size = 10 176 | 177 | """ 178 | 179 | #Create a Dataset object 180 | dataset = Data() 181 | trainloader = DataLoader(dataset = dataset, batch_size = 10) #This time we change batch size to 10 182 | print("The length of dataset: ", len(dataset)) #Using the len method in the Data class 183 | 184 | # Define the parameters w, b for y = wx + b 185 | w = torch.tensor(-15.0, requires_grad = True) #Requires_grad = true because torch needs to learn it. 186 | b = torch.tensor(-10.0, requires_grad = True) #Requires_grad = true because torch needs to learn it. 187 | 188 | #Define a learning rate and create an empty list LOSS to store the loss for each iteration 189 | lr = 0.1 #Our first choice learning rate 190 | LOSS_MINI10 = [] #Create the MiniBatch Loss empty list 191 | 192 | def train_model_Mini10(epochs): 193 | 194 | for epoch in range(epochs): 195 | Yhat = forward(X) 196 | get_surface.set_para_loss(w.data.tolist(), b.data.tolist(), criterion(Yhat, Y).tolist()) 197 | get_surface.plot_ps() 198 | LOSS_MINI10.append(criterion(forward(X), Y).tolist()) 199 | 200 | for x, y in trainloader: 201 | yhat = forward(x) 202 | loss = criterion(yhat, y) 203 | get_surface.set_para_loss(w.data.tolist(), b.data.tolist(), loss.tolist()) 204 | loss.backward() 205 | w.data = w.data - lr * w.grad.data 206 | b.data = b.data - lr * b.grad.data 207 | w.grad.data.zero_() 208 | b.grad.data.zero_() 209 | 210 | train_model_Mini10(5) 211 | 212 | 213 | 214 | #Lets Compare the two batch sizes 215 | #Plot out the LOSS_MINI10 and LOSS_MINI5 216 | 217 | plt.plot(LOSS_MINI10,label = " Mini Batch Gradient Descent 10") 218 | plt.plot(LOSS_MINI5, label = " Mini Batch Gradient Descent 5", linestyle='dashed') 219 | plt.xlabel('epoch') 220 | plt.ylabel('Cost/ total loss') 221 | plt.legend() 222 | plt.show() 223 | 224 | 225 | 226 | 227 | -------------------------------------------------------------------------------- /Week 2/Simple Linear Regression 1D Training One Parameter.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Linear Regression 1D: Training One Parameter 4 | 5 | """ 6 | 7 | import torch 8 | import numpy as np 9 | import matplotlib.pyplot as plt 10 | 11 | #Lets use this class for plotting and visualizing the parameter training 12 | class plot_diagram(): 13 | 14 | # Constructor 15 | def __init__(self, X, Y, w, stop, go = False): 16 | start = w.data 17 | self.error = [] 18 | self.parameter = [] 19 | self.X = X.numpy() 20 | self.Y = Y.numpy() 21 | self.parameter_values = torch.arange(start, stop) 22 | self.Loss_function = [criterion(forward(X), Y) for w.data in self.parameter_values] 23 | w.data = start 24 | 25 | # Executor 26 | def __call__(self, Yhat, w, error, n): 27 | self.error.append(error) 28 | self.parameter.append(w.data) 29 | plt.subplot(212) 30 | plt.plot(self.X, Yhat.detach().numpy()) 31 | plt.plot(self.X, self.Y,'ro') 32 | plt.xlabel("A") 33 | plt.ylim(-20, 20) 34 | plt.subplot(211) 35 | plt.title("Data Space (top) Estimated Line (bottom) Iteration " + str(n)) 36 | plt.plot(self.parameter_values.numpy(), self.Loss_function) 37 | plt.plot(self.parameter, self.error, 'ro') 38 | plt.xlabel("B") 39 | plt.figure() 40 | 41 | # Destructor 42 | def __del__(self): 43 | plt.close('all') 44 | 45 | 46 | #Lets generate some values that create a line with a slope of -3 47 | X = torch.arange(-3, 3, 0.1).view(-1, 1) #View changes the shape of the tensor 48 | f = -3 * X #Function of the line 49 | 50 | #Lets plot the line 51 | plt.plot(X.numpy(), f.numpy(), label = 'f') #We need to convert tensor to numpy 52 | plt.xlabel('x') 53 | plt.ylabel('y') 54 | plt.legend() 55 | plt.show() 56 | 57 | #Lets add noise to the data to simulate 'real data'. We use torch.randn to generate Gaussian noise. 58 | Y = f + 0.1 * torch.randn(X.size()) #The noise must be the same size as X 59 | 60 | #Lets plot Y 61 | plt.plot(X.numpy(), Y.numpy(), label = 'Y') #We need to convert tensor to numpy 62 | plt.xlabel('x') 63 | plt.ylabel('y') 64 | plt.legend() 65 | plt.show() 66 | 67 | 68 | 69 | #Create the model and cost function(total loss) 70 | #Create a forward function for prediction 71 | def forward(x): 72 | return w * x 73 | 74 | #Mean Squared Error function for evaluating the result 75 | def criterion(yhat, y): 76 | return torch.mean((yhat - y)**2) 77 | 78 | #Create a learning rate 79 | lr = 0.1 80 | 81 | #Create an empty list to append loss results for each iteration 82 | LOSS = [] 83 | 84 | #We create a parameter w with requires_grad = True to indicate that torch must learn it 85 | w = torch.tensor(-10.0, requires_grad = True) #Initialize w as -10.0 86 | 87 | #Create a plot diagram object to visualize the data and the parameter for each iteration 88 | gradient_plot = plot_diagram(X, Y, w, stop = 5) 89 | 90 | 91 | #Creating a function to train the model 92 | def train_model(iter): 93 | for epoch in range (iter): 94 | 95 | #Prediction Yhat using forward function 96 | Yhat = forward(X) 97 | 98 | #loss calculation using criterion loss function on Yhat, Y 99 | loss = criterion(Yhat,Y) 100 | 101 | #Plotting diagram for visualization 102 | gradient_plot(Yhat, w, loss.item(), epoch) 103 | 104 | #Appending Loss to LOSS list 105 | LOSS.append(loss.item()) 106 | 107 | #Compute the gradient of the loss wrt all parameters 108 | loss.backward() 109 | 110 | #Update parameters 111 | w.data = w.data - lr * w.grad.data 112 | 113 | #Zero the gradients before running the backward pass 114 | w.grad.data.zero_() 115 | 116 | #Lets train the model for 4 iterations 117 | train_model(4) 118 | 119 | #Plotting the list LOSS (loss per iteration) 120 | plt.plot(LOSS) 121 | plt.tight_layout() 122 | plt.xlabel("Epoch/Iterations") 123 | plt.ylabel("Cost") 124 | 125 | 126 | #Lets try a new parameter w 127 | w = torch.tensor(-15.0, requires_grad=True) #Initialize w as -15.0 128 | 129 | #Create an empty list to store the loss 130 | LOSS2 = [] 131 | gradient_plot1 = plot_diagram(X, Y, w, stop = 15) 132 | 133 | 134 | def my_train_model(iter): 135 | for epoch in range (iter): 136 | 137 | #Prediction Yhat using forward function 138 | Yhat = forward(X) 139 | 140 | #loss calculation using criterion loss function on Yhat, Y 141 | loss = criterion(Yhat,Y) 142 | 143 | #Plotting diagram for visualization 144 | gradient_plot1(Yhat, w, loss.item(), epoch) 145 | 146 | #Appending Loss to LOSS list 147 | LOSS2.append(loss.item()) 148 | 149 | #Compute the gradient of the loss wrt all parameters 150 | loss.backward() 151 | 152 | #Update parameters 153 | w.data = w.data - lr * w.grad.data 154 | 155 | #Zero the gradients before running the backward pass 156 | w.grad.data.zero_() 157 | 158 | 159 | my_train_model(4) 160 | 161 | plt.plot(LOSS, label = "LOSS") 162 | plt.plot(LOSS2, label = "LOSS2") 163 | plt.tight_layout() 164 | plt.xlabel("Epoch/Iterations") 165 | plt.ylabel("Cost") 166 | plt.legend() 167 | 168 | 169 | #We notice that the parameter value is sensitive to initialization. -------------------------------------------------------------------------------- /Week 2/Simple Linear Regression.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Simple Linear Regression 4 | 5 | 6 | """ 7 | 8 | ''' 9 | Manual Method 10 | 11 | ''' 12 | import torch 13 | import torch.nn as nn 14 | 15 | w = torch.tensor(2.0, requires_grad = True) #Slope tensor. req_grad = true because they need to be learnt 16 | b = torch.tensor(-1.0, requires_grad = True) #Bias tensor. req_grad = true because they need to be learnt 17 | def forward(x): #Function that will predict y with a given x 18 | y = w * x + b 19 | return y 20 | 21 | x = torch.tensor([1.0]) 22 | yhat = forward(x) #yhat is an estimate of the real value of y 23 | print(yhat) 24 | 25 | x = torch.tensor([[1],[2]]) #2D tensor x 26 | yhat = forward(x) 27 | print(yhat) 28 | 29 | ''' 30 | Using torch.nn Linear 31 | ''' 32 | 33 | #Lets try another method 34 | from torch.nn import Linear 35 | 36 | torch.manual_seed(1) #The seed for generating random numbers = 1 37 | model = Linear(in_features = 1, out_features = 1) 38 | x = torch.tensor([0.0]) 39 | yhat = model(x.float()) 40 | print(yhat) 41 | 42 | #Lets see the model parameters 43 | print(list(model.parameters())) 44 | 45 | x = torch.tensor([[1.0],[2.0]]) 46 | yhat = model(x) 47 | print(yhat) 48 | print(list(model.parameters())) 49 | 50 | 51 | 52 | ''' 53 | Custom class 54 | ''' 55 | 56 | #Lets create a sublass Linear Regression LR within nn.Module 57 | #We user super to call a method from nn.Module without needing to intitalize it 58 | class LR(nn.Module): #LR is defined as a child class of nn.Module 59 | def __init__(self, in_size, out_size): #Constructor 60 | #Inheriting methods from parent class nn.Module 61 | super(LR,self).__init__() 62 | self.linear = nn.Linear(in_size, out_size) 63 | 64 | def forward(self, x): #Prediction funciton 65 | out = self.linear(x) 66 | return out 67 | 68 | model = LR(1,1) 69 | 70 | #We use state_dict to initialize the weight and bias of the model. 71 | #State_dict is a python dictionary. It can be used to map the relationship of linear layers to their parameters. 72 | model.state_dict()['linear.weight'].data[0] = torch.tensor([0.5153]) #Initializing weight 73 | model.state_dict()['linear.bias'].data[0] = torch.tensor([-0.4414]) #initializing bias 74 | print("The Parameters: ", list(model.parameters())) #Lets see the initialized weight and bias 75 | #The above function works because the class LR, is a subclass of Modules. 76 | 77 | #Lets use the custom model to make predictions 78 | x = torch.tensor([[1.0], [2.0]]) #2D tensor 79 | print("The shape of x: ", x.shape) 80 | yhat = model(x) 81 | print(yhat) 82 | 83 | x = torch.tensor([[1.0], [2.0], [3.0]]) 84 | print("The shape of x: ", x.shape) 85 | print("The dimension of x: ", x.ndimension()) 86 | yhat = model(x) 87 | print(yhat) 88 | 89 | 90 | print("Python dictionary:", model.state_dict()) 91 | print("Keys:", model.state_dict().keys()) 92 | print("Values:", model.state_dict().values()) 93 | 94 | 95 | 96 | 97 | 98 | -------------------------------------------------------------------------------- /Week 2/Simple Linear regression 1D Training Two Parameter.py: -------------------------------------------------------------------------------- 1 | 2 | """ 3 | 4 | Linear regression 1D: Training Two Parameter 5 | 6 | """ 7 | 8 | import torch 9 | import numpy as np 10 | import matplotlib.pyplot as plt 11 | from mpl_toolkits import mplot3d 12 | 13 | 14 | class plot_error_surfaces(object): #This class will help visualize the data space and parameters during training. 15 | 16 | # Constructor 17 | def __init__(self, w_range, b_range, X, Y, n_samples = 30, go = True): 18 | W = np.linspace(-w_range, w_range, n_samples) 19 | B = np.linspace(-b_range, b_range, n_samples) 20 | w, b = np.meshgrid(W, B) 21 | Z = np.zeros((30,30)) 22 | count1 = 0 23 | self.y = Y.numpy() 24 | self.x = X.numpy() 25 | for w1, b1 in zip(w, b): 26 | count2 = 0 27 | for w2, b2 in zip(w1, b1): 28 | Z[count1, count2] = np.mean((self.y - w2 * self.x + b2) ** 2) 29 | count2 += 1 30 | count1 += 1 31 | self.Z = Z 32 | self.w = w 33 | self.b = b 34 | self.W = [] 35 | self.B = [] 36 | self.LOSS = [] 37 | self.n = 0 38 | if go == True: 39 | plt.figure() 40 | plt.figure(figsize = (7.5, 5)) 41 | plt.axes(projection='3d').plot_surface(self.w, self.b, self.Z, rstride = 1, cstride = 1,cmap = 'viridis', edgecolor = 'none') 42 | plt.title('Cost/Total Loss Surface') 43 | plt.xlabel('w') 44 | plt.ylabel('b') 45 | plt.show() 46 | plt.figure() 47 | plt.title('Cost/Total Loss Surface Contour') 48 | plt.xlabel('w') 49 | plt.ylabel('b') 50 | plt.contour(self.w, self.b, self.Z) 51 | plt.show() 52 | 53 | # Setter 54 | def set_para_loss(self, W, B, loss): 55 | self.n = self.n + 1 56 | self.W.append(W) 57 | self.B.append(B) 58 | self.LOSS.append(loss) 59 | 60 | # Plot diagram 61 | def final_plot(self): 62 | ax = plt.axes(projection = '3d') 63 | ax.plot_wireframe(self.w, self.b, self.Z) 64 | ax.scatter(self.W,self.B, self.LOSS, c = 'r', marker = 'x', s = 200, alpha = 1) 65 | plt.figure() 66 | plt.contour(self.w,self.b, self.Z) 67 | plt.scatter(self.W, self.B, c = 'r', marker = 'x') 68 | plt.xlabel('w') 69 | plt.ylabel('b') 70 | plt.show() 71 | 72 | # Plot diagram 73 | def plot_ps(self): 74 | plt.subplot(121) 75 | plt.ylim 76 | plt.plot(self.x, self.y, 'ro', label="training points") 77 | plt.plot(self.x, self.W[-1] * self.x + self.B[-1], label = "estimated line") 78 | plt.xlabel('x') 79 | plt.ylabel('y') 80 | plt.ylim((-10, 15)) 81 | plt.title('Data Space Iteration: ' + str(self.n)) 82 | 83 | plt.subplot(122) 84 | plt.contour(self.w, self.b, self.Z) 85 | plt.scatter(self.W, self.B, c = 'r', marker = 'x') 86 | plt.title('Total Loss Surface Contour Iteration' + str(self.n)) 87 | plt.xlabel('w') 88 | plt.ylabel('b') 89 | plt.show() 90 | 91 | 92 | #Create f(X) with a slope of 1 and a bias of -1 93 | X = torch.arange(-3, 3, 0.1).view(-1, 1) 94 | X.size() 95 | f = 1 * X - 1 96 | 97 | 98 | #Add noise 99 | Y = f + 0.1 * torch.randn(X.size()) 100 | 101 | 102 | # Plot out the line and the points with noise 103 | plt.plot(X.numpy(), Y.numpy(), label = 'y') 104 | plt.plot(X.numpy(), f.numpy(), label = 'f') 105 | plt.xlabel('x') 106 | plt.ylabel('y') 107 | plt.legend() 108 | 109 | #Define the forward function 110 | def forward(x): 111 | return w * x + b 112 | 113 | #Define the MSE Loss function 114 | def criterion(yhat,y): 115 | return torch.mean((yhat-y)**2) 116 | 117 | 118 | 119 | #Create plot_error_surfaces for viewing the data 120 | get_surface = plot_error_surfaces(15, 15, X, Y, 30) 121 | 122 | 123 | #Define the parameters w, b for y = wx + b 124 | w = torch.tensor(-15.0, requires_grad = True) 125 | b = torch.tensor(-10.0, requires_grad = True) 126 | 127 | 128 | #Define learning rate and create an empty list for containing the loss for each iteration. 129 | lr = 0.1 130 | LOSS = [] 131 | 132 | # The function for training the model 133 | def train_model(iter): 134 | 135 | #Loop 136 | for epoch in range(iter): 137 | 138 | #make a prediction 139 | Yhat = forward(X) 140 | 141 | #calculate the loss 142 | loss = criterion(Yhat, Y) 143 | 144 | #Section for plotting 145 | get_surface.set_para_loss(w.data.tolist(), b.data.tolist(), loss.tolist()) 146 | if epoch % 3 == 0: 147 | get_surface.plot_ps() 148 | 149 | #store the loss in the list LOSS 150 | LOSS.append(loss) 151 | 152 | #backward pass: compute gradient of the loss with respect to all the learnable parameters 153 | loss.backward() 154 | 155 | #update parameters slope and bias 156 | w.data = w.data - lr * w.grad.data 157 | b.data = b.data - lr * b.grad.data 158 | 159 | #zero the gradients before running the backward pass 160 | w.grad.data.zero_() 161 | b.grad.data.zero_() 162 | 163 | 164 | #Train the model with 15 iterations 165 | train_model(15) 166 | 167 | 168 | #Plot out the Loss Result 169 | get_surface.final_plot() 170 | plt.plot(LOSS) 171 | plt.tight_layout() 172 | plt.xlabel("Epoch/Iterations") 173 | plt.ylabel("Cost") 174 | -------------------------------------------------------------------------------- /Week 3/Logistic Regression Training (Cross-Entropy).py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Logistic Regression Training Negative Log likelihood (Cross-Entropy) 4 | 5 | """ 6 | 7 | import torch 8 | import torch.nn as nn 9 | import numpy as np 10 | import matplotlib.pyplot as plt 11 | from mpl_toolkits import mplot3d 12 | from torch.utils.data import Dataset, DataLoader 13 | 14 | torch.manual_seed(1) 15 | 16 | #Create class and function for plotting 17 | class plot_error_surfaces(object): 18 | 19 | #Construstor 20 | def __init__(self, w_range, b_range, X, Y, n_samples = 30, go = True): 21 | W = np.linspace(-w_range, w_range, n_samples) 22 | B = np.linspace(-b_range, b_range, n_samples) 23 | w, b = np.meshgrid(W, B) 24 | Z = np.zeros((30, 30)) 25 | count1 = 0 26 | self.y = Y.numpy() 27 | self.x = X.numpy() 28 | for w1, b1 in zip(w, b): 29 | count2 = 0 30 | for w2, b2 in zip(w1, b1): 31 | Z[count1, count2] = np.mean((self.y - (1 / (1 + np.exp(-1*w2 * self.x - b2)))) ** 2) 32 | count2 += 1 33 | count1 += 1 34 | self.Z = Z 35 | self.w = w 36 | self.b = b 37 | self.W = [] 38 | self.B = [] 39 | self.LOSS = [] 40 | self.n = 0 41 | if go == True: 42 | plt.figure() 43 | plt.figure(figsize=(7.5, 5)) 44 | plt.axes(projection='3d').plot_surface(self.w, self.b, self.Z, rstride=1, cstride=1, cmap='viridis', edgecolor='none') 45 | plt.title('Loss Surface') 46 | plt.xlabel('w') 47 | plt.ylabel('b') 48 | plt.show() 49 | plt.figure() 50 | plt.title('Loss Surface Contour') 51 | plt.xlabel('w') 52 | plt.ylabel('b') 53 | plt.contour(self.w, self.b, self.Z) 54 | plt.show() 55 | 56 | #Setter 57 | def set_para_loss(self, model, loss): 58 | self.n = self.n + 1 59 | self.W.append(list(model.parameters())[0].item()) 60 | self.B.append(list(model.parameters())[1].item()) 61 | self.LOSS.append(loss) 62 | 63 | #Plot diagram 64 | def final_plot(self): 65 | ax = plt.axes(projection='3d') 66 | ax.plot_wireframe(self.w, self.b, self.Z) 67 | ax.scatter(self.W, self.B, self.LOSS, c='r', marker='x', s=200, alpha=1) 68 | plt.figure() 69 | plt.contour(self.w, self.b, self.Z) 70 | plt.scatter(self.W, self.B, c='r', marker='x') 71 | plt.xlabel('w') 72 | plt.ylabel('b') 73 | plt.show() 74 | 75 | #Plot diagram 76 | def plot_ps(self): 77 | plt.subplot(121) 78 | plt.ylim 79 | plt.plot(self.x, self.y, 'ro', label="training points") 80 | plt.plot(self.x, self.W[-1] * self.x + self.B[-1], label="estimated line") 81 | plt.plot(self.x, 1 / (1 + np.exp(-1 * (self.W[-1] * self.x + self.B[-1]))), label='sigmoid') 82 | plt.xlabel('x') 83 | plt.ylabel('y') 84 | plt.ylim((-0.1, 2)) 85 | plt.title('Data Space Iteration: ' + str(self.n)) 86 | plt.show() 87 | plt.subplot(122) 88 | plt.contour(self.w, self.b, self.Z) 89 | plt.scatter(self.W, self.B, c='r', marker='x') 90 | plt.title('Loss Surface Contour Iteration' + str(self.n)) 91 | plt.xlabel('w') 92 | plt.ylabel('b') 93 | 94 | #Plot the diagram 95 | def PlotStuff(X, Y, model, epoch, leg=True): 96 | plt.plot(X.numpy(), model(X).detach().numpy(), label=('epoch ' + str(epoch))) 97 | plt.plot(X.numpy(), Y.numpy(), 'r') 98 | if leg == True: 99 | plt.legend() 100 | else: 101 | pass 102 | 103 | #Create a Data class 104 | class Data(Dataset): 105 | #Constructor 106 | def __init__(self): 107 | self.x = torch.arange(-1, 1, 0.1).view(-1, 1) #Create a column of values between -1 and 1 with 0.1 step 108 | self.y = torch.zeros(self.x.shape[0], 1) #Create a column of zeros with length of x 109 | self.y[self.x[:, 0] > 0.2 ] = 1 #y = 1 when x > 0.2 110 | self.len = self.x.shape[0] 111 | 112 | #Getter 113 | def __getitem__(self, index): 114 | return self.x[index], self.y[index] 115 | 116 | #Length 117 | def __len__(self): 118 | return self.len 119 | 120 | #Create the dataset 121 | data_set = Data() 122 | print(data_set.x) #Lets see x 123 | print(data_set.y) #Lets see y 124 | 125 | #Create a logistic regression custom class using nn.Module 126 | class logistic_regression(nn.Module): 127 | 128 | #Constructor 129 | def __init__( self, n_inputs): 130 | super().__init__() 131 | self.linear = nn.Linear(n_inputs, 1) 132 | 133 | #Prediction 134 | def forward(self, x): 135 | yhat = torch.sigmoid(self.linear(x)) 136 | 137 | return yhat 138 | 139 | 140 | #Initialize model 141 | model = logistic_regression(1) #1 input model 142 | 143 | #Manually setting weight and bias 144 | model.state_dict() ['linear.weight'].data[0] = torch.tensor([[-5]]) 145 | model.state_dict() ['linear.bias'].data[0] = torch.tensor([[-10]]) 146 | print("The parameters: ", model.state_dict()) 147 | 148 | 149 | #Create the plot_error_surfaces object 150 | get_surface = plot_error_surfaces(15, 13, data_set[:][0], data_set[:][1], 30) 151 | 152 | 153 | #Create a DataLoader object 154 | trainloader = DataLoader(dataset = data_set, batch_size = 3) 155 | 156 | #Create the error metric. Using RMS will cause the manually initialized parameters not to converge 157 | #Cross Entropy Loss error will be used to solve this problem 158 | #We can either manually create a loss function or import from torch 159 | 160 | #Lets create a cross entropy loss function 161 | def criterion(yhat, y): 162 | out = -1 * torch.mean(y * torch.log(yhat) + (1 - y) * torch.log(1 - yhat)) #negative log cross entropy function 163 | 164 | return out 165 | 166 | #This is the built in cross entropy function in torch 167 | #criterion = nn.BCELoss() 168 | 169 | #Set the learning rate 170 | learning_rate = 2 171 | 172 | #Initialize the optimizer 173 | optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate) 174 | 175 | #Create the training loop 176 | def train_model(epochs): 177 | for epoch in range(epochs): 178 | 179 | for x, y in trainloader: 180 | yhat = model(x) #predict 181 | loss = criterion(yhat, y) #calculate loss 182 | optimizer.zero_grad() #Set gradients to zero 183 | loss.backward() #Calculate gradients 184 | optimizer.step() #adjust new parameters 185 | get_surface.set_para_loss(model, loss.tolist()) #plot 186 | 187 | if epoch % 20 == 0: 188 | get_surface.plot_ps() 189 | 190 | train_model(100) 191 | 192 | #Prediction 193 | yhat = model(data_set.x) 194 | label = yhat > 0.5 195 | print("The accuracy: ", torch.mean((label == data_set.y.type(torch.ByteTensor)).type(torch.float))) #Accuracy = 1 or 100% 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | -------------------------------------------------------------------------------- /Week 3/Logistic Regression and Bad Initialization Value.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Logistic Regression and Bad Initialization Value 4 | 5 | """ 6 | 7 | import torch 8 | import torch.nn as nn 9 | import numpy as np 10 | import matplotlib.pyplot as plt 11 | from mpl_toolkits import mplot3d 12 | from torch.utils.data import Dataset, DataLoader 13 | 14 | torch.manual_seed(1) 15 | 16 | #Create class and function for plotting 17 | class plot_error_surfaces(object): 18 | 19 | #Construstor 20 | def __init__(self, w_range, b_range, X, Y, n_samples = 30, go = True): 21 | W = np.linspace(-w_range, w_range, n_samples) 22 | B = np.linspace(-b_range, b_range, n_samples) 23 | w, b = np.meshgrid(W, B) 24 | Z = np.zeros((30, 30)) 25 | count1 = 0 26 | self.y = Y.numpy() 27 | self.x = X.numpy() 28 | for w1, b1 in zip(w, b): 29 | count2 = 0 30 | for w2, b2 in zip(w1, b1): 31 | Z[count1, count2] = np.mean((self.y - (1 / (1 + np.exp(-1*w2 * self.x - b2)))) ** 2) 32 | count2 += 1 33 | count1 += 1 34 | self.Z = Z 35 | self.w = w 36 | self.b = b 37 | self.W = [] 38 | self.B = [] 39 | self.LOSS = [] 40 | self.n = 0 41 | if go == True: 42 | plt.figure() 43 | plt.figure(figsize=(7.5, 5)) 44 | plt.axes(projection='3d').plot_surface(self.w, self.b, self.Z, rstride=1, cstride=1, cmap='viridis', edgecolor='none') 45 | plt.title('Loss Surface') 46 | plt.xlabel('w') 47 | plt.ylabel('b') 48 | plt.show() 49 | plt.figure() 50 | plt.title('Loss Surface Contour') 51 | plt.xlabel('w') 52 | plt.ylabel('b') 53 | plt.contour(self.w, self.b, self.Z) 54 | plt.show() 55 | 56 | #Setter 57 | def set_para_loss(self, model, loss): 58 | self.n = self.n + 1 59 | self.W.append(list(model.parameters())[0].item()) 60 | self.B.append(list(model.parameters())[1].item()) 61 | self.LOSS.append(loss) 62 | 63 | #Plot diagram 64 | def final_plot(self): 65 | ax = plt.axes(projection='3d') 66 | ax.plot_wireframe(self.w, self.b, self.Z) 67 | ax.scatter(self.W, self.B, self.LOSS, c='r', marker='x', s=200, alpha=1) 68 | plt.figure() 69 | plt.contour(self.w, self.b, self.Z) 70 | plt.scatter(self.W, self.B, c='r', marker='x') 71 | plt.xlabel('w') 72 | plt.ylabel('b') 73 | plt.show() 74 | 75 | #Plot diagram 76 | def plot_ps(self): 77 | plt.subplot(121) 78 | plt.ylim 79 | plt.plot(self.x, self.y, 'ro', label="training points") 80 | plt.plot(self.x, self.W[-1] * self.x + self.B[-1], label="estimated line") 81 | plt.plot(self.x, 1 / (1 + np.exp(-1 * (self.W[-1] * self.x + self.B[-1]))), label='sigmoid') 82 | plt.xlabel('x') 83 | plt.ylabel('y') 84 | plt.ylim((-0.1, 2)) 85 | plt.title('Data Space Iteration: ' + str(self.n)) 86 | plt.show() 87 | plt.subplot(122) 88 | plt.contour(self.w, self.b, self.Z) 89 | plt.scatter(self.W, self.B, c='r', marker='x') 90 | plt.title('Loss Surface Contour Iteration' + str(self.n)) 91 | plt.xlabel('w') 92 | plt.ylabel('b') 93 | 94 | #Plot the diagram 95 | def PlotStuff(X, Y, model, epoch, leg=True): 96 | plt.plot(X.numpy(), model(X).detach().numpy(), label=('epoch ' + str(epoch))) 97 | plt.plot(X.numpy(), Y.numpy(), 'r') 98 | if leg == True: 99 | plt.legend() 100 | else: 101 | pass 102 | 103 | #Create a Data class 104 | class Data(Dataset): 105 | #Constructor 106 | def __init__(self): 107 | self.x = torch.arange(-1, 1, 0.1).view(-1, 1) #Create a column of values between -1 and 1 with 0.1 step 108 | self.y = torch.zeros(self.x.shape[0], 1) #Create a column of zeros with length of x 109 | self.y[self.x[:, 0] > 0.2 ] = 1 #y = 1 when x > 0.2 110 | self.len = self.x.shape[0] 111 | 112 | #Getter 113 | def __getitem__(self, index): 114 | return self.x[index], self.y[index] 115 | 116 | #Length 117 | def __len__(self): 118 | return self.len 119 | 120 | #Create the dataset 121 | data_set = Data() 122 | print(data_set.x) #Lets see x 123 | print(data_set.y) #Lets see y 124 | 125 | #Create a logistic regression custom class using nn.Module 126 | class logistic_regression(nn.Module): 127 | 128 | #Constructor 129 | def __init__( self, n_inputs): 130 | super().__init__() 131 | self.linear = nn.Linear(n_inputs, 1) 132 | 133 | #Prediction 134 | def forward(self, x): 135 | yhat = torch.sigmoid(self.linear(x)) 136 | 137 | return yhat 138 | 139 | 140 | #Initialize model 141 | model = logistic_regression(1) #1 input model 142 | 143 | #Manually setting weight and bias 144 | model.state_dict() ['linear.weight'].data[0] = torch.tensor([[-5]]) 145 | model.state_dict() ['linear.bias'].data[0] = torch.tensor([[-10]]) 146 | print("The parameters: ", model.state_dict()) 147 | 148 | 149 | #Create the plot_error_surfaces object 150 | get_surface = plot_error_surfaces(15, 13, data_set[:][0], data_set[:][1], 30) 151 | 152 | 153 | #Create a DataLoader object 154 | trainloader = DataLoader(dataset = data_set, batch_size = 3) 155 | 156 | #Create the error metric 157 | criterion_rms = nn.MSELoss() 158 | 159 | #Set the learning rate 160 | learning_rate = 2 161 | 162 | #Initialize the optimizer 163 | optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate) 164 | 165 | #Create the training loop 166 | def train_model(epochs): 167 | for epoch in range(epochs): 168 | 169 | for x, y in trainloader: 170 | yhat = model(x) #predict 171 | loss = criterion_rms(yhat, y) #calculate loss 172 | optimizer.zero_grad() #Set gradients to zero 173 | loss.backward() #Calculate gradients 174 | optimizer.step() #adjust new parameters 175 | get_surface.set_para_loss(model, loss.tolist()) #plot 176 | 177 | if epoch % 20 == 0: 178 | get_surface.plot_ps() 179 | 180 | train_model(100) 181 | 182 | #Notice how using RMS Loss causes the parameters manually not to converge 183 | 184 | 185 | #Make the Prediction 186 | yhat = model(data_set.x) 187 | label = yhat > 0.5 188 | print("The accuracy: ", torch.mean((label == data_set.y.type(torch.ByteTensor)).type(torch.float))) 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | 221 | 222 | 223 | 224 | 225 | -------------------------------------------------------------------------------- /Week 3/Logistic Regression.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Logistic Regression 4 | 5 | """ 6 | 7 | import torch 8 | import torch.nn as nn 9 | import matplotlib.pyplot as plt 10 | 11 | torch.manual_seed(1) 12 | 13 | #Create a tensor ranging from -100 to 100 14 | z = torch.arange(-100,100,0.1).view(-1,1) 15 | print("The tensor: ", z) 16 | 17 | #create a sigmoid object from torch.nn 18 | sig = nn.Sigmoid() 19 | 20 | #Calculate yhat using sigmoid object 21 | yhat = sig(z) 22 | print("The prediction yhat: ", yhat) 23 | 24 | #Plot results 25 | plt.plot(z.numpy(), yhat.numpy()) 26 | plt.xlabel('z') 27 | plt.ylabel('yhat') 28 | 29 | #Lets try the sigmoid model in torch 30 | yhat1 = torch.sigmoid(z) #Element-wise Sigmoid from torch 31 | 32 | #Plot results 33 | plt.plot(z.numpy(), yhat1.numpy()) 34 | plt.xlabel('z') 35 | plt.ylabel('yhat1') 36 | 37 | #The plots are identical. 38 | 39 | """ 40 | Build a logistic Regression object using nn.sequential with 1D input 41 | """ 42 | 43 | #Create x and X tensor 44 | x = torch.tensor([[1.0]]) 45 | X = torch.tensor([[1.0], [100]]) 46 | print('x = ', x) 47 | print('X = ', X) 48 | 49 | 50 | #We call nn.Sequential with Linear first then sigmoid 51 | model = nn.Sequential(nn.Linear(1, 1), nn.Sigmoid()) 52 | 53 | 54 | #Print the parameters 55 | print("list(model.parameters()):\n ", list(model.parameters())) 56 | print("\nmodel.state_dict():\n ", model.state_dict()) 57 | 58 | #Predict x using model created 59 | yhat = model(x) 60 | print("The prediction: ", yhat) 61 | 62 | 63 | #Lets predict X 64 | yhat1 = model(X) 65 | print("The prediction: ", yhat1) 66 | 67 | 68 | 69 | """ 70 | Custom Logistic Regression Class using nn.Module 71 | """ 72 | 73 | class logistic_regression(nn.Module): 74 | 75 | def __init__(self, n_inputs): 76 | super().__init__() 77 | self.linear = nn.Linear(n_inputs, 1) 78 | 79 | def forward(self, x): 80 | yhat = torch.sigmoid(self.linear(x)) 81 | 82 | return yhat 83 | 84 | #Create x and X tensor with 1 input 85 | x = torch.tensor([[1.0]]) 86 | X = torch.tensor([[-100], [0], [100.0]]) 87 | print('x = ', x) 88 | print('X = ', X) 89 | 90 | 91 | model = logistic_regression(1) #Initializing model that will predict 1 dimension 92 | 93 | #Lets view the randomly intialized parameters 94 | print("list(model.parameters()):\n", list(model.parameters())) 95 | print("\nmodel.state_dict():\n", model.state_dict()) 96 | 97 | #Predict x and X 98 | yhat = model(x) 99 | yhat1 = model(X) 100 | 101 | 102 | #Lets create a new model that takes 2 inputs 103 | model = logistic_regression(2) 104 | 105 | #Create x and X tensor with 2 inputs 106 | x = torch.tensor([[1.0, 2.0]]) 107 | X = torch.tensor([[-100, -100], [0.0, 0.0], [-100, 100]]) 108 | print('x = ', x) 109 | print('X = ', X) 110 | 111 | #Predict x and X with 2 inputs 112 | yhat = model(x) 113 | print(yhat) 114 | 115 | yhat1 = model(X) 116 | print(yhat1) 117 | 118 | 119 | 120 | 121 | 122 | -------------------------------------------------------------------------------- /Week 3/Multiple Linear Regression Training.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Multiple Linear Regression Training 4 | 5 | 6 | """ 7 | 8 | import torch 9 | from torch import nn, optim 10 | import numpy as np 11 | import matplotlib.pyplot as plt 12 | from mpl_toolkits.mplot3d import Axes3D 13 | from torch.utils.data import Dataset, DataLoader 14 | 15 | 16 | #Set random seed to 1 17 | torch.manual_seed(1) 18 | 19 | #The function for plotting 2D 20 | def Plot_2D_Plane(model, dataset, n=0): 21 | w1 = model.state_dict()['linear.weight'].numpy()[0][0] 22 | w2 = model.state_dict()['linear.weight'].numpy()[0][1] 23 | b = model.state_dict()['linear.bias'].numpy() 24 | 25 | # Data 26 | x1 = data_set.x[:, 0].view(-1, 1).numpy() 27 | x2 = data_set.x[:, 1].view(-1, 1).numpy() 28 | y = data_set.y.numpy() 29 | 30 | # Make plane 31 | X, Y = np.meshgrid(np.arange(x1.min(), x1.max(), 0.05), np.arange(x2.min(), x2.max(), 0.05)) 32 | yhat = w1 * X + w2 * Y + b 33 | 34 | # Plotting 35 | fig = plt.figure() 36 | ax = fig.gca(projection='3d') 37 | 38 | ax.plot(x1[:, 0], x2[:, 0], y[:, 0],'ro', label='y') # Scatter plot 39 | 40 | ax.plot_surface(X, Y, yhat) # Plane plot 41 | 42 | ax.set_xlabel('x1 ') 43 | ax.set_ylabel('x2 ') 44 | ax.set_zlabel('y') 45 | plt.title('estimated plane iteration:' + str(n)) 46 | ax.legend() 47 | 48 | plt.show() 49 | 50 | 51 | #Create Data class from torch Dataset import 52 | class Data2D(Dataset): 53 | 54 | #Constructor 55 | def __init__(self): 56 | self.x = torch.zeros(20, 2) #Returns a tensor filled with the scalar value 0 with the shape 20. (20 samples, 2D) 57 | self.x[:, 0] = torch.arange(-1, 1, 0.1) #fill first column of x with values interval -1 to 1 with 0.1 step 58 | self.x[:, 1] = torch.arange(-1, 1, 0.1) #fill second column of x with values interval -1 to 1 with 0.1 step 59 | self.w = torch.tensor([[1.0], [1.0]]) 60 | self.b = 1 61 | self.f = torch.mm(self.x, self.w) + self.b 62 | self.y = self.f + 0.1 * torch.randn((self.x.shape[0],1)) 63 | self.len = self.x.shape[0] 64 | 65 | #Getter 66 | def __getitem__(self, index): 67 | return self.x[index], self.y[index] 68 | 69 | #Get Length 70 | def __len__(self): 71 | return self.len 72 | 73 | 74 | #Create a Data2D object data_set 75 | data_set = Data2D() 76 | 77 | 78 | #Create a customized linear regression class from nn.Module class 79 | class linear_regression(nn.Module): 80 | 81 | # Constructor 82 | def __init__(self, input_size, output_size): 83 | super().__init__() 84 | self.linear = nn.Linear(input_size, output_size) 85 | 86 | # Prediction 87 | def forward(self, x): 88 | yhat = self.linear(x) 89 | return yhat 90 | 91 | #Create linear regression model 92 | model = linear_regression(2,1) 93 | print("The initialized model parameters: ", list(model.parameters())) #Print model parameters in a list 94 | 95 | #Create optimizer 96 | optimizer = optim.SGD(model.parameters(), lr = 0.1) #Create optimizer with SGD and learning rate 0.1 with model intialized parameters 97 | 98 | #Create loss function for torch.nn 99 | criterion = nn.MSELoss() 100 | 101 | #Create the dataloader using DataLoader and the data_set created and batch size 2 102 | train_loader = DataLoader(dataset = data_set, batch_size = 2) 103 | 104 | 105 | LOSS = [] #Empty list to store the error 106 | 107 | print("Before Training: ") 108 | Plot_2D_Plane(model, data_set) #The fit Plane with auto initialized parameters before training 109 | 110 | epochs = 100 111 | 112 | #Create train model 113 | def train_model(epochs): 114 | 115 | for epoch in range(epochs): 116 | 117 | for x,y in train_loader: 118 | 119 | yhat = model(x) 120 | 121 | loss = criterion(yhat, y) 122 | 123 | LOSS.append(loss.item()) 124 | 125 | optimizer.zero_grad() 126 | 127 | loss.backward() 128 | 129 | optimizer.step() 130 | 131 | #Train model for specified number of epochs. 132 | train_model(epochs) 133 | 134 | print("After Training: ") 135 | Plot_2D_Plane(model, data_set, epochs) #Fit Plane after training 136 | 137 | 138 | 139 | #Lets visualize the Loss for each epoch 140 | plt.plot(LOSS) 141 | plt.xlabel("Iterations ") 142 | plt.ylabel("Cost/total loss ") 143 | 144 | 145 | 146 | """ 147 | 148 | Lets try a new model with different hyper parameters 149 | 150 | """ 151 | 152 | #Create linear regression model 153 | model1 = linear_regression(2,1) 154 | print("The initialized model parameters: ", list(model.parameters())) #Print model parameters in a list 155 | 156 | #Create optimizer 157 | optimizer = optim.SGD(model1.parameters(), lr = 0.1) #Create optimizer with SGD and learning rate 0.1 with model intialized parameters 158 | 159 | #Create the dataloader using DataLoader and the data_set created and batch size 2 160 | train_loader = DataLoader(dataset = data_set, batch_size = 30) 161 | 162 | LOSS1 = [] #Empty list to store the error 163 | 164 | print("Before Training: ") 165 | Plot_2D_Plane(model1, data_set) #The fit Plane with auto initialized parameters before training 166 | 167 | epochs = 100 168 | 169 | #Create train model 170 | def train_model1(epochs): 171 | 172 | for epoch in range(epochs): 173 | 174 | for x,y in train_loader: 175 | 176 | yhat = model1(x) 177 | 178 | loss = criterion(yhat, y) 179 | 180 | LOSS1.append(loss.item()) 181 | 182 | optimizer.zero_grad() 183 | 184 | loss.backward() 185 | 186 | optimizer.step() 187 | 188 | #Train model for specified number of epochs. 189 | train_model1(epochs) 190 | 191 | print("After Training: ") 192 | Plot_2D_Plane(model1, data_set, epochs) #Fit Plane after training 193 | 194 | 195 | 196 | #Lets visualize the Loss for each epoch 197 | plt.plot(LOSS1) 198 | plt.xlabel("Iterations ") 199 | plt.ylabel("Cost/total loss ") 200 | 201 | 202 | #Lets create a validation dataset to calculate the total loss or cost for both models 203 | torch.manual_seed(2) 204 | val_data = Data2D() 205 | Y = val_data.y 206 | X = val_data.x 207 | 208 | print("total loss or cost for model: ",criterion(model(X),Y)) 209 | print("total loss or cost for model: ",criterion(model1(X),Y)) 210 | -------------------------------------------------------------------------------- /Week 3/Multiple Linear Regression with Multiple Outputs training.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Multiple Linear Regression with Multiple Outputs training 4 | 5 | 6 | """ 7 | 8 | import torch 9 | from torch import nn, optim 10 | import numpy as np 11 | import matplotlib.pyplot as plt 12 | from mpl_toolkits.mplot3d import Axes3D 13 | from torch.utils.data import Dataset, DataLoader 14 | 15 | 16 | #Set random seed to 1 17 | torch.manual_seed(1) 18 | 19 | #Create Data class from torch Dataset import 20 | class Data2D(Dataset): 21 | 22 | #Constructor 23 | def __init__(self): 24 | self.x = torch.zeros(20, 2) #Returns a tensor filled with the scalar value 0 with the shape 20. (20 samples, 2D) 25 | self.x[:, 0] = torch.arange(-1, 1, 0.1) #fill first column of x with values interval -1 to 1 with 0.1 step 26 | self.x[:, 1] = torch.arange(-1, 1, 0.1) #fill second column of x with values interval -1 to 1 with 0.1 step 27 | self.w = torch.tensor([ [1.0,-1.0],[1.0,3.0]]) #Because we have 2 outputs, we need a second set of w parameters 28 | self.b = torch.tensor([[1.0,-1.0]]) #Because we have 2 outputs, we need a second b parameter 29 | self.f = torch.mm(self.x, self.w) + self.b 30 | self.y = self.f + 0.01 * torch.randn((self.x.shape[0],1)) 31 | self.len = self.x.shape[0] 32 | 33 | #Getter 34 | def __getitem__(self, index): 35 | return self.x[index], self.y[index] 36 | 37 | #Get Length 38 | def __len__(self): 39 | return self.len 40 | 41 | #Create a Data2D object data_set 42 | data_set = Data2D() 43 | 44 | #Create a customized linear regression class from nn.Module class 45 | class linear_regression(nn.Module): 46 | 47 | # Constructor 48 | def __init__(self, input_size, output_size): 49 | super().__init__() 50 | self.linear = nn.Linear(input_size, output_size) 51 | 52 | # Prediction 53 | def forward(self, x): 54 | yhat = self.linear(x) 55 | return yhat 56 | 57 | #Create linear regression model with 2 inputs and 2 outputs 58 | model = linear_regression(2,2) 59 | print("The initialized model parameters: ", list(model.parameters())) #Print model parameters in a list 60 | 61 | 62 | #Create the optimizer and give it model parameters and learning rate 63 | optimizer = optim.SGD(model.parameters(), lr = 0.01) 64 | 65 | #Create loss function for torch.nn 66 | criterion = nn.MSELoss() 67 | 68 | #Create the dataloader using DataLoader and the data_set created and batch size 2 69 | train_loader = DataLoader(dataset = data_set, batch_size = 1) 70 | 71 | 72 | LOSS = [] #Empty list to store the error 73 | 74 | #number of epochs 75 | epochs = 100 76 | 77 | def train_model(epochs): 78 | for epochs in range(epochs): 79 | 80 | for x,y in train_loader: 81 | 82 | #make a prediction 83 | yhat=model(x) 84 | 85 | #calculate the loss 86 | loss=criterion(yhat,y) 87 | 88 | #store loss/cost in LOSS list 89 | LOSS.append(loss.item()) 90 | 91 | #clear gradient 92 | optimizer.zero_grad() 93 | 94 | #Backward pass: compute gradient of the loss with respect to all the learnable parameters 95 | loss.backward() 96 | 97 | #Update parameters using gradient and learning rate 98 | optimizer.step() 99 | 100 | #Train the model 101 | train_model(100) 102 | 103 | plt.plot(LOSS) 104 | plt.xlabel("iterations ") 105 | plt.ylabel("Cost/total loss ") 106 | plt.show() 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | -------------------------------------------------------------------------------- /Week 3/Multiple Linear Regression.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Multiple Linear Regression 4 | 5 | 6 | """ 7 | from torch import nn 8 | import torch 9 | torch.manual_seed(1) 10 | 11 | #Setting weight and bias manually 12 | w = torch.tensor([[2.0], [3.0]], requires_grad=True) 13 | b = torch.tensor([[1.0]], requires_grad=True) 14 | print(w) 15 | print(b) 16 | 17 | #Define Prediction Function 18 | def forward(x): 19 | yhat = torch.mm(x, w) + b #torch.mm is matrix multiplicaiton. torch.matmul can also be used here. 20 | return yhat 21 | 22 | x = torch.tensor([[1.0, 2.0]]) 23 | print("Dimensions of x: ", x.ndimension()) 24 | 25 | yhat = forward(x) 26 | print("The result: ", yhat) 27 | 28 | #Sample tensor X. Lets create a 2D X with 3 samples (or rows) 29 | X = torch.tensor([[1.0, 1.0], [1.0, 2.0], [1.0, 3.0]]) 30 | 31 | #Make the prediction of X 32 | yhat = forward(X) 33 | print("The result: ", yhat) 34 | 35 | #Make a linear regression model using build-in function 36 | model = nn.Linear(2, 1) 37 | 38 | #Make a prediction of x 39 | yhat = model(x) 40 | print("The result: ", yhat) 41 | 42 | #Model auto initialized parameters 43 | print("model initialized parameters: ", model.state_dict()) 44 | 45 | #Make a prediction of X 46 | yhat = model(X) 47 | print("The result: ", yhat) 48 | 49 | 50 | #Create linear_regression Class using nn.Module Class 51 | class linear_regression(nn.Module): 52 | 53 | # Constructor 54 | def __init__(self, input_size, output_size): 55 | super().__init__() 56 | self.linear = nn.Linear(input_size, output_size) 57 | 58 | # Prediction function 59 | def forward(self, x): 60 | yhat = self.linear(x) 61 | return yhat 62 | 63 | #Create multiple linear regression model with 2 inputs and 1 output 64 | model = linear_regression(2, 1) 65 | 66 | #Print model parameters (auto initialized) 67 | print("The parameters: ", list(model.parameters())) 68 | 69 | #Print model parameters 70 | print("The parameters: ", model.state_dict()) 71 | 72 | 73 | #Make a prediction of x 74 | yhat = model(x) 75 | print("The result: ", yhat) 76 | 77 | 78 | #Make a prediction of X 79 | yhat = model(X) 80 | print("The result: ", yhat) 81 | 82 | 83 | 84 | #Lets try a different X 85 | X = torch.tensor([[11.0, 12.0, 13, 14], [11, 12, 13, 14]]) 86 | 87 | #Since the X created has x1, x2, x3, x4 and 2 samples. We need to redo our model to be 4inputs and 1 ouput. 88 | model = linear_regression(4,1) 89 | 90 | #Prediction 91 | yhat = model(X) 92 | print(yhat) 93 | 94 | 95 | 96 | 97 | 98 | 99 | -------------------------------------------------------------------------------- /Week 4/Activation Functions.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Activation Functions 4 | 5 | """ 6 | 7 | # In[1]: Imports 8 | 9 | # Import the libraries we need for this lab 10 | 11 | import torch.nn as nn 12 | import torch 13 | 14 | import matplotlib.pyplot as plt 15 | torch.manual_seed(0) 16 | 17 | # In[2]: 18 | # Create a tensor 19 | 20 | z = torch.arange(-10, 10, 0.1,).view(-1, 1) 21 | 22 | # Create a sigmoid object with torch.nn 23 | 24 | sig = nn.Sigmoid() 25 | 26 | # Make a prediction of sigmoid function 27 | 28 | yhat = sig(z) 29 | 30 | # Plot the result 31 | 32 | 33 | def plot_sig(): 34 | plt.plot(z.numpy(), yhat.detach().numpy()) 35 | plt.xlabel('z') 36 | plt.ylabel('yhat') 37 | plt.show() 38 | 39 | 40 | plot_sig() 41 | 42 | # Use the built-in torch function to predict the result 43 | yhat = torch.sigmoid(z) 44 | plt.plot(z.numpy(), yhat.numpy()) 45 | plt.show() 46 | 47 | 48 | # Create a tanh object with torch.nn 49 | TANH = nn.Tanh() 50 | 51 | 52 | # Make the prediction using tanh object 53 | 54 | 55 | def plot_tanh(): 56 | yhat = TANH(z) 57 | plt.plot(z.numpy(), yhat.numpy()) 58 | plt.xlabel('z') 59 | plt.ylabel('yhat') 60 | plt.show() 61 | 62 | 63 | plot_tanh() 64 | # Make the prediction using the build-in tanh object 65 | yhat = torch.tanh(z) 66 | plt.plot(z.numpy(), yhat.numpy()) 67 | plt.show() 68 | 69 | 70 | # Create a relu object and make the prediction 71 | 72 | def ReLU(): 73 | RELU = nn.ReLU() 74 | yhat = RELU(z) 75 | plt.plot(z.numpy(), yhat.numpy()) 76 | plt.xlabel('z') 77 | plt.ylabel('yhat') 78 | plt.show() 79 | 80 | 81 | ReLU() 82 | 83 | # Use the build-in function to make the prediction 84 | yhat = torch.relu(z) 85 | plt.plot(z.numpy(), yhat.numpy()) 86 | plt.show() 87 | 88 | # Plot the results to compare the activation functions 89 | x = torch.arange(-2, 2, 0.1).view(-1, 1) 90 | 91 | 92 | def Plot_all(): 93 | plt.plot(x.numpy(), torch.relu(x).numpy(), label='relu') 94 | plt.plot(x.numpy(), torch.sigmoid(x).numpy(), label='sigmoid') 95 | plt.plot(x.numpy(), torch.tanh(x).numpy(), label='tanh') 96 | plt.legend() 97 | plt.show() 98 | 99 | 100 | Plot_all() 101 | 102 | # Compare the activation functions again using a tensor in the range (-1, 1) 103 | x = torch.arange(-1, 1, 0.1).view(-1, 1) 104 | 105 | Plot_all() 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | -------------------------------------------------------------------------------- /Week 4/Neural Networks More Hidden Neutrons.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Neural Networks More Hidden Neutrons 4 | 5 | """ 6 | 7 | 8 | # In[1] Imports 9 | 10 | import torch 11 | import torch.nn as nn 12 | import torch.nn.functional as F 13 | from torch.utils.data import DataLoader, Dataset 14 | import matplotlib.pylab as plt 15 | import numpy as np 16 | torch.manual_seed(0) 17 | 18 | 19 | # In[2] 20 | 21 | #Functions for plotting the model 22 | def get_hist(model, data_set): 23 | 24 | activations = model.activation(data_set.x) 25 | for i, activation in enumerate(activations): 26 | 27 | plt.hist(activation.numpy(), 4, density = True) 28 | 29 | plt.title("Activation layer " + str(i+1)) 30 | 31 | plt.xlabel("Activation") 32 | 33 | plt.legend() 34 | 35 | plt.show() 36 | 37 | def PlotStuff(X,Y,model = None, leg = False): 38 | 39 | plt.plot(X[Y==0].numpy(),Y[Y==0].numpy(),'or', label = 'training points y = 0 ' ) 40 | plt.plot(X[Y==1].numpy(),Y[Y==1].numpy(),'ob', label = 'training points y = 1 ' ) 41 | 42 | if model != None: 43 | plt.plot(X.numpy(),model(X).detach().numpy(), label = 'Neural Network ') 44 | 45 | plt.legend() 46 | plt.show() 47 | 48 | 49 | 50 | # In[3] 51 | 52 | #Neutral Network Module Class and Training Function 53 | class SimpleNet(nn.Module): 54 | 55 | #Constructor 56 | def __init__(self, D_in, H, D_out): 57 | super().__init__() 58 | 59 | #Hidden layer 60 | self.linear1 = nn.Linear(D_in, H) #Input dimension, Number of neurons. 61 | self.linear2 = nn.Linear(H, D_out) # number of neurons H and output dimension 62 | 63 | #Prediction forward function 64 | def forward(self, x): 65 | 66 | x = torch.sigmoid(self.linear1(x)) 67 | x = torch.sigmoid(self.linear2(x)) 68 | 69 | return x 70 | 71 | 72 | 73 | # In[4] 74 | 75 | #Define the training function 76 | def train_model(data_set, model, criterion, train_loader, optimizer, epochs = 10, plot_number = 10): 77 | 78 | cost = [] #Empty cost list to accumulate the cost 79 | 80 | for epoch in range(epochs): 81 | total = 0 82 | 83 | for x, y in train_loader: 84 | 85 | yhat = model(x) 86 | 87 | loss = criterion(yhat, y) 88 | 89 | loss.backward() 90 | 91 | optimizer.step() 92 | 93 | optimizer.zero_grad() 94 | 95 | #Cumulative loss 96 | total += loss.item() 97 | 98 | cost.append(total) 99 | 100 | if epoch % plot_number == 0: 101 | PlotStuff(data_set.x, data_set.y, model) 102 | 103 | plt.figure() 104 | plt.plot(cost) 105 | plt.xlabel('epoch') 106 | plt.ylabel('cost') 107 | plt.show() 108 | 109 | return cost 110 | 111 | 112 | # In[5] 113 | 114 | #Create Data Class 115 | class Data(Dataset): 116 | def __init__(self): 117 | self.x = torch.linspace(-20, 20, 100).view(-1,1) 118 | 119 | self.y = torch.zeros(self.x.shape[0]) 120 | self.y[(self.x[:,0]>-10)& (self.x[:,0]<-5)]=1 121 | self.y[(self.x[:,0]>5)& (self.x[:,0]<10)]=1 122 | self.y = self.y.view(-1,1) 123 | self.len = self.x.shape[0] 124 | def __getitem__(self,index): 125 | 126 | return self.x[index],self.y[index] 127 | def __len__(self): 128 | return self.len 129 | 130 | data_set = Data() 131 | PlotStuff(data_set.x, data_set.y, leg = False) 132 | 133 | # In[6] 134 | #Criterion BCE Loss, Optimizer Adam, train_loader 135 | criterion = nn.BCELoss() 136 | 137 | #Initialize the model with 9 hidden layer neurons. 138 | model = SimpleNet(1, 9, 1) 139 | 140 | #Learning rate 141 | learning_rate = 0.1 142 | 143 | #train_loader 144 | train_loader = DataLoader(dataset = data_set, batch_size = 100) 145 | 146 | #Optimizer 147 | optimizer = torch.optim.Adam(model.parameters(), lr = learning_rate) 148 | 149 | # In[7] 150 | 151 | #Training 152 | 153 | COST = train_model(data_set, model, criterion, train_loader, optimizer, epochs = 1000, plot_number = 200) 154 | 155 | plt.plot(COST) 156 | plt.xlabel('epoch') 157 | plt.title('BCE loss') 158 | 159 | 160 | -------------------------------------------------------------------------------- /Week 4/Neural Networks with One Hidden Layer1.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Neural Networks with One Hidden Layer 4 | 5 | """ 6 | 7 | # In[1]: Imports 8 | 9 | import torch 10 | import torch.nn as nn 11 | import torchvision.transforms as transforms 12 | import torchvision.datasets as dsets 13 | from torch.utils.data import DataLoader, Dataset 14 | import matplotlib.pylab as plt 15 | import numpy as np 16 | torch.manual_seed(0) 17 | 18 | # In[2] 19 | #Plotting function 20 | def plot_accuracy_loss(training_results): 21 | plt.subplot(2, 1, 1) 22 | plt.plot(training_results['training_loss'], 'r') 23 | plt.ylabel('loss') 24 | plt.title('training loss iterations') 25 | plt.subplot(2, 1, 2) 26 | plt.plot(training_results['validation_accuracy']) 27 | plt.ylabel('accuracy') 28 | plt.xlabel('epochs') 29 | plt.show() 30 | 31 | # In[3] 32 | #Model Parameters function 33 | def print_model_parameters(model): 34 | count = 0 35 | 36 | for element in model.state_dict(): 37 | count += 1 38 | if count %2 != 0: 39 | print ("The following are the parameters for the layer ", count // 2 + 1) 40 | if element.find("bias") != -1: 41 | print("The size of bias: ", model.state_dict()[element].size()) 42 | else: 43 | print("The size of weights: ", model.state_dict()[element].size()) 44 | 45 | # In[4] 46 | #Function to show data (images) 47 | def show_data(data_sample): 48 | plt.imshow(data_sample.numpy().reshape(28, 28), cmap = 'gray') 49 | plt.show() 50 | 51 | # In[5] 52 | #Class Neural Net with 1 hidden layer 53 | class SimpleNet(nn.Module): 54 | #Constructor 55 | def __init__(self, D_in, H, D_out): 56 | super().__init__() 57 | self.linear1 = nn.Linear(D_in, H) #input layer 58 | self.linear2 = nn.Linear(H, D_out) #Output layer 59 | 60 | #Prediction 61 | def forward(self, x): 62 | x = torch.sigmoid(self.linear1(x)) 63 | x = self.linear2(x) #We want to predict multiple classes so sigmoid function shouldnt be used 64 | 65 | return x 66 | 67 | # In[6] 68 | #Training function 69 | def train_model(model, criterion, train_loader, validation_loader, optimizer, epochs = 100): 70 | i = 0 71 | useful_stuff = {'training_loss': [], 'validation_accuracy': []} 72 | 73 | for epoch in range(epochs): 74 | 75 | for i, (x, y) in enumerate(train_loader): 76 | 77 | optimizer.zero_grad() 78 | 79 | z = model(x.view(-1, 28*28)) #reshaping to 28*28 80 | 81 | loss = criterion(z, y) 82 | 83 | loss.backward() 84 | 85 | optimizer.step() 86 | 87 | useful_stuff['training_loss'].append(loss.data.item()) 88 | 89 | correct = 0 90 | 91 | for x, y in validation_loader: 92 | 93 | z = model(x.view(-1, 28*28)) #reshaping to 28*28 94 | 95 | _, label = torch.max(z, 1) #Take class with max probability 96 | 97 | correct += (label == y).sum().item() 98 | 99 | accuracy = 100 * (correct / len(validation_dataset)) 100 | useful_stuff['validation_accuracy'].append(accuracy) 101 | return useful_stuff 102 | 103 | 104 | # In[7] 105 | 106 | #Load Data 107 | train_dataset = dsets.MNIST(root='./data', train = True, download = True, transform = transforms.ToTensor()) #Load train MNIST set and transform to tensor 108 | validation_dataset = dsets.MNIST(root='./data', train = False, download = True, transform = transforms.ToTensor()) #Load non-train MNIST set and transform to tensor 109 | 110 | # In[8] 111 | #Create dataloader, criterion function, optimizer, learning rate 112 | 113 | #traind and validation loader 114 | train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=2000, shuffle=True) 115 | validation_loader = torch.utils.data.DataLoader(dataset=validation_dataset, batch_size=5000, shuffle=False) 116 | 117 | #Criterion function 118 | criterion = nn.CrossEntropyLoss() 119 | 120 | #learning rate 121 | learning_rate = 0.01 122 | 123 | #Create model with input dimension of the images WxH 28*28, 100 neurons and 10 output dim. 124 | model = SimpleNet(784, 100, 10) 125 | 126 | #Model parameters 127 | print_model_parameters(model) 128 | 129 | #optimizer 130 | optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate) 131 | 132 | # In[9] 133 | 134 | #Training the model 135 | 136 | training_results = train_model(model, criterion, train_loader, validation_loader, optimizer, epochs=30) 137 | 138 | 139 | # In[10] 140 | #Plot accuracy and loss 141 | 142 | plot_accuracy_loss(training_results) 143 | 144 | 145 | #Plot the 1st 5 misclassified items 146 | count = 0 147 | for x, y in validation_dataset: 148 | z = model(x.reshape(-1, 28 * 28)) 149 | _,yhat = torch.max(z, 1) 150 | if yhat != y: 151 | show_data(x) 152 | count += 1 153 | if count >= 5: 154 | break 155 | 156 | 157 | 158 | # In[11] 159 | 160 | #Use nn.Sequential to build to same model, train it and plot 161 | model = torch.nn.Sequential(nn.Linear(784, 100), nn.Sigmoid(), nn.Linear(100, 10)) 162 | 163 | training_results = train_model(model, criterion, train_loader, validation_loader, optimizer, epochs = 10) 164 | 165 | plot_accuracy_loss(training_results) 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | -------------------------------------------------------------------------------- /Week 4/Neural Networks with one Hidden Layer.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Neural Networks with one Hidden Layer 4 | 5 | """ 6 | 7 | # In[1]: Imports 8 | 9 | import torch 10 | import torch.nn as nn 11 | from torch.utils.data import DataLoader, Dataset 12 | import matplotlib.pylab as plt 13 | from matplotlib.colors import ListedColormap 14 | import numpy as np 15 | torch.manual_seed(0) 16 | 17 | # In[2] 18 | #Plotting function 19 | def plot_decision_regions_2class(model, dataset): 20 | cmap_light = ListedColormap(['#FFAAAA', '#AAFFAA', '#00AAFF']) 21 | cmap_bold = ListedColormap(['#FF0000', '#00FF00', '#00AAFF']) 22 | X = dataset.x.numpy() 23 | y = dataset.y.numpy() 24 | h = .02 25 | x_min, x_max = X[:, 0].min() - 0.1 , X[:, 0].max() + 0.1 26 | y_min, y_max = X[:, 1].min() - 0.1 , X[:, 1].max() + 0.1 27 | xx, yy = np.meshgrid(np.arange(x_min, x_max, h),np.arange(y_min, y_max, h)) 28 | XX = torch.Tensor(np.c_[xx.ravel(), yy.ravel()]) 29 | 30 | yhat = np.logical_not((model(XX)[:, 0] > 0.5).numpy()).reshape(xx.shape) 31 | plt.pcolormesh(xx, yy, yhat, cmap=cmap_light) 32 | plt.plot(X[y[:, 0] == 0, 0], X[y[:, 0] == 0, 1], 'o', label='y=0') 33 | plt.plot(X[y[:, 0] == 1, 0], X[y[:, 0] == 1, 1], 'ro', label='y=1') 34 | plt.title("decision region") 35 | plt.legend() 36 | 37 | 38 | # In[3] 39 | #Accuracy function 40 | def accuracy(model, dataset): 41 | return np.mean(dataset.y.view(-1).numpy() == (model(dataset.x)[:, 0] > 0.5).numpy()) 42 | 43 | # In[4] 44 | #Class Neural Net with 1 hidden layer 45 | class SimpleNet(nn.Module): 46 | #Constructor 47 | def __init__(self, D_in, H, D_out): 48 | super().__init__() 49 | self.linear1 = nn.Linear(D_in, H) #input layer 50 | self.linear2 = nn.Linear(H, D_out) #Output layer 51 | 52 | #Prediction 53 | def forward(self, x): 54 | x = torch.sigmoid(self.linear1(x)) 55 | x = torch.sigmoid(self.linear2(x)) 56 | 57 | return x 58 | 59 | # In[5] 60 | #Training function 61 | def train_model(data_set, model, criterion, train_loader, optimizer, epochs = 5): 62 | COST = [] 63 | ACC = [] 64 | for epoch in range(epochs): 65 | total=0 66 | for x, y in train_loader: 67 | 68 | optimizer.zero_grad() 69 | 70 | yhat = model(x) 71 | 72 | loss = criterion(yhat, y) 73 | 74 | optimizer.zero_grad() 75 | 76 | loss.backward() 77 | 78 | optimizer.step() 79 | 80 | #cumulative loss 81 | total+=loss.item() 82 | ACC.append(accuracy(model, data_set)) 83 | COST.append(total) 84 | 85 | fig, ax1 = plt.subplots() 86 | color = 'tab:red' 87 | ax1.plot(COST, color = color) 88 | ax1.set_xlabel('epoch', color = color) 89 | ax1.set_ylabel('total loss', color = color) 90 | ax1.tick_params(axis = 'y', color = color) 91 | 92 | ax2 = ax1.twinx() 93 | color = 'tab:blue' 94 | ax2.set_ylabel('accuracy', color = color) # we already handled the x-label with ax1 95 | ax2.plot(ACC, color=color) 96 | ax2.tick_params(axis='y', color = color) 97 | fig.tight_layout() # otherwise the right y-label is slightly clipped 98 | 99 | plt.show() 100 | 101 | return COST 102 | 103 | # In[6] 104 | #Data Class 105 | 106 | class XOR_Data(Dataset): 107 | 108 | #Constructor 109 | def __init__(self, N_s = 100): 110 | self.x = torch.zeros((N_s, 2)) 111 | self.y = torch.zeros((N_s, 1)) 112 | for i in range(N_s // 4): 113 | self.x[i, :] = torch.Tensor([0.0, 0.0]) 114 | self.y[i, 0] = torch.Tensor([0.0]) 115 | 116 | self.x[i + N_s // 4, :] = torch.Tensor([0.0, 1.0]) 117 | self.y[i + N_s // 4, 0] = torch.Tensor([1.0]) 118 | 119 | self.x[i + N_s // 2, :] = torch.Tensor([1.0, 0.0]) 120 | self.y[i + N_s // 2, 0] = torch.Tensor([1.0]) 121 | 122 | self.x[i + 3 * N_s // 4, :] = torch.Tensor([1.0, 1.0]) 123 | self.y[i + 3 * N_s // 4, 0] = torch.Tensor([0.0]) 124 | 125 | self.x = self.x + 0.01 * torch.randn((N_s, 2)) 126 | self.len = N_s 127 | 128 | #Get 129 | def __getitem__(self, index): 130 | return self.x[index],self.y[index] 131 | 132 | #Length 133 | def __len__(self): 134 | return self.len 135 | 136 | #Plot the data 137 | def plot_stuff(self): 138 | plt.plot(self.x[self.y[:, 0] == 0, 0].numpy(), self.x[self.y[:, 0] == 0, 1].numpy(), 'o', label="y=0") 139 | plt.plot(self.x[self.y[:, 0] == 1, 0].numpy(), self.x[self.y[:, 0] == 1, 1].numpy(), 'ro', label="y=1") 140 | plt.legend() 141 | 142 | 143 | # In[7] 144 | #Create XOR_Data object 145 | dataset = XOR_Data() 146 | dataset.plot_stuff() 147 | 148 | # In[8] 149 | #Creating model and train 150 | 151 | model = SimpleNet(2,1,1) #Model with 2 inputs, 1 neuron and 1 output 152 | 153 | #Train the model 154 | learning_rate = 0.001 155 | criterion = nn.BCELoss() 156 | optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate) 157 | train_loader = DataLoader(dataset = dataset, batch_size = 1) 158 | LOSS12 = train_model(dataset, model, criterion, train_loader, optimizer, epochs = 500) 159 | plot_decision_regions_2class(model, dataset) 160 | 161 | model = SimpleNet(2,2,1) #Model with 2 inputs, 2 neuron and 1 output 162 | #Train the model 163 | learning_rate = 0.1 164 | criterion = nn.BCELoss() 165 | optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate) 166 | train_loader = DataLoader(dataset = dataset, batch_size = 1) 167 | LOSS12 = train_model(dataset, model, criterion, train_loader, optimizer, epochs = 500) 168 | plot_decision_regions_2class(model, dataset) 169 | 170 | 171 | model = SimpleNet(2,6,1) #Model with 2 inputs, 6 neuron and 1 output 172 | #Train the model 173 | learning_rate = 0.1 174 | criterion = nn.BCELoss() 175 | optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate) 176 | train_loader = DataLoader(dataset = dataset, batch_size = 1) 177 | LOSS12 = train_model(dataset, model, criterion, train_loader, optimizer, epochs = 500) 178 | plot_decision_regions_2class(model, dataset) 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | -------------------------------------------------------------------------------- /Week 4/Simple One Hidden Layer Neural Network.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Simple One Hidden Layer Neural Network 4 | 5 | """ 6 | 7 | # In[1] Imports 8 | 9 | import torch 10 | import torch.nn as nn 11 | from torch import sigmoid 12 | import matplotlib.pylab as plt 13 | import numpy as np 14 | from torch import optim 15 | torch.manual_seed(0) 16 | 17 | 18 | # In[2] 19 | 20 | #Function for plotting the model 21 | def PlotStuff(X, Y, model, epoch, leg=True): 22 | 23 | plt.plot(X.numpy(), model(X).detach().numpy(), label=('epoch ' + str(epoch))) 24 | plt.plot(X.numpy(), Y.numpy(), 'r') 25 | plt.xlabel('x') 26 | if leg == True: 27 | plt.legend() 28 | else: 29 | pass 30 | 31 | 32 | # In[3] 33 | 34 | #Neutral Network Module Class and Training Function 35 | class SimpleNet(nn.Module): 36 | 37 | #Constructor 38 | def __init__(self, D_in, H, D_out): 39 | super().__init__() 40 | 41 | #Hidden layer 42 | self.linear1 = nn.Linear(D_in, H) #Input dimension, Number of neurons. 43 | self.linear2 = nn.Linear(H, D_out) # number of neurons H and output dimension 44 | 45 | #Define the first linear layer as an attribute. This is not good practice 46 | self.a1 = None 47 | self.l1 = None 48 | self.l2 = None 49 | 50 | #Prediction forward function 51 | def forward(self, x): 52 | 53 | self.l1 = self.linear1(x) 54 | self.a1 = sigmoid(self.l1) 55 | self.l2 = self.linear2(self.a1) 56 | 57 | yhat = sigmoid(self.linear2(self.a1)) 58 | 59 | return yhat 60 | 61 | 62 | 63 | # In[4] 64 | 65 | #Define the training function 66 | def train_model(X, Y, model, optimizer, criterion, epochs = 1000): 67 | 68 | cost = [] #Empty cost list to accumulate the cost 69 | total = 0 70 | 71 | for epoch in range(epochs): 72 | total = 0 73 | 74 | for x, y in zip(X, Y): 75 | 76 | yhat = model(x) 77 | 78 | loss = criterion(yhat, y) 79 | 80 | loss.backward() 81 | 82 | optimizer.step() 83 | 84 | optimizer.zero_grad() 85 | 86 | #Cumulative loss 87 | total += loss.item() 88 | 89 | cost.append(total) 90 | 91 | if epoch % 500 == 0: 92 | 93 | PlotStuff(X, Y, model, epoch, leg = True) 94 | plt.show() 95 | model(X) 96 | plt.scatter(model.a1.detach().numpy()[:, 0], model.a1.detach().numpy()[:, 1], c=Y.numpy().reshape(-1)) 97 | plt.title('activations') 98 | plt.show() 99 | 100 | return cost 101 | 102 | 103 | # In[5] 104 | 105 | #Create Data 106 | X = torch.arange(-20, 20, 1).view(-1, 1).type(torch.FloatTensor) 107 | 108 | Y = torch.zeros(X.shape[0]) 109 | 110 | Y[(X[:, 0] > -4) & (X[:, 0] < 4)] = 1.0 111 | 112 | 113 | # In[6] 114 | #Criterion function Cross-Entropy, Optimizer 115 | def criterion_cross(outputs, labels): 116 | out = -1 * torch.mean(labels * torch.log(outputs) + (1 - labels) * torch.log(1 - outputs)) 117 | return out 118 | 119 | 120 | #Initialize the model 121 | model = SimpleNet(1, 2, 1) 122 | 123 | #Learning rate 124 | learning_rate = 0.1 125 | 126 | #Optimizer 127 | optimizer = optim.SGD(model.parameters(), lr = learning_rate) 128 | 129 | # In[7] 130 | 131 | #Training 132 | 133 | cost_cross = train_model(X, Y, model, optimizer, criterion_cross, epochs = 1000) 134 | 135 | plt.plot(cost_cross) 136 | plt.xlabel('epoch') 137 | plt.title('cross entropy loss') 138 | 139 | # In[8] 140 | #Prediction 141 | x = torch.tensor([0.0]) 142 | yhat = model(x) 143 | yhat 144 | 145 | X_=torch.tensor([[0.0],[2.0],[3.0]]) 146 | Yhat=model(X_) 147 | Yhat 148 | 149 | Yhat=Yhat>0.5 150 | Yhat 151 | 152 | # In[9] 153 | 154 | #Trying a model with MSE loss function. #Not good 155 | 156 | criterion_mse = nn.MSELoss() 157 | 158 | model = SimpleNet(1, 2, 1) 159 | 160 | cost_mse = train_model(X, Y, model, optimizer, criterion_mse, epochs = 1000) 161 | 162 | plt.plot(cost_mse) 163 | plt.xlabel('epoch') 164 | plt.title('MSE loss ') 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | -------------------------------------------------------------------------------- /Week 4/Softmax 1D.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Softmax Classifier 1D 4 | 5 | """ 6 | 7 | import torch 8 | import torch.nn as nn 9 | import matplotlib.pyplot as plt 10 | import numpy as np 11 | from torch.utils.data import DataLoader, Dataset 12 | 13 | #Create a plotting function 14 | def plot_data(data_set, model = None, n = 1, color = False): 15 | X = data_set[:][0] 16 | Y = data_set[:][1] 17 | plt.plot(X[Y == 0, 0].numpy(), Y[Y == 0].numpy(), 'bo', label = 'y = 0') 18 | plt.plot(X[Y == 1, 0].numpy(), 0 * Y[Y == 1].numpy(), 'ro', label = 'y = 1') 19 | plt.plot(X[Y == 2, 0].numpy(), 0 * Y[Y == 2].numpy(), 'go', label = 'y = 2') 20 | plt.ylim((-0.1, 3)) 21 | plt.legend() 22 | if model != None: 23 | w = list(model.parameters())[0][0].detach() 24 | b = list(model.parameters())[1][0].detach() 25 | y_label = ['yhat=0', 'yhat=1', 'yhat=2'] 26 | y_color = ['b', 'r', 'g'] 27 | Y = [] 28 | for w, b, y_l, y_c in zip(model.state_dict()['0.weight'], model.state_dict()['0.bias'], y_label, y_color): 29 | Y.append((w * X + b).numpy()) 30 | plt.plot(X.numpy(), (w * X + b).numpy(), y_c, label = y_l) 31 | if color == True: 32 | x = X.numpy() 33 | x = x.reshape(-1) 34 | top = np.ones(x.shape) 35 | y0 = Y[0].reshape(-1) 36 | y1 = Y[1].reshape(-1) 37 | y2 = Y[2].reshape(-1) 38 | plt.fill_between(x, y0, where = y1 > y1, interpolate = True, color = 'blue') 39 | plt.fill_between(x, y0, where = y1 > y2, interpolate = True, color = 'blue') 40 | plt.fill_between(x, y1, where = y1 > y0, interpolate = True, color = 'red') 41 | plt.fill_between(x, y1, where = ((y1 > y2) * (y1 > y0)),interpolate = True, color = 'red') 42 | plt.fill_between(x, y2, where = (y2 > y0) * (y0 > 0),interpolate = True, color = 'green') 43 | plt.fill_between(x, y2, where = (y2 > y1), interpolate = True, color = 'green') 44 | plt.legend() 45 | plt.show() 46 | 47 | 48 | torch.manual_seed(1) 49 | 50 | #Create Data class using torch Dataset 51 | class Data(Dataset): 52 | 53 | #constructor 54 | def __init__(self): 55 | self.x = torch.arange(-2, 2, 0.1).view(-1, 1) 56 | self.y = torch.zeros(self.x.shape[0]) 57 | self.y[(self.x > -1.0)[:, 0] * (self.x < 1.0)[:, 0]] = 1 58 | self.y[(self.x >= 1.0)[:, 0]] = 2 59 | self.y = self.y.type(torch.LongTensor) 60 | self.len = self.x.shape[0] 61 | 62 | #Get function 63 | def __getitem__(self, index): 64 | 65 | return self.x[index], self.y[index] 66 | 67 | def __len__(self): 68 | return self.len 69 | 70 | #Create Data object 71 | data_set = Data() 72 | 73 | #Plot data_set 74 | plot_data(data_set) 75 | 76 | """ 77 | Build a Softmax Classifier 78 | """ 79 | 80 | model = nn.Sequential(nn.Linear(1, 3)) #Create a Sequential model with 1 input and 3 outputs 81 | model.state_dict() #Show initialized parameters 82 | 83 | 84 | #Create criterion loss function, optimizer, and Dataloader object 85 | criterion = nn.CrossEntropyLoss() 86 | 87 | #Optimizer Stochastic Gradient Descent with learning rate = 0.01 88 | optimizer = torch.optim.SGD(model.parameters(), lr = 0.01) 89 | 90 | #Dataloader object with batch size 5 91 | trainloader = DataLoader(dataset = data_set, batch_size = 5) 92 | 93 | 94 | """ 95 | Training Model 96 | """ 97 | 98 | #Create empty Loss list 99 | LOSS = [] 100 | 101 | #Create training function 102 | def train_model(epochs): 103 | for epoch in range(epochs): 104 | 105 | if epoch %50 == 0: #When remainder of epoch/50 = 0, pass and plot 106 | pass 107 | plot_data(data_set, model) #Plot every 50 epochs 108 | 109 | for x, y in trainloader: 110 | optimizer.zero_grad() 111 | 112 | yhat = model(x) 113 | 114 | loss = criterion(yhat, y) 115 | 116 | LOSS.append(loss) 117 | 118 | loss.backward() 119 | 120 | optimizer.step() 121 | 122 | train_model(300) #train model for 300 epochs 123 | 124 | """ 125 | Analyze results and Predict 126 | """ 127 | 128 | #Model prediction z 129 | z = model(data_set.x) 130 | _, yhat = z.max(1) 131 | print("The prediction", yhat) 132 | 133 | 134 | #Accuracy 135 | correct = (data_set.y == yhat).sum().item() 136 | accuracy = correct / len(data_set) 137 | print("The accuracy: ", accuracy) 138 | 139 | 140 | #Softmax function can also convert 'z' to a probability 141 | Softmax_f = nn.Softmax(dim = -1) 142 | Probability = Softmax_f(z) 143 | 144 | for i in range(3): 145 | print("probability of class {} is given by {}".format(i, Probability[0,i]) ) 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | -------------------------------------------------------------------------------- /Week 4/Softmax MNIST Classifier.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Softmax Classifier using MNIST numbers dataset 4 | 5 | """ 6 | 7 | import torch 8 | import torch.nn as nn 9 | import torchvision.transforms as transforms 10 | import torchvision.datasets as dsets 11 | from torch.utils.data import DataLoader 12 | import matplotlib.pyplot as plt 13 | import numpy as np 14 | 15 | 16 | #Define a plotting function to plot parameters 17 | def PlotParameters(model): 18 | W = model.state_dict()['linear.weight'].data 19 | w_min = W.min().item() 20 | w_max = W.max().item() 21 | fig, axes = plt.subplots(2, 5) 22 | fig.subplots_adjust(hspace=0.01, wspace=0.1) 23 | for i, ax in enumerate(axes.flat): 24 | if i < 10: 25 | 26 | # Set the label for the sub-plot. 27 | ax.set_xlabel("class: {0}".format(i)) 28 | 29 | # Plot the image. 30 | ax.imshow(W[i, :].view(28, 28), vmin=w_min, vmax=w_max, cmap='seismic') 31 | 32 | ax.set_xticks([]) 33 | ax.set_yticks([]) 34 | 35 | # Ensure the plot is shown correctly with multiple plots 36 | # in a single Notebook cell. 37 | plt.show() 38 | 39 | def show_data(data_sample): 40 | plt.imshow(data_sample[0].numpy().reshape(28, 28), cmap='gray') #Showing image 28x28 in gray scale 41 | plt.title('y = ' + str(data_sample[1].item())) #Showing title as label corresponding to image 42 | 43 | 44 | 45 | 46 | #Lets Create some data (training set from mnist) 47 | train_dataset = dsets.MNIST(root='./data', train = True, download = True, transform = transforms.ToTensor()) 48 | print("The Training dataset:\n", train_dataset) 49 | 50 | #Create validation set 51 | val_dataset = dsets.MNIST(root = './data', train = False, download = True, transform = transforms.ToTensor()) 52 | print("The Validation dataset:\n", val_dataset) 53 | 54 | #Print datatype 55 | print("Type of data element: ", train_dataset[0][1].type()) 56 | 57 | #Plotting image samples 58 | print("The image: ", show_data(train_dataset[3])) 59 | print("The image: ", show_data(train_dataset[2])) 60 | 61 | 62 | 63 | """ 64 | Build a Softmax Classifier Class 65 | """ 66 | #Softmax Class from nn.Module 67 | class SoftMax(nn.Module): 68 | 69 | #Constructor 70 | def __init__(self, input_size, output_size): 71 | super().__init__() 72 | self.linear = nn.Linear(input_size, output_size) 73 | 74 | #Prediction 75 | def forward(self, x): 76 | 77 | z = self.linear(x) 78 | 79 | return z 80 | 81 | #train_dataset shape 82 | train_dataset[0][0].shape 83 | 84 | #The trainset needs to be flattened to 1 column and multiple rows (728) 85 | 86 | input_dim = 28 * 28 #728 rows 87 | output_dim = 10 #10 Categories 0,1,2,3,4,5,6,7,8,9 88 | 89 | 90 | #Create the model 91 | model = SoftMax(input_dim, output_dim) 92 | print("The Model:\n", model) 93 | 94 | 95 | #Lets see the initialized parameters and their size 96 | print('W: ',list(model.parameters())[0].size()) 97 | print('b: ',list(model.parameters())[1].size()) 98 | print("The Parameters are: \n", model.state_dict()) 99 | 100 | 101 | #Lets plot the parameters 102 | PlotParameters(model) 103 | 104 | #Load data into DataLoader 105 | train_loader = DataLoader(dataset = train_dataset, batch_size = 100) 106 | val_loader = DataLoader(dataset = val_dataset, batch_size = 5000) 107 | 108 | #Set the learning_rate 109 | learning_rate = 0.1 110 | 111 | #Define optimizer and criterion 112 | optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate) 113 | criterion = nn.CrossEntropyLoss() 114 | 115 | 116 | 117 | #Train the model 118 | epochs = 10 119 | LOSS_List = [] #empty list to store LOSS 120 | Accuracy_List = [] 121 | N_test = len(val_dataset) 122 | 123 | #Training function 124 | def train_model(n_epochs): 125 | for epoch in range(epochs): 126 | 127 | for x, y in train_loader: 128 | 129 | optimizer.zero_grad() 130 | 131 | z = model(x.view(-1, 28 * 28)) #reshaping to 28x28 132 | 133 | loss = criterion(z, y) 134 | 135 | loss.backward() 136 | 137 | optimizer.step() 138 | 139 | correct = 0 140 | print(correct) 141 | 142 | #Perform a prediction on the validation data 143 | for x_test, y_test in val_loader: 144 | 145 | z = model(x_test.view(-1, 28 * 28)) 146 | 147 | _, yhat = torch.max(z.data, 1) #Take max value from z 148 | 149 | correct += (yhat == y_test).sum().item() 150 | print(correct) 151 | 152 | accuracy = correct / N_test 153 | 154 | LOSS_List.append(loss.data) 155 | 156 | Accuracy_List.append(accuracy) 157 | 158 | train_model(epochs) 159 | 160 | 161 | """ 162 | Analyze the model 163 | """ 164 | 165 | #Plot the loss and accuracy 166 | fig, ax1 = plt.subplots() 167 | color = 'tab:red' 168 | ax1.plot(LOSS_List,color=color) 169 | ax1.set_xlabel('epoch',color=color) 170 | ax1.set_ylabel('total loss',color=color) 171 | ax1.tick_params(axis='y', color=color) 172 | 173 | ax2 = ax1.twinx() 174 | color = 'tab:blue' 175 | ax2.set_ylabel('accuracy', color=color) 176 | ax2.plot(Accuracy_List, color=color) 177 | ax2.tick_params(axis='y', color=color) 178 | fig.tight_layout() 179 | 180 | 181 | #Plot trained parameters 182 | PlotParameters(model) 183 | 184 | 185 | #Plot missclassified examples 186 | Softmax_function = nn.Softmax(dim = -1) 187 | count = 0 188 | 189 | #Plot the first 5 correctly classified samples and their respective probability by using torch.max 190 | for x,y in val_dataset: 191 | 192 | z = model(x.reshape(-1, 28*28)) 193 | _, yhat = torch.max(z, 1) 194 | 195 | if yhat == y: 196 | show_data((x, y)) 197 | plt.show() 198 | print("yhat: ". yhat) 199 | print("Probability of class: ", torch.max(Softmax_function(z).item())) 200 | count += 1 201 | 202 | if count >= 5: 203 | break 204 | 205 | 206 | #Plot the first 5 missclassified samples and their respective probability by using torch.max 207 | count = 0 208 | for x,y in val_dataset: 209 | 210 | z = model(x.reshape(-1, 28*28)) 211 | _, yhat = torch.max(z, 1) 212 | 213 | if yhat != y: 214 | show_data((x, y)) 215 | plt.show() 216 | print("yhat: ". yhat) 217 | print("Probability of class: ", torch.max(Softmax_function(z).item())) 218 | count += 1 219 | 220 | if count >= 5: 221 | break 222 | 223 | 224 | 225 | 226 | 227 | 228 | 229 | 230 | 231 | 232 | -------------------------------------------------------------------------------- /Week 4/Test Activation Functions on the MNIST Dataset.py: -------------------------------------------------------------------------------- 1 | 2 | """ 3 | 4 | Test Activation Functions on the MNIST Dataset 5 | 6 | """ 7 | # In[1] 8 | # Import the libraries 9 | 10 | # Using the following line code to install the torchvision library 11 | 12 | import torch 13 | import torch.nn as nn 14 | import torchvision.transforms as transforms 15 | import torchvision.datasets as dsets 16 | import torch.nn.functional as F 17 | 18 | import matplotlib.pylab as plt 19 | 20 | # use_cuda = torch.cuda.is_available() 21 | # device = torch.device("cuda:0" if use_cuda else "cpu") 22 | # torch.backends.cudnn.benchmark = True 23 | 24 | # In[2] 25 | 26 | # Build the model with sigmoid function 27 | 28 | 29 | class Net(nn.Module): 30 | 31 | # Constructor 32 | def __init__(self, D_in, H, D_out): 33 | super().__init__() 34 | self.linear1 = nn.Linear(D_in, H) 35 | self.linear2 = nn.Linear(H, D_out) 36 | 37 | # Prediction 38 | def forward(self, x): 39 | x = self.linear1(x) 40 | x = torch.sigmoid(x) 41 | x = self.linear2(x) 42 | return x 43 | 44 | 45 | # Build the model with Tanh function 46 | 47 | 48 | class NetTanh(nn.Module): 49 | 50 | # Constructor 51 | def __init__(self, D_in, H, D_out): 52 | super().__init__() 53 | self.linear1 = nn.Linear(D_in, H) 54 | self.linear2 = nn.Linear(H, D_out) 55 | 56 | # Prediction 57 | def forward(self, x): 58 | x = self.linear1(x) 59 | x = torch.tanh(x) 60 | x = self.linear2(x) 61 | return x 62 | 63 | 64 | # Build the model with Relu function 65 | 66 | class NetRelu(nn.Module): 67 | 68 | # Constructor 69 | def __init__(self, D_in, H, D_out): 70 | super().__init__() 71 | self.linear1 = nn.Linear(D_in, H) 72 | self.linear2 = nn.Linear(H, D_out) 73 | 74 | # Prediction 75 | def forward(self, x): 76 | x = self.linear1(x) 77 | x = torch.relu(x) 78 | x = self.linear2(x) 79 | return x 80 | 81 | # In[3] 82 | # Define the function for training the model 83 | 84 | 85 | def train(model, criterion, train_loader, validation_loader, optimizer, epochs=100): 86 | i = 0 87 | useful_stuff = {'training_loss': [], 'validation_accuracy': []} 88 | 89 | for epoch in range(epochs): 90 | for i, (x, y) in enumerate(train_loader): 91 | #x, y = x.to(device), x.to(device) 92 | optimizer.zero_grad() 93 | z = model(x.view(-1, 28 * 28)) 94 | loss = criterion(z, y) 95 | loss.backward() 96 | optimizer.step() 97 | useful_stuff['training_loss'].append(loss.item()) 98 | 99 | correct = 0 100 | for x, y in validation_loader: 101 | #x, y = x.to(device), y.to(device) 102 | z = model(x.view(-1, 28 * 28)) 103 | _, label = torch.max(z, 1) 104 | correct += (label == y).sum().item() 105 | accuracy = 100 * (correct / len(validation_dataset)) 106 | useful_stuff['validation_accuracy'].append(accuracy) 107 | 108 | return useful_stuff 109 | 110 | # In[4] 111 | # Create the training dataset 112 | 113 | 114 | train_dataset = dsets.MNIST(root='./data', train=True, download=True, transform=transforms.ToTensor()) 115 | print(len(train_dataset)) 116 | 117 | # Create the validation dataset 118 | 119 | validation_dataset = dsets.MNIST(root='./data', train=False, download=True, transform=transforms.ToTensor()) 120 | print(len(validation_dataset)) 121 | 122 | 123 | # Create the criterion function 124 | 125 | criterion = nn.CrossEntropyLoss() 126 | 127 | # Create the training data loader and validation data loader object 128 | 129 | train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=2000, shuffle=True) 130 | validation_loader = torch.utils.data.DataLoader(dataset=validation_dataset, batch_size=5000, shuffle=False) 131 | 132 | 133 | # Create the model object 134 | 135 | input_dim = 28 * 28 136 | hidden_dim = 100 137 | output_dim = 10 138 | 139 | model = Net(input_dim, hidden_dim, output_dim) 140 | #model = model.cuda() 141 | 142 | 143 | # Train a model with sigmoid function 144 | 145 | learning_rate = 0.01 146 | optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate) 147 | training_results = train(model, criterion, train_loader, validation_loader, optimizer, epochs=30) 148 | 149 | 150 | # Train a model with Tanh function 151 | model_Tanh = NetTanh(input_dim, hidden_dim, output_dim) 152 | #model = model_Tanh.cuda() 153 | optimizer = torch.optim.SGD(model_Tanh.parameters(), lr=learning_rate) 154 | training_results_tanch = train(model_Tanh, criterion, train_loader, validation_loader, optimizer, epochs=30) 155 | 156 | # Train a model with Relu function 157 | 158 | modelRelu = NetRelu(input_dim, hidden_dim, output_dim) 159 | #model = modelRelu.cuda() 160 | optimizer = torch.optim.SGD(modelRelu.parameters(), lr=learning_rate) 161 | training_results_relu = train(modelRelu, criterion, train_loader, validation_loader, optimizer, epochs=30) 162 | 163 | 164 | # Compare the training loss 165 | 166 | plt.plot(training_results_tanch['training_loss'], label='tanh') 167 | plt.plot(training_results['training_loss'], label='sigmoid') 168 | plt.plot(training_results_relu['training_loss'], label='relu') 169 | plt.ylabel('loss') 170 | plt.title('training loss iterations') 171 | plt.legend() 172 | plt.show() 173 | 174 | 175 | # Compare the validation loss 176 | 177 | plt.plot(training_results_tanch['validation_accuracy'], label='tanh') 178 | plt.plot(training_results['validation_accuracy'], label='sigmoid') 179 | plt.plot(training_results_relu['validation_accuracy'], label='relu') 180 | plt.ylabel('validation accuracy') 181 | plt.xlabel('epochs ') 182 | plt.legend() 183 | plt.show() 184 | -------------------------------------------------------------------------------- /Week 5/Batch Normalization with MNIST Dataset.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Batch Normalization with MNIST Dataset 4 | 5 | """ 6 | 7 | 8 | # In[1] Imports 9 | 10 | import torch 11 | import torch.nn as nn 12 | import torchvision.transforms as transforms 13 | import torchvision.datasets as dsets 14 | import torch.nn.functional as F 15 | import matplotlib.pylab as plt 16 | import numpy as np 17 | torch.manual_seed(0) 18 | 19 | 20 | # In[2] Create Data/Load MNIST Dataset 21 | 22 | # load the train dataset 23 | train_dataset = dsets.MNIST(root='./data', train=True, download=True, transform=transforms.ToTensor()) 24 | 25 | # load the train dataset 26 | validation_dataset = dsets.MNIST(root='./data', train=False, download=True, transform=transforms.ToTensor()) 27 | 28 | # Create Data Loader for both train and validating 29 | train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=2000, shuffle=True) 30 | validation_loader = torch.utils.data.DataLoader(dataset=validation_dataset, batch_size=5000, shuffle=False) 31 | 32 | 33 | # In[3] Create Neural Network with BatchNorm and without 34 | 35 | # NN with BatchNorm 36 | class NetworkBatchNorm(nn.Module): 37 | 38 | # Constructor 39 | def __init__(self, in_size, H1, H2, out_size): 40 | super().__init__() 41 | self.linear1 = nn.Linear(in_size, H1) 42 | self.linear2 = nn.Linear(H1, H2) 43 | self.linear3 = nn.Linear(H2, out_size) 44 | self.bn1 = nn.BatchNorm1d(H1) 45 | self.bn2 = nn.BatchNorm1d(H2) 46 | 47 | # Prediction 48 | def forward(self, x): 49 | x = self.linear1(x) 50 | x = torch.relu(x) 51 | x = self.bn1(x) 52 | x = self.linear2(x) 53 | x = torch.relu(x) 54 | x = self.bn2(x) 55 | x = self.linear3(x) 56 | return x 57 | 58 | # Activations to analyze the results 59 | def activation(self, x): 60 | out = [] 61 | z1 = self.bn1(self.linear1(x)) 62 | out.append(z1.detach().numpy().reshape(-1)) 63 | a1 = torch.relu(z1) 64 | out.append(a1.detach().numpy().reshape(-1)) 65 | z2 = self.bn2(self.linear2(a1)) 66 | out.append(z2.detach().numpy().reshape(-1)) 67 | a2 = torch.relu(z2) 68 | out.append(a2.detach().numpy().reshape(-1)) 69 | return out 70 | 71 | 72 | # NN without BatchNorm 73 | class Network(nn.Module): 74 | 75 | # Constructor 76 | def __init__(self, in_size, H1, H2, out_size): 77 | super().__init__() 78 | self.linear1 = nn.Linear(in_size, H1) 79 | self.linear2 = nn.Linear(H1, H2) 80 | self.linear3 = nn.Linear(H2, out_size) 81 | 82 | # Prediction 83 | def forward(self, x): 84 | x = self.linear1(x) 85 | x = torch.relu(x) 86 | x = self.linear2(x) 87 | x = torch.relu(x) 88 | x = self.linear3(x) 89 | return x 90 | 91 | # Activations to analyze the results 92 | def activation(self, x): 93 | out = [] 94 | z1 = self.linear1(x) 95 | out.append(z1.detach().numpy().reshape(-1)) 96 | a1 = torch.relu(z1) 97 | out.append(a1.detach().numpy().reshape(-1)) 98 | z2 = self.linear2(a1) 99 | out.append(z2.detach().numpy().reshape(-1)) 100 | a2 = torch.relu(z2) 101 | out.append(a2.detach().numpy().reshape(-1)) 102 | return out 103 | 104 | 105 | # In[4] Create train loop 106 | 107 | # Define the function to train model 108 | 109 | 110 | def train(model, criterion, train_loader, validation_loader, optimizer, epochs=100): 111 | i = 0 112 | useful_stuff = {'training_loss':[], 'validation_accuracy':[]} 113 | 114 | for epoch in range(epochs): 115 | for i, (x, y) in enumerate(train_loader): 116 | model.train() 117 | optimizer.zero_grad() 118 | z = model(x.view(-1, 28 * 28)) 119 | loss = criterion(z, y) 120 | loss.backward() 121 | optimizer.step() 122 | useful_stuff['training_loss'].append(loss.data.item()) 123 | 124 | correct = 0 125 | for x, y in validation_loader: 126 | model.eval() 127 | yhat = model(x.view(-1, 28 * 28)) 128 | _, label = torch.max(yhat, 1) 129 | correct += (label == y).sum().item() 130 | 131 | accuracy = 100 * (correct / len(validation_dataset)) 132 | useful_stuff['validation_accuracy'].append(accuracy) 133 | 134 | return useful_stuff 135 | 136 | 137 | # In[6] Initialize the model, optimizer, criterion 138 | 139 | # Loss function 140 | criterion = nn.CrossEntropyLoss() 141 | 142 | # Initializing both models 143 | model_batchnorm = NetworkBatchNorm(28*28, 100, 100, 10) 144 | model = Network(28*28, 100, 100, 10) 145 | 146 | # Optimizer and train 147 | optimizer = torch.optim.Adam(model_batchnorm.parameters(), lr=0.1) 148 | training_results_Norm = train(model_batchnorm, criterion, train_loader, validation_loader, optimizer, epochs=5) 149 | 150 | optimizer = torch.optim.Adam(model.parameters(), lr=0.1) 151 | training_results = train(model, criterion, train_loader, validation_loader, optimizer, epochs=5) 152 | 153 | 154 | # In[5] Model evaluation and Plotting 155 | 156 | # set models to evaluation so that batchnorm is put in eval mode. 157 | model.eval() 158 | model_batchnorm.eval() 159 | 160 | # Plot model activations 161 | out = model.activation(validation_dataset[0][0].reshape(-1,28*28)) 162 | plt.hist(out[2], label = 'model with no batch normalization' ) 163 | plt.xlabel("activation ") 164 | plt.legend() 165 | plt.show() 166 | 167 | out_batchnorm = model_batchnorm.activation(validation_dataset[0][0].reshape(-1,28*28)) 168 | plt.hist(out_batchnorm[2], label = 'model with normalization') 169 | plt.xlabel("activation ") 170 | plt.legend() 171 | plt.show() 172 | 173 | 174 | # Plot the diagram to show the loss 175 | plt.plot(training_results['training_loss'], label = 'No Batch Normalization') 176 | plt.plot(training_results_Norm['training_loss'], label = 'Batch Normalization') 177 | plt.ylabel('Cost') 178 | plt.xlabel('iterations ') 179 | plt.legend() 180 | plt.show() 181 | 182 | 183 | # Plot the diagram to show the accuracy 184 | plt.plot(training_results['validation_accuracy'], label = 'No Batch Normalization') 185 | plt.plot(training_results_Norm['validation_accuracy'], label = 'Batch Normalization') 186 | plt.ylabel('validation accuracy') 187 | plt.xlabel('epochs ') 188 | plt.legend() 189 | plt.show() 190 | -------------------------------------------------------------------------------- /Week 5/Creating a Neural Network with nn.Modulelist.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Creating a Neural Network with nn.Modulelist 4 | 5 | """ 6 | 7 | 8 | # In[1] Imports 9 | 10 | import matplotlib.pyplot as plt 11 | import numpy as np 12 | import torch 13 | import torch.nn as nn 14 | import torch.nn.functional as F 15 | from matplotlib.colors import ListedColormap 16 | from torch.utils.data import Dataset, DataLoader 17 | 18 | # In[2] Plotting function 19 | 20 | 21 | def plot_decision_regions_3class(model, data_set): 22 | cmap_light = ListedColormap(['#FFAAAA', '#AAFFAA', '#00AAFF']) 23 | cmap_bold = ListedColormap(['#FF0000', '#00FF00', '#00AAFF']) 24 | X = data_set.x.numpy() 25 | y = data_set.y.numpy() 26 | h = .02 27 | x_min, x_max = X[:, 0].min() - 0.1, X[:, 0].max() + 0.1 28 | y_min, y_max = X[:, 1].min() - 0.1, X[:, 1].max() + 0.1 29 | xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h)) 30 | XX = torch.Tensor(np.c_[xx.ravel(), yy.ravel()]) 31 | _, yhat = torch.max(model(XX), 1) 32 | yhat = yhat.numpy().reshape(xx.shape) 33 | plt.pcolormesh(xx, yy, yhat, cmap=cmap_light) 34 | plt.plot(X[y[:] == 0, 0], X[y[:] == 0, 1], 'ro', label='y=0') 35 | plt.plot(X[y[:] == 1, 0], X[y[:] == 1, 1], 'go', label='y=1') 36 | plt.plot(X[y[:] == 2, 0], X[y[:] == 2, 1], 'o', label='y=2') 37 | plt.title("decision region") 38 | plt.legend() 39 | 40 | # In[3] Creating Data 41 | 42 | class Data(Dataset): 43 | 44 | # Constructor 45 | def __init__(self, K=3, N=500): 46 | D = 2 47 | X = np.zeros((N * K, D)) # data matrix (each row = single example) 48 | y = np.zeros(N * K, dtype='uint8') # class labels 49 | for j in range(K): 50 | ix = range(N * j, N * (j + 1)) 51 | r = np.linspace(0.0, 1, N) # radius 52 | t = np.linspace(j * 4, (j + 1) * 4, N) + np.random.randn(N) * 0.2 # theta 53 | X[ix] = np.c_[r * np.sin(t), r*np.cos(t)] 54 | y[ix] = j 55 | self.y = torch.from_numpy(y).type(torch.LongTensor) 56 | self.x = torch.from_numpy(X).type(torch.FloatTensor) 57 | self.len = y.shape[0] 58 | 59 | # Getter 60 | def __getitem__(self, index): 61 | return self.x[index], self.y[index] 62 | 63 | # Get Length 64 | def __len__(self): 65 | return self.len 66 | 67 | # Plot the diagram 68 | def plot_stuff(self): 69 | plt.plot(self.x[self.y[:] == 0, 0].numpy(), self.x[self.y[:] == 0, 1].numpy(), 'o', label="y = 0") 70 | plt.plot(self.x[self.y[:] == 1, 0].numpy(), self.x[self.y[:] == 1, 1].numpy(), 'ro', label="y = 1") 71 | plt.plot(self.x[self.y[:] == 2, 0].numpy(), self.x[self.y[:] == 2, 1].numpy(), 'go', label="y = 2") 72 | plt.legend() 73 | 74 | # In[4] Creating Network using ModuleList 75 | 76 | 77 | class Network(nn.Module): 78 | 79 | # Constructor 80 | def __init__(self, Layers): 81 | super().__init__() 82 | self.hidden = nn.ModuleList() 83 | for input_size, output_size in zip(Layers, Layers[1:]): 84 | self.hidden.append(nn.Linear(input_size, output_size)) 85 | 86 | # Prediction 87 | def forward(self, activation): 88 | L = len(self.hidden) 89 | for (l, linear_transform) in zip(range(L), self.hidden): 90 | if l < L-1: 91 | activation = torch.relu(linear_transform(activation)) 92 | else: 93 | activation = linear_transform(activation) 94 | return activation 95 | 96 | 97 | # In[5] Training function 98 | 99 | 100 | def train(data_set, model, criterion, train_loader, optimizer, epochs=100): 101 | LOSS = [] 102 | ACC = [] 103 | for epoch in range(epochs): 104 | for x, y in train_loader: 105 | optimizer.zero_grad() 106 | yhat = model(x) 107 | loss = criterion(yhat, y) 108 | optimizer.zero_grad() 109 | loss.backward() 110 | optimizer.step() 111 | LOSS.append(loss.item()) 112 | ACC.append(accuracy(model, data_set)) 113 | 114 | fig, ax1 = plt.subplots() 115 | color = 'tab:red' 116 | ax1.plot(LOSS, color=color) 117 | ax1.set_xlabel('Iteration', color=color) 118 | ax1.set_ylabel('total loss', color=color) 119 | ax1.tick_params(axis='y', color=color) 120 | 121 | ax2 = ax1.twinx() 122 | color = 'tab:blue' 123 | ax2.set_ylabel('accuracy', color=color) # we already handled the x-label with ax1 124 | ax2.plot(ACC, color=color) 125 | ax2.tick_params(axis='y', color=color) 126 | fig.tight_layout() # otherwise the right y-label is slightly clipped 127 | 128 | plt.show() 129 | return LOSS 130 | 131 | 132 | # In[6] accuracy function 133 | 134 | 135 | def accuracy(model, data_set): 136 | _, yhat = torch.max(model(data_set.x), 1) 137 | return (yhat == data_set.y).numpy().mean() 138 | 139 | 140 | # In[7] Create Dataset object and plot 141 | 142 | data_set = Data() 143 | data_set.plot_stuff() 144 | data_set.y = data_set.y.view(-1) 145 | 146 | 147 | # In[8] Create model using ModuleList() with 1 hidden layer and train it 148 | 149 | 150 | Layers = [2, 50, 3] # 2 inputs (x,y), 1 hidden layer model with 50 neurons, 3 output neurons (R, G, B) (3 class model) 151 | model = Network(Layers) 152 | optimizer = torch.optim.SGD(model.parameters(), lr=0.1) 153 | criterion = nn.CrossEntropyLoss() 154 | train_loader = DataLoader(dataset=data_set, batch_size=20) 155 | 156 | # Train model 157 | LOSS = train(data_set, model, criterion, train_loader, optimizer, epochs=100) 158 | 159 | # Plot decision regions (3 class model) 160 | plot_decision_regions_3class(model, data_set) 161 | 162 | 163 | # In[9] Create model using ModuleList() with 2 hidden layer and train it 164 | 165 | 166 | Layers = [2, 10, 10, 3] 167 | model = Network(Layers) 168 | optimizer = torch.optim.SGD(model.parameters(), lr=0.1) 169 | criterion = nn.CrossEntropyLoss() 170 | train_loader = DataLoader(dataset=data_set, batch_size=20) 171 | 172 | # Train model 173 | LOSS = train(data_set, model, criterion, train_loader, optimizer, epochs=300) 174 | 175 | # Plot decision regions (3 class model) 176 | plot_decision_regions_3class(model, data_set) 177 | 178 | 179 | 180 | # In[10] Create model using ModuleList() with 2 hidden layer with adam optimizer and lr = 0.01 and train it 181 | 182 | 183 | Layers = [2, 25, 25, 3] 184 | model = Network(Layers) 185 | optimizer = torch.optim.Adam(model.parameters(), lr=0.01) 186 | criterion = nn.CrossEntropyLoss() 187 | train_loader = DataLoader(dataset=data_set, batch_size=20) 188 | 189 | # Train model 190 | LOSS = train(data_set, model, criterion, train_loader, optimizer, epochs=300) 191 | 192 | # Plot decision regions (3 class model) 193 | plot_decision_regions_3class(model, data_set) 194 | -------------------------------------------------------------------------------- /Week 5/Deep Network Sigmoid, Tanh and Relu on MNIST Dataset.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Hidden Layer Deep Network: Sigmoid, Tanh and Relu Activations Functions MNIST Dataset 4 | 5 | """ 6 | 7 | # In[1] Imports 8 | 9 | import torch 10 | import torch.nn as nn 11 | import torchvision.transforms as transforms 12 | import torchvision.datasets as dsets 13 | import matplotlib.pylab as plt 14 | torch.manual_seed(0) 15 | 16 | # In[2] Neural Network Classes for Sigmoid, Tanh and Relu 17 | # Neural Networks with two hidden layers 18 | 19 | 20 | class NetworkSigmoid(nn.Module): 21 | 22 | # Constructor 23 | def __init__(self, D_in, H1, H2, D_out): 24 | super().__init__() 25 | self.linear1 = nn.Linear(D_in, H1) 26 | self.linear2 = nn.Linear(H1, H2) 27 | self.linear3 = nn.Linear(H2, D_out) 28 | 29 | # Prediction 30 | def forward(self, x): 31 | x = torch.sigmoid(self.linear1(x)) 32 | x = torch.sigmoid(self.linear2(x)) 33 | x = self.linear3(x) 34 | return x 35 | 36 | 37 | class NetworkTanh(nn.Module): 38 | # Constructor 39 | def __init__(self, D_in, H1, H2, D_out): 40 | super().__init__() 41 | self.linear1 = nn.Linear(D_in, H1) 42 | self.linear2 = nn.Linear(H1, H2) 43 | self.linear3 = nn.Linear(H2, D_out) 44 | 45 | # Prediction 46 | def forward(self, x): 47 | x = torch.tanh(self.linear1(x)) 48 | x = torch.tanh(self.linear2(x)) 49 | x = self.linear3(x) 50 | return x 51 | 52 | 53 | class NetworkRelu(nn.Module): 54 | # Constructor 55 | def __init__(self, D_in, H1, H2, D_out): 56 | super().__init__() 57 | self.linear1 = nn.Linear(D_in, H1) 58 | self.linear2 = nn.Linear(H1, H2) 59 | self.linear3 = nn.Linear(H2, D_out) 60 | 61 | # Prediction 62 | def forward(self, x): 63 | x = torch.relu(self.linear1(x)) 64 | x = torch.relu(self.linear2(x)) 65 | x = self.linear3(x) 66 | return x 67 | 68 | 69 | # In[3] Training function 70 | 71 | 72 | def train(model, criterion, train_loader, validation_loader, optimizer, epochs=100): 73 | i = 0 74 | useful_stuff = {'training_loss': [], 'validation_accuracy': []} 75 | 76 | for epoch in range(epochs): 77 | for i, (x, y) in enumerate(train_loader): 78 | optimizer.zero_grad() 79 | z = model(x.view(-1, 28*28)) 80 | loss = criterion(z, y) 81 | loss.backward() 82 | optimizer.step() 83 | useful_stuff['training_loss'].append(loss.data.item()) 84 | 85 | correct = 0 86 | for x, y in validation_loader: 87 | z = model(x.view(-1, 28 * 28)) 88 | _, label = torch.max(z, 1) 89 | correct += (label == y).sum().item() 90 | 91 | accuracy = 100 * (correct / len(validation_dataset)) 92 | useful_stuff['validation_accuracy'].append(accuracy) 93 | 94 | return useful_stuff 95 | 96 | 97 | # In[4] Using MNIST dataset, creating Training and Validation sets 98 | 99 | # Creating the training_dataset 100 | train_dataset = dsets.MNIST(root='./data', train=True, download=True, transform=transforms.ToTensor()) 101 | 102 | # Creating the validation_dataset 103 | validation_dataset = dsets.MNIST(root='./data', train=False, download=True, transform=transforms.ToTensor()) 104 | 105 | # In[6] Creating loss function 106 | 107 | # criteiron loss function from torch.nn 108 | criterion = nn.CrossEntropyLoss() 109 | 110 | # In[7] Creating training and validation DataLoader objects 111 | 112 | train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=2000, shuffle=True) 113 | validation_loader = torch.utils.data.DataLoader(dataset=validation_dataset, batch_size=5000, shuffle=False) 114 | 115 | # In[8] Setting up optimizer, model, learning_rate 116 | 117 | input_dim = 28 * 28 118 | hidden_dim1 = 50 119 | hidden_dim2 = 50 120 | output_dim = 10 121 | 122 | # 28*28 pixels as input, 50 neurons in 1st layer and 50 in second, with 10 classes output 123 | model = NetworkSigmoid(28*28, 50, 50, 10) 124 | 125 | # 28*28 pixels as input, 50 neurons in 1st layer and 50 in second, with 10 classes output 126 | model1 = NetworkTanh(28*28, 50, 50, 10) 127 | 128 | # 28*28 pixels as input, 50 neurons in 1st layer and 50 in second, with 10 classes output 129 | model2 = NetworkRelu(28*28, 50, 50, 10) 130 | 131 | # Optimizer 132 | optimizer = torch.optim.SGD(model.parameters(), lr=0.01) 133 | optimizer1 = torch.optim.SGD(model1.parameters(), lr=0.01) 134 | optimizer2 = torch.optim.SGD(model2.parameters(), lr=0.01) 135 | 136 | # In[9] Training 137 | 138 | # train sigmoid model 139 | training_sigmoid = train(model, criterion, train_loader, validation_loader, optimizer, epochs=10) 140 | 141 | # train tanh model 142 | training_tanh = train(model1, criterion, train_loader, validation_loader, optimizer1, epochs=10) 143 | 144 | # train relu model 145 | training_relu = train(model2, criterion, train_loader, validation_loader, optimizer2, epochs=10) 146 | 147 | 148 | # In[10] Plotting results 149 | 150 | # Plotting training loss for different models 151 | plt.plot(training_sigmoid['training_loss'], label='sigmoid') 152 | plt.plot(training_tanh['training_loss'], label='tanh') 153 | plt.plot(training_relu['training_loss'], label='relu') 154 | plt.ylabel('loss') 155 | plt.title('training loss iterations') 156 | plt.legend() 157 | 158 | # Plotting validation accuracy for different models 159 | plt.plot(training_sigmoid['validation_accuracy'], label='sigmoid') 160 | plt.plot(training_tanh['validation_accuracy'], label='tanh') 161 | plt.plot(training_relu['validation_accuracy'], label='relu') 162 | plt.ylabel('validation accuracy') 163 | plt.xlabel('Iteration') 164 | plt.legend() 165 | -------------------------------------------------------------------------------- /Week 5/Dropout in Classification.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Comparing Neural Network Dropout vs Non in Classification 4 | 5 | """ 6 | 7 | 8 | # In[1] Imports 9 | 10 | import matplotlib.pyplot as plt 11 | import numpy as np 12 | import torch 13 | import torch.nn as nn 14 | from matplotlib.colors import ListedColormap 15 | from torch.utils.data import Dataset 16 | 17 | # In[2] Plotting function 18 | 19 | 20 | def plot_decision_regions_3class(data_set, model=None): 21 | cmap_light = ListedColormap([ '#0000FF','#FF0000']) 22 | cmap_bold = ListedColormap(['#FF0000', '#00FF00', '#00AAFF']) 23 | X = data_set.x.numpy() 24 | y = data_set.y.numpy() 25 | h = .02 26 | x_min, x_max = X[:, 0].min() - 0.1, X[:, 0].max() + 0.1 27 | y_min, y_max = X[:, 1].min() - 0.1, X[:, 1].max() + 0.1 28 | xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h)) 29 | newdata = np.c_[xx.ravel(), yy.ravel()] 30 | 31 | Z = data_set.multi_dim_poly(newdata).flatten() 32 | f = np.zeros(Z.shape) 33 | f[Z > 0] = 1 34 | f = f.reshape(xx.shape) 35 | if model != None: 36 | model.eval() 37 | XX = torch.Tensor(newdata) 38 | _, yhat = torch.max(model(XX), 1) 39 | yhat = yhat.numpy().reshape(xx.shape) 40 | plt.pcolormesh(xx, yy, yhat, cmap=cmap_light) 41 | plt.contour(xx, yy, f, cmap=plt.cm.Paired) 42 | else: 43 | plt.contour(xx, yy, f, cmap=plt.cm.Paired) 44 | plt.pcolormesh(xx, yy, f, cmap=cmap_light) 45 | 46 | plt.title("decision region vs True decision boundary") 47 | 48 | # In[3] Creating Data 49 | 50 | class Data(Dataset): 51 | 52 | # Constructor 53 | def __init__(self, N_SAMPLES=1000, noise_std=0.15, train=True): 54 | a = np.matrix([-1, 1, 2, 1, 1, -3, 1]).T 55 | self.x = np.matrix(np.random.rand(N_SAMPLES, 2)) 56 | self.f = np.array(a[0] + (self.x) * a[1:3] + np.multiply(self.x[:, 0], self.x[:, 1]) * a[4] + np.multiply(self.x, self.x) * a[5:7]).flatten() 57 | self.a = a 58 | 59 | self.y = np.zeros(N_SAMPLES) 60 | self.y[self.f > 0] = 1 61 | self.y = torch.from_numpy(self.y).type(torch.LongTensor) 62 | self.x = torch.from_numpy(self.x).type(torch.FloatTensor) 63 | self.x = self.x + noise_std * torch.randn(self.x.size()) 64 | self.f = torch.from_numpy(self.f) 65 | self.a = a 66 | if train == True: 67 | torch.manual_seed(1) 68 | self.x = self.x + noise_std * torch.randn(self.x.size()) 69 | torch.manual_seed(0) 70 | 71 | # Getter 72 | def __getitem__(self, index): 73 | return self.x[index], self.y[index] 74 | 75 | # Get Length 76 | def __len__(self): 77 | return self.len 78 | 79 | # Plot the diagram 80 | def plot(self): 81 | X = data_set.x.numpy() 82 | y = data_set.y.numpy() 83 | h = .02 84 | x_min, x_max = X[:, 0].min(), X[:, 0].max() 85 | y_min, y_max = X[:, 1].min(), X[:, 1].max() 86 | xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h)) 87 | Z = data_set.multi_dim_poly(np.c_[xx.ravel(), yy.ravel()]).flatten() 88 | f = np.zeros(Z.shape) 89 | f[Z > 0] = 1 90 | f = f.reshape(xx.shape) 91 | 92 | plt.title('True decision boundary and sample points with noise ') 93 | plt.plot(self.x[self.y == 0, 0].numpy(), self.x[self.y == 0,1].numpy(), 'bo', label='y=0') 94 | plt.plot(self.x[self.y == 1, 0].numpy(), self.x[self.y == 1,1].numpy(), 'ro', label='y=1') 95 | plt.contour(xx, yy, f,cmap=plt.cm.Paired) 96 | plt.xlim(0,1) 97 | plt.ylim(0,1) 98 | plt.legend() 99 | 100 | # Make a multidimension ploynomial function 101 | def multi_dim_poly(self, x): 102 | x = np.matrix(x) 103 | out = np.array(self.a[0] + (x) * self.a[1:3] + np.multiply(x[:, 0], x[:, 1]) * self.a[4] + np.multiply(x, x) * self.a[5:7]) 104 | out = np.array(out) 105 | return out 106 | 107 | # In[4] Creating Network 108 | 109 | 110 | class Network(nn.Module): 111 | 112 | # Constructor 113 | def __init__(self, D_in, H1, H2, D_out, p=0): # dropout disabled by default p=0 114 | super().__init__() 115 | self.drop = nn.Dropout(p=p) # Dropout with p 116 | self.linear1 = nn.Linear(D_in, H1) 117 | self.linear2 = nn.Linear(H1, H2) 118 | self.linear3 = nn.Linear(H2, D_out) 119 | 120 | # Prediction 121 | def forward(self, x): 122 | x = self.linear1(x) 123 | x = self.drop(x) 124 | x = torch.relu(x) 125 | x = self.linear2(x) 126 | x = self.drop(x) 127 | x = torch.relu(x) 128 | x = self.linear3(x) 129 | return x 130 | 131 | 132 | # In[5] Training function 133 | 134 | 135 | def train_model(epochs): 136 | 137 | for epoch in range(epochs): 138 | #all the samples are used for training 139 | yhat = model(data_set.x) 140 | yhat_dropout = model_dropout(data_set.x) 141 | loss = criterion(yhat, data_set.y) 142 | loss_dropout = criterion(yhat_dropout, data_set.y) 143 | 144 | #store the loss for both the training and validation data for both models 145 | LOSS['training data no dropout'].append(loss.item()) 146 | LOSS['validation data no dropout'].append(criterion(model(validation_set.x), validation_set.y).item()) 147 | LOSS['training data dropout'].append(loss_dropout.item()) 148 | model_dropout.eval() 149 | LOSS['validation data dropout'].append(criterion(model_dropout(validation_set.x), validation_set.y).item()) 150 | model_dropout.train() 151 | 152 | optimizer.zero_grad() 153 | optimizer_dropout.zero_grad() 154 | loss.backward() 155 | loss_dropout.backward() 156 | optimizer.step() 157 | optimizer_dropout.step() 158 | 159 | 160 | # In[6] accuracy function 161 | 162 | 163 | def accuracy(model, data_set): 164 | _, yhat = torch.max(model(data_set.x), 1) 165 | return (yhat == data_set.y).numpy().mean() 166 | 167 | 168 | # In[7] Create Dataset object and plot 169 | 170 | data_set = Data(noise_std=0.2) 171 | data_set.plot() 172 | validation_set = Data(train=False) 173 | 174 | # In[8] Create models with dropout and without, Create optimizer and criterion 175 | 176 | model = Network(2, 300, 300, 2) 177 | model_dropout = Network(2, 300, 300, 2, p=0.2) #dropout of probability 0.5 178 | 179 | model_dropout.train() # Set model to train model. It is trian by default but good practice to set it anyway 180 | 181 | optimizer = torch.optim.Adam(model.parameters(), lr=0.01) 182 | optimizer_dropout = torch.optim.Adam(model_dropout.parameters(), lr=0.01) 183 | criterion = torch.nn.CrossEntropyLoss() 184 | 185 | 186 | # In[9] Initialize the Loss dictionary to store Loss 187 | 188 | LOSS = {} 189 | LOSS['training data no dropout'] = [] 190 | LOSS['validation data no dropout'] = [] 191 | LOSS['training data dropout'] = [] 192 | LOSS['validation data dropout'] = [] 193 | 194 | # In[10] train models for 500 epochs and evaluate 195 | 196 | train_model(500) 197 | 198 | # Set model_dropout to evaluation mode. This is done to disable dropout 199 | 200 | model_dropout.eval() 201 | 202 | # Print out the accuracy of the model without dropout 203 | print("The accuracy of the model without dropout: ", accuracy(model, validation_set)) 204 | 205 | # Print out the accuracy of the model with dropout 206 | print("The accuracy of the model with dropout: ", accuracy(model_dropout, validation_set)) 207 | 208 | 209 | # In[11] Plot results to show decision model for model and model_dropout 210 | 211 | # Plot the decision boundary and the prediction 212 | plot_decision_regions_3class(data_set) # What the prediction should be 213 | 214 | # The model without dropout 215 | plot_decision_regions_3class(data_set, model) 216 | 217 | # The model with dropout 218 | plot_decision_regions_3class(data_set, model_dropout) 219 | 220 | 221 | # LOSS Plot 222 | plt.figure(figsize=(6.1, 10)) 223 | def plot_LOSS(): 224 | for key, value in LOSS.items(): 225 | plt.plot(np.log(np.array(value)), label=key) 226 | plt.legend() 227 | plt.xlabel("iterations") 228 | plt.ylabel("Log of cost or total loss") 229 | 230 | plot_LOSS() -------------------------------------------------------------------------------- /Week 5/Dropout in Regression.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Comparing Neural Network Dropout vs Non in Regression 4 | 5 | """ 6 | 7 | 8 | # In[1] Imports 9 | 10 | import matplotlib.pyplot as plt 11 | import numpy as np 12 | import torch 13 | import torch.nn as nn 14 | from torch.utils.data import Dataset 15 | 16 | 17 | # In[2] Creating Data 18 | 19 | class Data(Dataset): 20 | 21 | # Constructor 22 | def __init__(self, N_SAMPLES=40, noise_std=1, train=True): 23 | self.x = torch.linspace(-10, 10, N_SAMPLES).view(-1, 1) 24 | self.f = self.x ** 3 25 | if train != True: 26 | torch.manual_seed(1) 27 | self.y = self.f + noise_std * torch.randn(self.f.size()) 28 | self.y = self.y.view(-1, 1) 29 | torch.manual_seed(0) 30 | else: 31 | self.y = self.f + noise_std * torch.randn(self.f.size()) 32 | self.y = self.y.view(-1, 1) 33 | 34 | # Getter 35 | def __getitem__(self, index): 36 | return self.x[index], self.y[index] 37 | 38 | # Get Length 39 | def __len__(self): 40 | return self.len 41 | 42 | # Plot the data 43 | def plot(self): 44 | plt.figure(figsize = (9, 14)) 45 | plt.scatter(self.x.numpy(), self.y.numpy(), label="Samples") 46 | plt.plot(self.x.numpy(), self.f.numpy() ,label="True Function", color='orange') 47 | plt.xlabel("x") 48 | plt.ylabel("y") 49 | plt.xlim((-10, 10)) 50 | plt.ylim((-20, 20)) 51 | plt.legend(loc="best") 52 | plt.show() 53 | 54 | # In[3] Creating Network 55 | 56 | 57 | class Network(nn.Module): 58 | 59 | # Constructor 60 | def __init__(self, D_in, H1, H2, H3, H4, D_out, p=0): # dropout disabled by default p=0 61 | super().__init__() 62 | self.drop = nn.Dropout(p=p) # Dropout with p 63 | self.linear1 = nn.Linear(D_in, H1) 64 | self.linear2 = nn.Linear(H1, H2) 65 | self.linear3 = nn.Linear(H2, H3) 66 | self.linear4 = nn.Linear(H3, H4) 67 | self.linear5 = nn.Linear(H4, D_out) 68 | 69 | # Prediction 70 | def forward(self, x): 71 | x = self.linear1(x) 72 | x = self.drop(x) 73 | x = torch.relu(x) 74 | x = self.linear2(x) 75 | x = self.drop(x) 76 | x = torch.relu(x) 77 | x = self.linear3(x) 78 | x = self.drop(x) 79 | x = torch.relu(x) 80 | x = self.linear4(x) 81 | x = torch.relu(x) 82 | x = self.linear5(x) 83 | return x 84 | 85 | 86 | # In[4] Training function 87 | 88 | 89 | def train_model(epochs): 90 | 91 | for epoch in range(epochs): 92 | #all the samples are used for training 93 | yhat = model(data_set.x) 94 | yhat_dropout = model_dropout(data_set.x) 95 | loss = criterion(yhat, data_set.y) 96 | loss_dropout = criterion(yhat_dropout, data_set.y) 97 | 98 | #store the loss for both the training and validation data for both models 99 | LOSS['training data no dropout'].append(loss.item()) 100 | LOSS['validation data no dropout'].append(criterion(model(validation_set.x), validation_set.y).item()) 101 | LOSS['training data dropout'].append(loss_dropout.item()) 102 | model_dropout.eval() 103 | LOSS['validation data dropout'].append(criterion(model_dropout(validation_set.x), validation_set.y).item()) 104 | model_dropout.train() 105 | 106 | optimizer.zero_grad() 107 | optimizer_dropout.zero_grad() 108 | loss.backward() 109 | loss_dropout.backward() 110 | optimizer.step() 111 | optimizer_dropout.step() 112 | 113 | 114 | # In[5] Create Dataset object and plot 115 | 116 | data_set = Data(N_SAMPLES=1000, noise_std=10, train=True) 117 | data_set.plot() 118 | validation_set = Data(train=False) 119 | 120 | # In[6] Create models with dropout and without, Create optimizer and criterion 121 | 122 | model = Network(1, 250, 250, 250, 250, 1) 123 | model_dropout = Network(1, 250, 250, 250, 250, 1, p=0.2) #dropout of probability 0.5 124 | 125 | model_dropout.train() # Set model to train model. It is trian by default but good practice to set it anyway 126 | 127 | optimizer = torch.optim.Adam(model.parameters(), lr=0.01) 128 | optimizer_dropout = torch.optim.Adam(model_dropout.parameters(), lr=0.01) 129 | criterion = torch.nn.MSELoss() 130 | 131 | 132 | # In[7] Initialize the Loss dictionary to store Loss 133 | 134 | LOSS = {} 135 | LOSS['training data no dropout'] = [] 136 | LOSS['validation data no dropout'] = [] 137 | LOSS['training data dropout'] = [] 138 | LOSS['validation data dropout'] = [] 139 | 140 | # In[8] train models for 500 epochs and evaluate 141 | 142 | train_model(10000) 143 | 144 | # Set model_dropout to evaluation mode. This is done to disable dropout 145 | 146 | model_dropout.eval() 147 | yhat = model(data_set.x) 148 | yhat_drop = model_dropout(data_set.x) 149 | 150 | 151 | # In[9] Plot results to show decision model for model and model_dropout 152 | 153 | # Plot predictions for model and model_dropout 154 | plt.figure(figsize=(6.1, 10)) 155 | plt.scatter(data_set.x.numpy(), data_set.y.numpy(), label="Samples") 156 | plt.plot(data_set.x.numpy(), data_set.f.numpy(), label="True function", color='orange') 157 | plt.plot(data_set.x.numpy(), yhat.detach().numpy(), label='no dropout', c='r') 158 | plt.plot(data_set.x.numpy(), yhat_drop.detach().numpy(), label="dropout", c ='g') 159 | # Plot labels and limits 160 | plt.xlabel("x") 161 | plt.ylabel("y") 162 | plt.xlim((-10, 10)) 163 | plt.ylim((-20, 20)) 164 | plt.legend(loc = "best") 165 | plt.show() 166 | 167 | # LOSS Plot 168 | # Plot the loss 169 | plt.figure(figsize=(9, 14)) 170 | for key, value in LOSS.items(): 171 | plt.plot(np.log(np.array(value)), label=key) 172 | plt.legend() 173 | plt.xlabel("iterations") 174 | plt.ylabel("Log of cost or total loss") 175 | 176 | -------------------------------------------------------------------------------- /Week 5/Neural Networks with Momentum.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Neural Networks with Momentum 4 | 5 | """ 6 | 7 | 8 | # In[1] Imports 9 | 10 | import matplotlib.pyplot as plt 11 | import numpy as np 12 | import torch 13 | import torch.nn as nn 14 | import torch.nn.functional as F 15 | from matplotlib.colors import ListedColormap 16 | from torch.utils.data import Dataset, DataLoader 17 | 18 | torch.manual_seed(0) 19 | np.random.seed(0) 20 | 21 | 22 | # In[2] Plotting function 23 | 24 | # Define a function for plot the decision region 25 | 26 | 27 | def plot_decision_regions_3class(model, data_set): 28 | cmap_light = ListedColormap(['#FFAAAA', '#AAFFAA','#00AAFF']) 29 | cmap_bold = ListedColormap(['#FF0000', '#00FF00','#00AAFF']) 30 | X=data_set.x.numpy() 31 | y=data_set.y.numpy() 32 | h = .02 33 | x_min, x_max = X[:, 0].min() - 0.1 , X[:, 0].max() + 0.1 34 | y_min, y_max = X[:, 1].min() - 0.1 , X[:, 1].max() + 0.1 35 | xx, yy = np.meshgrid(np.arange(x_min, x_max, h),np.arange(y_min, y_max, h)) 36 | XX=torch.torch.Tensor(np.c_[xx.ravel(), yy.ravel()]) 37 | _,yhat=torch.max(model(XX),1) 38 | yhat=yhat.numpy().reshape(xx.shape) 39 | plt.pcolormesh(xx, yy, yhat, cmap=cmap_light) 40 | plt.plot(X[y[:]==0,0], X[y[:]==0,1], 'ro', label='y=0') 41 | plt.plot(X[y[:]==1,0], X[y[:]==1,1], 'go', label='y=1') 42 | plt.plot(X[y[:]==2,0], X[y[:]==2,1], 'o', label='y=2') 43 | plt.title("decision region") 44 | plt.legend() 45 | 46 | 47 | # In[3] Create Dataset Class 48 | 49 | 50 | class Data(Dataset): 51 | 52 | # Constructor 53 | def __init__(self, K=3, N=500): 54 | D = 2 55 | X = np.zeros((N * K, D)) # data matrix (each row = single example) 56 | y = np.zeros(N * K, dtype='uint8') # class labels 57 | for j in range(K): 58 | ix = range(N * j, N * (j + 1)) 59 | r = np.linspace(0.0, 1, N) # radius 60 | t = np.linspace(j * 4, (j + 1) * 4, N) + np.random.randn(N) * 0.2 # theta 61 | X[ix] = np.c_[r * np.sin(t), r * np.cos(t)] 62 | y[ix] = j 63 | 64 | self.y = torch.from_numpy(y).type(torch.LongTensor) 65 | self.x = torch.from_numpy(X).type(torch.FloatTensor) 66 | self.len = y.shape[0] 67 | 68 | # Getter 69 | def __getitem__(self, index): 70 | return self.x[index], self.y[index] 71 | 72 | # Get Length 73 | def __len__(self): 74 | return self.len 75 | 76 | # Plot the diagram 77 | def plot_data(self): 78 | plt.plot(self.x[self.y[:] == 0, 0].numpy(), self.x[self.y[:] == 0, 1].numpy(), 'o', label="y=0") 79 | plt.plot(self.x[self.y[:] == 1, 0].numpy(), self.x[self.y[:] == 1, 1].numpy(), 'ro', label="y=1") 80 | plt.plot(self.x[self.y[:] == 2, 0].numpy(),self.x[self.y[:] == 2, 1].numpy(), 'go',label="y=2") 81 | plt.legend() 82 | 83 | 84 | # In[4] Create Neural Network with ModuleList 85 | 86 | 87 | class Network(nn.Module): 88 | 89 | # Constructor 90 | def __init__(self, Layers): 91 | super().__init__() 92 | self.hidden = nn.ModuleList() 93 | for input_size, output_size in zip(Layers, Layers[1:]): 94 | self.hidden.append(nn.Linear(input_size, output_size)) 95 | 96 | 97 | # Prediction 98 | def forward(self, activation): 99 | L = len(self.hidden) 100 | for (l, linear_transform) in zip(range(L), self.hidden): 101 | if l < L-1: 102 | activation = torch.relu(linear_transform(activation)) 103 | else: 104 | activation = linear_transform(activation) 105 | return activation 106 | 107 | 108 | # In[5] Create train loop 109 | 110 | 111 | def train(data_set, model, criterion, train_loader, optimizer, epochs=100): 112 | LOSS = [] 113 | ACC = [] 114 | for epoch in range(epochs): 115 | for x, y in train_loader: 116 | optimizer.zero_grad() 117 | yhat = model(x) 118 | loss = criterion(yhat, y) 119 | loss.backward() 120 | optimizer.step() 121 | LOSS.append(loss.item()) 122 | ACC.append(accuracy(model, data_set)) 123 | 124 | results = {"Loss": LOSS, "Accuracy": ACC} 125 | fig, ax1 = plt.subplots() 126 | color = 'tab:red' 127 | ax1.plot(LOSS,color=color) 128 | ax1.set_xlabel('epoch', color=color) 129 | ax1.set_ylabel('total loss', color=color) 130 | ax1.tick_params(axis = 'y', color=color) 131 | 132 | ax2 = ax1.twinx() 133 | color = 'tab:blue' 134 | ax2.set_ylabel('accuracy', color=color) # we already handled the x-label with ax1 135 | ax2.plot(ACC, color=color) 136 | ax2.tick_params(axis='y', color=color) 137 | fig.tight_layout() # otherwise the right y-label is slightly clipped 138 | 139 | plt.show() 140 | return results 141 | 142 | 143 | # In[5] Accuracy function 144 | 145 | 146 | # Define a function for calculating accuracy 147 | def accuracy(model, data_set): 148 | _, yhat = torch.max(model(data_set.x), 1) 149 | return (yhat == data_set.y).numpy().mean() 150 | 151 | 152 | # In[6] Create Dataset object and dictionary to store loss and accuracy 153 | 154 | data_set = Data() 155 | data_set.plot_data() 156 | data_set.y = data_set.y.view(-1) 157 | 158 | # Initialize a dictionary to contain the cost and accuracy 159 | Results = {"momentum 0": {"Loss": 0, "Accuracy:": 0}, "momentum 0.1": {"Loss": 0, "Accuracy:": 0}} 160 | 161 | # In[6] Initialize the model, optimizer, criterion, momentum 162 | 163 | # model with 1 hidden layer and 50 neurons, momentum = 0 (default) 164 | Layers = [2, 50, 3] 165 | model = Network(Layers) 166 | optimizer = torch.optim.SGD(model.parameters(), lr=0.1) 167 | train_loader = DataLoader(dataset = data_set, batch_size=20) 168 | criterion = nn.CrossEntropyLoss() 169 | Results["momentum 0"] = train(data_set, model, criterion, train_loader, optimizer, epochs=100) 170 | plot_decision_regions_3class(model, data_set) 171 | 172 | 173 | # model with 1 hidden layer and 50 neurons, momentum = 0.1 174 | Layers = [2, 50, 3] 175 | model = Network(Layers) 176 | optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.1) 177 | train_loader = DataLoader(dataset = data_set, batch_size=20) 178 | criterion = nn.CrossEntropyLoss() 179 | Results["momentum 0.1"] = train(data_set, model, criterion, train_loader, optimizer, epochs=100) 180 | plot_decision_regions_3class(model, data_set) 181 | 182 | 183 | # model with 1 hidden layer and 50 neurons, momentum = 0.2 184 | Layers = [2, 50, 3] 185 | model = Network(Layers) 186 | optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.2) 187 | train_loader = DataLoader(dataset = data_set, batch_size=20) 188 | criterion = nn.CrossEntropyLoss() 189 | Results["momentum 0.2"] = train(data_set, model, criterion, train_loader, optimizer, epochs=100) 190 | plot_decision_regions_3class(model, data_set) 191 | 192 | 193 | # model with 1 hidden layer and 50 neurons, momentum = 0.3 194 | Layers = [2, 50, 3] 195 | model = Network(Layers) 196 | optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.3) 197 | train_loader = DataLoader(dataset = data_set, batch_size=20) 198 | criterion = nn.CrossEntropyLoss() 199 | Results["momentum 0.3"] = train(data_set, model, criterion, train_loader, optimizer, epochs=100) 200 | plot_decision_regions_3class(model, data_set) 201 | 202 | 203 | # model with 1 hidden layer and 50 neurons, momentum = 0.4 204 | Layers = [2, 50, 3] 205 | model = Network(Layers) 206 | optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.4) 207 | train_loader = DataLoader(dataset = data_set, batch_size=20) 208 | criterion = nn.CrossEntropyLoss() 209 | Results["momentum 0.4"] = train(data_set, model, criterion, train_loader, optimizer, epochs=100) 210 | #plot_decision_regions_3class(model, data_set) 211 | 212 | 213 | # model with 1 hidden layer and 50 neurons, momentum = 0.5 214 | Layers = [2, 50, 3] 215 | model = Network(Layers) 216 | optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.5) 217 | train_loader = DataLoader(dataset = data_set, batch_size=20) 218 | criterion = nn.CrossEntropyLoss() 219 | Results["momentum 0.5"] = train(data_set, model, criterion, train_loader, optimizer, epochs=100) 220 | #plot_decision_regions_3class(model, data_set) 221 | 222 | 223 | # In[5] Plotting results 224 | 225 | # Plot the Loss result for each term 226 | for key, value in Results.items(): 227 | plt.plot(value['Loss'], label=key) 228 | plt.legend() 229 | plt.xlabel('epoch') 230 | plt.ylabel('Total Loss or Cost') 231 | 232 | 233 | # Plot the Accuracy result for each term 234 | for key, value in Results.items(): 235 | plt.plot(value['Accuracy'], label=key) 236 | plt.legend() 237 | plt.xlabel('epoch') 238 | plt.ylabel('Accuracy') 239 | -------------------------------------------------------------------------------- /Week 5/Weights Experiment_Uniform_He_Xavier_Identical_Default.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Neural Network Weights Experiment Uniform He Xavier Identical Default 4 | 5 | """ 6 | 7 | 8 | # In[1] Imports 9 | 10 | import torch 11 | import torch.nn as nn 12 | import matplotlib.pylab as plt 13 | torch.manual_seed(0) 14 | import torchvision.datasets as dsets 15 | import torchvision.transforms as transforms 16 | from torch.utils.data import DataLoader 17 | 18 | 19 | # In[2] Creating Data 20 | 21 | # Create the train dataset 22 | train_dataset = dsets.MNIST(root='./data', train=True, download=True, transform=transforms.ToTensor()) 23 | 24 | # Create the validation dataset 25 | validation_dataset = dsets.MNIST(root='./data', train=False, download=True, transform=transforms.ToTensor()) 26 | 27 | # Create Dataloader for both train dataset and validation dataset 28 | 29 | train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=2000, shuffle=True) 30 | validation_loader = torch.utils.data.DataLoader(dataset=validation_dataset, batch_size=5000, shuffle=False) 31 | 32 | 33 | # In[3] Creating Networks using ModuleList 34 | 35 | # Network with relu activation and He_initialization with relu 36 | class Network_He(nn.Module): 37 | 38 | # Constructor 39 | def __init__(self, Layers): 40 | super().__init__() 41 | self.hidden = nn.ModuleList() 42 | 43 | for input_size, output_size in zip(Layers, Layers[1:]): 44 | linear = nn.Linear(input_size, output_size) 45 | torch.nn.init.kaiming_uniform_(linear.weight, nonlinearity='relu') 46 | self.hidden.append(linear) 47 | 48 | # Prediction 49 | def forward(self, x): 50 | L = len(self.hidden) 51 | for (l, linear_transform) in zip(range(L), self.hidden): 52 | if l < L - 1: 53 | x = torch.relu(linear_transform(x)) 54 | else: 55 | x = linear_transform(x) 56 | return x 57 | 58 | 59 | # Network with relu activation and uniform initialization 60 | class Network_Uniform(nn.Module): 61 | 62 | # Constructor 63 | def __init__(self, Layers): 64 | super().__init__() 65 | self.hidden = nn.ModuleList() 66 | 67 | for input_size, output_size in zip(Layers, Layers[1:]): 68 | linear = nn.Linear(input_size,output_size) 69 | linear.weight.data.uniform_(0, 1) 70 | self.hidden.append(linear) 71 | 72 | # Prediction 73 | def forward(self, x): 74 | L = len(self.hidden) 75 | for (l, linear_transform) in zip(range(L), self.hidden): 76 | if l < L - 1: 77 | x = torch.relu(linear_transform(x)) 78 | else: 79 | x = linear_transform(x) 80 | 81 | return x 82 | 83 | 84 | # Network with relu and Default Initilization 85 | class Network(nn.Module): 86 | 87 | # Constructor 88 | def __init__(self, Layers): 89 | super().__init__() 90 | self.hidden = nn.ModuleList() 91 | 92 | for input_size, output_size in zip(Layers, Layers[1:]): 93 | linear = nn.Linear(input_size, output_size) 94 | self.hidden.append(linear) 95 | 96 | def forward(self, x): 97 | L=len(self.hidden) 98 | for (l, linear_transform) in zip(range(L), self.hidden): 99 | if l < L - 1: 100 | x = torch.relu(linear_transform(x)) 101 | else: 102 | x = linear_transform(x) 103 | 104 | return x 105 | 106 | # Network with relu and Same Initilization 107 | class Network_Same(nn.Module): 108 | 109 | # Constructor 110 | def __init__(self, Layers): 111 | super().__init__() 112 | self.hidden = nn.ModuleList() 113 | 114 | for input_size, output_size in zip(Layers, Layers[1:]): 115 | linear = nn.Linear(input_size, output_size) 116 | self.hidden.append(linear) 117 | 118 | def forward(self, x): 119 | L=len(self.hidden) 120 | for (l, linear_transform) in zip(range(L), self.hidden): 121 | if l < L - 1: 122 | x = torch.relu(linear_transform(x)) 123 | else: 124 | x = linear_transform(x) 125 | 126 | return x 127 | 128 | # Network with tanh activation and Xavier Initialization for Tanh 129 | class Network_Xavier(nn.Module): 130 | 131 | # Constructor 132 | def __init__(self, Layers): 133 | super().__init__() 134 | self.hidden = nn.ModuleList() 135 | 136 | for input_size, output_size in zip(Layers, Layers[1:]): 137 | linear = nn.Linear(input_size, output_size) 138 | torch.nn.init.xavier_uniform_(linear.weight) 139 | self.hidden.append(linear) 140 | 141 | # Prediction 142 | def forward(self, x): 143 | L = len(self.hidden) 144 | for (l, linear_transform) in zip(range(L), self.hidden): 145 | if l < L - 1: 146 | x = torch.tanh(linear_transform(x)) 147 | else: 148 | x = linear_transform(x) 149 | return x 150 | 151 | 152 | # In[5] Training function 153 | 154 | 155 | def train(model, criterion, train_loader, validation_loader, optimizer, epochs = 100): 156 | i = 0 157 | loss_accuracy = {'training_loss':[], 'validation_accuracy':[]} 158 | 159 | for epoch in range(epochs): 160 | for i,(x, y) in enumerate(train_loader): 161 | optimizer.zero_grad() 162 | z = model(x.view(-1, 28 * 28)) 163 | loss = criterion(z, y) 164 | loss.backward() 165 | optimizer.step() 166 | loss_accuracy['training_loss'].append(loss.data.item()) 167 | 168 | correct = 0 169 | for x, y in validation_loader: 170 | yhat = model(x.view(-1, 28 * 28)) 171 | _, label = torch.max(yhat, 1) 172 | correct += (label==y).sum().item() 173 | accuracy = 100 * (correct / len(validation_dataset)) 174 | loss_accuracy['validation_accuracy'].append(accuracy) 175 | 176 | return loss_accuracy 177 | 178 | 179 | # In[4] Creating the layer for ModuleList 180 | 181 | input_dim = 28*28 182 | output_dim = 10 183 | layers = [input_dim, 100, 10, 100, 10, 100, output_dim] 184 | epochs = 10 185 | 186 | 187 | # In[5] Creating loss function and Optimizers and Networks for comparison 188 | 189 | criterion = nn.CrossEntropyLoss() 190 | 191 | # First model with default initialization 192 | model = Network(layers) 193 | optimizer = torch.optim.SGD(model.parameters(), lr=0.01) 194 | training_results = train(model, criterion, train_loader, validation_loader, optimizer, epochs=epochs) 195 | 196 | # 2nd model with Uniform Initialization 197 | model_Uniform = Network_Uniform(layers) 198 | optimizer = torch.optim.SGD(model_Uniform.parameters(), lr=0.01) 199 | training_results_Uniform = train(model_Uniform, criterion, train_loader, validation_loader, optimizer, epochs=epochs) 200 | 201 | 202 | # 3rd model with Xavier_initialization 203 | model_Xavier = Network_Xavier(layers) 204 | optimizer = torch.optim.SGD(model_Xavier.parameters(), lr=0.01) 205 | training_results_Xavier = train(model_Xavier, criterion, train_loader, validation_loader, optimizer, epochs=epochs) 206 | 207 | 208 | # 4th model with He_initialization 209 | model_He = Network_He(layers) 210 | optimizer = torch.optim.SGD(model_He.parameters(), lr=0.01) 211 | training_results_He = train(model_He, criterion, train_loader, validation_loader, optimizer, epochs=epochs) 212 | 213 | # 5th model with the same initializations for all neurons 214 | model_Same = Network_Same(layers) 215 | model_Same.state_dict()['hidden.0.weight'][:]=1.0 216 | model_Same.state_dict()['hidden.1.weight'][:]=1.0 217 | model_Same.state_dict()['hidden.2.weight'][:]=1.0 218 | model_Same.state_dict()['hidden.3.weight'][:]=1.0 219 | model_Same.state_dict()['hidden.4.weight'][:]=1.0 220 | model_Same.state_dict()['hidden.5.weight'][:]=1.0 221 | model_Same.state_dict()['hidden.0.bias'][:]=0.0 222 | model_Same.state_dict()['hidden.1.bias'][:]=0.0 223 | model_Same.state_dict()['hidden.2.bias'][:]=0.0 224 | model_Same.state_dict()['hidden.3.bias'][:]=0.0 225 | model_Same.state_dict()['hidden.4.bias'][:]=0.0 226 | model_Same.state_dict()['hidden.5.bias'][:]=0.0 227 | model_Same.state_dict() 228 | 229 | optimizer = torch.optim.SGD(model_Same.parameters(), lr=0.01) 230 | training_results_Same = train(model_Same, criterion, train_loader, validation_loader, optimizer, epochs=epochs) 231 | 232 | 233 | 234 | # In[5] Plotting results 235 | 236 | # Plot the accuracy 237 | plt.plot(training_results_Xavier['validation_accuracy'], label='Xavier') 238 | plt.plot(training_results_He['validation_accuracy'], label='He') 239 | plt.plot(training_results['validation_accuracy'], label='Default') 240 | plt.plot(training_results_Uniform['validation_accuracy'], label='Uniform') 241 | plt.plot(training_results_Same['validation_accuracy'], label='Same') 242 | plt.ylabel('validation accuracy') 243 | plt.xlabel('epochs') 244 | plt.legend() 245 | 246 | -------------------------------------------------------------------------------- /Week 6/Convolution Multiple Inputs and Outputs.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Convolution 2D with Multiple Outputs 4 | 5 | """ 6 | 7 | # In[1] Imports 8 | 9 | import torch 10 | import torch.nn as nn 11 | import matplotlib.pyplot as plt 12 | import numpy as np 13 | 14 | # In[2] Conv 2D 15 | 16 | # Conv2D with in_channels = 1 and out_channels = 3 and Kernel = 3,3. Stride default(1,1) 17 | conv1 = nn.Conv2d(in_channels=1, out_channels=3, kernel_size=3) 18 | conv1 19 | 20 | # Creating tensors for kernel weights 21 | Gx = torch.tensor([[1.0,0,-1.0],[2.0,0,-2.0],[1.0,0.0,-1.0]]) 22 | Gy = torch.tensor([[1.0,2.0,1.0],[0.0,0.0,0.0],[-1.0,-2.0,-1.0]]) 23 | Gz = torch.ones(3,3) 24 | 25 | # Manual initialization of kernel weights and bias 26 | conv1.state_dict() # dictionary containing keys (weights, bias) 27 | conv1.state_dict()['weight'][0][0] = Gx 28 | conv1.state_dict()['weight'][1][0] = Gy 29 | conv1.state_dict()['weight'][2][0] = Gz 30 | conv1.state_dict()['weight'] 31 | conv1.state_dict()['bias'][:] = torch.tensor([0.0,0.0,0.0]) 32 | conv1.state_dict()['bias'] 33 | 34 | # Showing kernels weights 35 | for x in conv1.state_dict()['weight']: 36 | print(x) 37 | 38 | # Showing bias in kernels 39 | for x in conv1.state_dict()['bias']: 40 | print(x) 41 | 42 | # In[3] Create Data/Image 43 | 44 | image = torch.zeros(1, 1, 5, 5) # Vertical white block in the middle 45 | image[0,0,:,2] = 1 46 | image 47 | 48 | image1 = torch.zeros(1, 1, 5, 5) # Horizontal white block in the middle 49 | image1[0, 0, 2, :] = 1 50 | image1 51 | 52 | # plot image 53 | plt.imshow(image[0, 0, :, :].numpy(), cmap = 'gray') 54 | plt.colorbar() 55 | plt.show 56 | 57 | plt.imshow(image1[0, 0, :, :].numpy(), cmap = 'gray') 58 | plt.colorbar() 59 | plt.show 60 | 61 | 62 | # In[4] Perform convolution and Plot 63 | 64 | out = conv1(image) 65 | out1 = conv1(image1) 66 | 67 | out.shape 68 | out1.shape 69 | 70 | # Print channels as tensors/image and plot for image 71 | for channel, image in enumerate(out[0]): 72 | plt.imshow(image.detach().numpy(), cmap = 'gray') 73 | print(image) 74 | plt.title("channel {}".format(channel)) 75 | plt.colorbar() 76 | plt.show() 77 | 78 | # Print channels as tensors/image and plot for image1 79 | for channel, image1 in enumerate(out1[0]): 80 | plt.imshow(image1.detach().numpy(), cmap = 'gray') 81 | print(image1) 82 | plt.title("channel {}".format(channel)) 83 | plt.colorbar() 84 | plt.show() 85 | 86 | # It can be seen that different kernels can be used to detect various features in an image. 87 | 88 | 89 | # In[4] Create Data 90 | 91 | image2 = torch.zeros(1,2,5,5) 92 | image2[0,0,2,:] = -2 93 | image2[0,1,2,:] = 1 94 | image2 95 | image2.shape 96 | 97 | for channel,image in enumerate(image2[0]): 98 | plt.imshow(image.detach().numpy(), cmap = 'gray') 99 | print(image) 100 | plt.title("channel {}".format(channel)) 101 | plt.colorbar() 102 | plt.show() 103 | 104 | 105 | # In[5] Create Conv2D object 106 | 107 | # Conv2D object with 2 inputs and 1 ouput, kernel (3,3) 108 | 109 | conv3 = nn.Conv2d(in_channels=2, out_channels=1, kernel_size=1) 110 | 111 | # Manual kernel initialization 112 | 113 | Gx1 = torch.tensor([[0.0,0.0,0.0],[0,1.0,0],[0.0,0.0,0.0]]) 114 | conv3.state_dict()['weight'][0][0] = 1*Gx1 115 | conv3.state_dict()['weight'][0][1] = -2*Gx1 116 | conv3.state_dict()['bias'][:] = torch.tensor([0.0]) 117 | conv3.state_dict() 118 | conv3.state_dict()['weight'] 119 | 120 | # In[6] Perform convolution and Plot 121 | 122 | out3 = conv3(image2) 123 | 124 | # Print channels as tensors/image and plot for image1 125 | for channel, image2 in enumerate(out3[0]): 126 | plt.imshow(image2.detach().numpy(), cmap = 'gray') 127 | print(image1) 128 | plt.title("channel {}".format(channel)) 129 | plt.colorbar() 130 | plt.show() 131 | 132 | 133 | # In[7] Create Conv2D and Data with multiple inputs and outputs 134 | 135 | # Create conv2D with 2 in_channels and 3 out_channels, kernel_size (3,3) 136 | conv4 = nn.Conv2d(in_channels=2, out_channels=3, kernel_size=3) 137 | 138 | # Manual weight and bias initialization 139 | conv4.state_dict()['weight'][0][0] = torch.tensor([[0.0,0.0,0.0],[0,0.5,0],[0.0,0.0,0.0]]) 140 | conv4.state_dict()['weight'][0][1] = torch.tensor([[0.0,0.0,0.0],[0,0.5,0],[0.0,0.0,0.0]]) 141 | 142 | conv4.state_dict()['weight'][1][0] = torch.tensor([[0.0,0.0,0.0],[0,1,0],[0.0,0.0,0.0]]) 143 | conv4.state_dict()['weight'][1][1] = torch.tensor([[0.0,0.0,0.0],[0,-1,0],[0.0,0.0,0.0]]) 144 | 145 | conv4.state_dict()['weight'][2][0] = torch.tensor([[1.0,0,-1.0],[2.0,0,-2.0],[1.0,0.0,-1.0]]) 146 | conv4.state_dict()['weight'][2][1] = torch.tensor([[1.0,2.0,1.0],[0.0,0.0,0.0],[-1.0,-2.0,-1.0]]) 147 | 148 | conv4.state_dict()['bias'][:]=torch.tensor([0.0,0.0,0.0]) 149 | 150 | 151 | image4 = torch.zeros(1,2,5,5) 152 | 153 | image4[0][0] = torch.ones(5,5) 154 | 155 | image4[0][1][2][2] = 1 156 | 157 | for channel, image in enumerate(image4[0]): 158 | plt.imshow(image.detach().numpy(), cmap = 'gray' ) 159 | print(image) 160 | plt.title("channel {}".format(channel)) 161 | plt.colorbar() 162 | plt.show() 163 | 164 | 165 | # Apply conv to image 166 | z = conv4(image4) 167 | z 168 | 169 | -------------------------------------------------------------------------------- /Week 6/Convolution.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | What is Convolution? 4 | - Convolution is a linear operation similar to a linear equation, dot product, or matrix multiplication. 5 | - Convolution has several advantages for analyzing images. 6 | 7 | In convolution, the parameter w is called a kernel. 8 | You can perform convolution on images where you let the variable image denote the variable X and w denote the parameter. 9 | 10 | Max pooling simply takes the maximum value in each region. 11 | 12 | 13 | """ 14 | 15 | # In[1] Imports 16 | 17 | import torch 18 | import torch.nn as nn 19 | 20 | # In[2] Conv 2D 21 | 22 | # in and out channels = 1 and Kernel = 3,3. Stride default(1,1) 23 | conv = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=3) 24 | conv 25 | 26 | # 3x3 tensor for kernel weights 27 | Gx = torch.tensor([[1.0,0,-1.0],[2.0,0,-2.0],[1.0,0,-1.0]]) 28 | 29 | # Lets give the conv2d manually assigned values 30 | conv.state_dict()['weight'][0][0] = Gx 31 | conv.state_dict()['bias'][0]=0.0 32 | conv.state_dict() 33 | 34 | # Create image 35 | image=torch.zeros(1,1,5,5) 36 | image[0,0,:,2] = 1 37 | image 38 | 39 | # Create relu function 40 | relu = nn.ReLU() 41 | 42 | # Apply conv to image and then Relu 43 | Z = conv(image) 44 | Z 45 | Z = relu(Z) 46 | Z 47 | 48 | # Create a second image 49 | image1=torch.zeros(1,1,4,4) 50 | image1[0,0,0,:] = torch.tensor([1.0,2.0,3.0,-4.0]) 51 | image1[0,0,1,:] = torch.tensor([0.0,2.0,-3.0,0.0]) 52 | image1[0,0,2,:] = torch.tensor([0.0,2.0,3.0,1.0]) 53 | image1 54 | 55 | 56 | # Create a maxpooling object in 2D 57 | max1 = torch.nn.MaxPool2d(2, stride=1) 58 | 59 | # Apply maxpooling to image 60 | max1(image1) 61 | 62 | # Create another maxpooling object without setting stride 63 | # default stride is equal to kernel_size 64 | max2 = torch.nn.MaxPool2d(2) 65 | 66 | # Apply max2 to image 67 | max2(image1) 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | -------------------------------------------------------------------------------- /Week 6/Convolutional Neural Network with Small Images.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Convolutional Neural Network with Small Images 4 | 5 | """ 6 | 7 | # In[1] Imports 8 | 9 | import torch 10 | import torch.nn as nn 11 | import torchvision.transforms as transforms 12 | import torchvision.datasets as dsets 13 | from torch.utils.data import DataLoader 14 | import matplotlib.pyplot as plt 15 | import numpy as np 16 | 17 | 18 | # In[2] Plot function 19 | 20 | # Define the function for plotting the channels 21 | 22 | def show_data(data_sample): 23 | plt.imshow(data_sample[0].numpy().reshape(IMAGE_SIZE, IMAGE_SIZE), cmap='gray') 24 | plt.title('y = ' + str(data_sample[1].item())) 25 | 26 | 27 | # In[3] Create Data 28 | 29 | 30 | IMAGE_SIZE = 16 31 | 32 | # Create a transform to resize image and convert to tensor 33 | transformation = transforms.Compose([transforms.Resize((IMAGE_SIZE, IMAGE_SIZE)), transforms.ToTensor()]) 34 | 35 | # Create Dataset from MNIST and apply composed transformation 36 | train_dataset = dsets.MNIST(root='./data', train=True, download=True, transform=transformation) 37 | validation_dataset = dsets.MNIST(root='./data', train=False, download=True, transform=transformation) 38 | 39 | # Check Data 40 | train_dataset[0][1] # Label - int object 41 | train_dataset[0][0] 42 | train_dataset[0][0].size() 43 | train_dataset[0][0].type() 44 | 45 | train_dataset[1][1] 46 | train_dataset[1][0] 47 | train_dataset[1][0].size() 48 | train_dataset[1][0].type() 49 | 50 | # Lets plot some images. 51 | # We need to squeeze the (1, 16, 16) to (16, 16) and convert to numpy array 52 | # We can also use reshape(16, 16) 53 | plt.imshow(train_dataset[1][0].squeeze().numpy()) 54 | plt.imshow(train_dataset[5][0].reshape(16, 16).numpy()) 55 | # Plot in gray scale 56 | plt.imshow(train_dataset[5][0].reshape(16, 16).numpy(), cmap='gray') 57 | 58 | # We can use show_data function 59 | show_data(train_dataset[1][0]) 60 | 61 | 62 | # In[4] Create CNN Class 63 | 64 | 65 | class CNN(nn.Module): 66 | 67 | # Constructor 68 | def __init__(self, out_1=16, out_2=32): 69 | super().__init__() 70 | self.cnn1 = nn.Conv2d(in_channels=1, out_channels=out_1, kernel_size=5, padding=2) 71 | self.maxpool1 = nn.MaxPool2d(kernel_size=2) 72 | self.ELU = nn.ELU() 73 | self.cnn2 = nn.Conv2d(in_channels=out_1, out_channels=out_2, kernel_size=5, padding=2) 74 | self.maxpool2 = nn.MaxPool2d(kernel_size=2) 75 | self.fc1 = nn.Linear(out_2*4*4, 10) 76 | 77 | # Prediction 78 | def forward(self, x): 79 | x = self.cnn1(x) 80 | x = self.ELU(x) 81 | x = self.maxpool1(x) 82 | x = self.cnn2(x) 83 | x = self.ELU(x) 84 | x = self.maxpool2(x) 85 | x = x.view(x.size(0), -1) 86 | x = self.fc1(x) 87 | return x 88 | 89 | # Outputs in each step 90 | def activations(self, x): 91 | # This part is for visualization purposes 92 | z1 = self.cnn1(x) 93 | a1 = self.ELU(z1) 94 | out = self.maxpool1(a1) 95 | z2 = self.cnn2(out) 96 | a2 = self.ELU(z2) 97 | out1 = self.maxpool2(a2) 98 | out2 = out1.view(out1.size(0), -1) 99 | return z1, a1, out, z2, a2, out1, out2 100 | 101 | 102 | # In[5] Train loop and training 103 | 104 | 105 | def train_model(model, train_loader, validation_loader, optimizer, n_epochs=4): 106 | 107 | # Global variable 108 | N_test = len(validation_dataset) 109 | accuracy_list = [] 110 | loss_list = [] 111 | for epoch in range(n_epochs): 112 | for x, y in train_loader: 113 | model.train() 114 | optimizer.zero_grad() 115 | z = model(x) 116 | loss = criterion(z, y) 117 | loss.backward() 118 | optimizer.step() 119 | loss_list.append(loss.data) 120 | 121 | correct = 0 122 | # Perform a prediction on the validation data 123 | for x_test, y_test in validation_loader: 124 | model.eval() 125 | z = model(x_test) 126 | _, yhat = torch.max(z.data, 1) 127 | correct += (yhat == y_test).sum().item() 128 | accuracy = correct / N_test 129 | accuracy_list.append(accuracy) 130 | 131 | return accuracy_list, loss_list 132 | 133 | 134 | 135 | # In[6] Batch Normalization CNN class 136 | 137 | 138 | class CNN_BatchNorm(nn.Module): 139 | # Constructor 140 | def __init__(self, out_1=16, out_2=32): 141 | super().__init__() 142 | self.cnn1 = nn.Conv2d(in_channels=1, out_channels=out_1, kernel_size=5, padding=2) 143 | self.conv1_bn = nn.BatchNorm2d(out_1) # To normalize conv2D, we need BatchNorm2D 144 | self.maxpool1 = nn.MaxPool2d(kernel_size=2) 145 | self.ELU = nn.ELU() 146 | self.cnn2 = nn.Conv2d(in_channels=out_1, out_channels=out_2, kernel_size=5, padding=2) 147 | self.conv2_bn = nn.BatchNorm2d(out_2) 148 | self.maxpool2 = nn.MaxPool2d(kernel_size=2) 149 | self.fc1 = nn.Linear(out_2*4*4, 10) 150 | self.bn_fc1 = nn.BatchNorm1d(10) # To normalize linear layer, BatchNorm1D must be used 151 | 152 | # Prediction 153 | def forward(self, x): 154 | x = self.cnn1(x) 155 | x = self.conv1_bn(x) 156 | x = self.ELU(x) 157 | x = self.maxpool1(x) 158 | x = self.cnn2(x) 159 | x = self.conv2_bn(x) 160 | x = self.ELU(x) 161 | x = self.maxpool2(x) 162 | x = x.view(x.size(0), -1) 163 | x = self.fc1(x) 164 | x = self.bn_fc1(x) 165 | return x 166 | 167 | 168 | # In[7] Initialize, create loss function, optimizer and data loaders 169 | 170 | 171 | # Create model object from CNN class 172 | model = CNN(16, 32) 173 | 174 | # Optimizer 175 | optimizer = torch.optim.Adam(model.parameters(), lr=0.01) 176 | 177 | # Loss function criterion 178 | criterion = nn.CrossEntropyLoss() 179 | 180 | # train and val loader 181 | train_loader = DataLoader(dataset=train_dataset, batch_size=100) 182 | validation_loader = DataLoader(dataset=validation_dataset, batch_size=5000) 183 | 184 | accuracy_list_normal, loss_list_normal = train_model(model=model, 185 | n_epochs=10, train_loader=train_loader, validation_loader=validation_loader, 186 | optimizer=optimizer) 187 | 188 | 189 | # Create model_BatchNorm object from CNN_BatchNorm class 190 | model_BatchNorm = CNN_BatchNorm(16, 32) 191 | 192 | # Optimizer 193 | optimizer_BatchNorm = torch.optim.Adam(model_BatchNorm.parameters(), lr=0.01) 194 | 195 | accuracy_list_batch, loss_list_batch = train_model(model=model_BatchNorm, 196 | n_epochs=10, train_loader=train_loader, validation_loader=validation_loader, 197 | optimizer=optimizer_BatchNorm) 198 | 199 | 200 | # In[8] Analyze Results and Compare 201 | 202 | # Plot the loss and accuracy 203 | plt.plot(loss_list_normal, 'b', label='loss normal cnn') 204 | plt.plot(loss_list_batch, 'r', label='loss batch cnn') 205 | plt.xlabel('iteration') 206 | plt.title("loss") 207 | plt.legend() 208 | 209 | plt.plot(accuracy_list_normal, 'b', label='normal CNN') 210 | plt.plot(accuracy_list_batch, 'r', label='CNN with Batch Norm') 211 | plt.xlabel('Epoch') 212 | plt.title("Accuracy ") 213 | plt.legend() 214 | plt.show() 215 | -------------------------------------------------------------------------------- /Week 6/MNIST CNN BatchNorm vs Normal.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Convolutional Neural Network with Small Images 4 | 5 | """ 6 | 7 | # In[1] Imports 8 | 9 | import torch 10 | import torch.nn as nn 11 | import torchvision.transforms as transforms 12 | import torchvision.datasets as dsets 13 | from torch.utils.data import DataLoader 14 | import matplotlib.pyplot as plt 15 | import numpy as np 16 | 17 | 18 | # In[2] Plot function 19 | 20 | # Define the function for plotting the channels 21 | 22 | def show_data(data_sample): 23 | plt.imshow(data_sample[0].numpy().reshape(IMAGE_SIZE, IMAGE_SIZE), cmap='gray') 24 | plt.title('y = ' + str(data_sample[1].item())) 25 | 26 | 27 | # In[3] Create Data 28 | 29 | 30 | IMAGE_SIZE = 16 31 | 32 | # Create a transform to resize image and convert to tensor 33 | transformation = transforms.Compose([transforms.Resize((IMAGE_SIZE, IMAGE_SIZE)), transforms.ToTensor()]) 34 | 35 | # Create Dataset from MNIST and apply composed transformation 36 | train_dataset = dsets.MNIST(root='./data', train=True, download=True, transform=transformation) 37 | validation_dataset = dsets.MNIST(root='./data', train=False, download=True, transform=transformation) 38 | 39 | # Check Data 40 | train_dataset[0][1] # Label - int object 41 | train_dataset[0][0] 42 | train_dataset[0][0].size() 43 | train_dataset[0][0].type() 44 | 45 | train_dataset[1][1] 46 | train_dataset[1][0] 47 | train_dataset[1][0].size() 48 | train_dataset[1][0].type() 49 | 50 | # Lets plot some images. 51 | # We need to squeeze the (1, 16, 16) to (16, 16) and convert to numpy array 52 | # We can also use reshape(16, 16) 53 | plt.imshow(train_dataset[1][0].squeeze().numpy()) 54 | plt.imshow(train_dataset[5][0].reshape(16, 16).numpy()) 55 | # Plot in gray scale 56 | plt.imshow(train_dataset[5][0].reshape(16, 16).numpy(), cmap='gray') 57 | 58 | # We can use show_data function 59 | show_data(train_dataset[1][0]) 60 | 61 | 62 | # In[4] Create CNN Class 63 | 64 | 65 | class CNN(nn.Module): 66 | 67 | # Constructor 68 | def __init__(self, out_1=16, out_2=32): 69 | super().__init__() 70 | self.cnn1 = nn.Conv2d(in_channels=1, out_channels=out_1, kernel_size=5, padding=2) 71 | self.maxpool1 = nn.MaxPool2d(kernel_size=2) 72 | self.ELU = nn.ELU() 73 | self.cnn2 = nn.Conv2d(in_channels=out_1, out_channels=out_2, kernel_size=5, padding=2) 74 | self.maxpool2 = nn.MaxPool2d(kernel_size=2) 75 | self.fc1 = nn.Linear(out_2*4*4, 10) 76 | 77 | # Prediction 78 | def forward(self, x): 79 | x = self.cnn1(x) 80 | x = self.ELU(x) 81 | x = self.maxpool1(x) 82 | x = self.cnn2(x) 83 | x = self.ELU(x) 84 | x = self.maxpool2(x) 85 | x = x.view(x.size(0), -1) 86 | x = self.fc1(x) 87 | return x 88 | 89 | # Outputs in each step 90 | def activations(self, x): 91 | # This part is for visualization purposes 92 | z1 = self.cnn1(x) 93 | a1 = self.ELU(z1) 94 | out = self.maxpool1(a1) 95 | z2 = self.cnn2(out) 96 | a2 = self.ELU(z2) 97 | out1 = self.maxpool2(a2) 98 | out2 = out1.view(out1.size(0), -1) 99 | return z1, a1, out, z2, a2, out1, out2 100 | 101 | 102 | # In[5] Train loop and training 103 | 104 | 105 | def train_model(model, train_loader, validation_loader, optimizer, n_epochs=4): 106 | 107 | # Global variable 108 | N_test = len(validation_dataset) 109 | accuracy_list = [] 110 | loss_list = [] 111 | for epoch in range(n_epochs): 112 | for x, y in train_loader: 113 | model.train() 114 | optimizer.zero_grad() 115 | z = model(x) 116 | loss = criterion(z, y) 117 | loss.backward() 118 | optimizer.step() 119 | loss_list.append(loss.data) 120 | 121 | correct = 0 122 | # Perform a prediction on the validation data 123 | for x_test, y_test in validation_loader: 124 | model.eval() 125 | z = model(x_test) 126 | _, yhat = torch.max(z.data, 1) 127 | correct += (yhat == y_test).sum().item() 128 | accuracy = correct / N_test 129 | accuracy_list.append(accuracy) 130 | 131 | return accuracy_list, loss_list 132 | 133 | 134 | 135 | # In[6] Batch Normalization CNN class 136 | 137 | 138 | class CNN_BatchNorm(nn.Module): 139 | # Constructor 140 | def __init__(self, out_1=16, out_2=32): 141 | super().__init__() 142 | self.cnn1 = nn.Conv2d(in_channels=1, out_channels=out_1, kernel_size=5, padding=2) 143 | self.conv1_bn = nn.BatchNorm2d(out_1) # To normalize conv2D, we need BatchNorm2D 144 | self.maxpool1 = nn.MaxPool2d(kernel_size=2) 145 | self.ELU = nn.ELU() 146 | self.cnn2 = nn.Conv2d(in_channels=out_1, out_channels=out_2, kernel_size=5, padding=2) 147 | self.conv2_bn = nn.BatchNorm2d(out_2) 148 | self.maxpool2 = nn.MaxPool2d(kernel_size=2) 149 | self.fc1 = nn.Linear(out_2*4*4, 10) 150 | self.bn_fc1 = nn.BatchNorm1d(10) # To normalize linear layer, BatchNorm1D must be used 151 | 152 | # Prediction 153 | def forward(self, x): 154 | x = self.cnn1(x) 155 | x = self.conv1_bn(x) 156 | x = self.ELU(x) 157 | x = self.maxpool1(x) 158 | x = self.cnn2(x) 159 | x = self.conv2_bn(x) 160 | x = self.ELU(x) 161 | x = self.maxpool2(x) 162 | x = x.view(x.size(0), -1) 163 | x = self.fc1(x) 164 | x = self.bn_fc1(x) 165 | return x 166 | 167 | 168 | # In[7] Initialize, create loss function, optimizer and data loaders 169 | 170 | 171 | # Create model object from CNN class 172 | model = CNN(16, 32) 173 | 174 | # Optimizer 175 | optimizer = torch.optim.Adam(model.parameters(), lr=0.01) 176 | 177 | # Loss function criterion 178 | criterion = nn.CrossEntropyLoss() 179 | 180 | # train and val loader 181 | train_loader = DataLoader(dataset=train_dataset, batch_size=100) 182 | validation_loader = DataLoader(dataset=validation_dataset, batch_size=5000) 183 | 184 | accuracy_list_normal, loss_list_normal = train_model(model=model, 185 | n_epochs=10, train_loader=train_loader, validation_loader=validation_loader, 186 | optimizer=optimizer) 187 | 188 | 189 | # Create model_BatchNorm object from CNN_BatchNorm class 190 | model_BatchNorm = CNN_BatchNorm(16, 32) 191 | 192 | # Optimizer 193 | optimizer_BatchNorm = torch.optim.Adam(model_BatchNorm.parameters(), lr=0.01) 194 | 195 | accuracy_list_batch, loss_list_batch = train_model(model=model_BatchNorm, 196 | n_epochs=10, train_loader=train_loader, validation_loader=validation_loader, 197 | optimizer=optimizer_BatchNorm) 198 | 199 | 200 | # In[8] Analyze Results and Compare 201 | 202 | # Plot the loss and accuracy 203 | plt.plot(loss_list_normal, 'b', label='loss normal cnn') 204 | plt.plot(loss_list_batch, 'r', label='loss batch cnn') 205 | plt.xlabel('iteration') 206 | plt.title("loss") 207 | plt.legend() 208 | 209 | plt.plot(accuracy_list_normal, 'b', label='normal CNN') 210 | plt.plot(accuracy_list_batch, 'r', label='CNN with Batch Norm') 211 | plt.xlabel('Epoch') 212 | plt.title("Accuracy ") 213 | plt.legend() 214 | plt.show() 215 | -------------------------------------------------------------------------------- /Week 7/FashionMNIST Project Honors.py: -------------------------------------------------------------------------------- 1 | """ 2 | 3 | Convolutional Neural Network with Small Images 4 | 5 | """ 6 | 7 | # In[1] Imports 8 | 9 | import torch 10 | import torch.nn as nn 11 | import torchvision.transforms as transforms 12 | import torchvision.datasets as dsets 13 | from torch.utils.data import DataLoader, Dataset 14 | import matplotlib.pyplot as plt 15 | import numpy as np 16 | torch.manual_seed(0) 17 | from matplotlib.pyplot import imshow 18 | import matplotlib.pylab as plt 19 | from PIL import Image 20 | import time 21 | 22 | # In[2] Plot function 23 | 24 | # Define the function for plotting the channels 25 | 26 | 27 | def show_data(data_sample): 28 | plt.imshow(data_sample[0].numpy().reshape(IMAGE_SIZE, IMAGE_SIZE), cmap='gray') 29 | plt.title('y = ' + str(data_sample[1])) 30 | 31 | 32 | # In[3] Create Data 33 | 34 | 35 | IMAGE_SIZE = 16 36 | 37 | # Create a transform to resize image and convert to tensor 38 | composed = transforms.Compose([transforms.Resize((IMAGE_SIZE, IMAGE_SIZE)), transforms.ToTensor()]) 39 | 40 | # Create Dataset from MNIST and apply composed transformation 41 | dataset_train = dsets.FashionMNIST(root='.fashion/data', train=True, download=True, transform=composed) 42 | dataset_val = dsets.FashionMNIST(root='.fashion/data', train=False, download=True, transform=composed) 43 | 44 | for n, data_sample in enumerate(dataset_val): 45 | 46 | show_data(data_sample) 47 | plt.show() 48 | if n == 2: 49 | break 50 | 51 | """ Question: Take a screen shot of the first three images of the validation dataset from the code provided. 52 | The function show_data 53 | """ 54 | 55 | show_data(dataset_val[0]) 56 | show_data(dataset_val[1]) 57 | show_data(dataset_val[2]) 58 | 59 | # In[4] Create CNN Class 60 | 61 | 62 | class CNN(nn.Module): 63 | 64 | # Constructor 65 | def __init__(self, out_1=16, out_2=32): 66 | super().__init__() 67 | self.cnn1 = nn.Conv2d(in_channels=1, out_channels=out_1, kernel_size=5, padding=2) 68 | self.maxpool1 = nn.MaxPool2d(kernel_size=2) 69 | self.ELU = nn.ELU() 70 | self.cnn2 = nn.Conv2d(in_channels=out_1, out_channels=out_2, kernel_size=5, padding=2) 71 | self.maxpool2 = nn.MaxPool2d(kernel_size=2) 72 | self.fc1 = nn.Linear(out_2*4*4, 10) 73 | 74 | # Prediction 75 | def forward(self, x): 76 | x = self.cnn1(x) 77 | x = self.ELU(x) 78 | x = self.maxpool1(x) 79 | x = self.cnn2(x) 80 | x = self.ELU(x) 81 | x = self.maxpool2(x) 82 | x = x.view(x.size(0), -1) 83 | x = self.fc1(x) 84 | return x 85 | 86 | # Outputs in each step 87 | def activations(self, x): 88 | # This part is for visualization purposes 89 | z1 = self.cnn1(x) 90 | a1 = self.ELU(z1) 91 | out = self.maxpool1(a1) 92 | z2 = self.cnn2(out) 93 | a2 = self.ELU(z2) 94 | out1 = self.maxpool2(a2) 95 | out2 = out1.view(out1.size(0), -1) 96 | return z1, a1, out, z2, a2, out1, out2 97 | 98 | 99 | # In[5] Batch Normalization CNN class 100 | 101 | 102 | class CNN_BatchNorm(nn.Module): 103 | # Constructor 104 | def __init__(self, out_1=16, out_2=32): 105 | super().__init__() 106 | self.cnn1 = nn.Conv2d(in_channels=1, out_channels=out_1, kernel_size=5, padding=2) 107 | self.conv1_bn = nn.BatchNorm2d(out_1) # To normalize conv2D, we need BatchNorm2D 108 | self.maxpool1 = nn.MaxPool2d(kernel_size=2) 109 | self.ELU = nn.ELU() 110 | self.cnn2 = nn.Conv2d(in_channels=out_1, out_channels=out_2, kernel_size=5, padding=2) 111 | self.conv2_bn = nn.BatchNorm2d(out_2) 112 | self.maxpool2 = nn.MaxPool2d(kernel_size=2) 113 | self.fc1 = nn.Linear(out_2*4*4, 10) 114 | self.bn_fc1 = nn.BatchNorm1d(10) # To normalize linear layer, BatchNorm1D must be used 115 | 116 | # Prediction 117 | def forward(self, x): 118 | x = self.cnn1(x) 119 | x = self.conv1_bn(x) 120 | x = self.ELU(x) 121 | x = self.maxpool1(x) 122 | x = self.cnn2(x) 123 | x = self.conv2_bn(x) 124 | x = self.ELU(x) 125 | x = self.maxpool2(x) 126 | x = x.view(x.size(0), -1) 127 | x = self.fc1(x) 128 | x = self.bn_fc1(x) 129 | return x 130 | 131 | 132 | # In[6] Initialize, create loss function, optimizer and data loaders 133 | 134 | # Loss function criterion 135 | criterion = nn.CrossEntropyLoss() 136 | 137 | # train and val loader 138 | train_loader = DataLoader(dataset=dataset_train, batch_size=100) 139 | validation_loader = DataLoader(dataset=dataset_val, batch_size=100) 140 | 141 | # Create model_BatchNorm object from CNN_BatchNorm class 142 | model = CNN_BatchNorm(16, 32) 143 | 144 | # Optimizer 145 | optimizer = torch.optim.SGD(model.parameters(), lr=0.1) 146 | 147 | 148 | # In[6] Train loop and training 149 | 150 | 151 | start_time = time.time() 152 | 153 | cost_list = [] 154 | accuracy_list = [] 155 | N_test = len(dataset_val) 156 | n_epochs = 5 157 | for epoch in range(n_epochs): 158 | cost = 0 159 | model.train() 160 | for x, y in train_loader: 161 | optimizer.zero_grad() 162 | z = model(x) 163 | loss = criterion(z, y) 164 | loss.backward() 165 | optimizer.step() 166 | cost += loss.item() 167 | correct = 0 168 | # Perform a prediction on the validation data 169 | model.eval() 170 | for x_test, y_test in validation_loader: 171 | z = model(x_test) 172 | _, yhat = torch.max(z.data, 1) 173 | correct += (yhat == y_test).sum().item() 174 | accuracy = correct / N_test 175 | accuracy_list.append(accuracy) 176 | cost_list.append(cost) 177 | 178 | # In[7] Analyze Results and Compare 179 | 180 | # Plot the loss and accuracy 181 | fig, ax1 = plt.subplots() 182 | color = 'tab:red' 183 | ax1.plot(cost_list, color=color) 184 | ax1.set_xlabel('epoch', color=color) 185 | ax1.set_ylabel('Cost', color=color) 186 | ax1.tick_params(axis='y', color=color) 187 | 188 | ax2 = ax1.twinx() 189 | color = 'tab:blue' 190 | ax2.set_ylabel('accuracy', color=color) 191 | ax2.set_xlabel('epoch', color=color) 192 | ax2.plot(accuracy_list, color=color) 193 | ax2.tick_params(axis='y', color=color) 194 | fig.tight_layout() 195 | -------------------------------------------------------------------------------- /Week 7/First 3 Items.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AyadiGithub/Deep-Neural-Networks-with-PyTorch/824a98999143ede21ba9f0b7fb020fbd5676e603/Week 7/First 3 Items.png -------------------------------------------------------------------------------- /Week 7/Image1 Validation_set.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AyadiGithub/Deep-Neural-Networks-with-PyTorch/824a98999143ede21ba9f0b7fb020fbd5676e603/Week 7/Image1 Validation_set.png -------------------------------------------------------------------------------- /Week 7/Image2 Validation_set.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AyadiGithub/Deep-Neural-Networks-with-PyTorch/824a98999143ede21ba9f0b7fb020fbd5676e603/Week 7/Image2 Validation_set.png -------------------------------------------------------------------------------- /Week 7/Image3 Validation_set.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AyadiGithub/Deep-Neural-Networks-with-PyTorch/824a98999143ede21ba9f0b7fb020fbd5676e603/Week 7/Image3 Validation_set.png -------------------------------------------------------------------------------- /Week 7/Plot Cost and Accuracy.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AyadiGithub/Deep-Neural-Networks-with-PyTorch/824a98999143ede21ba9f0b7fb020fbd5676e603/Week 7/Plot Cost and Accuracy.png --------------------------------------------------------------------------------