├── README.md ├── Linear Support Vector Machines.py └── LogisticsRegressionPyTorch.py /README.md: -------------------------------------------------------------------------------- 1 | # Problem Description 2 | You are asked to implement and solve linear Logistic Regression model and Linear SVM model 3 | (without regularization term) on MNIST dataset. In this task, you only need to perform binary classification 4 | on digit 0 and 1. You need to implement custom loss function, Logistics Loss and Hinge Loss in PyTorch.You 5 | are required to optimize the model by using SGD and Momentum methods. 6 | # Data 7 | We use MNIST digit classication dataset. Pytorch/torchvision has provide a useful dataloader to automatically 8 | download and load the data into batches. In this homework, we need two class, digit 0 and digit 1, for binary 9 | classication. 10 | 11 | # Submit below results 12 | For each of the model, report the average loss for each Batch for each training epoch, where B 13 | is the total number of batches, fb is the model (Logistic regression or Linear SVM) after updated by b-th 14 | batch and Db is the number of data points in b-th batch. An epoch is defined as one iteration of all 15 | dataset. Essentially, during a training epoch, you record down the average training loss of that batch after 16 | you update the model, and then report the average of all such batch-averaged losses after one iteration 17 | of whole dataset. You could plot the results as a figure our simply list down. Please at least report 10 18 | epochs. 19 | 1. Report the final testing accuracy of trained model. 20 | 2. Please compare results for 2 optimizer (SGD and SGD-Momentum)). 21 | 3. Try different step sizes and discuss your findings. 22 | # How to run 23 | Logistics Regression and Support Vector Machine is implemented using PyTorch 24 | 25 | 1. Python code for Logistics Regression and SVM is provided 26 | 2. Open the python files in your favorite IDE 27 | 2. Run the file to see the average of batch-averaged loss and the test accuracy 28 | 3. Change the learning_rate and momentum hyper parameters on top section of the python file to see different results 29 | 30 | # Overview of approach: 31 | I started with reviewing data preparation steps,data was prepared using PyTorch data loader package and split into train and test data set.I printed shape of images, printed few example image to see if the processing is done correctly. During training i implemented custom loss functions for logistics and Linear SVM .For each algorithm, i ran through the training data for every epoch in forward pass we load batches of data, set gradient to zero and calculate the loss using custom loss function.I used PyTorch SGD optimizer to update the parameters using optimizer.step() for different learning rates and momentum till the loss converges. In the last, I used the test data to measure the accuracy of the model. 32 | -------------------------------------------------------------------------------- /Linear Support Vector Machines.py: -------------------------------------------------------------------------------- 1 | #Don't change batch size 2 | batch_size = 64 3 | #Hyper-parameters 4 | input_size = 784 #(dimension of image 28 * 28) 5 | num_classes = 1 #(just -1 and 1 image) 6 | num_epochs = 10 # number of times you will iterate through the full training data 7 | learning_rate = 0.0001 ## step size used by SGD 8 | momentum = 0.0 ## Momentum is a moving average of our gradients (helps to keep direction) 9 | 10 | 11 | import torch 12 | import torch.nn as nn 13 | from torch.autograd import Variable 14 | from torchvision import datasets, transforms, utils 15 | from torch.utils.data.sampler import SubsetRandomSampler 16 | from torch.nn import functional as F 17 | import matplotlib.pyplot as plt 18 | 19 | ## USE THIS SNIPPET TO GET BINARY TRAIN/TEST DATA 20 | 21 | train_data = datasets.MNIST('./data/anupam-data/pytorch/data/', train=True, download=True, 22 | transform=transforms.Compose([ 23 | transforms.ToTensor(), 24 | transforms.Normalize((0.1307,), (0.3081,)) 25 | ])) 26 | test_data = datasets.MNIST('./data/anupam-data/pytorch/data/', train=False, download=True, 27 | transform=transforms.Compose([ 28 | transforms.ToTensor(), 29 | transforms.Normalize((0.1307,), (0.3081,)) 30 | ])) 31 | 32 | subset_indices = ((train_data.train_labels == 0) + (train_data.train_labels == 1)).nonzero().view(-1) 33 | 34 | train_loader = torch.utils.data.DataLoader(dataset=train_data, 35 | batch_size=batch_size, 36 | shuffle=False, 37 | sampler=SubsetRandomSampler(subset_indices)) 38 | 39 | subset_indices = ((test_data.test_labels == 0) + (test_data.test_labels == 1)).nonzero().view(-1) 40 | 41 | test_loader = torch.utils.data.DataLoader(dataset=test_data, 42 | batch_size=batch_size, 43 | shuffle=False, 44 | sampler=SubsetRandomSampler(subset_indices)) 45 | 46 | ## Displaying some sample images in train_loader with its ground truth 47 | 48 | #samples = enumerate(train_loader) 49 | #batch_idx, (sample_data, sample_targets) = next(samples) 50 | #sample_data.shape 51 | 52 | #fig = plt.figure() 53 | #for i in range(6): 54 | # plt.subplot(2,3,i+1) 55 | # plt.tight_layout() 56 | # plt.imshow(sample_data[i][0], cmap='gray', interpolation='none') 57 | # plt.title("Ground Truth: {}".format(sample_targets[i])) 58 | # plt.xticks([]) 59 | # plt.yticks([]) 60 | # fig 61 | 62 | 63 | #total_step = len(train_loader) 64 | #print(total_step) 65 | 66 | class SVM_Loss(nn.modules.Module): 67 | def __init__(self): 68 | super(SVM_Loss,self).__init__() 69 | def forward(self, outputs, labels): 70 | return torch.sum(torch.clamp(1 - outputs.t()*labels, min=0))/batch_size 71 | 72 | #SVM regression model and Loss 73 | svm_model = nn.Linear(input_size,num_classes) 74 | #model = LogisticRegression(input_size,num_classes) 75 | 76 | ## Loss criteria and SGD optimizer 77 | svm_loss_criteria = SVM_Loss() 78 | 79 | #loss_criteria = nn.CrossEntropyLoss() 80 | 81 | svm_optimizer = torch.optim.SGD(svm_model.parameters(), lr=learning_rate, momentum=momentum) 82 | 83 | total_step = len(train_loader) 84 | for epoch in range(num_epochs): 85 | avg_loss_epoch = 0 86 | batch_loss = 0 87 | total_batches = 0 88 | for i, (images, labels) in enumerate(train_loader): 89 | # Reshape images to (batch_size, input_size) 90 | images = images.reshape(-1, 28*28) 91 | labels = Variable(2*(labels.float()-0.5)) 92 | 93 | # Forward pass 94 | outputs = svm_model(images) 95 | loss_svm = svm_loss_criteria(outputs, labels) 96 | 97 | 98 | # Backward and optimize 99 | svm_optimizer.zero_grad() 100 | loss_svm.backward() 101 | svm_optimizer.step() 102 | 103 | #print("Model's parameter after the update:") 104 | #for param2 in svm_model.parameters(): 105 | # print(param2) 106 | total_batches += 1 107 | batch_loss += loss_svm.item() 108 | 109 | avg_loss_epoch = batch_loss/total_batches 110 | print ('Epoch [{}/{}], Averge Loss:for epoch[{}, {:.4f}]' 111 | .format(epoch+1, num_epochs, epoch+1, avg_loss_epoch )) 112 | 113 | 114 | # Test the SVM Model 115 | correct = 0. 116 | total = 0. 117 | for images, labels in test_loader: 118 | images = images.reshape(-1, 28*28) 119 | 120 | outputs = svm_model(images) 121 | predicted = outputs.data >= 0 122 | total += labels.size(0) 123 | correct += (predicted.view(-1).long() == labels).sum() 124 | 125 | print('Accuracy of the model on the test images: %f %%' % (100 * (correct.float() / total))) 126 | print("the learning rate is ", learning_rate) 127 | print( "the momentum is", momentum) -------------------------------------------------------------------------------- /LogisticsRegressionPyTorch.py: -------------------------------------------------------------------------------- 1 | #Don't change batch size 2 | batch_size = 64 3 | #Hyper-parameters 4 | input_size = 784 #(dimension of image 28 * 28) 5 | num_classes = 1 #(just -1 and 1 image) 6 | num_epochs = 10 # number of times you will iterate through the full training data 7 | learning_rate = 0.0001 ## step size used by SGD 8 | momentum = 0.9 ## Momentum is a moving average of our gradients (helps to keep direction) 9 | 10 | 11 | import torch 12 | import torch.nn as nn 13 | from torch.autograd import Variable 14 | from torchvision import datasets, transforms, utils 15 | from torch.utils.data.sampler import SubsetRandomSampler 16 | from torch.nn import functional as F 17 | import matplotlib.pyplot as plt 18 | 19 | ## USE THIS SNIPPET TO GET BINARY TRAIN/TEST DATA 20 | 21 | train_data = datasets.MNIST('./data/anupam-data/pytorch/data/', train=True, download=True, 22 | transform=transforms.Compose([ 23 | transforms.ToTensor(), 24 | transforms.Normalize((0.1307,), (0.3081,)) 25 | ])) 26 | test_data = datasets.MNIST('./data/anupam-data/pytorch/data/', train=False, download=True, 27 | transform=transforms.Compose([ 28 | transforms.ToTensor(), 29 | transforms.Normalize((0.1307,), (0.3081,)) 30 | ])) 31 | 32 | subset_indices = ((train_data.train_labels == 0) + (train_data.train_labels == 1)).nonzero().view(-1) 33 | 34 | train_loader = torch.utils.data.DataLoader(dataset=train_data, 35 | batch_size=batch_size, 36 | shuffle=False, 37 | sampler=SubsetRandomSampler(subset_indices)) 38 | 39 | subset_indices = ((test_data.test_labels == 0) + (test_data.test_labels == 1)).nonzero().view(-1) 40 | 41 | test_loader = torch.utils.data.DataLoader(dataset=test_data, 42 | batch_size=batch_size, 43 | shuffle=False, 44 | sampler=SubsetRandomSampler(subset_indices)) 45 | 46 | ## Displaying some sample images in train_loader with its ground truth 47 | 48 | #samples = enumerate(train_loader) 49 | #batch_idx, (sample_data, sample_targets) = next(samples) 50 | #sample_data.shape 51 | 52 | #fig = plt.figure() 53 | #for i in range(6): 54 | # plt.subplot(2,3,i+1) 55 | # plt.tight_layout() 56 | # plt.imshow(sample_data[i][0], cmap='gray', interpolation='none') 57 | # plt.title("Ground Truth: {}".format(sample_targets[i])) 58 | # plt.xticks([]) 59 | # plt.yticks([]) 60 | # fig 61 | 62 | 63 | #total_step = len(train_loader) 64 | #print(total_step) 65 | 66 | class Regress_Loss(nn.modules.Module): 67 | def __init__(self): 68 | super(Regress_Loss,self).__init__() 69 | def forward(self, outputs, labels): 70 | batch_size = outputs.size()[0] 71 | return torch.sum(torch.log(1 + torch.exp(-(outputs.t()*labels))))/batch_size 72 | 73 | 74 | #Logistic regression model and Loss 75 | logistics_model = 0 76 | logistics_model = nn.Linear(input_size,num_classes) 77 | 78 | ## Custom Loss criteria and SGD optimizer 79 | loss_criteria = Regress_Loss() 80 | 81 | optimizer = torch.optim.SGD(logistics_model.parameters(), lr=learning_rate, momentum= momentum) 82 | total_step = len(train_loader) 83 | 84 | ## Train the model parameters 85 | 86 | for epoch in range(num_epochs): 87 | avg_loss_epoch = 0 88 | batch_loss = 0 89 | total_batches = 0 90 | 91 | for i, (images, labels) in enumerate(train_loader): 92 | # Reshape images to (batch_size, input_size) 93 | images = images.reshape(-1, 28*28) 94 | labels = Variable(2*(labels.float()-0.5)) 95 | 96 | outputs = logistics_model(images) 97 | loss = loss_criteria(outputs, labels) 98 | 99 | # Backward and optimize 100 | optimizer.zero_grad() 101 | loss.backward() 102 | optimizer.step() 103 | 104 | total_batches += 1 105 | batch_loss += loss.item() 106 | 107 | avg_loss_epoch = batch_loss/total_batches 108 | print ('Epoch [{}/{}], Averge Loss:for epoch[{}, {:.4f}]' 109 | .format(epoch+1, num_epochs, epoch+1, avg_loss_epoch )) 110 | 111 | 112 | ## Test the model 113 | 114 | # Test the Model 115 | correct = 0. 116 | total = 0. 117 | for images, labels in test_loader: 118 | images = images.reshape(-1, 28*28) 119 | 120 | outputs_test = torch.sigmoid(logistics_model(images)) 121 | predicted = outputs_test.data >= 0.5 122 | 123 | total += labels.size(0) 124 | 125 | correct += (predicted.view(-1).long() == labels).sum() 126 | 127 | 128 | print('Accuracy of the model on the test images: %f %%' % (100 * (correct.float() / total))) 129 | #print("number of total test images", total) 130 | #print 131 | #print("numbers of correctly predicted test images" ,correct ) 132 | print("the learning rate is ", learning_rate) 133 | print( "the momentum is", momentum) 134 | --------------------------------------------------------------------------------