├── save_model └── .gitkeep ├── imgs └── lstm.jpg ├── utils.py ├── data ├── download.py └── video2jpg.py ├── README.md ├── .gitignore ├── dataset.py ├── model.py ├── train.py └── LICENSE /save_model/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /imgs/lstm.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IDKiro/action-recognition/HEAD/imgs/lstm.jpg -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import cv2 3 | 4 | 5 | class AverageMeter(object): 6 | def __init__(self): 7 | self.reset() 8 | 9 | def reset(self): 10 | self.val = 0 11 | self.avg = 0 12 | self.sum = 0 13 | self.count = 0 14 | 15 | def update(self, val, n=1): 16 | self.val = val 17 | self.sum += val * n 18 | self.count += n 19 | self.avg = self.sum / self.count 20 | 21 | 22 | def read_img(filename): 23 | img = cv2.imread(filename) 24 | img = img[:,:,::-1] / 255.0 25 | img = np.array(img).astype('float32') 26 | 27 | return img 28 | 29 | 30 | def hwc_to_chw(img): 31 | return np.transpose(img, axes=[2, 0, 1]) 32 | 33 | 34 | def chw_to_hwc(img): 35 | return np.transpose(img, axes=[1, 2, 0]) 36 | -------------------------------------------------------------------------------- /data/download.py: -------------------------------------------------------------------------------- 1 | import requests 2 | import os 3 | import glob 4 | 5 | def download_file(URL, destination): 6 | session = requests.Session() 7 | response = session.get(URL, stream = True) 8 | 9 | save_response_content(response, destination) 10 | 11 | 12 | def save_response_content(response, destination): 13 | CHUNK_SIZE = 32768 14 | 15 | with open(destination, "wb") as f: 16 | for chunk in response.iter_content(CHUNK_SIZE): 17 | if chunk: # filter out keep-alive new chunks 18 | f.write(chunk) 19 | 20 | 21 | print('Downloading dataset...') 22 | if not os.path.isfile('data/hmdb51_org.rar'): 23 | download_file('http://serre-lab.clps.brown.edu/wp-content/uploads/2013/10/hmdb51_org.rar', 'data/hmdb51_org.rar') 24 | 25 | if not os.path.isdir('data/video'): 26 | os.makedirs('data/video') 27 | 28 | os.system('unrar e data/hmdb51_org.rar data/video') 29 | 30 | filenames = glob.glob('data/video/*.rar') 31 | for file_name in filenames: 32 | os.system('unrar x %s data/video' %file_name) 33 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Action Recognition 2 | 3 | **Three steps** to train your own model for action recognition based on CNN and LSTM. 4 | 5 | ## Environment 6 | 7 | Python 3.7.5 (Anaconda 5.3.0): 8 | 9 | [https://mirrors.tuna.tsinghua.edu.cn/anaconda/archive/](https://mirrors.tuna.tsinghua.edu.cn/anaconda/archive/) 10 | 11 | PyTorch 1.3.0 (CUDA 10.1): 12 | 13 | [https://pytorch.org/](https://pytorch.org/) 14 | 15 | ## Data 16 | 17 | 1. Download the dataset ([HMDB51](http://serre-lab.clps.brown.edu/wp-content/uploads/2013/10/hmdb51_org.rar)): 18 | 19 | ``` 20 | python data/download.py 21 | ``` 22 | 23 | 2. Convert videos to images: 24 | 25 | ``` 26 | python data/video2jpg.py 27 | ``` 28 | 29 | ## Train 30 | 31 | Use the following command to train the model: 32 | 33 | ``` 34 | python train.py 35 | ``` 36 | 37 | ## Network 38 | 39 | This project is based on CNNs and LSTMs. 40 | 41 |
42 | 43 |
44 | 45 | 46 | ## Tips 47 | 48 | > If you download the dataset by yourself, you need to move the `rar` file to `data` folder firstly. 49 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Add by user 2 | .vscode/ 3 | data/*.rar 4 | data/train 5 | data/valid 6 | data/video 7 | *.tar 8 | 9 | # Byte-compiled / optimized / DLL files 10 | __pycache__/ 11 | *.py[cod] 12 | *$py.class 13 | 14 | # C extensions 15 | *.so 16 | 17 | # Distribution / packaging 18 | .Python 19 | build/ 20 | develop-eggs/ 21 | dist/ 22 | downloads/ 23 | eggs/ 24 | .eggs/ 25 | lib/ 26 | lib64/ 27 | parts/ 28 | sdist/ 29 | var/ 30 | wheels/ 31 | *.egg-info/ 32 | .installed.cfg 33 | *.egg 34 | MANIFEST 35 | 36 | # PyInstaller 37 | # Usually these files are written by a python script from a template 38 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 39 | *.manifest 40 | *.spec 41 | 42 | # Installer logs 43 | pip-log.txt 44 | pip-delete-this-directory.txt 45 | 46 | # Unit test / coverage reports 47 | htmlcov/ 48 | .tox/ 49 | .nox/ 50 | .coverage 51 | .coverage.* 52 | .cache 53 | nosetests.xml 54 | coverage.xml 55 | *.cover 56 | .hypothesis/ 57 | .pytest_cache/ 58 | 59 | # Translations 60 | *.mo 61 | *.pot 62 | 63 | # Django stuff: 64 | *.log 65 | local_settings.py 66 | db.sqlite3 67 | 68 | # Flask stuff: 69 | instance/ 70 | .webassets-cache 71 | 72 | # Scrapy stuff: 73 | .scrapy 74 | 75 | # Sphinx documentation 76 | docs/_build/ 77 | 78 | # PyBuilder 79 | target/ 80 | 81 | # Jupyter Notebook 82 | .ipynb_checkpoints 83 | 84 | # IPython 85 | profile_default/ 86 | ipython_config.py 87 | 88 | # pyenv 89 | .python-version 90 | 91 | # celery beat schedule file 92 | celerybeat-schedule 93 | 94 | # SageMath parsed files 95 | *.sage.py 96 | 97 | # Environments 98 | .env 99 | .venv 100 | env/ 101 | venv/ 102 | ENV/ 103 | env.bak/ 104 | venv.bak/ 105 | 106 | # Spyder project settings 107 | .spyderproject 108 | .spyproject 109 | 110 | # Rope project settings 111 | .ropeproject 112 | 113 | # mkdocs documentation 114 | /site 115 | 116 | # mypy 117 | .mypy_cache/ 118 | .dmypy.json 119 | dmypy.json -------------------------------------------------------------------------------- /dataset.py: -------------------------------------------------------------------------------- 1 | import os 2 | import random 3 | import torch 4 | import numpy as np 5 | import PIL.Image as Image 6 | 7 | from torch.utils.data import Dataset 8 | from torchvision import transforms, utils 9 | 10 | class loadedDataset(Dataset): 11 | def __init__(self, root_dir, transform=None): 12 | self.root_dir = root_dir 13 | self.transform = transform 14 | self.classes = sorted(os.listdir(self.root_dir)) 15 | self.count = [len(os.listdir(self.root_dir + '/' + c)) for c in self.classes] 16 | self.acc_count = [self.count[0]] 17 | for i in range(1, len(self.count)): 18 | self.acc_count.append(self.acc_count[i-1] + self.count[i]) 19 | # self.acc_count = [self.count[i] + self.acc_count[i-1] for i in range(1, len(self.count))] 20 | 21 | def __len__(self): 22 | l = np.sum(np.array([len(os.listdir(self.root_dir + '/' + c)) for c in self.classes])) 23 | return l 24 | 25 | def __getitem__(self, idx): 26 | for i in range(len(self.acc_count)): 27 | if idx < self.acc_count[i]: 28 | label = i 29 | break 30 | 31 | class_path = self.root_dir + '/' + self.classes[label] 32 | 33 | if label: 34 | file_path = class_path + '/' + sorted(os.listdir(class_path))[idx-self.acc_count[label]] 35 | else: 36 | file_path = class_path + '/' + sorted(os.listdir(class_path))[idx] 37 | 38 | _, file_name = os.path.split(file_path) 39 | 40 | frames = [] 41 | 42 | # print os.listdir(file_path) 43 | file_list = sorted(os.listdir(file_path)) 44 | # print file_list 45 | 46 | # v: maximum translation in every step 47 | v = 2 48 | offset = 0 49 | for i, f in enumerate(file_list): 50 | frame = Image.open(file_path + '/' + f) 51 | #translation 52 | offset += random.randrange(-v, v) 53 | offset = min(offset, 3 * v) 54 | offset = max(offset, -3 * v) 55 | frame = frame.transform(frame.size, Image.AFFINE, (1, 0, offset, 0, 1, 0)) 56 | if self.transform is not None: 57 | frame = self.transform[0](frame) 58 | frames.append(frame) 59 | 60 | return frames, label, file_name 61 | -------------------------------------------------------------------------------- /data/video2jpg.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function, division 2 | import os 3 | import sys 4 | import subprocess 5 | import shutil 6 | 7 | def class_process(dir_path, dst_dir_path, class_name, maxSize=1024): 8 | class_path = os.path.join(dir_path, class_name) 9 | if not os.path.isdir(class_path): 10 | return 11 | 12 | dst_class_path = os.path.join(dst_dir_path, class_name) 13 | if not os.path.exists(dst_class_path): 14 | os.makedirs(dst_class_path) 15 | 16 | for file_name in os.listdir(class_path): 17 | if '.avi' not in file_name: 18 | continue 19 | name, ext = os.path.splitext(file_name) 20 | dst_directory_path = os.path.join(dst_class_path, name) 21 | 22 | video_file_path = os.path.join(class_path, file_name) 23 | 24 | # skip large files 25 | # if os.path.getsize(video_file_path) > maxSize * 1000: 26 | # continue 27 | 28 | try: 29 | if os.path.exists(dst_directory_path): 30 | if not os.path.exists(os.path.join(dst_directory_path, 'image_00001.jpg')): 31 | subprocess.call('rm -r \"{}\"'.format(dst_directory_path), shell=True) 32 | print('remove {}'.format(dst_directory_path)) 33 | os.makedirs(dst_directory_path) 34 | else: 35 | continue 36 | else: 37 | os.makedirs(dst_directory_path) 38 | except: 39 | print(dst_directory_path) 40 | continue 41 | 42 | # cmd = 'ffmpeg -i \"{}\" -vf scale=-1:240 \"{}/image_%05d.jpg\"'.format(video_file_path, dst_directory_path) 43 | cmd = 'ffmpeg -i \"{}\" -qscale:v 2 \"{}/image_%05d.jpg\"'.format(video_file_path, dst_directory_path) 44 | 45 | print(cmd) 46 | subprocess.call(cmd, shell=True) 47 | print('\n') 48 | 49 | 50 | def class_move(dir_path, valid_dir_path, class_name): 51 | class_path = os.path.join(dir_path, class_name) 52 | if not os.path.isdir(class_path): 53 | return 54 | 55 | valid_class_path = os.path.join(valid_dir_path, class_name) 56 | if not os.path.exists(valid_class_path): 57 | os.makedirs(valid_class_path) 58 | 59 | for i, (file_name) in enumerate(os.listdir(class_path)): 60 | name, ext = os.path.splitext(file_name) 61 | train_directory_path = os.path.join(class_path, name) 62 | valid_directory_path = os.path.join(valid_class_path, name) 63 | 64 | if i % 10 == 0: 65 | shutil.move(train_directory_path, valid_directory_path) 66 | 67 | 68 | if __name__=="__main__": 69 | dir_path = './data/video/' 70 | dst_dir_path = './data/train/' 71 | valid_dir_path = './data/valid/' 72 | 73 | for class_name in os.listdir(dir_path): 74 | class_process(dir_path, dst_dir_path, class_name) 75 | class_move(dst_dir_path, valid_dir_path, class_name) 76 | -------------------------------------------------------------------------------- /model.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import torchvision.models as models 4 | from torchvision import transforms, utils 5 | 6 | 7 | class LSTMModel(nn.Module): 8 | def __init__(self, original_model, arch, num_classes, lstm_layers, hidden_size, fc_size): 9 | super(LSTMModel, self).__init__() 10 | self.hidden_size = hidden_size 11 | self.num_classes = num_classes 12 | self.fc_size = fc_size 13 | 14 | # select a base model 15 | if arch.startswith('alexnet'): 16 | self.features = original_model.features 17 | for i, param in enumerate(self.features.parameters()): 18 | param.requires_grad = False 19 | self.fc_pre = nn.Sequential(nn.Linear(9216, fc_size), nn.Dropout()) 20 | self.rnn = nn.LSTM(input_size = fc_size, 21 | hidden_size = hidden_size, 22 | num_layers = lstm_layers, 23 | batch_first = True) 24 | self.fc = nn.Linear(hidden_size, num_classes) 25 | self.modelName = 'alexnet_lstm' 26 | 27 | elif arch.startswith('resnet18'): 28 | self.features = nn.Sequential(*list(original_model.children())[:-1]) 29 | for i, param in enumerate(self.features.parameters()): 30 | param.requires_grad = False 31 | self.fc_pre = nn.Sequential(nn.Linear(512, fc_size), nn.Dropout()) 32 | self.rnn = nn.LSTM(input_size = fc_size, 33 | hidden_size = hidden_size, 34 | num_layers = lstm_layers, 35 | batch_first = True) 36 | self.fc = nn.Linear(hidden_size, num_classes) 37 | self.modelName = 'resnet18_lstm' 38 | 39 | elif arch.startswith('resnet34'): 40 | self.features = nn.Sequential(*list(original_model.children())[:-1]) 41 | for i, param in enumerate(self.features.parameters()): 42 | param.requires_grad = False 43 | self.fc_pre = nn.Sequential(nn.Linear(512, fc_size), nn.Dropout()) 44 | self.rnn = nn.LSTM(input_size = fc_size, 45 | hidden_size = hidden_size, 46 | num_layers = lstm_layers, 47 | batch_first = True) 48 | self.fc = nn.Linear(hidden_size, num_classes) 49 | self.modelName = 'resnet34_lstm' 50 | 51 | elif arch.startswith('resnet50'): 52 | self.features = nn.Sequential(*list(original_model.children())[:-1]) 53 | for i, param in enumerate(self.features.parameters()): 54 | param.requires_grad = False 55 | self.fc_pre = nn.Sequential(nn.Linear(2048, fc_size), nn.Dropout()) 56 | self.rnn = nn.LSTM(input_size = fc_size, 57 | hidden_size = hidden_size, 58 | num_layers = lstm_layers, 59 | batch_first = True) 60 | self.fc = nn.Linear(hidden_size, num_classes) 61 | self.modelName = 'resnet50_lstm' 62 | 63 | else: 64 | raise Exception("This architecture has not been supported yet") 65 | 66 | def init_hidden(self, num_layers, batch_size): 67 | return (torch.zeros(num_layers, batch_size, self.hidden_size).cuda(), 68 | torch.zeros(num_layers, batch_size, self.hidden_size).cuda()) 69 | 70 | def forward(self, inputs, hidden=None, steps=0): 71 | length = len(inputs) 72 | fs = torch.zeros(inputs[0].size(0), length, self.rnn.input_size).cuda() 73 | 74 | for i in range(length): 75 | f = self.features(inputs[i]) 76 | f = f.view(f.size(0), -1) 77 | f = self.fc_pre(f) 78 | fs[:, i, :] = f 79 | 80 | outputs, hidden = self.rnn(fs, hidden) 81 | outputs = self.fc(outputs) 82 | return outputs 83 | -------------------------------------------------------------------------------- /train.py: -------------------------------------------------------------------------------- 1 | import os 2 | import shutil 3 | import argparse 4 | import torch 5 | import torch.nn as nn 6 | import torchvision.models as models 7 | from torchvision import transforms, utils 8 | import torch.nn.functional as F 9 | 10 | from dataset import loadedDataset 11 | from model import LSTMModel 12 | from utils import AverageMeter 13 | 14 | 15 | parser = argparse.ArgumentParser(description = 'Training') 16 | parser.add_argument('--model', default='./save_model/', type=str, help = 'path to model') 17 | parser.add_argument('--arch', default = 'resnet50', help = 'model architecture') 18 | parser.add_argument('--lstm-layers', default=2, type=int, help='number of lstm layers') 19 | parser.add_argument('--hidden-size', default=512, type=int, help='output size of LSTM hidden layers') 20 | parser.add_argument('--fc-size', default=1024, type=int, help='size of fully connected layer before LSTM') 21 | parser.add_argument('--epochs', default=200, type=int, help='manual epoch number') 22 | parser.add_argument('--lr', default=1e-4, type=float, help='initial learning rate') 23 | parser.add_argument('--lr-step', default=100, type=float, help='learning rate decay frequency') 24 | parser.add_argument('--batch-size', default=8, type=int, help='mini-batch size') 25 | parser.add_argument('--workers', default=8, type=int, help='number of data loading workers') 26 | args = parser.parse_args() 27 | 28 | 29 | def save_checkpoint(state, is_best, filename='checkpoint.pth.tar'): 30 | torch.save(state, os.path.join('./save_model/', filename)) 31 | if is_best: 32 | shutil.copyfile(os.path.join('./save_model/', filename), './save_model/model_best.pth.tar') 33 | 34 | def adjust_learning_rate(optimizer, epoch): 35 | if not epoch % args.lr_step and epoch: 36 | for param_group in optimizer.param_groups: 37 | param_group['lr'] = param_group['lr'] * 0.1 38 | return optimizer 39 | 40 | 41 | def accuracy(output, target, topk=(1,)): 42 | maxk = max(topk) 43 | batch_size = target.size(0) 44 | 45 | _, pred = output.topk(maxk, 1, True, True) 46 | pred = pred.t() 47 | correct = pred.eq(target.view(1, -1).expand_as(pred)) 48 | 49 | res = [] 50 | for k in topk: 51 | correct_k = correct[:k].view(-1).float().sum(0, keepdim=True) 52 | res.append(correct_k.mul_(100.0 / batch_size)) 53 | return res 54 | 55 | 56 | def train(train_loader, model, criterion, optimizer, epoch): 57 | losses = AverageMeter() 58 | top1 = AverageMeter() 59 | top5 = AverageMeter() 60 | 61 | model.train() # switch to train mode 62 | 63 | for i, (inputs, target, _) in enumerate(train_loader): 64 | input_var = [input.cuda() for input in inputs] 65 | target_var = target.cuda() 66 | 67 | # compute output 68 | output = model(input_var) 69 | output = output[:, -1, :] 70 | loss = criterion(output, target_var) 71 | losses.update(loss.item(), 1) 72 | 73 | # compute accuracy 74 | prec1, prec5 = accuracy(output.data.cpu(), target, topk=(1, 5)) 75 | top1.update(prec1[0].item(), 1) 76 | top5.update(prec5[0].item(), 1) 77 | 78 | # zero the parameter gradients 79 | optimizer.zero_grad() 80 | 81 | # compute gradient 82 | loss.backward() 83 | optimizer.step() 84 | 85 | print('Epoch: [{0}][{1}/{2}]\t' 86 | 'lr {lr:.5f}\t' 87 | 'Loss {loss.val:.4f} ({loss.avg:.4f})\t' 88 | 'Top1 {top1.val:.3f} ({top1.avg:.3f})\t' 89 | 'Top5 {top5.val:.3f} ({top5.avg:.3f})'.format( 90 | epoch, i, len(train_loader), 91 | lr=optimizer.param_groups[-1]['lr'], 92 | loss=losses, 93 | top1=top1, 94 | top5=top5)) 95 | 96 | 97 | def validate(val_loader, model, criterion): 98 | losses = AverageMeter() 99 | top1 = AverageMeter() 100 | top5 = AverageMeter() 101 | 102 | # switch to evaluate mode 103 | model.eval() 104 | 105 | for i, (inputs, target, _) in enumerate(val_loader): 106 | input_var = [input.cuda() for input in inputs] 107 | target_var = target.cuda() 108 | 109 | # compute output 110 | with torch.no_grad(): 111 | output = model(input_var) 112 | output = output[:, -1, :] 113 | loss = criterion(output, target_var) 114 | losses.update(loss.item(), 1) 115 | 116 | # compute accuracy 117 | prec1, prec5 = accuracy(output.data.cpu(), target, topk=(1, 5)) 118 | top1.update(prec1[0].item(), 1) 119 | top5.update(prec5[0].item(), 1) 120 | 121 | print ('Test: [{0}/{1}]\t' 122 | 'Loss {loss.val:.4f} ({loss.avg:.4f})\t' 123 | 'Top1 {top1.val:.3f} ({top1.avg:.3f})\t' 124 | 'Top5 {top5.val:.3f} ({top5.avg:.3f})'.format( 125 | i, len(val_loader), 126 | loss=losses, 127 | top1=top1, 128 | top5=top5)) 129 | 130 | return (top1.avg, top5.avg) 131 | 132 | 133 | if __name__ == '__main__': 134 | # Data Transform and data loading 135 | traindir = './data/train/' 136 | valdir = './data/valid/' 137 | 138 | normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], 139 | std=[0.339, 0.224, 0.225]) 140 | 141 | transform = (transforms.Compose([ 142 | transforms.Resize(224), 143 | transforms.CenterCrop(224), 144 | transforms.ToTensor(), 145 | normalize] 146 | ), 147 | transforms.Compose([ 148 | transforms.Resize(224), 149 | transforms.CenterCrop(224), 150 | transforms.ToTensor()] 151 | ) 152 | ) 153 | 154 | train_dataset = loadedDataset(traindir, transform) 155 | val_dataset = loadedDataset(valdir, transform) 156 | 157 | train_loader = torch.utils.data.DataLoader( 158 | train_dataset, 159 | batch_size=args.batch_size, shuffle=True, 160 | num_workers=args.workers, pin_memory=True) 161 | val_loader = torch.utils.data.DataLoader( 162 | val_dataset, 163 | batch_size=args.batch_size, shuffle=False, 164 | num_workers=args.workers, pin_memory=True) 165 | 166 | if os.path.exists(os.path.join(args.model, 'checkpoint.pth.tar')): 167 | # load existing model 168 | model_info = torch.load(os.path.join(args.model, 'checkpoint.pth.tar')) 169 | print("==> loading existing model '{}' ".format(model_info['arch'])) 170 | original_model = models.__dict__[model_info['arch']](pretrained=False) 171 | model = LSTMModel(original_model, model_info['arch'], 172 | model_info['num_classes'], model_info['lstm_layers'], model_info['hidden_size'], model_info['fc_size']) 173 | # print(model) 174 | model.cuda() 175 | model.load_state_dict(model_info['state_dict']) 176 | best_prec = model_info['best_prec'] 177 | cur_epoch = model_info['epoch'] 178 | else: 179 | if not os.path.isdir(args.model): 180 | os.makedirs(args.model) 181 | # load and create model 182 | print("==> creating model '{}' ".format(args.arch)) 183 | original_model = models.__dict__[args.arch](pretrained=True) 184 | model = LSTMModel(original_model, args.arch, 185 | len(train_dataset.classes), args.lstm_layers, args.hidden_size, args.fc_size) 186 | # print(model) 187 | model.cuda() 188 | cur_epoch = 0 189 | 190 | # loss criterion and optimizer 191 | criterion = nn.CrossEntropyLoss() 192 | criterion = criterion.cuda() 193 | 194 | optimizer = torch.optim.Adam([{'params': model.fc_pre.parameters()}, 195 | {'params': model.rnn.parameters()}, 196 | {'params': model.fc.parameters()}], 197 | lr=args.lr) 198 | 199 | best_prec = 0 200 | 201 | # Training on epochs 202 | for epoch in range(cur_epoch, args.epochs): 203 | 204 | optimizer = adjust_learning_rate(optimizer, epoch) 205 | 206 | print("---------------------------------------------------Training---------------------------------------------------") 207 | 208 | # train on one epoch 209 | train(train_loader, model, criterion, optimizer, epoch) 210 | 211 | print("--------------------------------------------------Validation--------------------------------------------------") 212 | 213 | # evaluate on validation set 214 | prec1, prec5 = validate(val_loader, model, criterion) 215 | 216 | print("------Validation Result------") 217 | print(" Top1 accuracy: {prec: .2f} %".format(prec=prec1)) 218 | print(" Top5 accuracy: {prec: .2f} %".format(prec=prec5)) 219 | print("-----------------------------") 220 | 221 | # remember best top1 accuracy and save checkpoint 222 | is_best = prec1 > best_prec 223 | best_prec = max(prec1, best_prec) 224 | save_checkpoint({ 225 | 'epoch': epoch + 1, 226 | 'arch': args.arch, 227 | 'num_classes': len(train_dataset.classes), 228 | 'lstm_layers': args.lstm_layers, 229 | 'hidden_size': args.hidden_size, 230 | 'fc_size': args.fc_size, 231 | 'state_dict': model.state_dict(), 232 | 'best_prec': best_prec, 233 | 'optimizer' : optimizer.state_dict(),}, is_best) 234 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | GNU LESSER GENERAL PUBLIC LICENSE 2 | Version 2.1, February 1999 3 | 4 | Copyright (C) 1991, 1999 Free Software Foundation, Inc. 5 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA 6 | Everyone is permitted to copy and distribute verbatim copies 7 | of this license document, but changing it is not allowed. 8 | 9 | [This is the first released version of the Lesser GPL. It also counts 10 | as the successor of the GNU Library Public License, version 2, hence 11 | the version number 2.1.] 12 | 13 | Preamble 14 | 15 | The licenses for most software are designed to take away your 16 | freedom to share and change it. By contrast, the GNU General Public 17 | Licenses are intended to guarantee your freedom to share and change 18 | free software--to make sure the software is free for all its users. 19 | 20 | This license, the Lesser General Public License, applies to some 21 | specially designated software packages--typically libraries--of the 22 | Free Software Foundation and other authors who decide to use it. You 23 | can use it too, but we suggest you first think carefully about whether 24 | this license or the ordinary General Public License is the better 25 | strategy to use in any particular case, based on the explanations below. 26 | 27 | When we speak of free software, we are referring to freedom of use, 28 | not price. Our General Public Licenses are designed to make sure that 29 | you have the freedom to distribute copies of free software (and charge 30 | for this service if you wish); that you receive source code or can get 31 | it if you want it; that you can change the software and use pieces of 32 | it in new free programs; and that you are informed that you can do 33 | these things. 34 | 35 | To protect your rights, we need to make restrictions that forbid 36 | distributors to deny you these rights or to ask you to surrender these 37 | rights. These restrictions translate to certain responsibilities for 38 | you if you distribute copies of the library or if you modify it. 39 | 40 | For example, if you distribute copies of the library, whether gratis 41 | or for a fee, you must give the recipients all the rights that we gave 42 | you. You must make sure that they, too, receive or can get the source 43 | code. If you link other code with the library, you must provide 44 | complete object files to the recipients, so that they can relink them 45 | with the library after making changes to the library and recompiling 46 | it. And you must show them these terms so they know their rights. 47 | 48 | We protect your rights with a two-step method: (1) we copyright the 49 | library, and (2) we offer you this license, which gives you legal 50 | permission to copy, distribute and/or modify the library. 51 | 52 | To protect each distributor, we want to make it very clear that 53 | there is no warranty for the free library. Also, if the library is 54 | modified by someone else and passed on, the recipients should know 55 | that what they have is not the original version, so that the original 56 | author's reputation will not be affected by problems that might be 57 | introduced by others. 58 | 59 | Finally, software patents pose a constant threat to the existence of 60 | any free program. We wish to make sure that a company cannot 61 | effectively restrict the users of a free program by obtaining a 62 | restrictive license from a patent holder. Therefore, we insist that 63 | any patent license obtained for a version of the library must be 64 | consistent with the full freedom of use specified in this license. 65 | 66 | Most GNU software, including some libraries, is covered by the 67 | ordinary GNU General Public License. This license, the GNU Lesser 68 | General Public License, applies to certain designated libraries, and 69 | is quite different from the ordinary General Public License. We use 70 | this license for certain libraries in order to permit linking those 71 | libraries into non-free programs. 72 | 73 | When a program is linked with a library, whether statically or using 74 | a shared library, the combination of the two is legally speaking a 75 | combined work, a derivative of the original library. The ordinary 76 | General Public License therefore permits such linking only if the 77 | entire combination fits its criteria of freedom. The Lesser General 78 | Public License permits more lax criteria for linking other code with 79 | the library. 80 | 81 | We call this license the "Lesser" General Public License because it 82 | does Less to protect the user's freedom than the ordinary General 83 | Public License. It also provides other free software developers Less 84 | of an advantage over competing non-free programs. These disadvantages 85 | are the reason we use the ordinary General Public License for many 86 | libraries. However, the Lesser license provides advantages in certain 87 | special circumstances. 88 | 89 | For example, on rare occasions, there may be a special need to 90 | encourage the widest possible use of a certain library, so that it becomes 91 | a de-facto standard. To achieve this, non-free programs must be 92 | allowed to use the library. A more frequent case is that a free 93 | library does the same job as widely used non-free libraries. In this 94 | case, there is little to gain by limiting the free library to free 95 | software only, so we use the Lesser General Public License. 96 | 97 | In other cases, permission to use a particular library in non-free 98 | programs enables a greater number of people to use a large body of 99 | free software. For example, permission to use the GNU C Library in 100 | non-free programs enables many more people to use the whole GNU 101 | operating system, as well as its variant, the GNU/Linux operating 102 | system. 103 | 104 | Although the Lesser General Public License is Less protective of the 105 | users' freedom, it does ensure that the user of a program that is 106 | linked with the Library has the freedom and the wherewithal to run 107 | that program using a modified version of the Library. 108 | 109 | The precise terms and conditions for copying, distribution and 110 | modification follow. Pay close attention to the difference between a 111 | "work based on the library" and a "work that uses the library". The 112 | former contains code derived from the library, whereas the latter must 113 | be combined with the library in order to run. 114 | 115 | GNU LESSER GENERAL PUBLIC LICENSE 116 | TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 117 | 118 | 0. This License Agreement applies to any software library or other 119 | program which contains a notice placed by the copyright holder or 120 | other authorized party saying it may be distributed under the terms of 121 | this Lesser General Public License (also called "this License"). 122 | Each licensee is addressed as "you". 123 | 124 | A "library" means a collection of software functions and/or data 125 | prepared so as to be conveniently linked with application programs 126 | (which use some of those functions and data) to form executables. 127 | 128 | The "Library", below, refers to any such software library or work 129 | which has been distributed under these terms. A "work based on the 130 | Library" means either the Library or any derivative work under 131 | copyright law: that is to say, a work containing the Library or a 132 | portion of it, either verbatim or with modifications and/or translated 133 | straightforwardly into another language. (Hereinafter, translation is 134 | included without limitation in the term "modification".) 135 | 136 | "Source code" for a work means the preferred form of the work for 137 | making modifications to it. For a library, complete source code means 138 | all the source code for all modules it contains, plus any associated 139 | interface definition files, plus the scripts used to control compilation 140 | and installation of the library. 141 | 142 | Activities other than copying, distribution and modification are not 143 | covered by this License; they are outside its scope. The act of 144 | running a program using the Library is not restricted, and output from 145 | such a program is covered only if its contents constitute a work based 146 | on the Library (independent of the use of the Library in a tool for 147 | writing it). Whether that is true depends on what the Library does 148 | and what the program that uses the Library does. 149 | 150 | 1. You may copy and distribute verbatim copies of the Library's 151 | complete source code as you receive it, in any medium, provided that 152 | you conspicuously and appropriately publish on each copy an 153 | appropriate copyright notice and disclaimer of warranty; keep intact 154 | all the notices that refer to this License and to the absence of any 155 | warranty; and distribute a copy of this License along with the 156 | Library. 157 | 158 | You may charge a fee for the physical act of transferring a copy, 159 | and you may at your option offer warranty protection in exchange for a 160 | fee. 161 | 162 | 2. You may modify your copy or copies of the Library or any portion 163 | of it, thus forming a work based on the Library, and copy and 164 | distribute such modifications or work under the terms of Section 1 165 | above, provided that you also meet all of these conditions: 166 | 167 | a) The modified work must itself be a software library. 168 | 169 | b) You must cause the files modified to carry prominent notices 170 | stating that you changed the files and the date of any change. 171 | 172 | c) You must cause the whole of the work to be licensed at no 173 | charge to all third parties under the terms of this License. 174 | 175 | d) If a facility in the modified Library refers to a function or a 176 | table of data to be supplied by an application program that uses 177 | the facility, other than as an argument passed when the facility 178 | is invoked, then you must make a good faith effort to ensure that, 179 | in the event an application does not supply such function or 180 | table, the facility still operates, and performs whatever part of 181 | its purpose remains meaningful. 182 | 183 | (For example, a function in a library to compute square roots has 184 | a purpose that is entirely well-defined independent of the 185 | application. Therefore, Subsection 2d requires that any 186 | application-supplied function or table used by this function must 187 | be optional: if the application does not supply it, the square 188 | root function must still compute square roots.) 189 | 190 | These requirements apply to the modified work as a whole. If 191 | identifiable sections of that work are not derived from the Library, 192 | and can be reasonably considered independent and separate works in 193 | themselves, then this License, and its terms, do not apply to those 194 | sections when you distribute them as separate works. But when you 195 | distribute the same sections as part of a whole which is a work based 196 | on the Library, the distribution of the whole must be on the terms of 197 | this License, whose permissions for other licensees extend to the 198 | entire whole, and thus to each and every part regardless of who wrote 199 | it. 200 | 201 | Thus, it is not the intent of this section to claim rights or contest 202 | your rights to work written entirely by you; rather, the intent is to 203 | exercise the right to control the distribution of derivative or 204 | collective works based on the Library. 205 | 206 | In addition, mere aggregation of another work not based on the Library 207 | with the Library (or with a work based on the Library) on a volume of 208 | a storage or distribution medium does not bring the other work under 209 | the scope of this License. 210 | 211 | 3. You may opt to apply the terms of the ordinary GNU General Public 212 | License instead of this License to a given copy of the Library. To do 213 | this, you must alter all the notices that refer to this License, so 214 | that they refer to the ordinary GNU General Public License, version 2, 215 | instead of to this License. (If a newer version than version 2 of the 216 | ordinary GNU General Public License has appeared, then you can specify 217 | that version instead if you wish.) Do not make any other change in 218 | these notices. 219 | 220 | Once this change is made in a given copy, it is irreversible for 221 | that copy, so the ordinary GNU General Public License applies to all 222 | subsequent copies and derivative works made from that copy. 223 | 224 | This option is useful when you wish to copy part of the code of 225 | the Library into a program that is not a library. 226 | 227 | 4. You may copy and distribute the Library (or a portion or 228 | derivative of it, under Section 2) in object code or executable form 229 | under the terms of Sections 1 and 2 above provided that you accompany 230 | it with the complete corresponding machine-readable source code, which 231 | must be distributed under the terms of Sections 1 and 2 above on a 232 | medium customarily used for software interchange. 233 | 234 | If distribution of object code is made by offering access to copy 235 | from a designated place, then offering equivalent access to copy the 236 | source code from the same place satisfies the requirement to 237 | distribute the source code, even though third parties are not 238 | compelled to copy the source along with the object code. 239 | 240 | 5. A program that contains no derivative of any portion of the 241 | Library, but is designed to work with the Library by being compiled or 242 | linked with it, is called a "work that uses the Library". Such a 243 | work, in isolation, is not a derivative work of the Library, and 244 | therefore falls outside the scope of this License. 245 | 246 | However, linking a "work that uses the Library" with the Library 247 | creates an executable that is a derivative of the Library (because it 248 | contains portions of the Library), rather than a "work that uses the 249 | library". The executable is therefore covered by this License. 250 | Section 6 states terms for distribution of such executables. 251 | 252 | When a "work that uses the Library" uses material from a header file 253 | that is part of the Library, the object code for the work may be a 254 | derivative work of the Library even though the source code is not. 255 | Whether this is true is especially significant if the work can be 256 | linked without the Library, or if the work is itself a library. The 257 | threshold for this to be true is not precisely defined by law. 258 | 259 | If such an object file uses only numerical parameters, data 260 | structure layouts and accessors, and small macros and small inline 261 | functions (ten lines or less in length), then the use of the object 262 | file is unrestricted, regardless of whether it is legally a derivative 263 | work. (Executables containing this object code plus portions of the 264 | Library will still fall under Section 6.) 265 | 266 | Otherwise, if the work is a derivative of the Library, you may 267 | distribute the object code for the work under the terms of Section 6. 268 | Any executables containing that work also fall under Section 6, 269 | whether or not they are linked directly with the Library itself. 270 | 271 | 6. As an exception to the Sections above, you may also combine or 272 | link a "work that uses the Library" with the Library to produce a 273 | work containing portions of the Library, and distribute that work 274 | under terms of your choice, provided that the terms permit 275 | modification of the work for the customer's own use and reverse 276 | engineering for debugging such modifications. 277 | 278 | You must give prominent notice with each copy of the work that the 279 | Library is used in it and that the Library and its use are covered by 280 | this License. You must supply a copy of this License. If the work 281 | during execution displays copyright notices, you must include the 282 | copyright notice for the Library among them, as well as a reference 283 | directing the user to the copy of this License. Also, you must do one 284 | of these things: 285 | 286 | a) Accompany the work with the complete corresponding 287 | machine-readable source code for the Library including whatever 288 | changes were used in the work (which must be distributed under 289 | Sections 1 and 2 above); and, if the work is an executable linked 290 | with the Library, with the complete machine-readable "work that 291 | uses the Library", as object code and/or source code, so that the 292 | user can modify the Library and then relink to produce a modified 293 | executable containing the modified Library. (It is understood 294 | that the user who changes the contents of definitions files in the 295 | Library will not necessarily be able to recompile the application 296 | to use the modified definitions.) 297 | 298 | b) Use a suitable shared library mechanism for linking with the 299 | Library. A suitable mechanism is one that (1) uses at run time a 300 | copy of the library already present on the user's computer system, 301 | rather than copying library functions into the executable, and (2) 302 | will operate properly with a modified version of the library, if 303 | the user installs one, as long as the modified version is 304 | interface-compatible with the version that the work was made with. 305 | 306 | c) Accompany the work with a written offer, valid for at 307 | least three years, to give the same user the materials 308 | specified in Subsection 6a, above, for a charge no more 309 | than the cost of performing this distribution. 310 | 311 | d) If distribution of the work is made by offering access to copy 312 | from a designated place, offer equivalent access to copy the above 313 | specified materials from the same place. 314 | 315 | e) Verify that the user has already received a copy of these 316 | materials or that you have already sent this user a copy. 317 | 318 | For an executable, the required form of the "work that uses the 319 | Library" must include any data and utility programs needed for 320 | reproducing the executable from it. However, as a special exception, 321 | the materials to be distributed need not include anything that is 322 | normally distributed (in either source or binary form) with the major 323 | components (compiler, kernel, and so on) of the operating system on 324 | which the executable runs, unless that component itself accompanies 325 | the executable. 326 | 327 | It may happen that this requirement contradicts the license 328 | restrictions of other proprietary libraries that do not normally 329 | accompany the operating system. Such a contradiction means you cannot 330 | use both them and the Library together in an executable that you 331 | distribute. 332 | 333 | 7. You may place library facilities that are a work based on the 334 | Library side-by-side in a single library together with other library 335 | facilities not covered by this License, and distribute such a combined 336 | library, provided that the separate distribution of the work based on 337 | the Library and of the other library facilities is otherwise 338 | permitted, and provided that you do these two things: 339 | 340 | a) Accompany the combined library with a copy of the same work 341 | based on the Library, uncombined with any other library 342 | facilities. This must be distributed under the terms of the 343 | Sections above. 344 | 345 | b) Give prominent notice with the combined library of the fact 346 | that part of it is a work based on the Library, and explaining 347 | where to find the accompanying uncombined form of the same work. 348 | 349 | 8. You may not copy, modify, sublicense, link with, or distribute 350 | the Library except as expressly provided under this License. Any 351 | attempt otherwise to copy, modify, sublicense, link with, or 352 | distribute the Library is void, and will automatically terminate your 353 | rights under this License. However, parties who have received copies, 354 | or rights, from you under this License will not have their licenses 355 | terminated so long as such parties remain in full compliance. 356 | 357 | 9. You are not required to accept this License, since you have not 358 | signed it. However, nothing else grants you permission to modify or 359 | distribute the Library or its derivative works. These actions are 360 | prohibited by law if you do not accept this License. Therefore, by 361 | modifying or distributing the Library (or any work based on the 362 | Library), you indicate your acceptance of this License to do so, and 363 | all its terms and conditions for copying, distributing or modifying 364 | the Library or works based on it. 365 | 366 | 10. Each time you redistribute the Library (or any work based on the 367 | Library), the recipient automatically receives a license from the 368 | original licensor to copy, distribute, link with or modify the Library 369 | subject to these terms and conditions. You may not impose any further 370 | restrictions on the recipients' exercise of the rights granted herein. 371 | You are not responsible for enforcing compliance by third parties with 372 | this License. 373 | 374 | 11. If, as a consequence of a court judgment or allegation of patent 375 | infringement or for any other reason (not limited to patent issues), 376 | conditions are imposed on you (whether by court order, agreement or 377 | otherwise) that contradict the conditions of this License, they do not 378 | excuse you from the conditions of this License. If you cannot 379 | distribute so as to satisfy simultaneously your obligations under this 380 | License and any other pertinent obligations, then as a consequence you 381 | may not distribute the Library at all. For example, if a patent 382 | license would not permit royalty-free redistribution of the Library by 383 | all those who receive copies directly or indirectly through you, then 384 | the only way you could satisfy both it and this License would be to 385 | refrain entirely from distribution of the Library. 386 | 387 | If any portion of this section is held invalid or unenforceable under any 388 | particular circumstance, the balance of the section is intended to apply, 389 | and the section as a whole is intended to apply in other circumstances. 390 | 391 | It is not the purpose of this section to induce you to infringe any 392 | patents or other property right claims or to contest validity of any 393 | such claims; this section has the sole purpose of protecting the 394 | integrity of the free software distribution system which is 395 | implemented by public license practices. Many people have made 396 | generous contributions to the wide range of software distributed 397 | through that system in reliance on consistent application of that 398 | system; it is up to the author/donor to decide if he or she is willing 399 | to distribute software through any other system and a licensee cannot 400 | impose that choice. 401 | 402 | This section is intended to make thoroughly clear what is believed to 403 | be a consequence of the rest of this License. 404 | 405 | 12. If the distribution and/or use of the Library is restricted in 406 | certain countries either by patents or by copyrighted interfaces, the 407 | original copyright holder who places the Library under this License may add 408 | an explicit geographical distribution limitation excluding those countries, 409 | so that distribution is permitted only in or among countries not thus 410 | excluded. In such case, this License incorporates the limitation as if 411 | written in the body of this License. 412 | 413 | 13. The Free Software Foundation may publish revised and/or new 414 | versions of the Lesser General Public License from time to time. 415 | Such new versions will be similar in spirit to the present version, 416 | but may differ in detail to address new problems or concerns. 417 | 418 | Each version is given a distinguishing version number. If the Library 419 | specifies a version number of this License which applies to it and 420 | "any later version", you have the option of following the terms and 421 | conditions either of that version or of any later version published by 422 | the Free Software Foundation. If the Library does not specify a 423 | license version number, you may choose any version ever published by 424 | the Free Software Foundation. 425 | 426 | 14. If you wish to incorporate parts of the Library into other free 427 | programs whose distribution conditions are incompatible with these, 428 | write to the author to ask for permission. For software which is 429 | copyrighted by the Free Software Foundation, write to the Free 430 | Software Foundation; we sometimes make exceptions for this. Our 431 | decision will be guided by the two goals of preserving the free status 432 | of all derivatives of our free software and of promoting the sharing 433 | and reuse of software generally. 434 | 435 | NO WARRANTY 436 | 437 | 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO 438 | WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW. 439 | EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR 440 | OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY 441 | KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE 442 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 443 | PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE 444 | LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME 445 | THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 446 | 447 | 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN 448 | WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY 449 | AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU 450 | FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR 451 | CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE 452 | LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING 453 | RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A 454 | FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF 455 | SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH 456 | DAMAGES. 457 | 458 | END OF TERMS AND CONDITIONS 459 | 460 | How to Apply These Terms to Your New Libraries 461 | 462 | If you develop a new library, and you want it to be of the greatest 463 | possible use to the public, we recommend making it free software that 464 | everyone can redistribute and change. You can do so by permitting 465 | redistribution under these terms (or, alternatively, under the terms of the 466 | ordinary General Public License). 467 | 468 | To apply these terms, attach the following notices to the library. It is 469 | safest to attach them to the start of each source file to most effectively 470 | convey the exclusion of warranty; and each file should have at least the 471 | "copyright" line and a pointer to where the full notice is found. 472 | 473 | 474 | Copyright (C) 475 | 476 | This library is free software; you can redistribute it and/or 477 | modify it under the terms of the GNU Lesser General Public 478 | License as published by the Free Software Foundation; either 479 | version 2.1 of the License, or (at your option) any later version. 480 | 481 | This library is distributed in the hope that it will be useful, 482 | but WITHOUT ANY WARRANTY; without even the implied warranty of 483 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 484 | Lesser General Public License for more details. 485 | 486 | You should have received a copy of the GNU Lesser General Public 487 | License along with this library; if not, write to the Free Software 488 | Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 489 | USA 490 | 491 | Also add information on how to contact you by electronic and paper mail. 492 | 493 | You should also get your employer (if you work as a programmer) or your 494 | school, if any, to sign a "copyright disclaimer" for the library, if 495 | necessary. Here is a sample; alter the names: 496 | 497 | Yoyodyne, Inc., hereby disclaims all copyright interest in the 498 | library `Frob' (a library for tweaking knobs) written by James Random 499 | Hacker. 500 | 501 | , 1 April 1990 502 | Ty Coon, President of Vice 503 | 504 | That's all there is to it! 505 | --------------------------------------------------------------------------------