├── LICENSE ├── README.md ├── ResNet18-SPD ├── conf │ ├── __init__.py │ └── global_settings.py ├── dataset.py ├── model_comparison.py ├── models │ └── resnet50_spd.py ├── test.py ├── tiny_imagenet_loader.py ├── train_tiny.py └── utils.py ├── ResNet50-SPD ├── conf │ ├── __init__.py │ └── global_settings.py ├── dataset.py ├── models │ ├── resnet.py │ └── resnet50_spd.py ├── test.py ├── train.py └── utils.py ├── YOLOv5-SPD ├── .gitattributes ├── .github │ ├── FUNDING.yml │ ├── ISSUE_TEMPLATE │ │ ├── bug-report.md │ │ ├── feature-request.md │ │ └── question.md │ ├── dependabot.yml │ └── workflows │ │ ├── ci-testing.yml │ │ ├── codeql-analysis.yml │ │ ├── greetings.yml │ │ ├── rebase.yml │ │ └── stale.yml ├── CONTRIBUTING.md ├── LICENSE ├── detect.py ├── experiment2.py ├── experimenting.py ├── export.py ├── hubconf.py ├── models │ ├── __init__.py │ ├── common.py │ ├── experimental.py │ ├── new_models.py │ ├── space_depth_l.yaml │ ├── space_depth_m.yaml │ ├── space_depth_n.yaml │ ├── space_depth_s.yaml │ ├── tf.py │ └── yolo.py ├── myoptims │ └── tanangulargrad.py ├── test_download.py ├── train.py ├── utils │ ├── __init__.py │ ├── activations.py │ ├── augmentations.py │ ├── autoanchor.py │ ├── aws │ │ ├── __init__.py │ │ ├── mime.sh │ │ ├── resume.py │ │ └── userdata.sh │ ├── callbacks.py │ ├── datasets.py │ ├── downloads.py │ ├── flask_rest_api │ │ ├── README.md │ │ ├── example_request.py │ │ └── restapi.py │ ├── general.py │ ├── google_app_engine │ │ ├── Dockerfile │ │ ├── additional_requirements.txt │ │ └── app.yaml │ ├── loggers │ │ ├── __init__.py │ │ └── wandb │ │ │ ├── README.md │ │ │ ├── __init__.py │ │ │ ├── log_dataset.py │ │ │ ├── sweep.py │ │ │ ├── sweep.yaml │ │ │ └── wandb_utils.py │ ├── loss.py │ ├── loss2.py │ ├── metrics.py │ ├── plots.py │ └── torch_utils.py └── val.py └── requirements.txt /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 LabSAINT 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ### ![red](https://via.placeholder.com/15/f03c15/000000?text=+) This repository is no longer maintained. It has been relocated to SPD-Conv, where active updates and maintenance will continue. 2 | 3 | ### This repo contains the source code and evaluation scripts for our ECML PKDD 2022 paper. 4 | 5 | ## No More Strided Convolutions or Pooling: A New CNN Building Block for Low-Resolution Images and Small Objects 6 | 7 | [Link to paper on publisher site](https://link.springer.com/chapter/10.1007/978-3-031-26409-2_27)
8 | [Direct PDF from publisher](https://link.springer.com/content/pdf/10.1007/978-3-031-26409-2_27.pdf?pdf=inline%20link)
9 | [arXiv](https://arxiv.org/abs/2208.03641) 10 | 11 | ### Abstract 12 | 13 | Convolutional neural networks (CNNs) have made resounding success in many computer vision tasks such as image classification and object detection. However, their performance degrades rapidly on tougher tasks where images are of low resolution or objects are small. In this paper, we point out that this roots in a defective yet common design in existing CNN architectures, namely the use of *strided convolution* and/or *pooling layers*, which results in a loss of fine-grained information and learning of less effective feature representations. To this end, we propose a new CNN building block called *SPD-Conv* in place of each strided convolution layer and each pooling layer (thus eliminates them altogether). SPD-Conv is comprised of a *space-to-depth* (SPD) layer followed by a *non-strided* convolution (Conv) layer, and can be applied in most if not all CNN architectures. We explain this new design under two most representative computer vision tasks: object detection and image classification. We then create new CNN architectures by applying SPD-Conv to YOLOv5 and ResNet, and empirically show that our approach significantly outperforms state-of-the-art deep learning models, especially on tougher tasks with low-resolution images and small objects. 14 | 15 | ### Citation 16 | 17 | ``` 18 | @inproceedings{spd-conv2022, 19 | title={No More Strided Convolutions or Pooling: A New {CNN} Building Block for Low-Resolution Images and Small Objects}, 20 | author={Raja Sunkara and Tie Luo}, 21 | booktitle={European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD)}, 22 | month=Sep, 23 | year={2022}, 24 | pages={443-459} 25 | } 26 | ``` 27 | 28 | 31 | 32 | ### SPD-Conv Building Block: 33 | 34 | 35 | 36 | ![losses](https://github.com/pavanraja753/pictures/blob/22d14c2045d99a239a0e544a08de93357b9816cb/spd.png?raw=true) 37 | 38 | 39 | 45 | 46 | 47 | 48 | 49 | 50 | ### Installation 51 | 52 | ``` 53 | # Download the code 54 | git clone https://github.com/LabSAINT/SPD-Conv 55 | 56 | # Create an environment 57 | cd SPD-Conv 58 | conda create -n myenv python==3.7 59 | conda activate myenv 60 | pip3 install -r requirements.txt 61 | ``` 62 | 63 | SPD-Conv is evaluated using two most representative computer vision tasks, object detection and image classification. Specifically, we construct YOLOv5-SPD, ResNet18-SPD and ResNet50-SPD, and evaluate them on COCO-2017, Tiny ImageNet, and CIFAR-10 datasets in comparison with several state-of-the-art deep learning models. 64 | 65 | ### YOLOV5-SPD 66 | 67 | ``` 68 | cd YOLOv5-SPD 69 | ``` 70 | 71 | 72 | ##### Pre-trained models 73 | 74 | The table below gives an overview of the results of our models 75 | 76 | 77 | | $$\textbf{Model}$$ | $$\textbf{AP}$$ | $$\textbf{AP}_\textbf{S}$$ | $$\textbf{Params (M)}$$ | $$\textbf{Latency (ms)}$$ | 78 | |---- |:-:|:-:|:-:|:-:| 79 | | [YOLOv5-spd-n](https://drive.google.com/drive/folders/1RqI5JELROohhxRen78W3hG6N9MMRD-6K?usp=sharing) | 31.0 | 16.0 | 2.2 | 7.3| 80 | | [YOLOv5-spd-s](https://drive.google.com/drive/folders/1RqI5JELROohhxRen78W3hG6N9MMRD-6K?usp=sharing) | 40.0 | 23.5 | 8.7 | 7.3 | 81 | | [YOLOv5-spd-m](https://drive.google.com/drive/folders/1RqI5JELROohhxRen78W3hG6N9MMRD-6K?usp=sharing) | 46.5|30.3|24.6|8.4 82 | | [YOLOv5-spd-l](https://drive.google.com/drive/folders/1RqI5JELROohhxRen78W3hG6N9MMRD-6K?usp=sharing) | 48.5|32.4|52.7|10.3 83 | 84 | 85 | 86 | 87 | ##### Evaluation 88 | 89 | The script `val.py` can be used to evaluate the pre-trained models 90 | 91 | ``` 92 | $ python val.py --weights './weights/nano_best.pt' --img 640 --iou 0.65 --half --batch-size 1 --data data/coco.yaml 93 | $ python val.py --weights './weights/small_best.pt' --img 640 --iou 0.65 --half --batch-size 1 --data data/coco.yaml 94 | $ python val.py --weights './weights/medium_best.pt' --img 640 --iou 0.65 --half --batch-size 1 --data data/coco.yaml 95 | $ python val.py --weights './weights/large_best.pt' --img 640 --iou 0.65 --half --batch-size 1 --data data/coco.yaml 96 | 97 | ``` 98 | 99 | ##### Training 100 | 101 | 102 | ``` 103 | python3 train.py --data coco.yaml --cfg ./models/space_depth_n.yaml --weights '' --batch-size 128 --epochs 300 --sync-bn --project space_depth --name space_depth_n 104 | 105 | python3 train.py --data coco.yaml --cfg ./models/space_depth_s.yaml --weights '' --batch-size 128 --epochs 300 --sync-bn --project space_depth --name space_depth_s 106 | 107 | python3 train.py --data coco.yaml --cfg ./models/space_depth_m.yaml --weights '' --batch-size 32 --epochs 200 --sync-bn --project space_depth --name space_depth_m 108 | 109 | python3 train.py --data coco.yaml --cfg ./models/space_depth_l.yaml --hyp hyp.scratch_large.yaml --weights '' --batch-size 20 --epochs 200 --sync-bn --project space_depth --name space_depth_l 110 | ``` 111 | 112 | 113 | 114 | ### ResNet18-SPD 115 | 116 | ResNet18-SPD model is evaluated on the TinyImageNet dataset 117 | 118 | ```bash 119 | cd ./../ResNet18-SPD 120 | ``` 121 | 122 | | $\textbf{Model}$ | $\textbf{Dataset}$ | $\textbf{Top-1 accuracy}$ (\%) | 123 | |----|:-:|:-:| 124 | | [ResNet18-SPD](https://drive.google.com/drive/folders/1RqI5JELROohhxRen78W3hG6N9MMRD-6K?usp=sharing) | TinyImageNet | 64.52| 125 | | [ResNet50-SPD](https://drive.google.com/drive/folders/1RqI5JELROohhxRen78W3hG6N9MMRD-6K?usp=sharing) | CIFAR-10| 95.03 | 126 | 127 | 128 | ##### Dataset 129 | 130 | Tiny-ImageNet-200 dataset can be downloaded from this link [tiny-imagenet-200.zip](https://drive.google.com/file/d/1xLcRyy7-jLV-ywaGwCHxymX9D05X0g5i/view?usp=sharing) 131 | 132 | ##### Evaluation 133 | 134 | ```bash 135 | $ python3 test.py -net resnet18_spd -weights ./weights/resnet18_spd.pt 136 | ``` 137 | 138 | ##### Training 139 | 140 | ```bash 141 | python3 train_tiny.py -net resnet18_spd -b 256 -lr 0.01793 -momentum 0.9447 -weight_decay 0.002113 -gpu -project SPD -name resnet18_spd 142 | ``` 143 | 144 | 145 | 146 | ### ResNet50-SPD 147 | 148 | ResNet50-SPD model is implemented on the CIFAR-10 dataset 149 | 150 | ```bash 151 | cd ./../ResNet50-SPD 152 | ``` 153 | 154 | ##### Dataset 155 | 156 | ```bash 157 | CIFAR-10 dataset will be downloaded automatically by the script 158 | ``` 159 | 160 | ##### Evaluation 161 | 162 | ```bash 163 | # Evaluating resnet50-SPD model 164 | python test.py -weights ./weights/resnet50_spd.pth -net resnet50_spd 165 | ``` 166 | 167 | ##### Training 168 | 169 | 170 | ```bash 171 | # Training resnet50-SPD model 172 | $ python3 train.py -net resnet50_spd -gpu 173 | ``` 174 | 175 | 176 | 177 | 178 | -------------------------------------------------------------------------------- /ResNet18-SPD/conf/__init__.py: -------------------------------------------------------------------------------- 1 | """ dynamically load settings 2 | 3 | author baiyu 4 | """ 5 | import conf.global_settings as settings 6 | 7 | class Settings: 8 | def __init__(self, settings): 9 | 10 | for attr in dir(settings): 11 | if attr.isupper(): 12 | setattr(self, attr, getattr(settings, attr)) 13 | 14 | settings = Settings(settings) -------------------------------------------------------------------------------- /ResNet18-SPD/conf/global_settings.py: -------------------------------------------------------------------------------- 1 | """ configurations for this project 2 | 3 | author baiyu 4 | """ 5 | import os 6 | from datetime import datetime 7 | 8 | #CIFAR100 dataset path (python version) 9 | #CIFAR100_PATH = '/nfs/private/cifar100/cifar-100-python' 10 | 11 | #mean and std of cifar100 dataset 12 | CIFAR100_TRAIN_MEAN = (0.5070751592371323, 0.48654887331495095, 0.4409178433670343) 13 | CIFAR100_TRAIN_STD = (0.2673342858792401, 0.2564384629170883, 0.27615047132568404) 14 | 15 | #CIFAR100_TEST_MEAN = (0.5088964127604166, 0.48739301317401956, 0.44194221124387256) 16 | #CIFAR100_TEST_STD = (0.2682515741720801, 0.2573637364478126, 0.2770957707973042) 17 | 18 | #directory to save weights file 19 | CHECKPOINT_PATH = 'SPD' 20 | 21 | #total training epoches 22 | EPOCH = 200 23 | MILESTONES = [60, 120, 160] 24 | 25 | #initial learning rate 26 | #INIT_LR = 0.1 27 | 28 | DATE_FORMAT = '%A_%d_%B_%Y_%Hh_%Mm_%Ss' 29 | #time of we run the script 30 | TIME_NOW = datetime.now().strftime(DATE_FORMAT) 31 | 32 | #tensorboard log dir 33 | LOG_DIR = 'runs' 34 | 35 | #save weights file per SAVE_EPOCH epoch 36 | SAVE_EPOCH = 100 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | -------------------------------------------------------------------------------- /ResNet18-SPD/dataset.py: -------------------------------------------------------------------------------- 1 | """ train and test dataset 2 | 3 | author baiyu 4 | """ 5 | import os 6 | import sys 7 | import pickle 8 | 9 | from skimage import io 10 | import matplotlib.pyplot as plt 11 | import numpy 12 | import torch 13 | from torch.utils.data import Dataset 14 | 15 | class CIFAR100Train(Dataset): 16 | """cifar100 test dataset, derived from 17 | torch.utils.data.DataSet 18 | """ 19 | 20 | def __init__(self, path, transform=None): 21 | #if transform is given, we transoform data using 22 | with open(os.path.join(path, 'train'), 'rb') as cifar100: 23 | self.data = pickle.load(cifar100, encoding='bytes') 24 | self.transform = transform 25 | 26 | def __len__(self): 27 | return len(self.data['fine_labels'.encode()]) 28 | 29 | def __getitem__(self, index): 30 | label = self.data['fine_labels'.encode()][index] 31 | r = self.data['data'.encode()][index, :1024].reshape(32, 32) 32 | g = self.data['data'.encode()][index, 1024:2048].reshape(32, 32) 33 | b = self.data['data'.encode()][index, 2048:].reshape(32, 32) 34 | image = numpy.dstack((r, g, b)) 35 | 36 | if self.transform: 37 | image = self.transform(image) 38 | return label, image 39 | 40 | class CIFAR100Test(Dataset): 41 | """cifar100 test dataset, derived from 42 | torch.utils.data.DataSet 43 | """ 44 | 45 | def __init__(self, path, transform=None): 46 | with open(os.path.join(path, 'test'), 'rb') as cifar100: 47 | self.data = pickle.load(cifar100, encoding='bytes') 48 | self.transform = transform 49 | 50 | def __len__(self): 51 | return len(self.data['data'.encode()]) 52 | 53 | def __getitem__(self, index): 54 | label = self.data['fine_labels'.encode()][index] 55 | r = self.data['data'.encode()][index, :1024].reshape(32, 32) 56 | g = self.data['data'.encode()][index, 1024:2048].reshape(32, 32) 57 | b = self.data['data'.encode()][index, 2048:].reshape(32, 32) 58 | image = numpy.dstack((r, g, b)) 59 | 60 | if self.transform: 61 | image = self.transform(image) 62 | return label, image 63 | 64 | -------------------------------------------------------------------------------- /ResNet18-SPD/model_comparison.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | 3 | from matplotlib import pyplot as plt 4 | 5 | import torch 6 | import torchvision.transforms as transforms 7 | from torch.utils.data import DataLoader 8 | 9 | from conf import settings 10 | from utils import get_network, get_test_dataloader 11 | 12 | if __name__ == '__main__': 13 | 14 | parser = argparse.ArgumentParser() 15 | parser.add_argument('-net', type=str, required=True, help='net type') 16 | parser.add_argument('-weights', type=str, required=True, help='the weights file you want to test') 17 | parser.add_argument('-gpu', action='store_true', default=False, help='use gpu or not') 18 | parser.add_argument('-b', type=int, default=16, help='batch size for dataloader') 19 | args = parser.parse_args() 20 | 21 | net = get_network(args) 22 | 23 | cifar100_test_loader = get_test_dataloader( 24 | settings.CIFAR100_TRAIN_MEAN, 25 | settings.CIFAR100_TRAIN_STD, 26 | #settings.CIFAR100_PATH, 27 | num_workers=4, 28 | batch_size=args.b, 29 | ) 30 | 31 | net.load_state_dict(torch.load(args.weights)) 32 | print(net) 33 | net.eval() 34 | 35 | correct_1 = 0.0 36 | correct_5 = 0.0 37 | total = 0 38 | 39 | with torch.no_grad(): 40 | for n_iter, (image, label) in enumerate(cifar100_test_loader): 41 | print("iteration: {}\ttotal {} iterations".format(n_iter + 1, len(cifar100_test_loader))) 42 | 43 | if args.gpu: 44 | image = image.cuda() 45 | label = label.cuda() 46 | print('GPU INFO.....') 47 | print(torch.cuda.memory_summary(), end='') 48 | 49 | 50 | output = net(image) 51 | _, pred = output.topk(5, 1, largest=True, sorted=True) 52 | 53 | label = label.view(label.size(0), -1).expand_as(pred) 54 | correct = pred.eq(label).float() 55 | 56 | #compute top 5 57 | correct_5 += correct[:, :5].sum() 58 | 59 | #compute top1 60 | correct_1 += correct[:, :1].sum() 61 | 62 | if args.gpu: 63 | print('GPU INFO.....') 64 | print(torch.cuda.memory_summary(), end='') 65 | 66 | print() 67 | print("Top 1 err: ", 1 - correct_1 / len(cifar100_test_loader.dataset)) 68 | print("Top 5 err: ", 1 - correct_5 / len(cifar100_test_loader.dataset)) 69 | print("Parameter numbers: {}".format(sum(p.numel() for p in net.parameters()))) 70 | -------------------------------------------------------------------------------- /ResNet18-SPD/models/resnet50_spd.py: -------------------------------------------------------------------------------- 1 | """resnet in pytorch 2 | 3 | 4 | 5 | [1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. 6 | 7 | Deep Residual Learning for Image Recognition 8 | https://arxiv.org/abs/1512.03385v1 9 | """ 10 | 11 | import torch 12 | import torch.nn as nn 13 | 14 | 15 | class space_to_depth(nn.Module): 16 | # Changing the dimension of the Tensor 17 | def __init__(self, dimension=1): 18 | super().__init__() 19 | self.d = dimension 20 | 21 | def forward(self, x): 22 | return torch.cat([x[..., ::2, ::2], x[..., 1::2, ::2], x[..., ::2, 1::2], x[..., 1::2, 1::2]], 1) 23 | 24 | def autopad(k, p=None): # kernel, padding 25 | # Pad to 'same' 26 | if p is None: 27 | p = k // 2 if isinstance(k, int) else [x // 2 for x in k] # auto-pad 28 | return p 29 | 30 | 31 | class Conv(nn.Module): 32 | # Standard convolution 33 | def __init__(self, c1, c2, k=1, s=1, p=None, g=1, act=True): # ch_in, ch_out, kernel, stride, padding, groups 34 | super().__init__() 35 | self.conv = nn.Conv2d(c1, c2, k, s, autopad(k, p), groups=g, bias=False) 36 | self.bn = nn.BatchNorm2d(c2) 37 | self.act = nn.SiLU() if act is True else (act if isinstance(act, nn.Module) else nn.Identity()) 38 | 39 | def forward(self, x): 40 | return self.act(self.bn(self.conv(x))) 41 | 42 | def forward_fuse(self, x): 43 | return self.act(self.conv(x)) 44 | 45 | 46 | class Focus(nn.Module): 47 | # Focus wh information into c-space 48 | def __init__(self, c1, c2, k=1, s=1, p=None, g=1, act=True): # ch_in, ch_out, kernel, stride, padding, groups 49 | super().__init__() 50 | self.conv = Conv(c1 * 4, c2, k, s, p, g, act) 51 | # self.contract = Contract(gain=2) 52 | 53 | def forward(self, x): # x(b,c,w,h) -> y(b,4c,w/2,h/2) 54 | return self.conv(torch.cat([x[..., ::2, ::2], x[..., 1::2, ::2], x[..., ::2, 1::2], x[..., 1::2, 1::2]], 1)) 55 | # return self.conv(self.contract(x)) 56 | 57 | 58 | class BasicBlock(nn.Module): 59 | """Basic Block for resnet 18 and resnet 34 60 | 61 | """ 62 | 63 | #BasicBlock and BottleNeck block 64 | #have different output size 65 | #we use class attribute expansion 66 | #to distinct 67 | expansion = 1 68 | 69 | def __init__(self, in_channels, out_channels, stride=1): 70 | super().__init__() 71 | 72 | #residual function 73 | 74 | # residual function 75 | 76 | if stride ==2: 77 | layers = [ 78 | nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=1, padding=1, bias=False), 79 | space_to_depth(), 80 | nn.BatchNorm2d(4*out_channels), 81 | nn.ReLU(inplace=True), 82 | nn.Conv2d(4*out_channels, out_channels * BasicBlock.expansion, kernel_size=3, padding=1, bias=False), 83 | nn.BatchNorm2d(out_channels * BasicBlock.expansion) 84 | ] 85 | else: 86 | layers = [ 87 | nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=stride, padding=1, bias=False), 88 | nn.BatchNorm2d(out_channels), 89 | nn.ReLU(inplace=True), 90 | nn.Conv2d(out_channels, out_channels * BasicBlock.expansion, kernel_size=3, padding=1, bias=False), 91 | nn.BatchNorm2d(out_channels * BasicBlock.expansion) 92 | ] 93 | 94 | #shortcut 95 | self.residual_function = torch.nn.Sequential(*layers) 96 | self.shortcut = nn.Sequential() 97 | 98 | #the shortcut output dimension is not the same with residual function 99 | #use 1*1 convolution to match the dimension 100 | if stride != 1 or in_channels != BasicBlock.expansion * out_channels: 101 | self.shortcut = nn.Sequential( 102 | nn.Conv2d(in_channels, out_channels * BasicBlock.expansion, kernel_size=1, stride=stride, bias=False), 103 | nn.BatchNorm2d(out_channels * BasicBlock.expansion) 104 | ) 105 | 106 | def forward(self, x): 107 | return nn.ReLU(inplace=True)(self.residual_function(x) + self.shortcut(x)) 108 | 109 | 110 | 111 | class BottleNeck(nn.Module): 112 | """Residual block for resnet over 50 layers 113 | 114 | """ 115 | expansion = 4 116 | def __init__(self, in_channels, out_channels, stride=1): 117 | super().__init__() 118 | layers = [ 119 | nn.Conv2d(in_channels, out_channels, kernel_size=1, bias = False), 120 | nn.BatchNorm2d(out_channels), 121 | nn.ReLU(inplace=True), 122 | ] 123 | 124 | if stride ==2: 125 | 126 | layers2 = [ 127 | nn.Conv2d(out_channels, out_channels,stride= 1, kernel_size=3, padding=1, bias= False), 128 | space_to_depth(), # the output of this will result in 4*out_channels 129 | nn.BatchNorm2d(4*out_channels), 130 | nn.ReLU(inplace=True), 131 | 132 | nn.Conv2d(4*out_channels, out_channels* BottleNeck.expansion, kernel_size=1, bias = False), 133 | nn.BatchNorm2d(out_channels * BottleNeck.expansion), 134 | ] 135 | 136 | else: 137 | 138 | layers2 = [ 139 | nn.Conv2d(out_channels, out_channels,stride= stride, kernel_size=3, padding=1, bias= False), 140 | nn.BatchNorm2d(out_channels), 141 | nn.ReLU(inplace=True), 142 | 143 | nn.Conv2d(out_channels, out_channels* BottleNeck.expansion, kernel_size=1, bias = False), 144 | nn.BatchNorm2d(out_channels * BottleNeck.expansion), 145 | ] 146 | 147 | layers.extend(layers2) 148 | 149 | self.residual_function = torch.nn.Sequential(*layers) 150 | 151 | 152 | # self.residual_function = nn.Sequential( 153 | # nn.Conv2d(in_channels, out_channels, kernel_size=1, bias=False), 154 | # nn.BatchNorm2d(out_channels), 155 | # nn.ReLU(inplace=True), 156 | # nn.Conv2d(out_channels, out_channels, stride=1, kernel_size=3, padding=1, bias=False), 157 | # space_to_depth(), # the output of this will result in 4*out_channels 158 | # nn.BatchNorm2d(4*out_channels), 159 | # nn.ReLU(inplace=True), 160 | # nn.Conv2d(4*out_channels, out_channels * BottleNeck.expansion, kernel_size=1, bias=False), 161 | # nn.BatchNorm2d(out_channels * BottleNeck.expansion), 162 | # ) 163 | 164 | self.shortcut = nn.Sequential() 165 | 166 | if stride != 1 or in_channels != out_channels * BottleNeck.expansion: 167 | self.shortcut = nn.Sequential( 168 | nn.Conv2d(in_channels, out_channels * BottleNeck.expansion, stride=stride, kernel_size=1, bias=False), 169 | nn.BatchNorm2d(out_channels * BottleNeck.expansion) 170 | ) 171 | 172 | def forward(self, x): 173 | return nn.ReLU(inplace=True)(self.residual_function(x) + self.shortcut(x)) 174 | 175 | class ResNet(nn.Module): 176 | 177 | def __init__(self, block, num_block, num_classes=200): 178 | super().__init__() 179 | 180 | self.in_channels = 64 181 | 182 | # self.conv1 = nn.Sequential( 183 | # nn.Conv2d(3, 64, kernel_size=3, padding=1, bias=False), 184 | # nn.BatchNorm2d(64), 185 | # nn.ReLU(inplace=True)) 186 | 187 | self.conv1 = Focus(3, 64, k=1,s=1) 188 | 189 | 190 | 191 | #we use a different inputsize than the original paper 192 | #so conv2_x's stride is 1 193 | self.conv2_x = self._make_layer(block, 64, num_block[0], 1) # Here in_channels = 64, and num_block[0] = 64 and s = 1 194 | self.conv3_x = self._make_layer(block, 128, num_block[1], 2) 195 | self.conv4_x = self._make_layer(block, 256, num_block[2], 2) 196 | self.conv5_x = self._make_layer(block, 512, num_block[3], 2) 197 | self.avg_pool = nn.AdaptiveAvgPool2d((1, 1)) 198 | self.fc = nn.Linear(512 * block.expansion, num_classes) 199 | 200 | def _make_layer(self, block, out_channels, num_blocks, stride): 201 | """make resnet layers(by layer i didnt mean this 'layer' was the 202 | same as a neuron netowork layer, ex. conv layer), one layer may 203 | contain more than one residual block 204 | 205 | Args: 206 | block: block type, basic block or bottle neck block 207 | out_channels: output depth channel number of this layer 208 | num_blocks: how many blocks per layer 209 | stride: the stride of the first block of this layer 210 | 211 | Return: 212 | return a resnet layer 213 | """ 214 | 215 | # we have num_block blocks per layer, the first block 216 | # could be 1 or 2, other blocks would always be 1 217 | strides = [stride] + [1] * (num_blocks - 1) 218 | layers = [] 219 | for stride in strides: 220 | layers.append(block(self.in_channels, out_channels, stride)) 221 | self.in_channels = out_channels * block.expansion 222 | 223 | return nn.Sequential(*layers) 224 | 225 | def forward(self, x): 226 | output = self.conv1(x) 227 | output = self.conv2_x(output) 228 | output = self.conv3_x(output) 229 | output = self.conv4_x(output) 230 | output = self.conv5_x(output) 231 | output = self.avg_pool(output) 232 | output = output.view(output.size(0), -1) 233 | output = self.fc(output) 234 | 235 | return output 236 | 237 | 238 | def resnet50(): 239 | """ return a ResNet 50 object 240 | """ 241 | return ResNet(BottleNeck, [3, 4, 6, 3]) 242 | 243 | def resnet101(): 244 | """ return a ResNet 101 object 245 | """ 246 | return ResNet(BottleNeck, [3, 4, 23, 3]) 247 | 248 | def resnet152(): 249 | """ return a ResNet 152 object 250 | """ 251 | return ResNet(BottleNeck, [3, 8, 36, 3]) 252 | 253 | 254 | def resnet18(): 255 | """ 256 | return a ResNet 18 object 257 | """ 258 | return ResNet(BottleNeck,[3,4,6,3]) 259 | 260 | 261 | #if __name__ =="__main__": 262 | # net = resnet50() 263 | # x = torch.empty((2,3,112,112)).normal_() 264 | # print(net(x).shape) 265 | 266 | 267 | 268 | -------------------------------------------------------------------------------- /ResNet18-SPD/test.py: -------------------------------------------------------------------------------- 1 | #test.py 2 | #!/usr/bin/env python3 3 | 4 | """ test neuron network performace 5 | print top1 and top5 err on test dataset 6 | of a model 7 | 8 | author baiyu 9 | """ 10 | 11 | import argparse 12 | 13 | from matplotlib import pyplot as plt 14 | 15 | import torch 16 | import torchvision.transforms as transforms 17 | from torch.utils.data import DataLoader 18 | 19 | from conf import settings 20 | from utils import get_network, get_test_dataloader 21 | 22 | from tiny_imagenet_loader import tiny_imagenet_test_loader 23 | 24 | if __name__ == '__main__': 25 | 26 | parser = argparse.ArgumentParser() 27 | parser.add_argument('-net', type=str, required=True, help='net type') 28 | parser.add_argument('-weights', type=str, required=True, help='the weights file you want to test') 29 | parser.add_argument('-gpu', action='store_true', default=False, help='use gpu or not') 30 | parser.add_argument('-b', type=int, default=16, help='batch size for dataloader') 31 | args = parser.parse_args() 32 | 33 | net = get_network(args) 34 | 35 | 36 | tiny_imagenet_data = tiny_imagenet_test_loader(args.b) 37 | 38 | net.load_state_dict(torch.load(args.weights,map_location=torch.device('cpu'))) 39 | print(net) 40 | net.eval() 41 | 42 | correct_1 = 0.0 43 | correct_5 = 0.0 44 | total = 0 45 | 46 | with torch.no_grad(): 47 | for n_iter, (image, label) in enumerate(tiny_imagenet_data): 48 | print("iteration: {}\ttotal {} iterations".format(n_iter + 1, len(tiny_imagenet_data))) 49 | 50 | if args.gpu: 51 | image = image.cuda() 52 | label = label.cuda() 53 | print('GPU INFO.....') 54 | print(torch.cuda.memory_summary(), end='') 55 | 56 | 57 | output = net(image) 58 | _, pred = output.topk(5, 1, largest=True, sorted=True) 59 | 60 | label = label.view(label.size(0), -1).expand_as(pred) 61 | correct = pred.eq(label).float() 62 | 63 | #compute top 5 64 | correct_5 += correct[:, :5].sum() 65 | 66 | #compute top1 67 | correct_1 += correct[:, :1].sum() 68 | 69 | if args.gpu: 70 | print('GPU INFO.....') 71 | print(torch.cuda.memory_summary(), end='') 72 | 73 | print() 74 | print("Top 1 err: ", 1 - correct_1 / len(tiny_imagenet_data.dataset)) 75 | print("Top 5 err: ", 1 - correct_5 / len(tiny_imagenet_data.dataset)) 76 | print("Parameter numbers: {}".format(sum(p.numel() for p in net.parameters()))) 77 | -------------------------------------------------------------------------------- /ResNet18-SPD/tiny_imagenet_loader.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """TinyImageNetLoader.ipynb 3 | Automatically generated by Colaboratory. 4 | """ 5 | 6 | 7 | import torch 8 | import torchvision 9 | import torchvision.transforms as transforms 10 | from torch.utils.data import DataLoader 11 | from torch.utils.data import Dataset 12 | import os, glob 13 | from torchvision.io import read_image, ImageReadMode 14 | 15 | batch_size = 64 16 | 17 | id_dict = {} 18 | for i, line in enumerate(open('./tiny-imagenet-200/wnids.txt', 'r')): 19 | id_dict[line.replace('\n', '')] = i 20 | 21 | class TrainTinyImageNetDataset(Dataset): 22 | def __init__(self, id, transform=None): 23 | self.filenames = glob.glob("./tiny-imagenet-200/train/*/*/*.JPEG") 24 | self.transform = transform 25 | self.id_dict = id 26 | 27 | def __len__(self): 28 | return len(self.filenames) 29 | 30 | def __getitem__(self, idx): 31 | img_path = self.filenames[idx] 32 | image = read_image(img_path) 33 | # print(image.shape) 34 | if image.shape[0] == 1: 35 | image = read_image(img_path,ImageReadMode.RGB) 36 | label = self.id_dict[img_path.split('/')[3]] 37 | if self.transform: 38 | image = self.transform(image.type(torch.FloatTensor)) 39 | return image, label 40 | 41 | class TestTinyImageNetDataset(Dataset): 42 | def __init__(self, id, transform=None): 43 | self.filenames = glob.glob("./tiny-imagenet-200/val/images/*.JPEG") 44 | self.transform = transform 45 | self.id_dict = id 46 | self.cls_dic = {} 47 | for i, line in enumerate(open('./tiny-imagenet-200/val/val_annotations.txt', 'r')): 48 | a = line.split('\t') 49 | img, cls_id = a[0],a[1] 50 | self.cls_dic[img] = self.id_dict[cls_id] 51 | 52 | 53 | def __len__(self): 54 | return len(self.filenames) 55 | 56 | def __getitem__(self, idx): 57 | img_path = self.filenames[idx] 58 | image = read_image(img_path) 59 | if image.shape[0] == 1: 60 | image = read_image(img_path,ImageReadMode.RGB) 61 | label = self.cls_dic[img_path.split('/')[-1]] 62 | if self.transform: 63 | image = self.transform(image.type(torch.FloatTensor)) 64 | return image, label 65 | 66 | transform = transforms.Normalize((122.4786, 114.2755, 101.3963), (70.4924, 68.5679, 71.8127)) 67 | 68 | train_transform = transforms.Compose([ 69 | transforms.RandomHorizontalFlip(), 70 | transforms.RandomCrop(64, padding=4), 71 | # transforms.ToTensor(), 72 | # transforms.RandomAffine(30), 73 | transforms.RandomVerticalFlip(), 74 | # transforms.RandomPerspective(distortion_scale=0.6, p=1.0), 75 | transforms.Normalize((122.4786, 114.2755, 101.3963), (70.4924, 68.5679, 71.8127)) 76 | ]) 77 | 78 | trainset = TrainTinyImageNetDataset(id=id_dict, transform = train_transform) 79 | #trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True, num_workers=2) 80 | 81 | test_transform = transforms.Compose( 82 | [transforms.Normalize((122.4786, 114.2755, 101.3963), (70.4924, 68.5679, 71.8127))] 83 | ) 84 | 85 | testset = TestTinyImageNetDataset(id=id_dict, transform=test_transform) 86 | #testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size, shuffle=False, num_workers=2) 87 | 88 | 89 | def tiny_imagenet_train_loader(batch_size): 90 | return torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True, num_workers=2) 91 | 92 | 93 | 94 | def tiny_imagenet_test_loader(batch_size): 95 | return torch.utils.data.DataLoader(testset, batch_size=batch_size, shuffle=False, num_workers=2) 96 | 97 | 98 | 99 | -------------------------------------------------------------------------------- /ResNet18-SPD/train_tiny.py: -------------------------------------------------------------------------------- 1 | # train.py 2 | #!/usr/bin/env python3 3 | 4 | """ train network using pytorch 5 | 6 | author baiyu 7 | """ 8 | from ptflops import get_model_complexity_info 9 | import os 10 | import sys 11 | import argparse 12 | import time 13 | from datetime import datetime 14 | 15 | import numpy as np 16 | import torch 17 | import torch.nn as nn 18 | import torch.optim as optim 19 | import torchvision 20 | import torchvision.transforms as transforms 21 | 22 | from torch.utils.data import DataLoader 23 | from torch.utils.tensorboard import SummaryWriter 24 | 25 | import wandb 26 | #wandb.init() 27 | 28 | from conf import settings 29 | from utils import get_network, WarmUpLR, \ 30 | most_recent_folder, most_recent_weights, last_epoch, best_acc_weights 31 | 32 | from tiny_imagenet_loader import tiny_imagenet_train_loader, tiny_imagenet_test_loader 33 | 34 | def train(epoch): 35 | 36 | start = time.time() 37 | net.train() 38 | for batch_index, (images, labels) in enumerate(cifar100_training_loader): 39 | print(images.shape) 40 | 41 | if args.gpu: 42 | labels = labels.cuda() 43 | images = images.cuda() 44 | 45 | optimizer.zero_grad() 46 | outputs = net(images) 47 | loss = loss_function(outputs, labels) 48 | loss.backward() 49 | optimizer.step() 50 | 51 | n_iter = (epoch - 1) * len(cifar100_training_loader) + batch_index + 1 52 | 53 | last_layer = list(net.children())[-1] 54 | for name, para in last_layer.named_parameters(): 55 | if 'weight' in name: 56 | writer.add_scalar('LastLayerGradients/grad_norm2_weights', para.grad.norm(), n_iter) 57 | if 'bias' in name: 58 | writer.add_scalar('LastLayerGradients/grad_norm2_bias', para.grad.norm(), n_iter) 59 | 60 | print('Training Epoch: {epoch} [{trained_samples}/{total_samples}]\tLoss: {:0.4f}\tLR: {:0.6f}'.format( 61 | loss.item(), 62 | optimizer.param_groups[0]['lr'], 63 | epoch=epoch, 64 | trained_samples=batch_index * args.b + len(images), 65 | total_samples=len(cifar100_training_loader.dataset) 66 | )) 67 | print("loss",loss.item()) 68 | wandb.log({'loss': loss.item()}) 69 | 70 | #update training loss for each iteration 71 | writer.add_scalar('Train/loss', loss.item(), n_iter) 72 | 73 | if epoch <= args.warm: 74 | warmup_scheduler.step() 75 | 76 | for name, param in net.named_parameters(): 77 | layer, attr = os.path.splitext(name) 78 | attr = attr[1:] 79 | writer.add_histogram("{}/{}".format(layer, attr), param, epoch) 80 | 81 | finish = time.time() 82 | 83 | print('epoch {} training time consumed: {:.2f}s'.format(epoch, finish - start)) 84 | 85 | 86 | @torch.no_grad() 87 | def eval_training(epoch=0, tb=True): 88 | 89 | start = time.time() 90 | net.eval() 91 | 92 | test_loss = 0.0 # cost function error 93 | correct = 0.0 94 | 95 | for (images, labels) in cifar100_test_loader: 96 | 97 | if args.gpu: 98 | images = images.cuda() 99 | labels = labels.cuda() 100 | 101 | outputs = net(images) 102 | loss = loss_function(outputs, labels) 103 | 104 | test_loss += loss.item() 105 | _, preds = outputs.max(1) 106 | correct += preds.eq(labels).sum() 107 | 108 | finish = time.time() 109 | if args.gpu: 110 | print('GPU INFO.....') 111 | # print(torch.cuda.memory_summary(), end='') 112 | print('Evaluating Network.....') 113 | print('Test set: Epoch: {}, Average loss: {:.4f}, Accuracy: {:.4f}, Time consumed:{:.2f}s'.format( 114 | epoch, 115 | test_loss / len(cifar100_test_loader.dataset), 116 | correct.float() / len(cifar100_test_loader.dataset), 117 | finish - start 118 | )) 119 | 120 | wandb.log({'Test loss': test_loss / len(cifar100_test_loader.dataset)}) 121 | wandb.log({'Test Accuracy':correct.float() / len(cifar100_test_loader.dataset)}) 122 | print() 123 | 124 | #add informations to tensorboard 125 | if tb: 126 | writer.add_scalar('Test/Average loss', test_loss / len(cifar100_test_loader.dataset), epoch) 127 | 128 | writer.add_scalar('Test/Accuracy', correct.float() / len(cifar100_test_loader.dataset), epoch) 129 | 130 | return correct.float() / len(cifar100_test_loader.dataset) 131 | 132 | if __name__ == '__main__': 133 | 134 | parser = argparse.ArgumentParser() 135 | parser.add_argument('-net', type=str, required=True, help='net type') 136 | parser.add_argument('-project', type=str, required = False, default = None) 137 | parser.add_argument('-name', type=str, required = False, default = None) 138 | parser.add_argument('-gpu', action='store_true', default=False, help='use gpu or not') 139 | parser.add_argument('-b', type=int, default=128, help='batch size for dataloader') 140 | parser.add_argument('-warm', type=int, default=1, help='warm up training phase') 141 | parser.add_argument('-lr', type=float, default=0.1, help='initial learning rate') 142 | parser.add_argument('-resume', action='store_true', default=False, help='resume training') 143 | parser.add_argument('-momentum',type=float, default=0.9) 144 | parser.add_argument('-weight_decay',type=float, default =0.05) 145 | args = parser.parse_args() 146 | 147 | net = get_network(args) 148 | 149 | macs, params = get_model_complexity_info(net, (3, 64, 64), as_strings=True, 150 | print_per_layer_stat=True, verbose=True) 151 | print('{:<30} {:<8}'.format('Computational complexity: ', macs)) 152 | print('{:<30} {:<8}'.format('Number of parameters: ', params)) 153 | 154 | #data preprocessing: 155 | cifar100_training_loader = tiny_imagenet_train_loader(args.b) 156 | 157 | wandb.init(project = args.project, name = args.name) 158 | 159 | cifar100_test_loader = tiny_imagenet_test_loader(args.b) 160 | 161 | loss_function = nn.CrossEntropyLoss() 162 | optimizer = optim.SGD(net.parameters(), lr=args.lr, momentum=args.momentum, weight_decay=args.weight_decay) 163 | # optimizer = optim.Adam(net.parameters(), 164 | # lr=args.lr) 165 | train_scheduler = optim.lr_scheduler.MultiStepLR(optimizer, milestones=settings.MILESTONES, gamma=0.2) #learning rate decay 166 | iter_per_epoch = len(cifar100_training_loader) 167 | warmup_scheduler = WarmUpLR(optimizer, iter_per_epoch * args.warm) 168 | 169 | if args.resume: 170 | recent_folder = most_recent_folder(os.path.join(settings.CHECKPOINT_PATH, args.net), fmt=settings.DATE_FORMAT) 171 | if not recent_folder: 172 | raise Exception('no recent folder were found') 173 | 174 | checkpoint_path = os.path.join(settings.CHECKPOINT_PATH, args.net, recent_folder) 175 | 176 | else: 177 | checkpoint_path = os.path.join(settings.CHECKPOINT_PATH, args.net, settings.TIME_NOW) 178 | 179 | #use tensorboard 180 | if not os.path.exists(settings.LOG_DIR): 181 | os.mkdir(settings.LOG_DIR) 182 | 183 | #since tensorboard can't overwrite old values 184 | #so the only way is to create a new tensorboard log 185 | writer = SummaryWriter(log_dir=os.path.join( 186 | settings.LOG_DIR, args.net, settings.TIME_NOW)) 187 | input_tensor = torch.Tensor(1, 3, 32, 32) 188 | if args.gpu: 189 | input_tensor = input_tensor.cuda() 190 | writer.add_graph(net, input_tensor) 191 | 192 | #create checkpoint folder to save model 193 | if not os.path.exists(checkpoint_path): 194 | os.makedirs(checkpoint_path) 195 | checkpoint_path = os.path.join(checkpoint_path, '{net}-{epoch}-{type}.pth') 196 | 197 | best_acc = 0.0 198 | if args.resume: 199 | best_weights = best_acc_weights(os.path.join(settings.CHECKPOINT_PATH, args.net, recent_folder)) 200 | if best_weights: 201 | weights_path = os.path.join(settings.CHECKPOINT_PATH, args.net, recent_folder, best_weights) 202 | print('found best acc weights file:{}'.format(weights_path)) 203 | print('load best training file to test acc...') 204 | net.load_state_dict(torch.load(weights_path)) 205 | best_acc = eval_training(tb=False) 206 | print('best acc is {:0.2f}'.format(best_acc)) 207 | 208 | recent_weights_file = most_recent_weights(os.path.join(settings.CHECKPOINT_PATH, args.net, recent_folder)) 209 | if not recent_weights_file: 210 | raise Exception('no recent weights file were found') 211 | weights_path = os.path.join(settings.CHECKPOINT_PATH, args.net, recent_folder, recent_weights_file) 212 | print('loading weights file {} to resume training.....'.format(weights_path)) 213 | net.load_state_dict(torch.load(weights_path)) 214 | 215 | resume_epoch = last_epoch(os.path.join(settings.CHECKPOINT_PATH, args.net, recent_folder)) 216 | 217 | 218 | for epoch in range(1, settings.EPOCH + 1): 219 | if epoch > args.warm: 220 | train_scheduler.step(epoch) 221 | 222 | if args.resume: 223 | if epoch <= resume_epoch: 224 | continue 225 | 226 | train(epoch) 227 | acc = eval_training(epoch) 228 | 229 | #start to save best performance model after learning rate decay to 0.01 230 | # if epoch > settings.MILESTONES[1] and best_acc < acc: 231 | # weights_path = checkpoint_path.format(net=args.net, epoch=epoch, type='best') 232 | # print('saving weights file to {}'.format(weights_path)) 233 | # torch.save(net.state_dict(), weights_path) 234 | # best_acc = acc 235 | # continue 236 | 237 | if not epoch % settings.SAVE_EPOCH: 238 | weights_path = checkpoint_path.format(net=args.net, epoch=epoch, type='regular') 239 | print('saving weights file to {}'.format(weights_path)) 240 | torch.save(net.state_dict(), "./saved_models/best.pt") 241 | 242 | writer.close() 243 | -------------------------------------------------------------------------------- /ResNet50-SPD/conf/__init__.py: -------------------------------------------------------------------------------- 1 | """ dynamically load settings 2 | 3 | author baiyu 4 | """ 5 | import conf.global_settings as settings 6 | 7 | class Settings: 8 | def __init__(self, settings): 9 | 10 | for attr in dir(settings): 11 | if attr.isupper(): 12 | setattr(self, attr, getattr(settings, attr)) 13 | 14 | settings = Settings(settings) -------------------------------------------------------------------------------- /ResNet50-SPD/conf/global_settings.py: -------------------------------------------------------------------------------- 1 | """ configurations for this project 2 | 3 | author baiyu 4 | """ 5 | import os 6 | from datetime import datetime 7 | 8 | #CIFAR100 dataset path (python version) 9 | #CIFAR100_PATH = '/nfs/private/cifar100/cifar-100-python' 10 | 11 | #mean and std of cifar100 dataset 12 | CIFAR100_TRAIN_MEAN = (0.5070751592371323, 0.48654887331495095, 0.4409178433670343) 13 | CIFAR100_TRAIN_STD = (0.2673342858792401, 0.2564384629170883, 0.27615047132568404) 14 | 15 | #CIFAR100_TEST_MEAN = (0.5088964127604166, 0.48739301317401956, 0.44194221124387256) 16 | #CIFAR100_TEST_STD = (0.2682515741720801, 0.2573637364478126, 0.2770957707973042) 17 | 18 | #directory to save weights file 19 | CHECKPOINT_PATH = 'checkpoint' 20 | 21 | #total training epoches 22 | EPOCH = 500 23 | MILESTONES = [60, 120, 160] 24 | 25 | #initial learning rate 26 | #INIT_LR = 0.1 27 | 28 | DATE_FORMAT = '%A_%d_%B_%Y_%Hh_%Mm_%Ss' 29 | #time of we run the script 30 | TIME_NOW = datetime.now().strftime(DATE_FORMAT) 31 | 32 | #tensorboard log dir 33 | LOG_DIR = 'runs' 34 | 35 | #save weights file per SAVE_EPOCH epoch 36 | SAVE_EPOCH = 500 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | -------------------------------------------------------------------------------- /ResNet50-SPD/dataset.py: -------------------------------------------------------------------------------- 1 | """ train and test dataset 2 | 3 | author baiyu 4 | """ 5 | import os 6 | import sys 7 | import pickle 8 | 9 | from skimage import io 10 | import matplotlib.pyplot as plt 11 | import numpy 12 | import torch 13 | from torch.utils.data import Dataset 14 | 15 | class CIFAR100Train(Dataset): 16 | """cifar100 test dataset, derived from 17 | torch.utils.data.DataSet 18 | """ 19 | 20 | def __init__(self, path, transform=None): 21 | #if transform is given, we transoform data using 22 | with open(os.path.join(path, 'train'), 'rb') as cifar100: 23 | self.data = pickle.load(cifar100, encoding='bytes') 24 | self.transform = transform 25 | 26 | def __len__(self): 27 | return len(self.data['fine_labels'.encode()]) 28 | 29 | def __getitem__(self, index): 30 | label = self.data['fine_labels'.encode()][index] 31 | r = self.data['data'.encode()][index, :1024].reshape(32, 32) 32 | g = self.data['data'.encode()][index, 1024:2048].reshape(32, 32) 33 | b = self.data['data'.encode()][index, 2048:].reshape(32, 32) 34 | image = numpy.dstack((r, g, b)) 35 | 36 | if self.transform: 37 | image = self.transform(image) 38 | return label, image 39 | 40 | class CIFAR100Test(Dataset): 41 | """cifar100 test dataset, derived from 42 | torch.utils.data.DataSet 43 | """ 44 | 45 | def __init__(self, path, transform=None): 46 | with open(os.path.join(path, 'test'), 'rb') as cifar100: 47 | self.data = pickle.load(cifar100, encoding='bytes') 48 | self.transform = transform 49 | 50 | def __len__(self): 51 | return len(self.data['data'.encode()]) 52 | 53 | def __getitem__(self, index): 54 | label = self.data['fine_labels'.encode()][index] 55 | r = self.data['data'.encode()][index, :1024].reshape(32, 32) 56 | g = self.data['data'.encode()][index, 1024:2048].reshape(32, 32) 57 | b = self.data['data'.encode()][index, 2048:].reshape(32, 32) 58 | image = numpy.dstack((r, g, b)) 59 | 60 | if self.transform: 61 | image = self.transform(image) 62 | return label, image 63 | 64 | -------------------------------------------------------------------------------- /ResNet50-SPD/models/resnet.py: -------------------------------------------------------------------------------- 1 | """resnet in pytorch 2 | 3 | 4 | 5 | [1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. 6 | 7 | Deep Residual Learning for Image Recognition 8 | https://arxiv.org/abs/1512.03385v1 9 | """ 10 | 11 | import torch 12 | import torch.nn as nn 13 | 14 | class BasicBlock(nn.Module): 15 | """Basic Block for resnet 18 and resnet 34 16 | 17 | """ 18 | 19 | #BasicBlock and BottleNeck block 20 | #have different output size 21 | #we use class attribute expansion 22 | #to distinct 23 | expansion = 1 24 | 25 | def __init__(self, in_channels, out_channels, stride=1): 26 | super().__init__() 27 | 28 | #residual function 29 | self.residual_function = nn.Sequential( 30 | nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=stride, padding=1, bias=False), 31 | nn.BatchNorm2d(out_channels), 32 | nn.ReLU(inplace=True), 33 | nn.Conv2d(out_channels, out_channels * BasicBlock.expansion, kernel_size=3, padding=1, bias=False), 34 | nn.BatchNorm2d(out_channels * BasicBlock.expansion) 35 | ) 36 | 37 | #shortcut 38 | self.shortcut = nn.Sequential() 39 | 40 | #the shortcut output dimension is not the same with residual function 41 | #use 1*1 convolution to match the dimension 42 | if stride != 1 or in_channels != BasicBlock.expansion * out_channels: 43 | self.shortcut = nn.Sequential( 44 | nn.Conv2d(in_channels, out_channels * BasicBlock.expansion, kernel_size=1, stride=stride, bias=False), 45 | nn.BatchNorm2d(out_channels * BasicBlock.expansion) 46 | ) 47 | 48 | def forward(self, x): 49 | return nn.ReLU(inplace=True)(self.residual_function(x) + self.shortcut(x)) 50 | 51 | class BottleNeck(nn.Module): 52 | """Residual block for resnet over 50 layers 53 | 54 | """ 55 | expansion = 4 56 | def __init__(self, in_channels, out_channels, stride=1): 57 | super().__init__() 58 | self.residual_function = nn.Sequential( 59 | nn.Conv2d(in_channels, out_channels, kernel_size=1, bias=False), 60 | nn.BatchNorm2d(out_channels), 61 | nn.ReLU(inplace=True), 62 | nn.Conv2d(out_channels, out_channels, stride=stride, kernel_size=3, padding=1, bias=False), 63 | nn.BatchNorm2d(out_channels), 64 | nn.ReLU(inplace=True), 65 | nn.Conv2d(out_channels, out_channels * BottleNeck.expansion, kernel_size=1, bias=False), 66 | nn.BatchNorm2d(out_channels * BottleNeck.expansion), 67 | ) 68 | 69 | self.shortcut = nn.Sequential() 70 | 71 | if stride != 1 or in_channels != out_channels * BottleNeck.expansion: 72 | self.shortcut = nn.Sequential( 73 | nn.Conv2d(in_channels, out_channels * BottleNeck.expansion, stride=stride, kernel_size=1, bias=False), 74 | nn.BatchNorm2d(out_channels * BottleNeck.expansion) 75 | ) 76 | 77 | def forward(self, x): 78 | return nn.ReLU(inplace=True)(self.residual_function(x) + self.shortcut(x)) 79 | 80 | class ResNet(nn.Module): 81 | 82 | def __init__(self, block, num_block, num_classes=100): 83 | super().__init__() 84 | 85 | self.in_channels = 64 86 | 87 | self.conv1 = nn.Sequential( 88 | nn.Conv2d(3, 64, kernel_size=3, padding=1, bias=False), 89 | nn.BatchNorm2d(64), 90 | nn.ReLU(inplace=True)) 91 | #we use a different inputsize than the original paper 92 | #so conv2_x's stride is 1 93 | self.conv2_x = self._make_layer(block, 64, num_block[0], 1) 94 | self.conv3_x = self._make_layer(block, 128, num_block[1], 2) 95 | self.conv4_x = self._make_layer(block, 256, num_block[2], 2) 96 | self.conv5_x = self._make_layer(block, 512, num_block[3], 2) 97 | self.avg_pool = nn.AdaptiveAvgPool2d((1, 1)) 98 | self.fc = nn.Linear(512 * block.expansion, num_classes) 99 | 100 | def _make_layer(self, block, out_channels, num_blocks, stride): 101 | """make resnet layers(by layer i didnt mean this 'layer' was the 102 | same as a neuron netowork layer, ex. conv layer), one layer may 103 | contain more than one residual block 104 | 105 | Args: 106 | block: block type, basic block or bottle neck block 107 | out_channels: output depth channel number of this layer 108 | num_blocks: how many blocks per layer 109 | stride: the stride of the first block of this layer 110 | 111 | Return: 112 | return a resnet layer 113 | """ 114 | 115 | # we have num_block blocks per layer, the first block 116 | # could be 1 or 2, other blocks would always be 1 117 | strides = [stride] + [1] * (num_blocks - 1) 118 | layers = [] 119 | for stride in strides: 120 | layers.append(block(self.in_channels, out_channels, stride)) 121 | self.in_channels = out_channels * block.expansion 122 | 123 | return nn.Sequential(*layers) 124 | 125 | def forward(self, x): 126 | output = self.conv1(x) 127 | output = self.conv2_x(output) 128 | output = self.conv3_x(output) 129 | output = self.conv4_x(output) 130 | output = self.conv5_x(output) 131 | output = self.avg_pool(output) 132 | output = output.view(output.size(0), -1) 133 | output = self.fc(output) 134 | 135 | return output 136 | 137 | def resnet18(): 138 | """ return a ResNet 18 object 139 | """ 140 | return ResNet(BasicBlock, [2, 2, 2, 2]) 141 | 142 | def resnet34(): 143 | """ return a ResNet 34 object 144 | """ 145 | return ResNet(BasicBlock, [3, 4, 6, 3]) 146 | 147 | def resnet50(): 148 | """ return a ResNet 50 object 149 | """ 150 | return ResNet(BottleNeck, [3, 4, 6, 3]) 151 | 152 | def resnet101(): 153 | """ return a ResNet 101 object 154 | """ 155 | return ResNet(BottleNeck, [3, 4, 23, 3]) 156 | 157 | def resnet152(): 158 | """ return a ResNet 152 object 159 | """ 160 | return ResNet(BottleNeck, [3, 8, 36, 3]) 161 | 162 | 163 | 164 | -------------------------------------------------------------------------------- /ResNet50-SPD/models/resnet50_spd.py: -------------------------------------------------------------------------------- 1 | """resnet in pytorch 2 | 3 | 4 | 5 | [1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. 6 | 7 | Deep Residual Learning for Image Recognition 8 | https://arxiv.org/abs/1512.03385v1 9 | """ 10 | 11 | import torch 12 | import torch.nn as nn 13 | 14 | 15 | class space_to_depth(nn.Module): 16 | # Changing the dimension of the Tensor 17 | def __init__(self, dimension=1): 18 | super().__init__() 19 | self.d = dimension 20 | 21 | def forward(self, x): 22 | return torch.cat([x[..., ::2, ::2], x[..., 1::2, ::2], x[..., ::2, 1::2], x[..., 1::2, 1::2]], 1) 23 | 24 | def autopad(k, p=None): # kernel, padding 25 | # Pad to 'same' 26 | if p is None: 27 | p = k // 2 if isinstance(k, int) else [x // 2 for x in k] # auto-pad 28 | return p 29 | 30 | 31 | class Conv(nn.Module): 32 | # Standard convolution 33 | def __init__(self, c1, c2, k=1, s=1, p=None, g=1, act=True): # ch_in, ch_out, kernel, stride, padding, groups 34 | super().__init__() 35 | self.conv = nn.Conv2d(c1, c2, k, s, autopad(k, p), groups=g, bias=False) 36 | self.bn = nn.BatchNorm2d(c2) 37 | self.act = nn.SiLU() if act is True else (act if isinstance(act, nn.Module) else nn.Identity()) 38 | 39 | def forward(self, x): 40 | return self.act(self.bn(self.conv(x))) 41 | 42 | def forward_fuse(self, x): 43 | return self.act(self.conv(x)) 44 | 45 | 46 | class Focus(nn.Module): 47 | # Focus wh information into c-space 48 | def __init__(self, c1, c2, k=1, s=1, p=None, g=1, act=True): # ch_in, ch_out, kernel, stride, padding, groups 49 | super().__init__() 50 | self.conv = Conv(c1 * 4, c2, k, s, p, g, act) 51 | # self.contract = Contract(gain=2) 52 | 53 | def forward(self, x): # x(b,c,w,h) -> y(b,4c,w/2,h/2) 54 | return self.conv(torch.cat([x[..., ::2, ::2], x[..., 1::2, ::2], x[..., ::2, 1::2], x[..., 1::2, 1::2]], 1)) 55 | # return self.conv(self.contract(x)) 56 | 57 | 58 | class BottleNeck(nn.Module): 59 | """Residual block for resnet over 50 layers 60 | 61 | """ 62 | expansion = 4 63 | def __init__(self, in_channels, out_channels, stride=1): 64 | super().__init__() 65 | layers = [ 66 | nn.Conv2d(in_channels, out_channels, kernel_size=1, bias = False), 67 | nn.BatchNorm2d(out_channels), 68 | nn.ReLU(inplace=True), 69 | ] 70 | 71 | if stride ==2: 72 | 73 | layers2 = [ 74 | nn.Conv2d(out_channels, out_channels,stride= 1, kernel_size=3, padding=1, bias= False), 75 | space_to_depth(), # the output of this will result in 4*out_channels 76 | nn.BatchNorm2d(4*out_channels), 77 | nn.ReLU(inplace=True), 78 | 79 | nn.Conv2d(4*out_channels, out_channels* BottleNeck.expansion, kernel_size=1, bias = False), 80 | nn.BatchNorm2d(out_channels * BottleNeck.expansion), 81 | ] 82 | 83 | else: 84 | 85 | layers2 = [ 86 | nn.Conv2d(out_channels, out_channels,stride= stride, kernel_size=3, padding=1, bias= False), 87 | nn.BatchNorm2d(out_channels), 88 | nn.ReLU(inplace=True), 89 | 90 | nn.Conv2d(out_channels, out_channels* BottleNeck.expansion, kernel_size=1, bias = False), 91 | nn.BatchNorm2d(out_channels * BottleNeck.expansion), 92 | ] 93 | 94 | layers.extend(layers2) 95 | 96 | self.residual_function = torch.nn.Sequential(*layers) 97 | 98 | 99 | # self.residual_function = nn.Sequential( 100 | # nn.Conv2d(in_channels, out_channels, kernel_size=1, bias=False), 101 | # nn.BatchNorm2d(out_channels), 102 | # nn.ReLU(inplace=True), 103 | # nn.Conv2d(out_channels, out_channels, stride=1, kernel_size=3, padding=1, bias=False), 104 | # space_to_depth(), # the output of this will result in 4*out_channels 105 | # nn.BatchNorm2d(4*out_channels), 106 | # nn.ReLU(inplace=True), 107 | # nn.Conv2d(4*out_channels, out_channels * BottleNeck.expansion, kernel_size=1, bias=False), 108 | # nn.BatchNorm2d(out_channels * BottleNeck.expansion), 109 | # ) 110 | 111 | self.shortcut = nn.Sequential() 112 | 113 | if stride != 1 or in_channels != out_channels * BottleNeck.expansion: 114 | self.shortcut = nn.Sequential( 115 | nn.Conv2d(in_channels, out_channels * BottleNeck.expansion, stride=stride, kernel_size=1, bias=False), 116 | nn.BatchNorm2d(out_channels * BottleNeck.expansion) 117 | ) 118 | 119 | def forward(self, x): 120 | return nn.ReLU(inplace=True)(self.residual_function(x) + self.shortcut(x)) 121 | 122 | class ResNet(nn.Module): 123 | 124 | def __init__(self, block, num_block, num_classes=100): 125 | super().__init__() 126 | 127 | self.in_channels = 64 128 | 129 | # self.conv1 = nn.Sequential( 130 | # nn.Conv2d(3, 64, kernel_size=3, padding=1, bias=False), 131 | # nn.BatchNorm2d(64), 132 | # nn.ReLU(inplace=True)) 133 | 134 | self.conv1 = Focus(3, 64, k=1,s=1) 135 | 136 | 137 | 138 | #we use a different inputsize than the original paper 139 | #so conv2_x's stride is 1 140 | self.conv2_x = self._make_layer(block, 64, num_block[0], 1) # Here in_channels = 64, and num_block[0] = 64 and s = 1 141 | self.conv3_x = self._make_layer(block, 128, num_block[1], 2) 142 | self.conv4_x = self._make_layer(block, 256, num_block[2], 2) 143 | self.conv5_x = self._make_layer(block, 512, num_block[3], 2) 144 | self.avg_pool = nn.AdaptiveAvgPool2d((1, 1)) 145 | self.fc = nn.Linear(512 * block.expansion, num_classes) 146 | 147 | def _make_layer(self, block, out_channels, num_blocks, stride): 148 | """make resnet layers(by layer i didnt mean this 'layer' was the 149 | same as a neuron netowork layer, ex. conv layer), one layer may 150 | contain more than one residual block 151 | 152 | Args: 153 | block: block type, basic block or bottle neck block 154 | out_channels: output depth channel number of this layer 155 | num_blocks: how many blocks per layer 156 | stride: the stride of the first block of this layer 157 | 158 | Return: 159 | return a resnet layer 160 | """ 161 | 162 | # we have num_block blocks per layer, the first block 163 | # could be 1 or 2, other blocks would always be 1 164 | strides = [stride] + [1] * (num_blocks - 1) 165 | layers = [] 166 | for stride in strides: 167 | layers.append(block(self.in_channels, out_channels, stride)) 168 | self.in_channels = out_channels * block.expansion 169 | 170 | return nn.Sequential(*layers) 171 | 172 | def forward(self, x): 173 | output = self.conv1(x) 174 | output = self.conv2_x(output) 175 | output = self.conv3_x(output) 176 | output = self.conv4_x(output) 177 | output = self.conv5_x(output) 178 | output = self.avg_pool(output) 179 | output = output.view(output.size(0), -1) 180 | output = self.fc(output) 181 | 182 | return output 183 | 184 | 185 | def resnet50(): 186 | """ return a ResNet 50 object 187 | """ 188 | return ResNet(BottleNeck, [3, 4, 6, 3]) 189 | 190 | def resnet101(): 191 | """ return a ResNet 101 object 192 | """ 193 | return ResNet(BottleNeck, [3, 4, 23, 3]) 194 | 195 | def resnet152(): 196 | """ return a ResNet 152 object 197 | """ 198 | return ResNet(BottleNeck, [3, 8, 36, 3]) 199 | 200 | 201 | 202 | if __name__ =="__main__": 203 | net = resnet50() 204 | x = torch.empty((2,3,112,112)).normal_() 205 | print(net(x).shape) 206 | 207 | 208 | 209 | -------------------------------------------------------------------------------- /ResNet50-SPD/test.py: -------------------------------------------------------------------------------- 1 | #test.py 2 | #!/usr/bin/env python3 3 | 4 | """ test neuron network performace 5 | print top1 and top5 err on test dataset 6 | of a model 7 | 8 | author baiyu 9 | """ 10 | 11 | import argparse 12 | 13 | from matplotlib import pyplot as plt 14 | 15 | import torch 16 | import torchvision.transforms as transforms 17 | from torch.utils.data import DataLoader 18 | 19 | from conf import settings 20 | from utils import get_network, get_test_dataloader 21 | 22 | if __name__ == '__main__': 23 | 24 | parser = argparse.ArgumentParser() 25 | parser.add_argument('-net', type=str, required=True, help='net type') 26 | parser.add_argument('-weights', type=str, required=True, help='the weights file you want to test') 27 | parser.add_argument('-gpu', action='store_true', default=False, help='use gpu or not') 28 | parser.add_argument('-b', type=int, default=16, help='batch size for dataloader') 29 | args = parser.parse_args() 30 | 31 | net = get_network(args) 32 | 33 | cifar10_test_loader = get_test_dataloader( 34 | settings.CIFAR100_TRAIN_MEAN, 35 | settings.CIFAR100_TRAIN_STD, 36 | num_workers=4, 37 | batch_size=args.b, 38 | shuffle=True 39 | ) 40 | 41 | net.load_state_dict(torch.load(args.weights,map_location=torch.device('cpu'))) 42 | net.eval() 43 | 44 | correct_1 = 0.0 45 | correct_5 = 0.0 46 | total = 0 47 | 48 | with torch.no_grad(): 49 | for n_iter, (image, label) in enumerate(cifar10_test_loader): 50 | print("iteration: {}\ttotal {} iterations".format(n_iter + 1, len(cifar10_test_loader))) 51 | 52 | if args.gpu: 53 | image = image.cuda() 54 | label = label.cuda() 55 | print('GPU INFO.....') 56 | print(torch.cuda.memory_summary(), end='') 57 | 58 | 59 | output = net(image) 60 | _, pred = output.topk(5, 1, largest=True, sorted=True) 61 | 62 | label = label.view(label.size(0), -1).expand_as(pred) 63 | correct = pred.eq(label).float() 64 | 65 | #compute top 5 66 | correct_5 += correct[:, :5].sum() 67 | 68 | #compute top1 69 | correct_1 += correct[:, :1].sum() 70 | 71 | if args.gpu: 72 | print('GPU INFO.....') 73 | print(torch.cuda.memory_summary(), end='') 74 | 75 | print() 76 | print("Top 1 err: ", 1 - correct_1 / len(cifar10_test_loader.dataset)) 77 | print("Top 5 err: ", 1 - correct_5 / len(cifar10_test_loader.dataset)) 78 | print("Parameter numbers: {}".format(sum(p.numel() for p in net.parameters()))) 79 | -------------------------------------------------------------------------------- /ResNet50-SPD/train.py: -------------------------------------------------------------------------------- 1 | # train.py 2 | #!/usr/bin/env python3 3 | 4 | """ train network using pytorch 5 | 6 | author baiyu 7 | """ 8 | from ptflops import get_model_complexity_info 9 | import os 10 | import sys 11 | import argparse 12 | import time 13 | from datetime import datetime 14 | 15 | import numpy as np 16 | import torch 17 | import torch.nn as nn 18 | import torch.optim as optim 19 | import torchvision 20 | import torchvision.transforms as transforms 21 | 22 | import wandb 23 | wandb.init() 24 | 25 | from torch.utils.data import DataLoader 26 | from torch.utils.tensorboard import SummaryWriter 27 | 28 | from conf import settings 29 | from utils import get_network, get_training_dataloader, get_test_dataloader, WarmUpLR, \ 30 | most_recent_folder, most_recent_weights, last_epoch, best_acc_weights 31 | 32 | def train(epoch): 33 | 34 | start = time.time() 35 | net.train() 36 | for batch_index, (images, labels) in enumerate(cifar10_training_loader): 37 | 38 | if args.gpu: 39 | labels = labels.cuda() 40 | images = images.cuda() 41 | 42 | optimizer.zero_grad() 43 | outputs = net(images) 44 | loss = loss_function(outputs, labels) 45 | loss.backward() 46 | optimizer.step() 47 | 48 | n_iter = (epoch - 1) * len(cifar10_training_loader) + batch_index + 1 49 | 50 | last_layer = list(net.children())[-1] 51 | for name, para in last_layer.named_parameters(): 52 | if 'weight' in name: 53 | writer.add_scalar('LastLayerGradients/grad_norm2_weights', para.grad.norm(), n_iter) 54 | if 'bias' in name: 55 | writer.add_scalar('LastLayerGradients/grad_norm2_bias', para.grad.norm(), n_iter) 56 | 57 | print('Training Epoch: {epoch} [{trained_samples}/{total_samples}]\tLoss: {:0.4f}\tLR: {:0.6f}'.format( 58 | loss.item(), 59 | optimizer.param_groups[0]['lr'], 60 | epoch=epoch, 61 | trained_samples=batch_index * args.b + len(images), 62 | total_samples=len(cifar10_training_loader.dataset) 63 | )) 64 | 65 | wandb.log({'loss': loss.item()}) 66 | 67 | #update training loss for each iteration 68 | writer.add_scalar('Train/loss', loss.item(), n_iter) 69 | 70 | if epoch <= args.warm: 71 | warmup_scheduler.step() 72 | 73 | for name, param in net.named_parameters(): 74 | layer, attr = os.path.splitext(name) 75 | attr = attr[1:] 76 | writer.add_histogram("{}/{}".format(layer, attr), param, epoch) 77 | 78 | finish = time.time() 79 | 80 | print('epoch {} training time consumed: {:.2f}s'.format(epoch, finish - start)) 81 | 82 | @torch.no_grad() 83 | def eval_training(epoch=0, tb=True): 84 | 85 | start = time.time() 86 | net.eval() 87 | 88 | test_loss = 0.0 # cost function error 89 | correct = 0.0 90 | 91 | for (images, labels) in cifar10_test_loader: 92 | 93 | if args.gpu: 94 | images = images.cuda() 95 | labels = labels.cuda() 96 | 97 | outputs = net(images) 98 | loss = loss_function(outputs, labels) 99 | 100 | test_loss += loss.item() 101 | _, preds = outputs.max(1) 102 | correct += preds.eq(labels).sum() 103 | 104 | finish = time.time() 105 | if args.gpu: 106 | print('GPU INFO.....') 107 | print(torch.cuda.memory_summary(), end='') 108 | print('Evaluating Network.....') 109 | print('Test set: Epoch: {}, Average loss: {:.4f}, Accuracy: {:.4f}, Time consumed:{:.2f}s'.format( 110 | epoch, 111 | test_loss / len(cifar10_test_loader.dataset), 112 | correct.float() / len(cifar10_test_loader.dataset), 113 | finish - start 114 | )) 115 | 116 | wandb.log({'Test loss': test_loss / len(cifar10_test_loader.dataset)}) 117 | wandb.log({'Test Accuracy':correct.float() / len(cifar10_test_loader.dataset)}) 118 | 119 | print() 120 | 121 | #add informations to tensorboard 122 | if tb: 123 | writer.add_scalar('Test/Average loss', test_loss / len(cifar10_test_loader.dataset), epoch) 124 | writer.add_scalar('Test/Accuracy', correct.float() / len(cifar10_test_loader.dataset), epoch) 125 | 126 | return correct.float() / len(cifar10_test_loader.dataset) 127 | 128 | if __name__ == '__main__': 129 | 130 | parser = argparse.ArgumentParser() 131 | parser.add_argument('-net', type=str, required=True, help='net type') 132 | parser.add_argument('-gpu', action='store_true', default=False, help='use gpu or not') 133 | parser.add_argument('-b', type=int, default=128, help='batch size for dataloader') 134 | parser.add_argument('-warm', type=int, default=1, help='warm up training phase') 135 | parser.add_argument('-lr', type=float, default=0.1, help='initial learning rate') 136 | parser.add_argument('-resume', action='store_true', default=False, help='resume training') 137 | args = parser.parse_args() 138 | 139 | net = get_network(args) 140 | 141 | macs, params = get_model_complexity_info(net, (3, 32, 32), as_strings=True, 142 | print_per_layer_stat=True, verbose=True) 143 | print('{:<30} {:<8}'.format('Computational complexity: ', macs)) 144 | print('{:<30} {:<8}'.format('Number of parameters: ', params)) 145 | 146 | 147 | #data preprocessing: 148 | cifar10_training_loader = get_training_dataloader( 149 | settings.CIFAR100_TRAIN_MEAN, 150 | settings.CIFAR100_TRAIN_STD, 151 | num_workers=4, 152 | batch_size=args.b, 153 | shuffle=True 154 | ) 155 | 156 | cifar10_test_loader = get_test_dataloader( 157 | settings.CIFAR100_TRAIN_MEAN, 158 | settings.CIFAR100_TRAIN_STD, 159 | num_workers=4, 160 | batch_size=args.b, 161 | shuffle=True 162 | ) 163 | 164 | loss_function = nn.CrossEntropyLoss() 165 | optimizer = optim.SGD(net.parameters(), lr=args.lr, momentum=0.9, weight_decay=5e-4) 166 | train_scheduler = optim.lr_scheduler.MultiStepLR(optimizer, milestones=settings.MILESTONES, gamma=0.2) #learning rate decay 167 | iter_per_epoch = len(cifar10_training_loader) 168 | warmup_scheduler = WarmUpLR(optimizer, iter_per_epoch * args.warm) 169 | 170 | if args.resume: 171 | recent_folder = most_recent_folder(os.path.join(settings.CHECKPOINT_PATH, args.net), fmt=settings.DATE_FORMAT) 172 | if not recent_folder: 173 | raise Exception('no recent folder were found') 174 | 175 | checkpoint_path = os.path.join(settings.CHECKPOINT_PATH, args.net, recent_folder) 176 | 177 | else: 178 | checkpoint_path = os.path.join(settings.CHECKPOINT_PATH, args.net, settings.TIME_NOW) 179 | 180 | #use tensorboard 181 | if not os.path.exists(settings.LOG_DIR): 182 | os.mkdir(settings.LOG_DIR) 183 | 184 | #since tensorboard can't overwrite old values 185 | #so the only way is to create a new tensorboard log 186 | writer = SummaryWriter(log_dir=os.path.join( 187 | settings.LOG_DIR, args.net, settings.TIME_NOW)) 188 | input_tensor = torch.Tensor(1, 3, 32, 32) 189 | if args.gpu: 190 | input_tensor = input_tensor.cuda() 191 | writer.add_graph(net, input_tensor) 192 | 193 | #create checkpoint folder to save model 194 | if not os.path.exists(checkpoint_path): 195 | os.makedirs(checkpoint_path) 196 | checkpoint_path = os.path.join(checkpoint_path, '{net}-{epoch}-{type}.pth') 197 | 198 | best_acc = 0.0 199 | if args.resume: 200 | best_weights = best_acc_weights(os.path.join(settings.CHECKPOINT_PATH, args.net, recent_folder)) 201 | if best_weights: 202 | weights_path = os.path.join(settings.CHECKPOINT_PATH, args.net, recent_folder, best_weights) 203 | print('found best acc weights file:{}'.format(weights_path)) 204 | print('load best training file to test acc...') 205 | net.load_state_dict(torch.load(weights_path)) 206 | best_acc = eval_training(tb=False) 207 | print('best acc is {:0.2f}'.format(best_acc)) 208 | 209 | recent_weights_file = most_recent_weights(os.path.join(settings.CHECKPOINT_PATH, args.net, recent_folder)) 210 | if not recent_weights_file: 211 | raise Exception('no recent weights file were found') 212 | weights_path = os.path.join(settings.CHECKPOINT_PATH, args.net, recent_folder, recent_weights_file) 213 | print('loading weights file {} to resume training.....'.format(weights_path)) 214 | net.load_state_dict(torch.load(weights_path)) 215 | 216 | resume_epoch = last_epoch(os.path.join(settings.CHECKPOINT_PATH, args.net, recent_folder)) 217 | 218 | 219 | for epoch in range(1, settings.EPOCH + 1): 220 | if epoch > args.warm: 221 | train_scheduler.step(epoch) 222 | 223 | if args.resume: 224 | if epoch <= resume_epoch: 225 | continue 226 | 227 | train(epoch) 228 | acc = eval_training(epoch) 229 | 230 | #start to save best performance model after learning rate decay to 0.01 231 | if epoch > settings.MILESTONES[1] and best_acc < acc: 232 | weights_path = checkpoint_path.format(net=args.net, epoch=epoch, type='best') 233 | print('saving weights file to {}'.format(weights_path)) 234 | torch.save(net.state_dict(), weights_path) 235 | best_acc = acc 236 | continue 237 | 238 | if not epoch % settings.SAVE_EPOCH: 239 | weights_path = checkpoint_path.format(net=args.net, epoch=epoch, type='regular') 240 | print('saving weights file to {}'.format(weights_path)) 241 | torch.save(net.state_dict(), weights_path) 242 | 243 | writer.close() 244 | -------------------------------------------------------------------------------- /ResNet50-SPD/utils.py: -------------------------------------------------------------------------------- 1 | """ helper function 2 | 3 | author baiyu 4 | """ 5 | import os 6 | import sys 7 | import re 8 | import datetime 9 | 10 | import numpy 11 | 12 | import torch 13 | from torch.optim.lr_scheduler import _LRScheduler 14 | import torchvision 15 | import torchvision.transforms as transforms 16 | from torch.utils.data import DataLoader 17 | 18 | 19 | def get_network(args): 20 | """ return given network 21 | """ 22 | 23 | if args.net == 'vgg16': 24 | from models.vgg import vgg16_bn 25 | net = vgg16_bn() 26 | elif args.net == 'vgg13': 27 | from models.vgg import vgg13_bn 28 | net = vgg13_bn() 29 | elif args.net == 'vgg11': 30 | from models.vgg import vgg11_bn 31 | net = vgg11_bn() 32 | elif args.net == 'vgg19': 33 | from models.vgg import vgg19_bn 34 | net = vgg19_bn() 35 | elif args.net == 'densenet121': 36 | from models.densenet import densenet121 37 | net = densenet121() 38 | elif args.net == 'densenet161': 39 | from models.densenet import densenet161 40 | net = densenet161() 41 | elif args.net == 'densenet169': 42 | from models.densenet import densenet169 43 | net = densenet169() 44 | elif args.net == 'densenet201': 45 | from models.densenet import densenet201 46 | net = densenet201() 47 | elif args.net == 'googlenet': 48 | from models.googlenet import googlenet 49 | net = googlenet() 50 | elif args.net == 'inceptionv3': 51 | from models.inceptionv3 import inceptionv3 52 | net = inceptionv3() 53 | elif args.net == 'inceptionv4': 54 | from models.inceptionv4 import inceptionv4 55 | net = inceptionv4() 56 | elif args.net == 'inceptionresnetv2': 57 | from models.inceptionv4 import inception_resnet_v2 58 | net = inception_resnet_v2() 59 | elif args.net == 'xception': 60 | from models.xception import xception 61 | net = xception() 62 | elif args.net == 'resnet18': 63 | from models.resnet import resnet18 64 | net = resnet18() 65 | elif args.net == 'resnet34': 66 | from models.resnet import resnet34 67 | net = resnet34() 68 | elif args.net == 'resnet50': 69 | from models.resnet import resnet50 70 | net = resnet50() 71 | elif args.net == 'resnet50_spd': 72 | from models.resnet50_spd import resnet50 73 | net = resnet50() 74 | 75 | elif args.net == 'resnet101': 76 | from models.resnet import resnet101 77 | net = resnet101() 78 | elif args.net == 'resnet152': 79 | from models.resnet import resnet152 80 | net = resnet152() 81 | elif args.net == 'preactresnet18': 82 | from models.preactresnet import preactresnet18 83 | net = preactresnet18() 84 | elif args.net == 'preactresnet34': 85 | from models.preactresnet import preactresnet34 86 | net = preactresnet34() 87 | elif args.net == 'preactresnet50': 88 | from models.preactresnet import preactresnet50 89 | net = preactresnet50() 90 | elif args.net == 'preactresnet101': 91 | from models.preactresnet import preactresnet101 92 | net = preactresnet101() 93 | elif args.net == 'preactresnet152': 94 | from models.preactresnet import preactresnet152 95 | net = preactresnet152() 96 | elif args.net == 'resnext50': 97 | from models.resnext import resnext50 98 | net = resnext50() 99 | elif args.net == 'resnext101': 100 | from models.resnext import resnext101 101 | net = resnext101() 102 | elif args.net == 'resnext152': 103 | from models.resnext import resnext152 104 | net = resnext152() 105 | elif args.net == 'shufflenet': 106 | from models.shufflenet import shufflenet 107 | net = shufflenet() 108 | elif args.net == 'shufflenetv2': 109 | from models.shufflenetv2 import shufflenetv2 110 | net = shufflenetv2() 111 | elif args.net == 'squeezenet': 112 | from models.squeezenet import squeezenet 113 | net = squeezenet() 114 | elif args.net == 'mobilenet': 115 | from models.mobilenet import mobilenet 116 | net = mobilenet() 117 | elif args.net == 'mobilenetv2': 118 | from models.mobilenetv2 import mobilenetv2 119 | net = mobilenetv2() 120 | elif args.net == 'nasnet': 121 | from models.nasnet import nasnet 122 | net = nasnet() 123 | elif args.net == 'attention56': 124 | from models.attention import attention56 125 | net = attention56() 126 | elif args.net == 'attention92': 127 | from models.attention import attention92 128 | net = attention92() 129 | elif args.net == 'seresnet18': 130 | from models.senet import seresnet18 131 | net = seresnet18() 132 | elif args.net == 'seresnet34': 133 | from models.senet import seresnet34 134 | net = seresnet34() 135 | elif args.net == 'seresnet50': 136 | from models.senet import seresnet50 137 | net = seresnet50() 138 | elif args.net == 'seresnet101': 139 | from models.senet import seresnet101 140 | net = seresnet101() 141 | elif args.net == 'seresnet152': 142 | from models.senet import seresnet152 143 | net = seresnet152() 144 | elif args.net == 'wideresnet': 145 | from models.wideresidual import wideresnet 146 | net = wideresnet() 147 | elif args.net == 'stochasticdepth18': 148 | from models.stochasticdepth import stochastic_depth_resnet18 149 | net = stochastic_depth_resnet18() 150 | elif args.net == 'stochasticdepth34': 151 | from models.stochasticdepth import stochastic_depth_resnet34 152 | net = stochastic_depth_resnet34() 153 | elif args.net == 'stochasticdepth50': 154 | from models.stochasticdepth import stochastic_depth_resnet50 155 | net = stochastic_depth_resnet50() 156 | elif args.net == 'stochasticdepth101': 157 | from models.stochasticdepth import stochastic_depth_resnet101 158 | net = stochastic_depth_resnet101() 159 | 160 | else: 161 | print('the network name you have entered is not supported yet') 162 | sys.exit() 163 | 164 | if args.gpu: #use_gpu 165 | net = net.cuda() 166 | 167 | return net 168 | 169 | 170 | def get_training_dataloader(mean, std, batch_size=16, num_workers=2, shuffle=True): 171 | """ return training dataloader 172 | Args: 173 | mean: mean of cifar100 training dataset 174 | std: std of cifar100 training dataset 175 | path: path to cifar100 training python dataset 176 | batch_size: dataloader batchsize 177 | num_workers: dataloader num_works 178 | shuffle: whether to shuffle 179 | Returns: train_data_loader:torch dataloader object 180 | """ 181 | 182 | transform_train = transforms.Compose([ 183 | #transforms.ToPILImage(), 184 | transforms.RandomCrop(32, padding=4), 185 | transforms.RandomHorizontalFlip(), 186 | transforms.RandomRotation(15), 187 | transforms.ToTensor(), 188 | transforms.Normalize(mean, std) 189 | ]) 190 | #cifar100_training = CIFAR100Train(path, transform=transform_train) 191 | cifar10_training = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform_train) 192 | cifar10_training_loader = DataLoader( 193 | cifar10_training, shuffle=shuffle, num_workers=num_workers, batch_size=batch_size) 194 | 195 | return cifar10_training_loader 196 | 197 | def get_test_dataloader(mean, std, batch_size=16, num_workers=2, shuffle=True): 198 | """ return training dataloader 199 | Args: 200 | mean: mean of cifar100 test dataset 201 | std: std of cifar100 test dataset 202 | path: path to cifar100 test python dataset 203 | batch_size: dataloader batchsize 204 | num_workers: dataloader num_works 205 | shuffle: whether to shuffle 206 | Returns: cifar100_test_loader:torch dataloader object 207 | """ 208 | 209 | transform_test = transforms.Compose([ 210 | transforms.ToTensor(), 211 | transforms.Normalize(mean, std) 212 | ]) 213 | #cifar100_test = CIFAR100Test(path, transform=transform_test) 214 | cifar10_test = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform_test) 215 | cifar10_test_loader = DataLoader( 216 | cifar10_test, shuffle=shuffle, num_workers=num_workers, batch_size=batch_size) 217 | 218 | return cifar10_test_loader 219 | 220 | def compute_mean_std(cifar10_dataset): 221 | """compute the mean and std of cifar100 dataset 222 | Args: 223 | cifar100_training_dataset or cifar100_test_dataset 224 | witch derived from class torch.utils.data 225 | 226 | Returns: 227 | a tuple contains mean, std value of entire dataset 228 | """ 229 | 230 | data_r = numpy.dstack([cifar10_dataset[i][1][:, :, 0] for i in range(len(cifar100_dataset))]) 231 | data_g = numpy.dstack([cifar10_dataset[i][1][:, :, 1] for i in range(len(cifar100_dataset))]) 232 | data_b = numpy.dstack([cifar10_dataset[i][1][:, :, 2] for i in range(len(cifar100_dataset))]) 233 | mean = numpy.mean(data_r), numpy.mean(data_g), numpy.mean(data_b) 234 | std = numpy.std(data_r), numpy.std(data_g), numpy.std(data_b) 235 | 236 | return mean, std 237 | 238 | class WarmUpLR(_LRScheduler): 239 | """warmup_training learning rate scheduler 240 | Args: 241 | optimizer: optimzier(e.g. SGD) 242 | total_iters: totoal_iters of warmup phase 243 | """ 244 | def __init__(self, optimizer, total_iters, last_epoch=-1): 245 | 246 | self.total_iters = total_iters 247 | super().__init__(optimizer, last_epoch) 248 | 249 | def get_lr(self): 250 | """we will use the first m batches, and set the learning 251 | rate to base_lr * m / total_iters 252 | """ 253 | return [base_lr * self.last_epoch / (self.total_iters + 1e-8) for base_lr in self.base_lrs] 254 | 255 | 256 | def most_recent_folder(net_weights, fmt): 257 | """ 258 | return most recent created folder under net_weights 259 | if no none-empty folder were found, return empty folder 260 | """ 261 | # get subfolders in net_weights 262 | folders = os.listdir(net_weights) 263 | 264 | # filter out empty folders 265 | folders = [f for f in folders if len(os.listdir(os.path.join(net_weights, f)))] 266 | if len(folders) == 0: 267 | return '' 268 | 269 | # sort folders by folder created time 270 | folders = sorted(folders, key=lambda f: datetime.datetime.strptime(f, fmt)) 271 | return folders[-1] 272 | 273 | def most_recent_weights(weights_folder): 274 | """ 275 | return most recent created weights file 276 | if folder is empty return empty string 277 | """ 278 | weight_files = os.listdir(weights_folder) 279 | if len(weights_folder) == 0: 280 | return '' 281 | 282 | regex_str = r'([A-Za-z0-9]+)-([0-9]+)-(regular|best)' 283 | 284 | # sort files by epoch 285 | weight_files = sorted(weight_files, key=lambda w: int(re.search(regex_str, w).groups()[1])) 286 | 287 | return weight_files[-1] 288 | 289 | def last_epoch(weights_folder): 290 | weight_file = most_recent_weights(weights_folder) 291 | if not weight_file: 292 | raise Exception('no recent weights were found') 293 | resume_epoch = int(weight_file.split('-')[1]) 294 | 295 | return resume_epoch 296 | 297 | def best_acc_weights(weights_folder): 298 | """ 299 | return the best acc .pth file in given folder, if no 300 | best acc weights file were found, return empty string 301 | """ 302 | files = os.listdir(weights_folder) 303 | if len(files) == 0: 304 | return '' 305 | 306 | regex_str = r'([A-Za-z0-9]+)-([0-9]+)-(regular|best)' 307 | best_files = [w for w in files if re.search(regex_str, w).groups()[2] == 'best'] 308 | if len(best_files) == 0: 309 | return '' 310 | 311 | best_files = sorted(best_files, key=lambda w: int(re.search(regex_str, w).groups()[1])) 312 | return best_files[-1] 313 | -------------------------------------------------------------------------------- /YOLOv5-SPD/.gitattributes: -------------------------------------------------------------------------------- 1 | # this drop notebooks from GitHub language stats 2 | *.ipynb linguist-vendored 3 | -------------------------------------------------------------------------------- /YOLOv5-SPD/.github/FUNDING.yml: -------------------------------------------------------------------------------- 1 | # These are supported funding model platforms 2 | 3 | github: glenn-jocher 4 | patreon: ultralytics 5 | open_collective: ultralytics 6 | -------------------------------------------------------------------------------- /YOLOv5-SPD/.github/ISSUE_TEMPLATE/bug-report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: "🐛 Bug report" 3 | about: Create a report to help us improve 4 | title: '' 5 | labels: bug 6 | assignees: '' 7 | 8 | --- 9 | 10 | Before submitting a bug report, please be aware that your issue **must be reproducible** with all of the following, 11 | otherwise it is non-actionable, and we can not help you: 12 | 13 | - **Current repo**: run `git fetch && git status -uno` to check and `git pull` to update repo 14 | - **Common dataset**: coco.yaml or coco128.yaml 15 | - **Common environment**: Colab, Google Cloud, or Docker image. See https://github.com/ultralytics/yolov5#environments 16 | 17 | If this is a custom dataset/training question you **must include** your `train*.jpg`, `val*.jpg` and `results.png` 18 | figures, or we can not help you. You can generate these with `utils.plot_results()`. 19 | 20 | ## 🐛 Bug 21 | 22 | A clear and concise description of what the bug is. 23 | 24 | ## To Reproduce (REQUIRED) 25 | 26 | Input: 27 | 28 | ``` 29 | import torch 30 | 31 | a = torch.tensor([5]) 32 | c = a / 0 33 | ``` 34 | 35 | Output: 36 | 37 | ``` 38 | Traceback (most recent call last): 39 | File "/Users/glennjocher/opt/anaconda3/envs/env1/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3331, in run_code 40 | exec(code_obj, self.user_global_ns, self.user_ns) 41 | File "", line 5, in 42 | c = a / 0 43 | RuntimeError: ZeroDivisionError 44 | ``` 45 | 46 | ## Expected behavior 47 | 48 | A clear and concise description of what you expected to happen. 49 | 50 | ## Environment 51 | 52 | If applicable, add screenshots to help explain your problem. 53 | 54 | - OS: [e.g. Ubuntu] 55 | - GPU [e.g. 2080 Ti] 56 | 57 | ## Additional context 58 | 59 | Add any other context about the problem here. 60 | -------------------------------------------------------------------------------- /YOLOv5-SPD/.github/ISSUE_TEMPLATE/feature-request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: "🚀 Feature request" 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: enhancement 6 | assignees: '' 7 | 8 | --- 9 | 10 | ## 🚀 Feature 11 | 12 | 13 | 14 | ## Motivation 15 | 16 | 18 | 19 | ## Pitch 20 | 21 | 22 | 23 | ## Alternatives 24 | 25 | 26 | 27 | ## Additional context 28 | 29 | 30 | -------------------------------------------------------------------------------- /YOLOv5-SPD/.github/ISSUE_TEMPLATE/question.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: "❓Question" 3 | about: Ask a general question 4 | title: '' 5 | labels: question 6 | assignees: '' 7 | 8 | --- 9 | 10 | ## ❔Question 11 | 12 | ## Additional context 13 | -------------------------------------------------------------------------------- /YOLOv5-SPD/.github/dependabot.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | updates: 3 | - package-ecosystem: pip 4 | directory: "/" 5 | schedule: 6 | interval: weekly 7 | time: "04:00" 8 | open-pull-requests-limit: 10 9 | reviewers: 10 | - glenn-jocher 11 | labels: 12 | - dependencies 13 | -------------------------------------------------------------------------------- /YOLOv5-SPD/.github/workflows/ci-testing.yml: -------------------------------------------------------------------------------- 1 | # YOLOv5 🚀 by Ultralytics, GPL-3.0 license 2 | 3 | name: CI CPU testing 4 | 5 | on: # https://help.github.com/en/actions/reference/events-that-trigger-workflows 6 | push: 7 | branches: [ master ] 8 | pull_request: 9 | # The branches below must be a subset of the branches above 10 | branches: [ master ] 11 | schedule: 12 | - cron: '0 0 * * *' # Runs at 00:00 UTC every day 13 | 14 | jobs: 15 | cpu-tests: 16 | 17 | runs-on: ${{ matrix.os }} 18 | strategy: 19 | fail-fast: false 20 | matrix: 21 | os: [ ubuntu-latest, macos-latest, windows-latest ] 22 | python-version: [ 3.8 ] 23 | model: [ 'yolov5n' ] # models to test 24 | 25 | # Timeout: https://stackoverflow.com/a/59076067/4521646 26 | timeout-minutes: 50 27 | steps: 28 | - uses: actions/checkout@v2 29 | - name: Set up Python ${{ matrix.python-version }} 30 | uses: actions/setup-python@v2 31 | with: 32 | python-version: ${{ matrix.python-version }} 33 | 34 | # Note: This uses an internal pip API and may not always work 35 | # https://github.com/actions/cache/blob/master/examples.md#multiple-oss-in-a-workflow 36 | - name: Get pip cache 37 | id: pip-cache 38 | run: | 39 | python -c "from pip._internal.locations import USER_CACHE_DIR; print('::set-output name=dir::' + USER_CACHE_DIR)" 40 | 41 | - name: Cache pip 42 | uses: actions/cache@v1 43 | with: 44 | path: ${{ steps.pip-cache.outputs.dir }} 45 | key: ${{ runner.os }}-${{ matrix.python-version }}-pip-${{ hashFiles('requirements.txt') }} 46 | restore-keys: | 47 | ${{ runner.os }}-${{ matrix.python-version }}-pip- 48 | 49 | - name: Install dependencies 50 | run: | 51 | python -m pip install --upgrade pip 52 | pip install -qr requirements.txt -f https://download.pytorch.org/whl/cpu/torch_stable.html 53 | pip install -q onnx tensorflow-cpu # for export 54 | python --version 55 | pip --version 56 | pip list 57 | shell: bash 58 | 59 | - name: Download data 60 | run: | 61 | # curl -L -o tmp.zip https://github.com/ultralytics/yolov5/releases/download/v1.0/coco128.zip 62 | # unzip -q tmp.zip -d ../ 63 | # rm tmp.zip 64 | 65 | - name: Tests workflow 66 | run: | 67 | # export PYTHONPATH="$PWD" # to run '$ python *.py' files in subdirectories 68 | di=cpu # device 69 | 70 | # Train 71 | python train.py --img 64 --batch 32 --weights ${{ matrix.model }}.pt --cfg ${{ matrix.model }}.yaml --epochs 1 --device $di 72 | # Val 73 | python val.py --img 64 --batch 32 --weights ${{ matrix.model }}.pt --device $di 74 | python val.py --img 64 --batch 32 --weights runs/train/exp/weights/last.pt --device $di 75 | # Detect 76 | python detect.py --weights ${{ matrix.model }}.pt --device $di 77 | python detect.py --weights runs/train/exp/weights/last.pt --device $di 78 | python hubconf.py # hub 79 | # Export 80 | python models/yolo.py --cfg ${{ matrix.model }}.yaml # build PyTorch model 81 | python models/tf.py --weights ${{ matrix.model }}.pt # build TensorFlow model 82 | python export.py --img 64 --batch 1 --weights ${{ matrix.model }}.pt --include torchscript onnx # export 83 | # Python 84 | python - <=3.6.0**](https://www.python.org/) with all [requirements.txt](https://github.com/ultralytics/yolov5/blob/master/requirements.txt) installed including [**PyTorch>=1.7**](https://pytorch.org/get-started/locally/). To get started: 39 | ```bash 40 | $ git clone https://github.com/ultralytics/yolov5 41 | $ cd yolov5 42 | $ pip install -r requirements.txt 43 | ``` 44 | 45 | ## Environments 46 | 47 | YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including [CUDA](https://developer.nvidia.com/cuda)/[CUDNN](https://developer.nvidia.com/cudnn), [Python](https://www.python.org/) and [PyTorch](https://pytorch.org/) preinstalled): 48 | 49 | - **Google Colab and Kaggle** notebooks with free GPU: Open In Colab Open In Kaggle 50 | - **Google Cloud** Deep Learning VM. See [GCP Quickstart Guide](https://github.com/ultralytics/yolov5/wiki/GCP-Quickstart) 51 | - **Amazon** Deep Learning AMI. See [AWS Quickstart Guide](https://github.com/ultralytics/yolov5/wiki/AWS-Quickstart) 52 | - **Docker Image**. See [Docker Quickstart Guide](https://github.com/ultralytics/yolov5/wiki/Docker-Quickstart) Docker Pulls 53 | 54 | 55 | ## Status 56 | 57 | CI CPU testing 58 | 59 | If this badge is green, all [YOLOv5 GitHub Actions](https://github.com/ultralytics/yolov5/actions) Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training ([train.py](https://github.com/ultralytics/yolov5/blob/master/train.py)), validation ([val.py](https://github.com/ultralytics/yolov5/blob/master/val.py)), inference ([detect.py](https://github.com/ultralytics/yolov5/blob/master/detect.py)) and export ([export.py](https://github.com/ultralytics/yolov5/blob/master/export.py)) on MacOS, Windows, and Ubuntu every 24 hours and on every commit. 60 | 61 | -------------------------------------------------------------------------------- /YOLOv5-SPD/.github/workflows/rebase.yml: -------------------------------------------------------------------------------- 1 | name: Automatic Rebase 2 | # https://github.com/marketplace/actions/automatic-rebase 3 | 4 | on: 5 | issue_comment: 6 | types: [created] 7 | 8 | jobs: 9 | rebase: 10 | name: Rebase 11 | if: github.event.issue.pull_request != '' && contains(github.event.comment.body, '/rebase') 12 | runs-on: ubuntu-latest 13 | steps: 14 | - name: Checkout the latest code 15 | uses: actions/checkout@v2 16 | with: 17 | fetch-depth: 0 18 | - name: Automatic Rebase 19 | uses: cirrus-actions/rebase@1.3.1 20 | env: 21 | GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} 22 | -------------------------------------------------------------------------------- /YOLOv5-SPD/.github/workflows/stale.yml: -------------------------------------------------------------------------------- 1 | # YOLOv5 🚀 by Ultralytics, GPL-3.0 license 2 | 3 | name: Close stale issues 4 | on: 5 | schedule: 6 | - cron: "0 0 * * *" 7 | 8 | jobs: 9 | stale: 10 | runs-on: ubuntu-latest 11 | steps: 12 | - uses: actions/stale@v3 13 | with: 14 | repo-token: ${{ secrets.GITHUB_TOKEN }} 15 | stale-issue-message: | 16 | 👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs. 17 | 18 | Access additional [YOLOv5](https://ultralytics.com/yolov5) 🚀 resources: 19 | - **Wiki** – https://github.com/ultralytics/yolov5/wiki 20 | - **Tutorials** – https://github.com/ultralytics/yolov5#tutorials 21 | - **Docs** – https://docs.ultralytics.com 22 | 23 | Access additional [Ultralytics](https://ultralytics.com) ⚡ resources: 24 | - **Ultralytics HUB** – https://ultralytics.com/hub 25 | - **Vision API** – https://ultralytics.com/yolov5 26 | - **About Us** – https://ultralytics.com/about 27 | - **Join Our Team** – https://ultralytics.com/work 28 | - **Contact Us** – https://ultralytics.com/contact 29 | 30 | Feel free to inform us of any other **issues** you discover or **feature requests** that come to mind in the future. Pull Requests (PRs) are also always welcomed! 31 | 32 | Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐! 33 | 34 | stale-pr-message: 'This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions YOLOv5 🚀 and Vision AI ⭐.' 35 | days-before-stale: 30 36 | days-before-close: 5 37 | exempt-issue-labels: 'documentation,tutorial' 38 | operations-per-run: 100 # The maximum number of operations per run, used to control rate limiting. 39 | -------------------------------------------------------------------------------- /YOLOv5-SPD/CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | ## Contributing to YOLOv5 🚀 2 | 3 | We love your input! We want to make contributing to YOLOv5 as easy and transparent as possible, whether it's: 4 | 5 | - Reporting a bug 6 | - Discussing the current state of the code 7 | - Submitting a fix 8 | - Proposing a new feature 9 | - Becoming a maintainer 10 | 11 | YOLOv5 works so well due to our combined community effort, and for every small improvement you contribute you will be 12 | helping push the frontiers of what's possible in AI 😃! 13 | 14 | ## Submitting a Pull Request (PR) 🛠️ 15 | 16 | Submitting a PR is easy! This example shows how to submit a PR for updating `requirements.txt` in 4 steps: 17 | 18 | ### 1. Select File to Update 19 | 20 | Select `requirements.txt` to update by clicking on it in GitHub. 21 |

PR_step1

22 | 23 | ### 2. Click 'Edit this file' 24 | 25 | Button is in top-right corner. 26 |

PR_step2

27 | 28 | ### 3. Make Changes 29 | 30 | Change `matplotlib` version from `3.2.2` to `3.3`. 31 |

PR_step3

32 | 33 | ### 4. Preview Changes and Submit PR 34 | 35 | Click on the **Preview changes** tab to verify your updates. At the bottom of the screen select 'Create a **new branch** 36 | for this commit', assign your branch a descriptive name such as `fix/matplotlib_version` and click the green **Propose 37 | changes** button. All done, your PR is now submitted to YOLOv5 for review and approval 😃! 38 |

PR_step4

39 | 40 | ### PR recommendations 41 | 42 | To allow your work to be integrated as seamlessly as possible, we advise you to: 43 | 44 | - ✅ Verify your PR is **up-to-date with origin/master.** If your PR is behind origin/master an 45 | automatic [GitHub actions](https://github.com/ultralytics/yolov5/blob/master/.github/workflows/rebase.yml) rebase may 46 | be attempted by including the /rebase command in a comment body, or by running the following code, replacing 'feature' 47 | with the name of your local branch: 48 | 49 | ```bash 50 | git remote add upstream https://github.com/ultralytics/yolov5.git 51 | git fetch upstream 52 | git checkout feature # <----- replace 'feature' with local branch name 53 | git merge upstream/master 54 | git push -u origin -f 55 | ``` 56 | 57 | - ✅ Verify all Continuous Integration (CI) **checks are passing**. 58 | - ✅ Reduce changes to the absolute **minimum** required for your bug fix or feature addition. _"It is not daily increase 59 | but daily decrease, hack away the unessential. The closer to the source, the less wastage there is."_ -Bruce Lee 60 | 61 | ## Submitting a Bug Report 🐛 62 | 63 | If you spot a problem with YOLOv5 please submit a Bug Report! 64 | 65 | For us to start investigating a possibel problem we need to be able to reproduce it ourselves first. We've created a few 66 | short guidelines below to help users provide what we need in order to get started. 67 | 68 | When asking a question, people will be better able to provide help if you provide **code** that they can easily 69 | understand and use to **reproduce** the problem. This is referred to by community members as creating 70 | a [minimum reproducible example](https://stackoverflow.com/help/minimal-reproducible-example). Your code that reproduces 71 | the problem should be: 72 | 73 | * ✅ **Minimal** – Use as little code as possible that still produces the same problem 74 | * ✅ **Complete** – Provide **all** parts someone else needs to reproduce your problem in the question itself 75 | * ✅ **Reproducible** – Test the code you're about to provide to make sure it reproduces the problem 76 | 77 | In addition to the above requirements, for [Ultralytics](https://ultralytics.com/) to provide assistance your code 78 | should be: 79 | 80 | * ✅ **Current** – Verify that your code is up-to-date with current 81 | GitHub [master](https://github.com/ultralytics/yolov5/tree/master), and if necessary `git pull` or `git clone` a new 82 | copy to ensure your problem has not already been resolved by previous commits. 83 | * ✅ **Unmodified** – Your problem must be reproducible without any modifications to the codebase in this 84 | repository. [Ultralytics](https://ultralytics.com/) does not provide support for custom code ⚠️. 85 | 86 | If you believe your problem meets all of the above criteria, please close this issue and raise a new one using the 🐛 ** 87 | Bug Report** [template](https://github.com/ultralytics/yolov5/issues/new/choose) and providing 88 | a [minimum reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) to help us better 89 | understand and diagnose your problem. 90 | 91 | ## License 92 | 93 | By contributing, you agree that your contributions will be licensed under 94 | the [GPL-3.0 license](https://choosealicense.com/licenses/gpl-3.0/) 95 | -------------------------------------------------------------------------------- /YOLOv5-SPD/experiment2.py: -------------------------------------------------------------------------------- 1 | """ 2 | Experimental Modules 3 | """ 4 | 5 | import logging 6 | import math 7 | import warnings 8 | from copy import copy 9 | from pathlib import Path 10 | import sys 11 | from models.yolo import Model 12 | 13 | import numpy as np 14 | import pandas as pd 15 | import requests 16 | import torch 17 | import torch.nn as nn 18 | from PIL import Image 19 | from torch.cuda import amp 20 | 21 | FILE = Path(__file__).resolve() 22 | ROOT = FILE.parents[1] # YOLOv5 root directory 23 | if str(ROOT) not in sys.path: 24 | sys.path.append(str(ROOT)) # add ROOT to PATH 25 | # ROOT = ROOT.relative_to(Path.cwd()) # relative 26 | 27 | 28 | from utils.datasets import exif_transpose, letterbox 29 | from utils.general import colorstr, increment_path, make_divisible, non_max_suppression, save_one_box, \ 30 | scale_coords, xyxy2xywh 31 | from utils.plots import Annotator, colors 32 | from utils.torch_utils import time_sync 33 | 34 | LOGGER = logging.getLogger(__name__) 35 | 36 | 37 | def autopad(k, p=None): # kernel, padding 38 | # Pad to 'same' 39 | if p is None: 40 | p = k //2 if isinstance(k, int) else [x//2 for x in k] # autopad 41 | 42 | return p 43 | 44 | class Conv(nn.Module): 45 | # standard convolution 46 | def __init__(self, c1, c2, k=1, s=1, p=None, g=1, act=True): 47 | #ch_in, ch_ out, kernel, stride, padding, groups 48 | super().__init__() 49 | self.conv = nn.Conv2d(c1, c2, k, s, autopad(k, p), groups=g, bias=False) 50 | self.bn = nn.BatchNorm2d(c2) 51 | self.act = nn.SiLU() if act is True else (act if isinstance(act, nn.Model) else nn.Identity()) 52 | 53 | def forward(self,x): 54 | return self.act(self.bn(self.conv(x))) 55 | 56 | def forward_fuse(self,x): 57 | return self.act(self.conv(x)) 58 | 59 | class DWConv(Conv): 60 | # Depth-wise convolution class 61 | 62 | def __init_(self, c1, c2, k=1, s=1, act=True): 63 | super().__init__(c1,c2,k,s,g=math.gcd(c1,c2),act=act) 64 | 65 | 66 | class TransformerLayer(nn.Module): 67 | # Transforment layer 68 | 69 | def __init__(self, c, num_heads): 70 | super().__init__() 71 | self.q = nn.Linear(c, c, bias = False) 72 | self.k = nn.Linear(c, c, bias=False) 73 | self.v = nn.Linear(c, c, bias=False) 74 | self.ma = nn.MultiheadAttention(embed_dim=c, num_heads=num_heads) 75 | self.fc1 = nn.Linear(c, c, bias=False) 76 | self.fc2 = nn.Linear(c, c, bias=False) 77 | 78 | def forward(self,x): 79 | x= self.ma(self.q(x),self.k(x),self.v(x))[0] + x 80 | x = self.fc2(self.fc1(x)) + x 81 | return x 82 | 83 | class TransformerBlock(nn.Module): 84 | 85 | def __init__(self, c1, c2, num_heads, num_layers): 86 | super().__init__() 87 | self.conv = None 88 | if c1 != c2: 89 | self.conv = Conv(c1,c2) 90 | self.linear = nn.Linear(c2, c2) # learnable position embedding 91 | self.tr=nn.Sequential(*[TransformerLayer(c2, num_heads) for _ in range(num_layers)]) 92 | self.c2=c2 93 | 94 | def forward(self,x): 95 | if self.conv is not None: 96 | x= self.conv(x) 97 | 98 | b, _, w, h = x.shape 99 | 100 | p = x.flatten(2).unsqueeze(0).transpose(0, 3).squeeze(3) 101 | return self.tr(p + self.linear(p)).unsqueeze(3).transpose(0, 3).reshape(b, self.c2, w, h) 102 | 103 | 104 | class Bottleneck(nn.Module): 105 | # Standard bottleneck 106 | 107 | def __init__(self, c1, c2, shortcut= True, g=1, e=0.5): 108 | # ch_in, ch_out, shortcut, groups, expansion 109 | super().__init__() 110 | c_ = int(c2*e) # hidden channels 111 | 112 | self.cv1 = Conv(c1, c_, 1, 1) 113 | self.cv2 = Conv(c_, c2, 3, 1, g=g) 114 | self.add = shortcut and c1==c2 115 | 116 | def forward(self, x): 117 | 118 | return x + self.cv2(self.cv1(x)) if self.add else self.cv2(self.cv1(x)) 119 | 120 | 121 | class BottleneckCSP(nn.Module): 122 | # CSP Bottleneck 123 | 124 | def __init__(self, c1, c2, n=1, shortcut=True, g=1, e=0.5): 125 | # ch_in, ch_out, number, shortcut, groups, expansion 126 | super().__init__() 127 | c_ = int(c2*e) # hidden channels 128 | self.cv1 = Conv(c1, c_, 1, 1) 129 | self.cv2 = nn.Conv2d(c1, c_, 1, 1, bias = False) 130 | self.cv3 = nn.Conv2d(c_, c_, 1, 1, bias=False) 131 | self.cv4 = Conv(2 * c_, c2, 1, 1) 132 | self.bn = nn.BatchNorm2d(2 * c_) # applied to cat(cv2, cv3) 133 | self.act = nn.LeakyReLU(0.1, inplace=True) 134 | self.m = nn.Sequential(*[Bottleneck(c_,c_,shortcut, g, e=1.0) for _ in range(n)]) 135 | 136 | def forward(self,x): 137 | y1 = self.cv3(self.m(self.cv1(x))) 138 | y2 = self.cv2(x) 139 | 140 | return self.cv4(self.act(self.bn(torch.cat((y1,y2),dim=1)))) 141 | 142 | class C3(nn.Module): 143 | # CSP Bottleneck with 3 convolutions 144 | def __init__(self, c1, c2, n=1, shortcut=True, g=1, e=0.5): 145 | # ch_in, ch_out, number, shortcut, groups, expansion 146 | super().__init__() 147 | c_ = int(c2 * e) # Hidden channels 148 | self.cv1 = Conv(c1, c_, 1, 1) 149 | self.cv2 = Conv(c1, c_, 1, 1) 150 | self.cv3 = Conv(2* c_, c2, 1) 151 | self.m = nn.Sequential(*[Bottleneck(c_, c_, shortcut, g, e=1.0) for _ in range(n)]) 152 | 153 | def forward(self,x): 154 | return self.cv3(torch.cat((self.m(self.cv1(x)), self.cv2(x)), dim=1)) 155 | 156 | 157 | class C3TR(C3): 158 | # C3 module with TransformerBlock() 159 | 160 | def __init__(self,c1, c2, n=1, shortcut= True, g=1, e=0.5): 161 | super().__init__(c1, c2, n=1, shortcut, g=1, e) 162 | c_ = int(c2*e) 163 | self.m = TransformerBlock(c_, c_, 4, n) 164 | 165 | class SPP(nn.Module): 166 | # spatial pyramid pooling layer 167 | 168 | def __init__(self, c1, c2, k=(5,9,13)): 169 | super().__init__() 170 | c_ = c1 //2 # hidden channels 171 | 172 | self.cv1 = Conv(c1, c_, 1, 1) 173 | self.cv2 = Conv(c_ *(len(k)+1), c2, 1, 1) 174 | self.m = nn.ModuleList([nn.MaxPool2d(kernel_size=x, stride=1, padding = x//2 ) for x in k]) 175 | 176 | def forward(self, x): 177 | x = self.cv1(x) 178 | with warnings.catch_warnings(): 179 | warnings.simplefilter('ignore') # suppress torch 1.9.0 max_pool2d() warning 180 | return self.cv2(torch.cat([x] + [m(x) for m in self.m], 1)) 181 | 182 | class SPPF(nn.Module): 183 | # Spatial Pyramid Pooling - Fast (SPPF) layer for YOLOV5 184 | 185 | def __init__(self,c1,c2, k=5): 186 | super().__init__() 187 | 188 | c_ = c1 // 2 189 | self.cv1 = Conv(c1, c_, 1, 1) 190 | self.cv2 = Conv(c_ *4, c2, 1, 1) 191 | self.m = nn.MaxPool2d(kernel_size=k, stride=1, padding=k //2) 192 | 193 | def forward(self,x): 194 | x = self.cv1(x) 195 | with warnings.catch_warnings(): 196 | warnings.simplefilter('ignore') 197 | y1= self.m(x) 198 | y2 = self.m(y1) 199 | return self.cv2(torch.cat([x, y1, y2, self.m(y2)], 1)) 200 | 201 | class Focus(nn.Module): 202 | # Focus wh information into c-space 203 | def __init__(self, c1, c2, k=1, s=1, p=None, g=1, act=True): 204 | super().__init__ 205 | self.conv(c1*4, c2, k, s, p, g, act) 206 | 207 | def forward(self,x): # x(b,c,w,h) -> y(b,4c,w/2,h/2) 208 | return self.conv(torch.cat([x[..., ::2, ::2], x[..., 1::2, ::2], x[..., ::2, 1::2], x[..., 1::2, 1::2]], 1)) 209 | 210 | class GhostBottleneck(nn.Module): 211 | # Chost bottleneck 212 | 213 | def __init__(self, c1, c2, k=3, s=1): 214 | super().__init__() 215 | c_ = c2 // 2 216 | self.conv = nn.Sequential(GhostConv(c1, c_, 1, 1), 217 | DWConv(c_,c_,k,s, act=False) if s==2 else nn.Identity(), 218 | GhostConv(c_, c2, 1, 1, act=False)) 219 | 220 | self.shortcut = nn.Sequential(DWConv(c2, c1, k, s, act= False), 221 | Conv(c1, c2, 1, 1, act=False) if s == 2 else nn.Identity()) 222 | 223 | 224 | def forward(self,x): 225 | return self.conv(x) + self.shortcut(x) 226 | 227 | 228 | class GhostConv(nn.Module): 229 | # Chost convolution 230 | def __init__(self, c1, c2, k=1, s=1, g=1, act= True): 231 | super().__init__() 232 | c_ = c1 // 2 233 | self.cv1 = Conv(c1, c_, k, s, None, g, act) 234 | self.cv2 = Conv(c_, c_, 5, 1, None, c_, act) 235 | 236 | def forward(self,x): 237 | y = self.cv1(x) 238 | return torch.cat([y, self.cv2(y)], 1) 239 | 240 | 241 | class Contract(nn.Module): 242 | # contract width=height into channels, i.e x(1,64,80,80) to x(1,256,40,40) 243 | 244 | def __init__(self,gain=2): 245 | super().__init__() 246 | self.gain = gain 247 | 248 | def forward(self,x): 249 | b, c, h, w = x.size() 250 | s = self.gain 251 | x = x.view(b, c, h //s, s, w //s, s) 252 | x =x.permute(0, 3, 5, 1, 2, 4).contiguous() 253 | return x.view(b, c * s *s, h//s, w//s) 254 | 255 | class Expand(nn.Module): 256 | # Expand channels into width-height, i.e. x(1,64,80,80) to x(1,16,160,160) 257 | def __init__(self, gain=2): 258 | super().__init__() 259 | self.gain = gain 260 | 261 | def forward(self, x): 262 | b, c, h, w = x.size() # assert C / s ** 2 == 0, 'Indivisible gain' 263 | s = self.gain 264 | x = x.view(b, s, s, c // s ** 2, h, w) # x(1,2,2,16,80,80) 265 | x = x.permute(0, 3, 4, 1, 5, 2).contiguous() # x(1,16,80,2,80,2) 266 | return x.view(b, c // s ** 2, h * s, w * s) # x(1,16,160,160) 267 | 268 | 269 | class Concat(nn.Module): 270 | # Concatenate a list of tensors along dimension 271 | def __init__(self, dimension=1): 272 | super().__init__() 273 | self.d = dimension 274 | 275 | def forward(self, x): 276 | return torch.cat(x, self.d) 277 | 278 | 279 | class AutoShape(nn.Module): 280 | # YOLOv5 input- robust model warapper for passing cv2/np/PIL/torch inputs. Includes preprocessing, inference and NMS 281 | conf = 0.25 # NMS confidence threshold 282 | iou = 0.45 # NMS IoU threshold 283 | classes = None 284 | multi_label = False 285 | mx_det = 1000 # maximum number of detection per image 286 | 287 | def __init__(self, model): 288 | super().__init__() 289 | self.model = model.eval() 290 | 291 | def autoshape(self): 292 | LOGGER.info('AutoShape already enables, skiping...') 293 | 294 | def _apply(self,fn): 295 | 296 | self = super()._apply(fn) 297 | m = self.model.model[-1] 298 | m.stride = fn(m.stride) 299 | m.grid = list(map(fn, m.grid)) 300 | if isinstance(m.anchor_grid, list): 301 | m.anchor_grid = list(map(fn, m.anchor_grid)) 302 | return self 303 | 304 | 305 | @torch.no_grad() 306 | def forward(self, imgs, size=640, augment = False, profile= False): 307 | # Inference from various sources. For height=640, width=1280, RGB images example inputs are: 308 | # file: imgs = 'data/images/zidane.jpg' # str or PosixPath 309 | # URI: = 'https://ultralytics.com/images/zidane.jpg' 310 | # OpenCV: = cv2.imread('image.jpg')[:,:,::-1] # HWC BGR to RGB x(640,1280,3) 311 | # PIL: = Image.open('image.jpg') or ImageGrab.grab() # HWC x(640,1280,3) 312 | # numpy: = np.zeros((640,1280,3)) # HWC 313 | # torch: = torch.zeros(16,3,320,640) # BCHW (scaled to size=640, 0-1 values) 314 | # multiple: = [Image.open('image1.jpg'), Image.open('image2.jpg'), ...] # list of images 315 | 316 | t = [time_sync()] 317 | p = next(self.model.parameters()) # for device and type 318 | 319 | if isinstance(imgs,torch.Tensor): # torch 320 | with amp.autocast(enabled=p.device.type != 'cpu'): 321 | return self.model(imgs.to(p.device).type_as(p), augment, profile) 322 | 323 | 324 | 325 | 326 | 327 | 328 | 329 | 330 | 331 | 332 | 333 | 334 | 335 | 336 | 337 | 338 | 339 | 340 | -------------------------------------------------------------------------------- /YOLOv5-SPD/hubconf.py: -------------------------------------------------------------------------------- 1 | # YOLOv5 🚀 by Ultralytics, GPL-3.0 license 2 | """ 3 | PyTorch Hub models https://pytorch.org/hub/ultralytics_yolov5/ 4 | 5 | Usage: 6 | import torch 7 | model = torch.hub.load('ultralytics/yolov5', 'yolov5s') 8 | """ 9 | 10 | import torch 11 | 12 | 13 | def _create(name, pretrained=True, channels=3, classes=80, autoshape=True, verbose=True, device=None): 14 | """Creates a specified YOLOv5 model 15 | 16 | Arguments: 17 | name (str): name of model, i.e. 'yolov5s' 18 | pretrained (bool): load pretrained weights into the model 19 | channels (int): number of input channels 20 | classes (int): number of model classes 21 | autoshape (bool): apply YOLOv5 .autoshape() wrapper to model 22 | verbose (bool): print all information to screen 23 | device (str, torch.device, None): device to use for model parameters 24 | 25 | Returns: 26 | YOLOv5 pytorch model 27 | """ 28 | from pathlib import Path 29 | 30 | from models.yolo import Model 31 | from models.experimental import attempt_load 32 | from utils.general import check_requirements, set_logging 33 | from utils.downloads import attempt_download 34 | from utils.torch_utils import select_device 35 | 36 | file = Path(__file__).resolve() 37 | check_requirements(exclude=('tensorboard', 'thop', 'opencv-python')) 38 | set_logging(verbose=verbose) 39 | 40 | save_dir = Path('') if str(name).endswith('.pt') else file.parent 41 | path = (save_dir / name).with_suffix('.pt') # checkpoint path 42 | try: 43 | device = select_device(('0' if torch.cuda.is_available() else 'cpu') if device is None else device) 44 | 45 | if pretrained and channels == 3 and classes == 80: 46 | model = attempt_load(path, map_location=device) # download/load FP32 model 47 | else: 48 | cfg = list((Path(__file__).parent / 'models').rglob(f'{name}.yaml'))[0] # model.yaml path 49 | model = Model(cfg, channels, classes) # create model 50 | if pretrained: 51 | ckpt = torch.load(attempt_download(path), map_location=device) # load 52 | msd = model.state_dict() # model state_dict 53 | csd = ckpt['model'].float().state_dict() # checkpoint state_dict as FP32 54 | csd = {k: v for k, v in csd.items() if msd[k].shape == v.shape} # filter 55 | model.load_state_dict(csd, strict=False) # load 56 | if len(ckpt['model'].names) == classes: 57 | model.names = ckpt['model'].names # set class names attribute 58 | if autoshape: 59 | model = model.autoshape() # for file/URI/PIL/cv2/np inputs and NMS 60 | return model.to(device) 61 | 62 | except Exception as e: 63 | help_url = 'https://github.com/ultralytics/yolov5/issues/36' 64 | s = 'Cache may be out of date, try `force_reload=True`. See %s for help.' % help_url 65 | raise Exception(s) from e 66 | 67 | 68 | def custom(path='path/to/model.pt', autoshape=True, verbose=True, device=None): 69 | # YOLOv5 custom or local model 70 | return _create(path, autoshape=autoshape, verbose=verbose, device=device) 71 | 72 | 73 | def yolov5n(pretrained=True, channels=3, classes=80, autoshape=True, verbose=True, device=None): 74 | # YOLOv5-nano model https://github.com/ultralytics/yolov5 75 | return _create('yolov5n', pretrained, channels, classes, autoshape, verbose, device) 76 | 77 | 78 | def yolov5s(pretrained=True, channels=3, classes=80, autoshape=True, verbose=True, device=None): 79 | # YOLOv5-small model https://github.com/ultralytics/yolov5 80 | return _create('yolov5s', pretrained, channels, classes, autoshape, verbose, device) 81 | 82 | 83 | def yolov5m(pretrained=True, channels=3, classes=80, autoshape=True, verbose=True, device=None): 84 | # YOLOv5-medium model https://github.com/ultralytics/yolov5 85 | return _create('yolov5m', pretrained, channels, classes, autoshape, verbose, device) 86 | 87 | 88 | def yolov5l(pretrained=True, channels=3, classes=80, autoshape=True, verbose=True, device=None): 89 | # YOLOv5-large model https://github.com/ultralytics/yolov5 90 | return _create('yolov5l', pretrained, channels, classes, autoshape, verbose, device) 91 | 92 | 93 | def yolov5x(pretrained=True, channels=3, classes=80, autoshape=True, verbose=True, device=None): 94 | # YOLOv5-xlarge model https://github.com/ultralytics/yolov5 95 | return _create('yolov5x', pretrained, channels, classes, autoshape, verbose, device) 96 | 97 | 98 | def yolov5n6(pretrained=True, channels=3, classes=80, autoshape=True, verbose=True, device=None): 99 | # YOLOv5-nano-P6 model https://github.com/ultralytics/yolov5 100 | return _create('yolov5n6', pretrained, channels, classes, autoshape, verbose, device) 101 | 102 | 103 | def yolov5s6(pretrained=True, channels=3, classes=80, autoshape=True, verbose=True, device=None): 104 | # YOLOv5-small-P6 model https://github.com/ultralytics/yolov5 105 | return _create('yolov5s6', pretrained, channels, classes, autoshape, verbose, device) 106 | 107 | 108 | def yolov5m6(pretrained=True, channels=3, classes=80, autoshape=True, verbose=True, device=None): 109 | # YOLOv5-medium-P6 model https://github.com/ultralytics/yolov5 110 | return _create('yolov5m6', pretrained, channels, classes, autoshape, verbose, device) 111 | 112 | 113 | def yolov5l6(pretrained=True, channels=3, classes=80, autoshape=True, verbose=True, device=None): 114 | # YOLOv5-large-P6 model https://github.com/ultralytics/yolov5 115 | return _create('yolov5l6', pretrained, channels, classes, autoshape, verbose, device) 116 | 117 | 118 | def yolov5x6(pretrained=True, channels=3, classes=80, autoshape=True, verbose=True, device=None): 119 | # YOLOv5-xlarge-P6 model https://github.com/ultralytics/yolov5 120 | return _create('yolov5x6', pretrained, channels, classes, autoshape, verbose, device) 121 | 122 | 123 | if __name__ == '__main__': 124 | model = _create(name='yolov5s', pretrained=True, channels=3, classes=80, autoshape=True, verbose=True) # pretrained 125 | # model = custom(path='path/to/model.pt') # custom 126 | 127 | # Verify inference 128 | import cv2 129 | import numpy as np 130 | from PIL import Image 131 | from pathlib import Path 132 | 133 | imgs = ['data/images/zidane.jpg', # filename 134 | Path('data/images/zidane.jpg'), # Path 135 | 'https://ultralytics.com/images/zidane.jpg', # URI 136 | cv2.imread('data/images/bus.jpg')[:, :, ::-1], # OpenCV 137 | Image.open('data/images/bus.jpg'), # PIL 138 | np.zeros((320, 640, 3))] # numpy 139 | 140 | results = model(imgs) # batched inference 141 | results.print() 142 | results.save() 143 | -------------------------------------------------------------------------------- /YOLOv5-SPD/models/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LabSAINT/SPD-Conv/c09f7928b8d2c33034fc3e9889a866f8bf37b44d/YOLOv5-SPD/models/__init__.py -------------------------------------------------------------------------------- /YOLOv5-SPD/models/experimental.py: -------------------------------------------------------------------------------- 1 | """ 2 | Experimental modules 3 | """ 4 | 5 | import numpy as np 6 | import torch 7 | import torch.nn as nn 8 | 9 | from models.common import Conv 10 | from utils.downloads import attempt_download 11 | 12 | 13 | class CrossConv(nn.Module): 14 | # Cross Convolution Downsample 15 | def __init__(self, c1, c2, k=3, s=1, g=1, e=1.0, shortcut=False): 16 | # ch_in, ch_out, kernel, stride, groups, expansion, shortcut 17 | super().__init__() 18 | c_ = int(c2 * e) # hidden channels 19 | self.cv1 = Conv(c1, c_, (1, k), (1, s)) 20 | self.cv2 = Conv(c_, c2, (k, 1), (s, 1), g=g) 21 | self.add = shortcut and c1 == c2 22 | 23 | def forward(self, x): 24 | return x + self.cv2(self.cv1(x)) if self.add else self.cv2(self.cv1(x)) 25 | 26 | 27 | class Sum(nn.Module): 28 | # Weighted sum of 2 or more layers https://arxiv.org/abs/1911.09070 29 | def __init__(self, n, weight=False): # n: number of inputs 30 | super().__init__() 31 | self.weight = weight # apply weights boolean 32 | self.iter = range(n - 1) # iter object 33 | if weight: 34 | self.w = nn.Parameter(-torch.arange(1., n) / 2, requires_grad=True) # layer weights 35 | 36 | def forward(self, x): 37 | y = x[0] # no weight 38 | if self.weight: 39 | w = torch.sigmoid(self.w) * 2 40 | for i in self.iter: 41 | y = y + x[i + 1] * w[i] 42 | else: 43 | for i in self.iter: 44 | y = y + x[i + 1] 45 | return y 46 | 47 | 48 | class MixConv2d(nn.Module): 49 | # Mixed Depth-wise Conv https://arxiv.org/abs/1907.09595 50 | def __init__(self, c1, c2, k=(1, 3), s=1, equal_ch=True): 51 | super().__init__() 52 | groups = len(k) 53 | if equal_ch: # equal c_ per group 54 | i = torch.linspace(0, groups - 1E-6, c2).floor() # c2 indices 55 | c_ = [(i == g).sum() for g in range(groups)] # intermediate channels 56 | else: # equal weight.numel() per group 57 | b = [c2] + [0] * groups 58 | a = np.eye(groups + 1, groups, k=-1) 59 | a -= np.roll(a, 1, axis=1) 60 | a *= np.array(k) ** 2 61 | a[0] = 1 62 | c_ = np.linalg.lstsq(a, b, rcond=None)[0].round() # solve for equal weight indices, ax = b 63 | 64 | self.m = nn.ModuleList([nn.Conv2d(c1, int(c_[g]), k[g], s, k[g] // 2, bias=False) for g in range(groups)]) 65 | self.bn = nn.BatchNorm2d(c2) 66 | self.act = nn.LeakyReLU(0.1, inplace=True) 67 | 68 | def forward(self, x): 69 | return x + self.act(self.bn(torch.cat([m(x) for m in self.m], 1))) 70 | 71 | 72 | class Ensemble(nn.ModuleList): 73 | # Ensemble of models 74 | def __init__(self): 75 | super().__init__() 76 | 77 | def forward(self, x, augment=False, profile=False, visualize=False): 78 | y = [] 79 | for module in self: 80 | y.append(module(x, augment, profile, visualize)[0]) 81 | # y = torch.stack(y).max(0)[0] # max ensemble 82 | # y = torch.stack(y).mean(0) # mean ensemble 83 | y = torch.cat(y, 1) # nms ensemble 84 | return y, None # inference, train output 85 | 86 | 87 | def attempt_load(weights, map_location=None, inplace=True, fuse=True): 88 | from models.yolo import Detect, Model 89 | 90 | # Loads an ensemble of models weights=[a,b,c] or a single model weights=[a] or weights=a 91 | model = Ensemble() 92 | for w in weights if isinstance(weights, list) else [weights]: 93 | ckpt = torch.load(attempt_download(w), map_location=map_location) # load 94 | if fuse: 95 | model.append(ckpt['ema' if ckpt.get('ema') else 'model'].float().fuse().eval()) # FP32 model 96 | else: 97 | model.append(ckpt['ema' if ckpt.get('ema') else 'model'].float().eval()) # without layer fuse 98 | 99 | 100 | # Compatibility updates 101 | for m in model.modules(): 102 | if type(m) in [nn.Hardswish, nn.LeakyReLU, nn.ReLU, nn.ReLU6, nn.SiLU, Detect, Model]: 103 | m.inplace = inplace # pytorch 1.7.0 compatibility 104 | if type(m) is Detect: 105 | if not isinstance(m.anchor_grid, list): # new Detect Layer compatibility 106 | delattr(m, 'anchor_grid') 107 | setattr(m, 'anchor_grid', [torch.zeros(1)] * m.nl) 108 | elif type(m) is Conv: 109 | m._non_persistent_buffers_set = set() # pytorch 1.6.0 compatibility 110 | 111 | if len(model) == 1: 112 | return model[-1] # return model 113 | else: 114 | print(f'Ensemble created with {weights}\n') 115 | for k in ['names']: 116 | setattr(model, k, getattr(model[-1], k)) 117 | model.stride = model[torch.argmax(torch.tensor([m.stride.max() for m in model])).int()].stride # max stride 118 | return model # return ensemble 119 | -------------------------------------------------------------------------------- /YOLOv5-SPD/models/new_models.py: -------------------------------------------------------------------------------- 1 | from random import weibullvariate 2 | import torch 3 | import torch.nn as nn 4 | 5 | class AFF(nn.Module): 6 | """ 7 | Implimenting AFF module 8 | """ 9 | 10 | def __init__(self, channels=64, r=4): 11 | super(AFF, self).__init__() 12 | inter_channels = int(channels //r ) 13 | 14 | self.local_att = nn.Sequential( 15 | nn.Conv2d(channels, inter_channels, kernel_size=1, stride=1, padding=0), 16 | nn.BatchNorm2d(inter_channels), 17 | # nn.ReLU(inplace=True), 18 | nn.SiLU(), 19 | nn.Conv2d(inter_channels, channels, kernel_size=1, stride=1, padding=0), 20 | nn.BatchNorm2d(channels), 21 | ) 22 | 23 | self.global_att = nn.Sequential( 24 | nn.AdaptiveAvgPool2d(1), 25 | nn.Conv2d(channels, inter_channels, kernel_size=1, stride=1, padding=0), 26 | nn.BatchNorm2d(inter_channels), 27 | # nn.ReLU(inplace=True), 28 | nn.SiLU(), 29 | nn.Conv2d(inter_channels, channels, kernel_size=1, stride=1, padding=0), 30 | nn.BatchNorm2d(channels), 31 | ) 32 | 33 | self.sigmoid = nn.Sigmoid() 34 | 35 | def forward(self, input): 36 | x= input[0] 37 | y= input[1] 38 | xa = x + y 39 | xl = self.local_att(xa) 40 | xg= self.global_att(xa) 41 | xlg = xl + xg 42 | m = self.sigmoid(xlg) 43 | 44 | x_union_y = 2* x * m + 2* y * (1-m) 45 | 46 | return x_union_y 47 | 48 | 49 | 50 | class iAFF(nn.Module): 51 | 52 | """ 53 | implimenting iAFF module 54 | """ 55 | 56 | def __init__(self, channels=64, r=4): 57 | super(iAFF, self).__init__() 58 | inter_channels = int(channels // r) 59 | 60 | self.local_attention1 = nn.Sequential( 61 | nn.Conv2d(channels, inter_channels, kernel_size=1, stride=1, padding=0), 62 | nn.BatchNorm2d(inter_channels), 63 | # nn.ReLU(inplace=True), 64 | nn.SiLU(), 65 | nn.Conv2d(inter_channels, channels, kernel_size=1, stride=1, padding=0), 66 | nn.BatchNorm2d(channels), 67 | ) 68 | self.global_attention1 = nn.Sequential( 69 | nn.AdaptiveAvgPool2d(1), 70 | nn.Conv2d(channels, inter_channels, kernel_size=1, stride=1, padding=0), 71 | nn.BatchNorm2d(inter_channels), 72 | # nn.ReLU(inplace=True), 73 | nn.SiLU(), 74 | nn.Conv2d(inter_channels, channels, kernel_size=1, stride=1, padding=0), 75 | nn.BatchNorm2d(channels), 76 | ) 77 | 78 | self.local_attention2 = nn.Sequential( 79 | nn.Conv2d(channels, inter_channels, kernel_size=1, stride=1, padding=0), 80 | nn.BatchNorm2d(inter_channels), 81 | # nn.ReLU(inplace=True), 82 | nn.SiLU(), 83 | nn.Conv2d(inter_channels, channels, kernel_size=1, stride=1, padding=0), 84 | nn.BatchNorm2d(channels), 85 | ) 86 | self.global_attention2 = nn.Sequential( 87 | nn.AdaptiveAvgPool2d(1), 88 | nn.Conv2d(channels, inter_channels, kernel_size=1, stride=1, padding=0), 89 | nn.BatchNorm2d(inter_channels), 90 | # nn.ReLU(inplace=True), 91 | nn.SiLU(), 92 | nn.Conv2d(inter_channels, channels, kernel_size=1, stride=1, padding=0), 93 | nn.BatchNorm2d(channels), 94 | ) 95 | 96 | self.sigmoid = nn.Sigmoid() 97 | 98 | 99 | def forward(self, input): 100 | """ 101 | Implimenting the iAFF forward step 102 | """ 103 | x = input[0] 104 | y = input[1] 105 | xa = x+y 106 | xl = self.local_attention1(xa) 107 | xg = self.global_attention1(xa) 108 | xlg = xl+xg 109 | m1 = self.sigmoid(xlg) 110 | xuniony = x * m1 + y * (1-m1) 111 | 112 | xl2 = self.local_attention2(xuniony) 113 | xg2 = self.global_attention2(xuniony) 114 | xlg2 = xl2 + xg2 115 | m2 = self.sigmoid(xlg2) 116 | z = x * m2 + y * (1-m2) 117 | return z 118 | 119 | 120 | if __name__ == '__main__': 121 | import os 122 | x = torch.randn(8,64,32,32) 123 | y = torch.randn(8,64,32,32) 124 | channels = x.shape[1] 125 | 126 | model = iAFF(channels=channels) 127 | output = model(x,y) 128 | print(output) 129 | print(output.shape) 130 | 131 | 132 | 133 | 134 | -------------------------------------------------------------------------------- /YOLOv5-SPD/models/space_depth_l.yaml: -------------------------------------------------------------------------------- 1 | # Parameters 2 | nc: 80 # number of classes 3 | depth_multiple: 1 # model depth multiple 4 | width_multiple: 1 # layer channel multiple 5 | anchors: 6 | - [10,13, 16,30, 33,23] # P3/8 7 | - [30,61, 62,45, 59,119] # P4/16 8 | - [116,90, 156,198, 373,326] # P5/32 9 | 10 | # YOLOv5 v6.0 backbone 11 | backbone: 12 | # [from, number, module, args] 13 | [[-1, 1, Focus, [64, 3]], # 0-P1/2 14 | [-1, 1, Conv, [128, 3, 1]], # 1 15 | [-1,1,space_to_depth,[1]], # 2 -P2/4 16 | [-1, 3, C3, [128]], # 3 17 | [-1, 1, Conv, [256, 3, 1]], # 4 18 | [-1,1,space_to_depth,[1]], # 5 -P3/8 19 | [-1, 6, C3, [256]], # 6 20 | [-1, 1, Conv, [512, 3, 1]], # 7-P4/16 21 | [-1,1,space_to_depth,[1]], # 8 -P4/16 22 | [-1, 9, C3, [512]], # 9 23 | [-1, 1, Conv, [1024, 3, 1]], # 10-P5/32 24 | [-1,1,space_to_depth,[1]], # 11 -P5/32 25 | [-1, 3, C3, [1024]], # 12 26 | [-1, 1, SPPF, [1024, 5]], # 13 27 | ] 28 | 29 | # YOLOv5 v6.0 head 30 | head: 31 | [[-1, 1, Conv, [512, 1, 1]], # 14 32 | [-1, 1, nn.Upsample, [None, 2, 'nearest']], # 15 33 | [[-1, 9], 1, Concat, [1]], # 16 cat backbone P4 34 | [-1, 3, C3, [512, False]], # 17 35 | 36 | [-1, 1, Conv, [256, 1, 1]], # 18 37 | [-1, 1, nn.Upsample, [None, 2, 'nearest']], # 19 38 | [[-1, 6], 1, Concat, [1]], # 20 cat backbone P3 39 | [-1, 3, C3, [256, False]], # 21 (P3/8-small) 40 | 41 | [-1, 1, Conv, [256, 3, 1]], # 22 42 | [-1,1,space_to_depth,[1]], # 23 -P2/4 43 | [[-1, 18], 1, Concat, [1]], # 24 cat head P4 44 | [-1, 3, C3, [512, False]], # 25 (P4/16-medium) 45 | 46 | [-1, 1, Conv, [512, 3, 1]], # 26 47 | [-1,1,space_to_depth,[1]], # 27 -P2/4 48 | [[-1, 14], 1, Concat, [1]], # 28 cat head P5 49 | [-1, 3, C3, [1024, False]], # 29 (P5/32-large) 50 | 51 | [[21, 25, 29], 1, Detect, [nc, anchors]], # Detect(P3, P4, P5) 52 | ] 53 | -------------------------------------------------------------------------------- /YOLOv5-SPD/models/space_depth_m.yaml: -------------------------------------------------------------------------------- 1 | # Parameters 2 | nc: 80 # number of classes 3 | depth_multiple: 0.67 # model depth multiple 4 | width_multiple: 0.75 # layer channel multiple 5 | anchors: 6 | - [10,13, 16,30, 33,23] # P3/8 7 | - [30,61, 62,45, 59,119] # P4/16 8 | - [116,90, 156,198, 373,326] # P5/32 9 | 10 | # YOLOv5 v6.0 backbone 11 | backbone: 12 | # [from, number, module, args] 13 | [[-1, 1, Focus, [64, 3]], # 0-P1/2 14 | [-1, 1, Conv, [128, 3, 1]], # 1 15 | [-1,1,space_to_depth,[1]], # 2 -P2/4 16 | [-1, 3, C3, [128]], # 3 17 | [-1, 1, Conv, [256, 3, 1]], # 4 18 | [-1,1,space_to_depth,[1]], # 5 -P3/8 19 | [-1, 6, C3, [256]], # 6 20 | [-1, 1, Conv, [512, 3, 1]], # 7-P4/16 21 | [-1,1,space_to_depth,[1]], # 8 -P4/16 22 | [-1, 9, C3, [512]], # 9 23 | [-1, 1, Conv, [1024, 3, 1]], # 10-P5/32 24 | [-1,1,space_to_depth,[1]], # 11 -P5/32 25 | [-1, 3, C3, [1024]], # 12 26 | [-1, 1, SPPF, [1024, 5]], # 13 27 | ] 28 | 29 | # YOLOv5 v6.0 head 30 | head: 31 | [[-1, 1, Conv, [512, 1, 1]], # 14 32 | [-1, 1, nn.Upsample, [None, 2, 'nearest']], # 15 33 | [[-1, 9], 1, Concat, [1]], # 16 cat backbone P4 34 | [-1, 3, C3, [512, False]], # 17 35 | 36 | [-1, 1, Conv, [256, 1, 1]], # 18 37 | [-1, 1, nn.Upsample, [None, 2, 'nearest']], # 19 38 | [[-1, 6], 1, Concat, [1]], # 20 cat backbone P3 39 | [-1, 3, C3, [256, False]], # 21 (P3/8-small) 40 | 41 | [-1, 1, Conv, [256, 3, 1]], # 22 42 | [-1,1,space_to_depth,[1]], # 23 -P2/4 43 | [[-1, 18], 1, Concat, [1]], # 24 cat head P4 44 | [-1, 3, C3, [512, False]], # 25 (P4/16-medium) 45 | 46 | [-1, 1, Conv, [512, 3, 1]], # 26 47 | [-1,1,space_to_depth,[1]], # 27 -P2/4 48 | [[-1, 14], 1, Concat, [1]], # 28 cat head P5 49 | [-1, 3, C3, [1024, False]], # 29 (P5/32-large) 50 | 51 | [[21, 25, 29], 1, Detect, [nc, anchors]], # Detect(P3, P4, P5) 52 | ] 53 | -------------------------------------------------------------------------------- /YOLOv5-SPD/models/space_depth_n.yaml: -------------------------------------------------------------------------------- 1 | # Parameters 2 | nc: 80 # number of classes 3 | depth_multiple: 0.33 # model depth multiple 4 | width_multiple: 0.25 # layer channel multiple 5 | anchors: 6 | - [10,13, 16,30, 33,23] # P3/8 7 | - [30,61, 62,45, 59,119] # P4/16 8 | - [116,90, 156,198, 373,326] # P5/32 9 | 10 | # YOLOv5 v6.0 backbone 11 | backbone: 12 | # [from, number, module, args] 13 | [[-1, 1, Focus, [64, 3]], # 0-P1/2 14 | [-1, 1, Conv, [128, 3, 1]], # 1 15 | [-1,1,space_to_depth,[1]], # 2 -P2/4 16 | [-1, 3, C3, [128]], # 3 17 | [-1, 1, Conv, [256, 3, 1]], # 4 18 | [-1,1,space_to_depth,[1]], # 5 -P3/8 19 | [-1, 6, C3, [256]], # 6 20 | [-1, 1, Conv, [512, 3, 1]], # 7-P4/16 21 | [-1,1,space_to_depth,[1]], # 8 -P4/16 22 | [-1, 9, C3, [512]], # 9 23 | [-1, 1, Conv, [1024, 3, 1]], # 10-P5/32 24 | [-1,1,space_to_depth,[1]], # 11 -P5/32 25 | [-1, 3, C3, [1024]], # 12 26 | [-1, 1, SPPF, [1024, 5]], # 13 27 | ] 28 | 29 | # YOLOv5 v6.0 head 30 | head: 31 | [[-1, 1, Conv, [512, 1, 1]], # 14 32 | [-1, 1, nn.Upsample, [None, 2, 'nearest']], # 15 33 | [[-1, 9], 1, Concat, [1]], # 16 cat backbone P4 34 | [-1, 3, C3, [512, False]], # 17 35 | 36 | [-1, 1, Conv, [256, 1, 1]], # 18 37 | [-1, 1, nn.Upsample, [None, 2, 'nearest']], # 19 38 | [[-1, 6], 1, Concat, [1]], # 20 cat backbone P3 39 | [-1, 3, C3, [256, False]], # 21 (P3/8-small) 40 | 41 | [-1, 1, Conv, [256, 3, 1]], # 22 42 | [-1,1,space_to_depth,[1]], # 23 -P2/4 43 | [[-1, 18], 1, Concat, [1]], # 24 cat head P4 44 | [-1, 3, C3, [512, False]], # 25 (P4/16-medium) 45 | 46 | [-1, 1, Conv, [512, 3, 1]], # 26 47 | [-1,1,space_to_depth,[1]], # 27 -P2/4 48 | [[-1, 14], 1, Concat, [1]], # 28 cat head P5 49 | [-1, 3, C3, [1024, False]], # 29 (P5/32-large) 50 | 51 | [[21, 25, 29], 1, Detect, [nc, anchors]], # Detect(P3, P4, P5) 52 | ] 53 | -------------------------------------------------------------------------------- /YOLOv5-SPD/models/space_depth_s.yaml: -------------------------------------------------------------------------------- 1 | # Parameters 2 | nc: 80 # number of classes 3 | depth_multiple: 0.33 # model depth multiple 4 | width_multiple: 0.50 # layer channel multiple 5 | anchors: 6 | - [10,13, 16,30, 33,23] # P3/8 7 | - [30,61, 62,45, 59,119] # P4/16 8 | - [116,90, 156,198, 373,326] # P5/32 9 | 10 | # YOLOv5 v6.0 backbone 11 | backbone: 12 | # [from, number, module, args] 13 | [[-1, 1, Focus, [64, 3]], # 0-P1/2 14 | [-1, 1, Conv, [128, 3, 1]], # 1 15 | [-1,1,space_to_depth,[1]], # 2 -P2/4 16 | [-1, 3, C3, [128]], # 3 17 | [-1, 1, Conv, [256, 3, 1]], # 4 18 | [-1,1,space_to_depth,[1]], # 5 -P3/8 19 | [-1, 6, C3, [256]], # 6 20 | [-1, 1, Conv, [512, 3, 1]], # 7-P4/16 21 | [-1,1,space_to_depth,[1]], # 8 -P4/16 22 | [-1, 9, C3, [512]], # 9 23 | [-1, 1, Conv, [1024, 3, 1]], # 10-P5/32 24 | [-1,1,space_to_depth,[1]], # 11 -P5/32 25 | [-1, 3, C3, [1024]], # 12 26 | [-1, 1, SPPF, [1024, 5]], # 13 27 | ] 28 | 29 | # YOLOv5 v6.0 head 30 | head: 31 | [[-1, 1, Conv, [512, 1, 1]], # 14 32 | [-1, 1, nn.Upsample, [None, 2, 'nearest']], # 15 33 | [[-1, 9], 1, Concat, [1]], # 16 cat backbone P4 34 | [-1, 3, C3, [512, False]], # 17 35 | 36 | [-1, 1, Conv, [256, 1, 1]], # 18 37 | [-1, 1, nn.Upsample, [None, 2, 'nearest']], # 19 38 | [[-1, 6], 1, Concat, [1]], # 20 cat backbone P3 39 | [-1, 3, C3, [256, False]], # 21 (P3/8-small) 40 | 41 | [-1, 1, Conv, [256, 3, 1]], # 22 42 | [-1,1,space_to_depth,[1]], # 23 -P2/4 43 | [[-1, 18], 1, Concat, [1]], # 24 cat head P4 44 | [-1, 3, C3, [512, False]], # 25 (P4/16-medium) 45 | 46 | [-1, 1, Conv, [512, 3, 1]], # 26 47 | [-1,1,space_to_depth,[1]], # 27 -P2/4 48 | [[-1, 14], 1, Concat, [1]], # 28 cat head P5 49 | [-1, 3, C3, [1024, False]], # 29 (P5/32-large) 50 | 51 | [[21, 25, 29], 1, Detect, [nc, anchors]], # Detect(P3, P4, P5) 52 | ] 53 | -------------------------------------------------------------------------------- /YOLOv5-SPD/myoptims/tanangulargrad.py: -------------------------------------------------------------------------------- 1 | import math 2 | import torch 3 | from torch.optim.optimizer import Optimizer 4 | import numpy as np 5 | import torch.nn as nn 6 | 7 | 8 | class tanangulargrad(Optimizer): 9 | 10 | def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0): 11 | if not 0.0 <= lr: 12 | raise ValueError("Invalid learning rate: {}".format(lr)) 13 | if not 0.0 <= eps: 14 | raise ValueError("Invalid epsilon value: {}".format(eps)) 15 | if not 0.0 <= betas[0] < 1.0: 16 | raise ValueError("Invalid beta parameter at index 0: {}".format(betas[0])) 17 | if not 0.0 <= betas[1] < 1.0: 18 | raise ValueError("Invalid beta parameter at index 1: {}".format(betas[1])) 19 | defaults = dict(lr=lr, betas=betas, eps=eps, weight_decay=weight_decay) 20 | super(tanangulargrad, self).__init__(params, defaults) 21 | 22 | def __setstate__(self, state): 23 | super(tanangulargrad, self).__setstate__(state) 24 | 25 | def step(self, closure=None): 26 | """Performs a single optimization step. 27 | Arguments: 28 | closure (callable, optional): A closure that reevaluates the model 29 | and returns the loss. 30 | """ 31 | loss = None 32 | if closure is not None: 33 | loss = closure() 34 | 35 | for group in self.param_groups: 36 | for p in group['params']: 37 | if p.grad is None: 38 | continue 39 | grad = p.grad.data 40 | if grad.is_sparse: 41 | raise RuntimeError( 42 | 'tanangulargrad does not support sparse gradients, please consider SparseAdam instead') 43 | 44 | state = self.state[p] 45 | 46 | # State initialization 47 | if len(state) == 0: 48 | state['step'] = 0 49 | # Exponential moving average of gradient values 50 | state['exp_avg'] = torch.zeros_like(p.data) 51 | # Exponential moving average of squared gradient values 52 | state['exp_avg_sq'] = torch.zeros_like(p.data) 53 | # Previous gradient 54 | state['previous_grad'] = torch.zeros_like(p.data) 55 | # temporary minimum value for comparison 56 | state['min'] = torch.zeros_like(p.data) 57 | # temporary difference between gradients for comparison 58 | state['diff'] = torch.zeros_like(p.data) 59 | # final tan value to be used 60 | state['final_tan_theta'] = torch.zeros_like(p.data) 61 | 62 | exp_avg, exp_avg_sq, previous_grad, min, diff, final_tan_theta = state['exp_avg'], state['exp_avg_sq'], \ 63 | state['previous_grad'], state['min'], \ 64 | state['diff'], state['final_tan_theta'] 65 | beta1, beta2 = group['betas'] 66 | 67 | state['step'] += 1 68 | 69 | if group['weight_decay'] != 0: 70 | grad.add_(group['weight_decay'], p.data) 71 | 72 | # Decay the first and second moment running average coefficient 73 | exp_avg.mul_(beta1).add_(1 - beta1, grad) 74 | exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad) 75 | denom = exp_avg_sq.sqrt().add_(group['eps']) 76 | 77 | bias_correction1 = 1 - beta1 ** state['step'] 78 | bias_correction2 = 1 - beta2 ** state['step'] 79 | 80 | tan_theta = abs((previous_grad - grad) / (1 + previous_grad * grad)) 81 | 82 | angle = torch.atan(tan_theta) * (180 / 3.141592653589793238) 83 | ans = torch.gt(angle, min) 84 | ans1, count = torch.unique(ans, return_counts=True) 85 | 86 | try: 87 | if (count[1] < count[0]): 88 | min = angle 89 | diff = abs(previous_grad - grad) 90 | final_tan_theta = tan_theta.clone() 91 | except: 92 | if (ans1[0].item() == False): 93 | min = angle 94 | diff = abs(previous_grad - grad) 95 | final_tan_theta = tan_theta.clone() 96 | 97 | angular_coeff = torch.tanh(abs(final_tan_theta)) * 0.5 +0.5 # Calculating Angular coefficient 98 | 99 | state['previous_grad'] = grad.clone() 100 | state['min'] = min.clone() 101 | state['diff'] = diff.clone() 102 | state['final_tan_theta'] = final_tan_theta.clone() 103 | 104 | # update momentum with angular_coeff 105 | exp_avg1 = exp_avg * angular_coeff 106 | 107 | step_size = group['lr'] * math.sqrt(bias_correction2) / bias_correction1 108 | 109 | p.data.addcdiv_(-step_size, exp_avg1, denom) 110 | 111 | return loss 112 | -------------------------------------------------------------------------------- /YOLOv5-SPD/test_download.py: -------------------------------------------------------------------------------- 1 | 2 | # Download COCO test-dev2017 3 | 4 | import torch 5 | 6 | torch.hub.download_url_to_file('https://ultralytics.com/assets/coco2017labels.zip', 'tmp.zip') 7 | 8 | -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LabSAINT/SPD-Conv/c09f7928b8d2c33034fc3e9889a866f8bf37b44d/YOLOv5-SPD/utils/__init__.py -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/activations.py: -------------------------------------------------------------------------------- 1 | # YOLOv5 🚀 by Ultralytics, GPL-3.0 license 2 | """ 3 | Activation functions 4 | """ 5 | 6 | import torch 7 | import torch.nn as nn 8 | import torch.nn.functional as F 9 | 10 | 11 | # SiLU https://arxiv.org/pdf/1606.08415.pdf ---------------------------------------------------------------------------- 12 | class SiLU(nn.Module): # export-friendly version of nn.SiLU() 13 | @staticmethod 14 | def forward(x): 15 | return x * torch.sigmoid(x) 16 | 17 | 18 | class Hardswish(nn.Module): # export-friendly version of nn.Hardswish() 19 | @staticmethod 20 | def forward(x): 21 | # return x * F.hardsigmoid(x) # for torchscript and CoreML 22 | return x * F.hardtanh(x + 3, 0., 6.) / 6. # for torchscript, CoreML and ONNX 23 | 24 | 25 | # Mish https://github.com/digantamisra98/Mish -------------------------------------------------------------------------- 26 | class Mish(nn.Module): 27 | @staticmethod 28 | def forward(x): 29 | return x * F.softplus(x).tanh() 30 | 31 | 32 | class MemoryEfficientMish(nn.Module): 33 | class F(torch.autograd.Function): 34 | @staticmethod 35 | def forward(ctx, x): 36 | ctx.save_for_backward(x) 37 | return x.mul(torch.tanh(F.softplus(x))) # x * tanh(ln(1 + exp(x))) 38 | 39 | @staticmethod 40 | def backward(ctx, grad_output): 41 | x = ctx.saved_tensors[0] 42 | sx = torch.sigmoid(x) 43 | fx = F.softplus(x).tanh() 44 | return grad_output * (fx + x * sx * (1 - fx * fx)) 45 | 46 | def forward(self, x): 47 | return self.F.apply(x) 48 | 49 | 50 | # FReLU https://arxiv.org/abs/2007.11824 ------------------------------------------------------------------------------- 51 | class FReLU(nn.Module): 52 | def __init__(self, c1, k=3): # ch_in, kernel 53 | super().__init__() 54 | self.conv = nn.Conv2d(c1, c1, k, 1, 1, groups=c1, bias=False) 55 | self.bn = nn.BatchNorm2d(c1) 56 | 57 | def forward(self, x): 58 | return torch.max(x, self.bn(self.conv(x))) 59 | 60 | 61 | # ACON https://arxiv.org/pdf/2009.04759.pdf ---------------------------------------------------------------------------- 62 | class AconC(nn.Module): 63 | r""" ACON activation (activate or not). 64 | AconC: (p1*x-p2*x) * sigmoid(beta*(p1*x-p2*x)) + p2*x, beta is a learnable parameter 65 | according to "Activate or Not: Learning Customized Activation" . 66 | """ 67 | 68 | def __init__(self, c1): 69 | super().__init__() 70 | self.p1 = nn.Parameter(torch.randn(1, c1, 1, 1)) 71 | self.p2 = nn.Parameter(torch.randn(1, c1, 1, 1)) 72 | self.beta = nn.Parameter(torch.ones(1, c1, 1, 1)) 73 | 74 | def forward(self, x): 75 | dpx = (self.p1 - self.p2) * x 76 | return dpx * torch.sigmoid(self.beta * dpx) + self.p2 * x 77 | 78 | 79 | class MetaAconC(nn.Module): 80 | r""" ACON activation (activate or not). 81 | MetaAconC: (p1*x-p2*x) * sigmoid(beta*(p1*x-p2*x)) + p2*x, beta is generated by a small network 82 | according to "Activate or Not: Learning Customized Activation" . 83 | """ 84 | 85 | def __init__(self, c1, k=1, s=1, r=16): # ch_in, kernel, stride, r 86 | super().__init__() 87 | c2 = max(r, c1 // r) 88 | self.p1 = nn.Parameter(torch.randn(1, c1, 1, 1)) 89 | self.p2 = nn.Parameter(torch.randn(1, c1, 1, 1)) 90 | self.fc1 = nn.Conv2d(c1, c2, k, s, bias=True) 91 | self.fc2 = nn.Conv2d(c2, c1, k, s, bias=True) 92 | # self.bn1 = nn.BatchNorm2d(c2) 93 | # self.bn2 = nn.BatchNorm2d(c1) 94 | 95 | def forward(self, x): 96 | y = x.mean(dim=2, keepdims=True).mean(dim=3, keepdims=True) 97 | # batch-size 1 bug/instabilities https://github.com/ultralytics/yolov5/issues/2891 98 | # beta = torch.sigmoid(self.bn2(self.fc2(self.bn1(self.fc1(y))))) # bug/unstable 99 | beta = torch.sigmoid(self.fc2(self.fc1(y))) # bug patch BN layers removed 100 | dpx = (self.p1 - self.p2) * x 101 | return dpx * torch.sigmoid(beta * dpx) + self.p2 * x 102 | -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/autoanchor.py: -------------------------------------------------------------------------------- 1 | # YOLOv5 🚀 by Ultralytics, GPL-3.0 license 2 | """ 3 | Auto-anchor utils 4 | """ 5 | 6 | import random 7 | 8 | import numpy as np 9 | import torch 10 | import yaml 11 | from tqdm import tqdm 12 | 13 | from utils.general import colorstr 14 | 15 | 16 | def check_anchor_order(m): 17 | # Check anchor order against stride order for YOLOv5 Detect() module m, and correct if necessary 18 | a = m.anchors.prod(-1).view(-1) # anchor area 19 | da = a[-1] - a[0] # delta a 20 | ds = m.stride[-1] - m.stride[0] # delta s 21 | if da.sign() != ds.sign(): # same order 22 | print('Reversing anchor order') 23 | m.anchors[:] = m.anchors.flip(0) 24 | 25 | 26 | def check_anchors(dataset, model, thr=4.0, imgsz=640): 27 | # Check anchor fit to data, recompute if necessary 28 | prefix = colorstr('autoanchor: ') 29 | print(f'\n{prefix}Analyzing anchors... ', end='') 30 | m = model.module.model[-1] if hasattr(model, 'module') else model.model[-1] # Detect() 31 | shapes = imgsz * dataset.shapes / dataset.shapes.max(1, keepdims=True) 32 | scale = np.random.uniform(0.9, 1.1, size=(shapes.shape[0], 1)) # augment scale 33 | wh = torch.tensor(np.concatenate([l[:, 3:5] * s for s, l in zip(shapes * scale, dataset.labels)])).float() # wh 34 | 35 | def metric(k): # compute metric 36 | r = wh[:, None] / k[None] 37 | x = torch.min(r, 1. / r).min(2)[0] # ratio metric 38 | best = x.max(1)[0] # best_x 39 | aat = (x > 1. / thr).float().sum(1).mean() # anchors above threshold 40 | bpr = (best > 1. / thr).float().mean() # best possible recall 41 | return bpr, aat 42 | 43 | anchors = m.anchors.clone() * m.stride.to(m.anchors.device).view(-1, 1, 1) # current anchors 44 | bpr, aat = metric(anchors.cpu().view(-1, 2)) 45 | print(f'anchors/target = {aat:.2f}, Best Possible Recall (BPR) = {bpr:.4f}', end='') 46 | if bpr < 0.98: # threshold to recompute 47 | print('. Attempting to improve anchors, please wait...') 48 | na = m.anchors.numel() // 2 # number of anchors 49 | try: 50 | anchors = kmean_anchors(dataset, n=na, img_size=imgsz, thr=thr, gen=1000, verbose=False) 51 | except Exception as e: 52 | print(f'{prefix}ERROR: {e}') 53 | new_bpr = metric(anchors)[0] 54 | if new_bpr > bpr: # replace anchors 55 | anchors = torch.tensor(anchors, device=m.anchors.device).type_as(m.anchors) 56 | m.anchors[:] = anchors.clone().view_as(m.anchors) / m.stride.to(m.anchors.device).view(-1, 1, 1) # loss 57 | check_anchor_order(m) 58 | print(f'{prefix}New anchors saved to model. Update model *.yaml to use these anchors in the future.') 59 | else: 60 | print(f'{prefix}Original anchors better than new anchors. Proceeding with original anchors.') 61 | print('') # newline 62 | 63 | 64 | def kmean_anchors(dataset='./data/coco128.yaml', n=9, img_size=640, thr=4.0, gen=1000, verbose=True): 65 | """ Creates kmeans-evolved anchors from training dataset 66 | 67 | Arguments: 68 | dataset: path to data.yaml, or a loaded dataset 69 | n: number of anchors 70 | img_size: image size used for training 71 | thr: anchor-label wh ratio threshold hyperparameter hyp['anchor_t'] used for training, default=4.0 72 | gen: generations to evolve anchors using genetic algorithm 73 | verbose: print all results 74 | 75 | Return: 76 | k: kmeans evolved anchors 77 | 78 | Usage: 79 | from utils.autoanchor import *; _ = kmean_anchors() 80 | """ 81 | from scipy.cluster.vq import kmeans 82 | 83 | thr = 1. / thr 84 | prefix = colorstr('autoanchor: ') 85 | 86 | def metric(k, wh): # compute metrics 87 | r = wh[:, None] / k[None] 88 | x = torch.min(r, 1. / r).min(2)[0] # ratio metric 89 | # x = wh_iou(wh, torch.tensor(k)) # iou metric 90 | return x, x.max(1)[0] # x, best_x 91 | 92 | def anchor_fitness(k): # mutation fitness 93 | _, best = metric(torch.tensor(k, dtype=torch.float32), wh) 94 | return (best * (best > thr).float()).mean() # fitness 95 | 96 | def print_results(k): 97 | k = k[np.argsort(k.prod(1))] # sort small to large 98 | x, best = metric(k, wh0) 99 | bpr, aat = (best > thr).float().mean(), (x > thr).float().mean() * n # best possible recall, anch > thr 100 | print(f'{prefix}thr={thr:.2f}: {bpr:.4f} best possible recall, {aat:.2f} anchors past thr') 101 | print(f'{prefix}n={n}, img_size={img_size}, metric_all={x.mean():.3f}/{best.mean():.3f}-mean/best, ' 102 | f'past_thr={x[x > thr].mean():.3f}-mean: ', end='') 103 | for i, x in enumerate(k): 104 | print('%i,%i' % (round(x[0]), round(x[1])), end=', ' if i < len(k) - 1 else '\n') # use in *.cfg 105 | return k 106 | 107 | if isinstance(dataset, str): # *.yaml file 108 | with open(dataset, errors='ignore') as f: 109 | data_dict = yaml.safe_load(f) # model dict 110 | from utils.datasets import LoadImagesAndLabels 111 | dataset = LoadImagesAndLabels(data_dict['train'], augment=True, rect=True) 112 | 113 | # Get label wh 114 | shapes = img_size * dataset.shapes / dataset.shapes.max(1, keepdims=True) 115 | wh0 = np.concatenate([l[:, 3:5] * s for s, l in zip(shapes, dataset.labels)]) # wh 116 | 117 | # Filter 118 | i = (wh0 < 3.0).any(1).sum() 119 | if i: 120 | print(f'{prefix}WARNING: Extremely small objects found. {i} of {len(wh0)} labels are < 3 pixels in size.') 121 | wh = wh0[(wh0 >= 2.0).any(1)] # filter > 2 pixels 122 | # wh = wh * (np.random.rand(wh.shape[0], 1) * 0.9 + 0.1) # multiply by random scale 0-1 123 | 124 | # Kmeans calculation 125 | print(f'{prefix}Running kmeans for {n} anchors on {len(wh)} points...') 126 | s = wh.std(0) # sigmas for whitening 127 | k, dist = kmeans(wh / s, n, iter=30) # points, mean distance 128 | assert len(k) == n, f'{prefix}ERROR: scipy.cluster.vq.kmeans requested {n} points but returned only {len(k)}' 129 | k *= s 130 | wh = torch.tensor(wh, dtype=torch.float32) # filtered 131 | wh0 = torch.tensor(wh0, dtype=torch.float32) # unfiltered 132 | k = print_results(k) 133 | 134 | # Plot 135 | # k, d = [None] * 20, [None] * 20 136 | # for i in tqdm(range(1, 21)): 137 | # k[i-1], d[i-1] = kmeans(wh / s, i) # points, mean distance 138 | # fig, ax = plt.subplots(1, 2, figsize=(14, 7), tight_layout=True) 139 | # ax = ax.ravel() 140 | # ax[0].plot(np.arange(1, 21), np.array(d) ** 2, marker='.') 141 | # fig, ax = plt.subplots(1, 2, figsize=(14, 7)) # plot wh 142 | # ax[0].hist(wh[wh[:, 0]<100, 0],400) 143 | # ax[1].hist(wh[wh[:, 1]<100, 1],400) 144 | # fig.savefig('wh.png', dpi=200) 145 | 146 | # Evolve 147 | npr = np.random 148 | f, sh, mp, s = anchor_fitness(k), k.shape, 0.9, 0.1 # fitness, generations, mutation prob, sigma 149 | pbar = tqdm(range(gen), desc=f'{prefix}Evolving anchors with Genetic Algorithm:') # progress bar 150 | for _ in pbar: 151 | v = np.ones(sh) 152 | while (v == 1).all(): # mutate until a change occurs (prevent duplicates) 153 | v = ((npr.random(sh) < mp) * random.random() * npr.randn(*sh) * s + 1).clip(0.3, 3.0) 154 | kg = (k.copy() * v).clip(min=2.0) 155 | fg = anchor_fitness(kg) 156 | if fg > f: 157 | f, k = fg, kg.copy() 158 | pbar.desc = f'{prefix}Evolving anchors with Genetic Algorithm: fitness = {f:.4f}' 159 | if verbose: 160 | print_results(k) 161 | 162 | return print_results(k) 163 | -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/aws/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LabSAINT/SPD-Conv/c09f7928b8d2c33034fc3e9889a866f8bf37b44d/YOLOv5-SPD/utils/aws/__init__.py -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/aws/mime.sh: -------------------------------------------------------------------------------- 1 | # AWS EC2 instance startup 'MIME' script https://aws.amazon.com/premiumsupport/knowledge-center/execute-user-data-ec2/ 2 | # This script will run on every instance restart, not only on first start 3 | # --- DO NOT COPY ABOVE COMMENTS WHEN PASTING INTO USERDATA --- 4 | 5 | Content-Type: multipart/mixed; boundary="//" 6 | MIME-Version: 1.0 7 | 8 | --// 9 | Content-Type: text/cloud-config; charset="us-ascii" 10 | MIME-Version: 1.0 11 | Content-Transfer-Encoding: 7bit 12 | Content-Disposition: attachment; filename="cloud-config.txt" 13 | 14 | #cloud-config 15 | cloud_final_modules: 16 | - [scripts-user, always] 17 | 18 | --// 19 | Content-Type: text/x-shellscript; charset="us-ascii" 20 | MIME-Version: 1.0 21 | Content-Transfer-Encoding: 7bit 22 | Content-Disposition: attachment; filename="userdata.txt" 23 | 24 | #!/bin/bash 25 | # --- paste contents of userdata.sh here --- 26 | --// 27 | -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/aws/resume.py: -------------------------------------------------------------------------------- 1 | # Resume all interrupted trainings in yolov5/ dir including DDP trainings 2 | # Usage: $ python utils/aws/resume.py 3 | 4 | import os 5 | import sys 6 | from pathlib import Path 7 | 8 | import torch 9 | import yaml 10 | 11 | FILE = Path(__file__).resolve() 12 | ROOT = FILE.parents[2] # YOLOv5 root directory 13 | if str(ROOT) not in sys.path: 14 | sys.path.append(str(ROOT)) # add ROOT to PATH 15 | 16 | port = 0 # --master_port 17 | path = Path('').resolve() 18 | for last in path.rglob('*/**/last.pt'): 19 | ckpt = torch.load(last) 20 | if ckpt['optimizer'] is None: 21 | continue 22 | 23 | # Load opt.yaml 24 | with open(last.parent.parent / 'opt.yaml', errors='ignore') as f: 25 | opt = yaml.safe_load(f) 26 | 27 | # Get device count 28 | d = opt['device'].split(',') # devices 29 | nd = len(d) # number of devices 30 | ddp = nd > 1 or (nd == 0 and torch.cuda.device_count() > 1) # distributed data parallel 31 | 32 | if ddp: # multi-GPU 33 | port += 1 34 | cmd = f'python -m torch.distributed.run --nproc_per_node {nd} --master_port {port} train.py --resume {last}' 35 | else: # single-GPU 36 | cmd = f'python train.py --resume {last}' 37 | 38 | cmd += ' > /dev/null 2>&1 &' # redirect output to dev/null and run in daemon thread 39 | print(cmd) 40 | os.system(cmd) 41 | -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/aws/userdata.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # AWS EC2 instance startup script https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html 3 | # This script will run only once on first instance start (for a re-start script see mime.sh) 4 | # /home/ubuntu (ubuntu) or /home/ec2-user (amazon-linux) is working dir 5 | # Use >300 GB SSD 6 | 7 | cd home/ubuntu 8 | if [ ! -d yolov5 ]; then 9 | echo "Running first-time script." # install dependencies, download COCO, pull Docker 10 | git clone https://github.com/ultralytics/yolov5 -b master && sudo chmod -R 777 yolov5 11 | cd yolov5 12 | bash data/scripts/get_coco.sh && echo "COCO done." & 13 | sudo docker pull ultralytics/yolov5:latest && echo "Docker done." & 14 | python -m pip install --upgrade pip && pip install -r requirements.txt && python detect.py && echo "Requirements done." & 15 | wait && echo "All tasks done." # finish background tasks 16 | else 17 | echo "Running re-start script." # resume interrupted runs 18 | i=0 19 | list=$(sudo docker ps -qa) # container list i.e. $'one\ntwo\nthree\nfour' 20 | while IFS= read -r id; do 21 | ((i++)) 22 | echo "restarting container $i: $id" 23 | sudo docker start $id 24 | # sudo docker exec -it $id python train.py --resume # single-GPU 25 | sudo docker exec -d $id python utils/aws/resume.py # multi-scenario 26 | done <<<"$list" 27 | fi 28 | -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/callbacks.py: -------------------------------------------------------------------------------- 1 | # YOLOv5 🚀 by Ultralytics, GPL-3.0 license 2 | """ 3 | Callback utils 4 | """ 5 | 6 | 7 | class Callbacks: 8 | """" 9 | Handles all registered callbacks for YOLOv5 Hooks 10 | """ 11 | 12 | # Define the available callbacks 13 | _callbacks = { 14 | 'on_pretrain_routine_start': [], 15 | 'on_pretrain_routine_end': [], 16 | 17 | 'on_train_start': [], 18 | 'on_train_epoch_start': [], 19 | 'on_train_batch_start': [], 20 | 'optimizer_step': [], 21 | 'on_before_zero_grad': [], 22 | 'on_train_batch_end': [], 23 | 'on_train_epoch_end': [], 24 | 25 | 'on_val_start': [], 26 | 'on_val_batch_start': [], 27 | 'on_val_image_end': [], 28 | 'on_val_batch_end': [], 29 | 'on_val_end': [], 30 | 31 | 'on_fit_epoch_end': [], # fit = train + val 32 | 'on_model_save': [], 33 | 'on_train_end': [], 34 | 35 | 'teardown': [], 36 | } 37 | 38 | def register_action(self, hook, name='', callback=None): 39 | """ 40 | Register a new action to a callback hook 41 | 42 | Args: 43 | hook The callback hook name to register the action to 44 | name The name of the action for later reference 45 | callback The callback to fire 46 | """ 47 | assert hook in self._callbacks, f"hook '{hook}' not found in callbacks {self._callbacks}" 48 | assert callable(callback), f"callback '{callback}' is not callable" 49 | self._callbacks[hook].append({'name': name, 'callback': callback}) 50 | 51 | def get_registered_actions(self, hook=None): 52 | """" 53 | Returns all the registered actions by callback hook 54 | 55 | Args: 56 | hook The name of the hook to check, defaults to all 57 | """ 58 | if hook: 59 | return self._callbacks[hook] 60 | else: 61 | return self._callbacks 62 | 63 | def run(self, hook, *args, **kwargs): 64 | """ 65 | Loop through the registered actions and fire all callbacks 66 | 67 | Args: 68 | hook The name of the hook to check, defaults to all 69 | args Arguments to receive from YOLOv5 70 | kwargs Keyword Arguments to receive from YOLOv5 71 | """ 72 | 73 | assert hook in self._callbacks, f"hook '{hook}' not found in callbacks {self._callbacks}" 74 | 75 | for logger in self._callbacks[hook]: 76 | logger['callback'](*args, **kwargs) 77 | -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/downloads.py: -------------------------------------------------------------------------------- 1 | # YOLOv5 🚀 by Ultralytics, GPL-3.0 license 2 | """ 3 | Download utils 4 | """ 5 | 6 | import os 7 | import platform 8 | import subprocess 9 | import time 10 | import urllib 11 | from pathlib import Path 12 | from zipfile import ZipFile 13 | 14 | import requests 15 | import torch 16 | 17 | 18 | def gsutil_getsize(url=''): 19 | # gs://bucket/file size https://cloud.google.com/storage/docs/gsutil/commands/du 20 | s = subprocess.check_output(f'gsutil du {url}', shell=True).decode('utf-8') 21 | return eval(s.split(' ')[0]) if len(s) else 0 # bytes 22 | 23 | 24 | def safe_download(file, url, url2=None, min_bytes=1E0, error_msg=''): 25 | # Attempts to download file from url or url2, checks and removes incomplete downloads < min_bytes 26 | file = Path(file) 27 | assert_msg = f"Downloaded file '{file}' does not exist or size is < min_bytes={min_bytes}" 28 | try: # url1 29 | print(f'Downloading {url} to {file}...') 30 | torch.hub.download_url_to_file(url, str(file)) 31 | assert file.exists() and file.stat().st_size > min_bytes, assert_msg # check 32 | except Exception as e: # url2 33 | file.unlink(missing_ok=True) # remove partial downloads 34 | print(f'ERROR: {e}\nRe-attempting {url2 or url} to {file}...') 35 | os.system(f"curl -L '{url2 or url}' -o '{file}' --retry 3 -C -") # curl download, retry and resume on fail 36 | finally: 37 | if not file.exists() or file.stat().st_size < min_bytes: # check 38 | file.unlink(missing_ok=True) # remove partial downloads 39 | print(f"ERROR: {assert_msg}\n{error_msg}") 40 | print('') 41 | 42 | 43 | def attempt_download(file, repo='ultralytics/yolov5'): # from utils.downloads import *; attempt_download() 44 | # Attempt file download if does not exist 45 | file = Path(str(file).strip().replace("'", '')) 46 | 47 | if not file.exists(): 48 | # URL specified 49 | name = Path(urllib.parse.unquote(str(file))).name # decode '%2F' to '/' etc. 50 | if str(file).startswith(('http:/', 'https:/')): # download 51 | url = str(file).replace(':/', '://') # Pathlib turns :// -> :/ 52 | name = name.split('?')[0] # parse authentication https://url.com/file.txt?auth... 53 | safe_download(file=name, url=url, min_bytes=1E5) 54 | return name 55 | 56 | # GitHub assets 57 | file.parent.mkdir(parents=True, exist_ok=True) # make parent dir (if required) 58 | try: 59 | response = requests.get(f'https://api.github.com/repos/{repo}/releases/latest').json() # github api 60 | assets = [x['name'] for x in response['assets']] # release assets, i.e. ['yolov5s.pt', 'yolov5m.pt', ...] 61 | tag = response['tag_name'] # i.e. 'v1.0' 62 | except: # fallback plan 63 | assets = ['yolov5n.pt', 'yolov5s.pt', 'yolov5m.pt', 'yolov5l.pt', 'yolov5x.pt', 64 | 'yolov5n6.pt', 'yolov5s6.pt', 'yolov5m6.pt', 'yolov5l6.pt', 'yolov5x6.pt'] 65 | try: 66 | tag = subprocess.check_output('git tag', shell=True, stderr=subprocess.STDOUT).decode().split()[-1] 67 | except: 68 | tag = 'v6.0' # current release 69 | 70 | if name in assets: 71 | safe_download(file, 72 | url=f'https://github.com/{repo}/releases/download/{tag}/{name}', 73 | # url2=f'https://storage.googleapis.com/{repo}/ckpt/{name}', # backup url (optional) 74 | min_bytes=1E5, 75 | error_msg=f'{file} missing, try downloading from https://github.com/{repo}/releases/') 76 | 77 | return str(file) 78 | 79 | 80 | def gdrive_download(id='16TiPfZj7htmTyhntwcZyEEAejOUxuT6m', file='tmp.zip'): 81 | # Downloads a file from Google Drive. from yolov5.utils.downloads import *; gdrive_download() 82 | t = time.time() 83 | file = Path(file) 84 | cookie = Path('cookie') # gdrive cookie 85 | print(f'Downloading https://drive.google.com/uc?export=download&id={id} as {file}... ', end='') 86 | file.unlink(missing_ok=True) # remove existing file 87 | cookie.unlink(missing_ok=True) # remove existing cookie 88 | 89 | # Attempt file download 90 | out = "NUL" if platform.system() == "Windows" else "/dev/null" 91 | os.system(f'curl -c ./cookie -s -L "drive.google.com/uc?export=download&id={id}" > {out}') 92 | if os.path.exists('cookie'): # large file 93 | s = f'curl -Lb ./cookie "drive.google.com/uc?export=download&confirm={get_token()}&id={id}" -o {file}' 94 | else: # small file 95 | s = f'curl -s -L -o {file} "drive.google.com/uc?export=download&id={id}"' 96 | r = os.system(s) # execute, capture return 97 | cookie.unlink(missing_ok=True) # remove existing cookie 98 | 99 | # Error check 100 | if r != 0: 101 | file.unlink(missing_ok=True) # remove partial 102 | print('Download error ') # raise Exception('Download error') 103 | return r 104 | 105 | # Unzip if archive 106 | if file.suffix == '.zip': 107 | print('unzipping... ', end='') 108 | ZipFile(file).extractall(path=file.parent) # unzip 109 | file.unlink() # remove zip 110 | 111 | print(f'Done ({time.time() - t:.1f}s)') 112 | return r 113 | 114 | 115 | def get_token(cookie="./cookie"): 116 | with open(cookie) as f: 117 | for line in f: 118 | if "download" in line: 119 | return line.split()[-1] 120 | return "" 121 | 122 | # Google utils: https://cloud.google.com/storage/docs/reference/libraries ---------------------------------------------- 123 | # 124 | # 125 | # def upload_blob(bucket_name, source_file_name, destination_blob_name): 126 | # # Uploads a file to a bucket 127 | # # https://cloud.google.com/storage/docs/uploading-objects#storage-upload-object-python 128 | # 129 | # storage_client = storage.Client() 130 | # bucket = storage_client.get_bucket(bucket_name) 131 | # blob = bucket.blob(destination_blob_name) 132 | # 133 | # blob.upload_from_filename(source_file_name) 134 | # 135 | # print('File {} uploaded to {}.'.format( 136 | # source_file_name, 137 | # destination_blob_name)) 138 | # 139 | # 140 | # def download_blob(bucket_name, source_blob_name, destination_file_name): 141 | # # Uploads a blob from a bucket 142 | # storage_client = storage.Client() 143 | # bucket = storage_client.get_bucket(bucket_name) 144 | # blob = bucket.blob(source_blob_name) 145 | # 146 | # blob.download_to_filename(destination_file_name) 147 | # 148 | # print('Blob {} downloaded to {}.'.format( 149 | # source_blob_name, 150 | # destination_file_name)) 151 | -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/flask_rest_api/README.md: -------------------------------------------------------------------------------- 1 | # Flask REST API 2 | 3 | [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) [API](https://en.wikipedia.org/wiki/API)s are 4 | commonly used to expose Machine Learning (ML) models to other services. This folder contains an example REST API 5 | created using Flask to expose the YOLOv5s model from [PyTorch Hub](https://pytorch.org/hub/ultralytics_yolov5/). 6 | 7 | ## Requirements 8 | 9 | [Flask](https://palletsprojects.com/p/flask/) is required. Install with: 10 | 11 | ```shell 12 | $ pip install Flask 13 | ``` 14 | 15 | ## Run 16 | 17 | After Flask installation run: 18 | 19 | ```shell 20 | $ python3 restapi.py --port 5000 21 | ``` 22 | 23 | Then use [curl](https://curl.se/) to perform a request: 24 | 25 | ```shell 26 | $ curl -X POST -F image=@zidane.jpg 'http://localhost:5000/v1/object-detection/yolov5s' 27 | ``` 28 | 29 | The model inference results are returned as a JSON response: 30 | 31 | ```json 32 | [ 33 | { 34 | "class": 0, 35 | "confidence": 0.8900438547, 36 | "height": 0.9318675399, 37 | "name": "person", 38 | "width": 0.3264600933, 39 | "xcenter": 0.7438579798, 40 | "ycenter": 0.5207948685 41 | }, 42 | { 43 | "class": 0, 44 | "confidence": 0.8440024257, 45 | "height": 0.7155083418, 46 | "name": "person", 47 | "width": 0.6546785235, 48 | "xcenter": 0.427829951, 49 | "ycenter": 0.6334488392 50 | }, 51 | { 52 | "class": 27, 53 | "confidence": 0.3771208823, 54 | "height": 0.3902671337, 55 | "name": "tie", 56 | "width": 0.0696444362, 57 | "xcenter": 0.3675483763, 58 | "ycenter": 0.7991207838 59 | }, 60 | { 61 | "class": 27, 62 | "confidence": 0.3527112305, 63 | "height": 0.1540903747, 64 | "name": "tie", 65 | "width": 0.0336618312, 66 | "xcenter": 0.7814827561, 67 | "ycenter": 0.5065554976 68 | } 69 | ] 70 | ``` 71 | 72 | An example python script to perform inference using [requests](https://docs.python-requests.org/en/master/) is given 73 | in `example_request.py` 74 | -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/flask_rest_api/example_request.py: -------------------------------------------------------------------------------- 1 | """Perform test request""" 2 | import pprint 3 | 4 | import requests 5 | 6 | DETECTION_URL = "http://localhost:5000/v1/object-detection/yolov5s" 7 | TEST_IMAGE = "zidane.jpg" 8 | 9 | image_data = open(TEST_IMAGE, "rb").read() 10 | 11 | response = requests.post(DETECTION_URL, files={"image": image_data}).json() 12 | 13 | pprint.pprint(response) 14 | -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/flask_rest_api/restapi.py: -------------------------------------------------------------------------------- 1 | """ 2 | Run a rest API exposing the yolov5s object detection model 3 | """ 4 | import argparse 5 | import io 6 | 7 | import torch 8 | from PIL import Image 9 | from flask import Flask, request 10 | 11 | app = Flask(__name__) 12 | 13 | DETECTION_URL = "/v1/object-detection/yolov5s" 14 | 15 | 16 | @app.route(DETECTION_URL, methods=["POST"]) 17 | def predict(): 18 | if not request.method == "POST": 19 | return 20 | 21 | if request.files.get("image"): 22 | image_file = request.files["image"] 23 | image_bytes = image_file.read() 24 | 25 | img = Image.open(io.BytesIO(image_bytes)) 26 | 27 | results = model(img, size=640) # reduce size=320 for faster inference 28 | return results.pandas().xyxy[0].to_json(orient="records") 29 | 30 | 31 | if __name__ == "__main__": 32 | parser = argparse.ArgumentParser(description="Flask API exposing YOLOv5 model") 33 | parser.add_argument("--port", default=5000, type=int, help="port number") 34 | args = parser.parse_args() 35 | 36 | model = torch.hub.load("ultralytics/yolov5", "yolov5s", force_reload=True) # force_reload to recache 37 | app.run(host="0.0.0.0", port=args.port) # debug=True causes Restarting with stat 38 | -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/google_app_engine/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM gcr.io/google-appengine/python 2 | 3 | # Create a virtualenv for dependencies. This isolates these packages from 4 | # system-level packages. 5 | # Use -p python3 or -p python3.7 to select python version. Default is version 2. 6 | RUN virtualenv /env -p python3 7 | 8 | # Setting these environment variables are the same as running 9 | # source /env/bin/activate. 10 | ENV VIRTUAL_ENV /env 11 | ENV PATH /env/bin:$PATH 12 | 13 | RUN apt-get update && apt-get install -y python-opencv 14 | 15 | # Copy the application's requirements.txt and run pip to install all 16 | # dependencies into the virtualenv. 17 | ADD requirements.txt /app/requirements.txt 18 | RUN pip install -r /app/requirements.txt 19 | 20 | # Add the application source code. 21 | ADD . /app 22 | 23 | # Run a WSGI server to serve the application. gunicorn must be declared as 24 | # a dependency in requirements.txt. 25 | CMD gunicorn -b :$PORT main:app 26 | -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/google_app_engine/additional_requirements.txt: -------------------------------------------------------------------------------- 1 | # add these requirements in your app on top of the existing ones 2 | pip==19.2 3 | Flask==1.0.2 4 | gunicorn==19.9.0 5 | -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/google_app_engine/app.yaml: -------------------------------------------------------------------------------- 1 | runtime: custom 2 | env: flex 3 | 4 | service: yolov5app 5 | 6 | liveness_check: 7 | initial_delay_sec: 600 8 | 9 | manual_scaling: 10 | instances: 1 11 | resources: 12 | cpu: 1 13 | memory_gb: 4 14 | disk_size_gb: 20 -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/loggers/__init__.py: -------------------------------------------------------------------------------- 1 | # YOLOv5 🚀 by Ultralytics, GPL-3.0 license 2 | """ 3 | Logging utils 4 | """ 5 | 6 | import os 7 | import warnings 8 | from threading import Thread 9 | 10 | import pkg_resources as pkg 11 | import torch 12 | from torch.utils.tensorboard import SummaryWriter 13 | 14 | from utils.general import colorstr, emojis 15 | from utils.loggers.wandb.wandb_utils import WandbLogger 16 | from utils.plots import plot_images, plot_results 17 | from utils.torch_utils import de_parallel 18 | 19 | LOGGERS = ('csv', 'tb', 'wandb') # text-file, TensorBoard, Weights & Biases 20 | RANK = int(os.getenv('RANK', -1)) 21 | 22 | try: 23 | import wandb 24 | 25 | assert hasattr(wandb, '__version__') # verify package import not local dir 26 | if pkg.parse_version(wandb.__version__) >= pkg.parse_version('0.12.2') and RANK in [0, -1]: 27 | wandb_login_success = wandb.login(timeout=30) 28 | if not wandb_login_success: 29 | wandb = None 30 | except (ImportError, AssertionError): 31 | wandb = None 32 | 33 | 34 | class Loggers(): 35 | # YOLOv5 Loggers class 36 | def __init__(self, save_dir=None, weights=None, opt=None, hyp=None, logger=None, include=LOGGERS): 37 | self.save_dir = save_dir 38 | self.weights = weights 39 | self.opt = opt 40 | self.hyp = hyp 41 | self.logger = logger # for printing results to console 42 | self.include = include 43 | self.keys = ['train/box_loss', 'train/obj_loss', 'train/cls_loss', # train loss 44 | 'metrics/precision', 'metrics/recall', 'metrics/mAP_0.5', 'metrics/mAP_0.5:0.95', # metrics 45 | 'val/box_loss', 'val/obj_loss', 'val/cls_loss', # val loss 46 | 'x/lr0', 'x/lr1', 'x/lr2'] # params 47 | for k in LOGGERS: 48 | setattr(self, k, None) # init empty logger dictionary 49 | self.csv = True # always log to csv 50 | 51 | # Message 52 | if not wandb: 53 | prefix = colorstr('Weights & Biases: ') 54 | s = f"{prefix}run 'pip install wandb' to automatically track and visualize YOLOv5 🚀 runs (RECOMMENDED)" 55 | # print(emojis(s)) 56 | 57 | # TensorBoard 58 | s = self.save_dir 59 | if 'tb' in self.include and not self.opt.evolve: 60 | prefix = colorstr('TensorBoard: ') 61 | self.logger.info(f"{prefix}Start with 'tensorboard --logdir {s.parent}', view at http://localhost:6006/") 62 | self.tb = SummaryWriter(str(s)) 63 | 64 | # W&B 65 | if wandb and 'wandb' in self.include: 66 | wandb_artifact_resume = isinstance(self.opt.resume, str) and self.opt.resume.startswith('wandb-artifact://') 67 | run_id = torch.load(self.weights).get('wandb_id') if self.opt.resume and not wandb_artifact_resume else None 68 | self.opt.hyp = self.hyp # add hyperparameters 69 | self.wandb = WandbLogger(self.opt, run_id) 70 | else: 71 | self.wandb = None 72 | 73 | def on_pretrain_routine_end(self): 74 | # Callback runs on pre-train routine end 75 | paths = self.save_dir.glob('*labels*.jpg') # training labels 76 | if self.wandb: 77 | self.wandb.log({"Labels": [wandb.Image(str(x), caption=x.name) for x in paths]}) 78 | 79 | def on_train_batch_end(self, ni, model, imgs, targets, paths, plots, sync_bn): 80 | # Callback runs on train batch end 81 | if plots: 82 | if ni == 0: 83 | if not sync_bn: # tb.add_graph() --sync known issue https://github.com/ultralytics/yolov5/issues/3754 84 | with warnings.catch_warnings(): 85 | warnings.simplefilter('ignore') # suppress jit trace warning 86 | self.tb.add_graph(torch.jit.trace(de_parallel(model), imgs, strict=False), []) 87 | if ni < 3: 88 | f = self.save_dir / f'train_batch{ni}.jpg' # filename 89 | Thread(target=plot_images, args=(imgs, targets, paths, f), daemon=True).start() 90 | if self.wandb and ni == 10: 91 | files = sorted(self.save_dir.glob('train*.jpg')) 92 | self.wandb.log({'Mosaics': [wandb.Image(str(f), caption=f.name) for f in files if f.exists()]}) 93 | 94 | def on_train_epoch_end(self, epoch): 95 | # Callback runs on train epoch end 96 | if self.wandb: 97 | self.wandb.current_epoch = epoch + 1 98 | 99 | def on_val_image_end(self, pred, predn, path, names, im): 100 | # Callback runs on val image end 101 | if self.wandb: 102 | self.wandb.val_one_image(pred, predn, path, names, im) 103 | 104 | def on_val_end(self): 105 | # Callback runs on val end 106 | if self.wandb: 107 | files = sorted(self.save_dir.glob('val*.jpg')) 108 | self.wandb.log({"Validation": [wandb.Image(str(f), caption=f.name) for f in files]}) 109 | 110 | def on_fit_epoch_end(self, vals, epoch, best_fitness, fi): 111 | # Callback runs at the end of each fit (train+val) epoch 112 | x = {k: v for k, v in zip(self.keys, vals)} # dict 113 | if self.csv: 114 | file = self.save_dir / 'results.csv' 115 | n = len(x) + 1 # number of cols 116 | s = '' if file.exists() else (('%20s,' * n % tuple(['epoch'] + self.keys)).rstrip(',') + '\n') # add header 117 | with open(file, 'a') as f: 118 | f.write(s + ('%20.5g,' * n % tuple([epoch] + vals)).rstrip(',') + '\n') 119 | 120 | if self.tb: 121 | for k, v in x.items(): 122 | self.tb.add_scalar(k, v, epoch) 123 | 124 | if self.wandb: 125 | self.wandb.log(x) 126 | self.wandb.end_epoch(best_result=best_fitness == fi) 127 | 128 | def on_model_save(self, last, epoch, final_epoch, best_fitness, fi): 129 | # Callback runs on model save event 130 | if self.wandb: 131 | if ((epoch + 1) % self.opt.save_period == 0 and not final_epoch) and self.opt.save_period != -1: 132 | self.wandb.log_model(last.parent, self.opt, epoch, fi, best_model=best_fitness == fi) 133 | 134 | def on_train_end(self, last, best, plots, epoch): 135 | # Callback runs on training end 136 | if plots: 137 | plot_results(file=self.save_dir / 'results.csv') # save results.png 138 | files = ['results.png', 'confusion_matrix.png', *[f'{x}_curve.png' for x in ('F1', 'PR', 'P', 'R')]] 139 | files = [(self.save_dir / f) for f in files if (self.save_dir / f).exists()] # filter 140 | 141 | if self.tb: 142 | import cv2 143 | for f in files: 144 | self.tb.add_image(f.stem, cv2.imread(str(f))[..., ::-1], epoch, dataformats='HWC') 145 | 146 | if self.wandb: 147 | self.wandb.log({"Results": [wandb.Image(str(f), caption=f.name) for f in files]}) 148 | # Calling wandb.log. TODO: Refactor this into WandbLogger.log_model 149 | if not self.opt.evolve: 150 | wandb.log_artifact(str(best if best.exists() else last), type='model', 151 | name='run_' + self.wandb.wandb_run.id + '_model', 152 | aliases=['latest', 'best', 'stripped']) 153 | self.wandb.finish_run() 154 | else: 155 | self.wandb.finish_run() 156 | self.wandb = WandbLogger(self.opt) 157 | -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/loggers/wandb/README.md: -------------------------------------------------------------------------------- 1 | 📚 This guide explains how to use **Weights & Biases** (W&B) with YOLOv5 🚀. UPDATED 29 September 2021. 2 | * [About Weights & Biases](#about-weights-&-biases) 3 | * [First-Time Setup](#first-time-setup) 4 | * [Viewing runs](#viewing-runs) 5 | * [Advanced Usage: Dataset Versioning and Evaluation](#advanced-usage) 6 | * [Reports: Share your work with the world!](#reports) 7 | 8 | ## About Weights & Biases 9 | Think of [W&B](https://wandb.ai/site?utm_campaign=repo_yolo_wandbtutorial) like GitHub for machine learning models. With a few lines of code, save everything you need to debug, compare and reproduce your models — architecture, hyperparameters, git commits, model weights, GPU usage, and even datasets and predictions. 10 | 11 | Used by top researchers including teams at OpenAI, Lyft, Github, and MILA, W&B is part of the new standard of best practices for machine learning. How W&B can help you optimize your machine learning workflows: 12 | 13 | * [Debug](https://wandb.ai/wandb/getting-started/reports/Visualize-Debug-Machine-Learning-Models--VmlldzoyNzY5MDk#Free-2) model performance in real time 14 | * [GPU usage](https://wandb.ai/wandb/getting-started/reports/Visualize-Debug-Machine-Learning-Models--VmlldzoyNzY5MDk#System-4) visualized automatically 15 | * [Custom charts](https://wandb.ai/wandb/customizable-charts/reports/Powerful-Custom-Charts-To-Debug-Model-Peformance--VmlldzoyNzY4ODI) for powerful, extensible visualization 16 | * [Share insights](https://wandb.ai/wandb/getting-started/reports/Visualize-Debug-Machine-Learning-Models--VmlldzoyNzY5MDk#Share-8) interactively with collaborators 17 | * [Optimize hyperparameters](https://docs.wandb.com/sweeps) efficiently 18 | * [Track](https://docs.wandb.com/artifacts) datasets, pipelines, and production models 19 | 20 | ## First-Time Setup 21 |
22 | Toggle Details 23 | When you first train, W&B will prompt you to create a new account and will generate an **API key** for you. If you are an existing user you can retrieve your key from https://wandb.ai/authorize. This key is used to tell W&B where to log your data. You only need to supply your key once, and then it is remembered on the same device. 24 | 25 | W&B will create a cloud **project** (default is 'YOLOv5') for your training runs, and each new training run will be provided a unique run **name** within that project as project/name. You can also manually set your project and run name as: 26 | 27 | ```shell 28 | $ python train.py --project ... --name ... 29 | ``` 30 | 31 | YOLOv5 notebook example: Open In Colab Open In Kaggle 32 | Screen Shot 2021-09-29 at 10 23 13 PM 33 | 34 | 35 |
36 | 37 | ## Viewing Runs 38 |
39 | Toggle Details 40 | Run information streams from your environment to the W&B cloud console as you train. This allows you to monitor and even cancel runs in realtime . All important information is logged: 41 | 42 | * Training & Validation losses 43 | * Metrics: Precision, Recall, mAP@0.5, mAP@0.5:0.95 44 | * Learning Rate over time 45 | * A bounding box debugging panel, showing the training progress over time 46 | * GPU: Type, **GPU Utilization**, power, temperature, **CUDA memory usage** 47 | * System: Disk I/0, CPU utilization, RAM memory usage 48 | * Your trained model as W&B Artifact 49 | * Environment: OS and Python types, Git repository and state, **training command** 50 | 51 |

Weights & Biases dashboard

52 | 53 | 54 |
55 | 56 | ## Advanced Usage 57 | You can leverage W&B artifacts and Tables integration to easily visualize and manage your datasets, models and training evaluations. Here are some quick examples to get you started. 58 |
59 |

1. Visualize and Version Datasets

60 | Log, visualize, dynamically query, and understand your data with W&B Tables. You can use the following command to log your dataset as a W&B Table. This will generate a {dataset}_wandb.yaml file which can be used to train from dataset artifact. 61 |
62 | Usage 63 | Code $ python utils/logger/wandb/log_dataset.py --project ... --name ... --data .. 64 | 65 | ![Screenshot (64)](https://user-images.githubusercontent.com/15766192/128486078-d8433890-98a3-4d12-8986-b6c0e3fc64b9.png) 66 |
67 | 68 |

2: Train and Log Evaluation simultaneousy

69 | This is an extension of the previous section, but it'll also training after uploading the dataset. This also evaluation Table 70 | Evaluation table compares your predictions and ground truths across the validation set for each epoch. It uses the references to the already uploaded datasets, 71 | so no images will be uploaded from your system more than once. 72 |
73 | Usage 74 | Code $ python utils/logger/wandb/log_dataset.py --data .. --upload_data 75 | 76 | ![Screenshot (72)](https://user-images.githubusercontent.com/15766192/128979739-4cf63aeb-a76f-483f-8861-1c0100b938a5.png) 77 |
78 | 79 |

3: Train using dataset artifact

80 | When you upload a dataset as described in the first section, you get a new config file with an added `_wandb` to its name. This file contains the information that 81 | can be used to train a model directly from the dataset artifact. This also logs evaluation 82 |
83 | Usage 84 | Code $ python utils/logger/wandb/log_dataset.py --data {data}_wandb.yaml 85 | 86 | ![Screenshot (72)](https://user-images.githubusercontent.com/15766192/128979739-4cf63aeb-a76f-483f-8861-1c0100b938a5.png) 87 |
88 | 89 |

4: Save model checkpoints as artifacts

90 | To enable saving and versioning checkpoints of your experiment, pass `--save_period n` with the base cammand, where `n` represents checkpoint interval. 91 | You can also log both the dataset and model checkpoints simultaneously. If not passed, only the final model will be logged 92 | 93 |
94 | Usage 95 | Code $ python train.py --save_period 1 96 | 97 | ![Screenshot (68)](https://user-images.githubusercontent.com/15766192/128726138-ec6c1f60-639d-437d-b4ee-3acd9de47ef3.png) 98 |
99 | 100 |
101 | 102 |

5: Resume runs from checkpoint artifacts.

103 | Any run can be resumed using artifacts if the --resume argument starts with wandb-artifact:// prefix followed by the run path, i.e, wandb-artifact://username/project/runid . This doesn't require the model checkpoint to be present on the local system. 104 | 105 |
106 | Usage 107 | Code $ python train.py --resume wandb-artifact://{run_path} 108 | 109 | ![Screenshot (70)](https://user-images.githubusercontent.com/15766192/128728988-4e84b355-6c87-41ae-a591-14aecf45343e.png) 110 |
111 | 112 |

6: Resume runs from dataset artifact & checkpoint artifacts.

113 | Local dataset or model checkpoints are not required. This can be used to resume runs directly on a different device 114 | The syntax is same as the previous section, but you'll need to lof both the dataset and model checkpoints as artifacts, i.e, set bot --upload_dataset or 115 | train from _wandb.yaml file and set --save_period 116 | 117 |
118 | Usage 119 | Code $ python train.py --resume wandb-artifact://{run_path} 120 | 121 | ![Screenshot (70)](https://user-images.githubusercontent.com/15766192/128728988-4e84b355-6c87-41ae-a591-14aecf45343e.png) 122 |
123 | 124 | 125 | 126 | 127 |

Reports

128 | W&B Reports can be created from your saved runs for sharing online. Once a report is created you will receive a link you can use to publically share your results. Here is an example report created from the COCO128 tutorial trainings of all four YOLOv5 models ([link](https://wandb.ai/glenn-jocher/yolov5_tutorial/reports/YOLOv5-COCO128-Tutorial-Results--VmlldzozMDI5OTY)). 129 | 130 | Weights & Biases Reports 131 | 132 | 133 | ## Environments 134 | 135 | YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including [CUDA](https://developer.nvidia.com/cuda)/[CUDNN](https://developer.nvidia.com/cudnn), [Python](https://www.python.org/) and [PyTorch](https://pytorch.org/) preinstalled): 136 | 137 | - **Google Colab and Kaggle** notebooks with free GPU: Open In Colab Open In Kaggle 138 | - **Google Cloud** Deep Learning VM. See [GCP Quickstart Guide](https://github.com/ultralytics/yolov5/wiki/GCP-Quickstart) 139 | - **Amazon** Deep Learning AMI. See [AWS Quickstart Guide](https://github.com/ultralytics/yolov5/wiki/AWS-Quickstart) 140 | - **Docker Image**. See [Docker Quickstart Guide](https://github.com/ultralytics/yolov5/wiki/Docker-Quickstart) Docker Pulls 141 | 142 | 143 | ## Status 144 | 145 | ![CI CPU testing](https://github.com/ultralytics/yolov5/workflows/CI%20CPU%20testing/badge.svg) 146 | 147 | If this badge is green, all [YOLOv5 GitHub Actions](https://github.com/ultralytics/yolov5/actions) Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training ([train.py](https://github.com/ultralytics/yolov5/blob/master/train.py)), validation ([val.py](https://github.com/ultralytics/yolov5/blob/master/val.py)), inference ([detect.py](https://github.com/ultralytics/yolov5/blob/master/detect.py)) and export ([export.py](https://github.com/ultralytics/yolov5/blob/master/export.py)) on MacOS, Windows, and Ubuntu every 24 hours and on every commit. 148 | -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/loggers/wandb/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LabSAINT/SPD-Conv/c09f7928b8d2c33034fc3e9889a866f8bf37b44d/YOLOv5-SPD/utils/loggers/wandb/__init__.py -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/loggers/wandb/log_dataset.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | 3 | from wandb_utils import WandbLogger 4 | 5 | WANDB_ARTIFACT_PREFIX = 'wandb-artifact://' 6 | 7 | 8 | def create_dataset_artifact(opt): 9 | logger = WandbLogger(opt, None, job_type='Dataset Creation') # TODO: return value unused 10 | 11 | 12 | if __name__ == '__main__': 13 | parser = argparse.ArgumentParser() 14 | parser.add_argument('--data', type=str, default='data/coco128.yaml', help='data.yaml path') 15 | parser.add_argument('--single-cls', action='store_true', help='train as single-class dataset') 16 | parser.add_argument('--project', type=str, default='YOLOv5', help='name of W&B Project') 17 | parser.add_argument('--entity', default=None, help='W&B entity') 18 | parser.add_argument('--name', type=str, default='log dataset', help='name of W&B run') 19 | 20 | opt = parser.parse_args() 21 | opt.resume = False # Explicitly disallow resume check for dataset upload job 22 | 23 | create_dataset_artifact(opt) 24 | -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/loggers/wandb/sweep.py: -------------------------------------------------------------------------------- 1 | import sys 2 | from pathlib import Path 3 | 4 | import wandb 5 | 6 | FILE = Path(__file__).resolve() 7 | ROOT = FILE.parents[3] # YOLOv5 root directory 8 | if str(ROOT) not in sys.path: 9 | sys.path.append(str(ROOT)) # add ROOT to PATH 10 | 11 | from train import train, parse_opt 12 | from utils.general import increment_path 13 | from utils.torch_utils import select_device 14 | from utils.callbacks import Callbacks 15 | 16 | 17 | def sweep(): 18 | wandb.init() 19 | # Get hyp dict from sweep agent 20 | hyp_dict = vars(wandb.config).get("_items") 21 | 22 | # Workaround: get necessary opt args 23 | opt = parse_opt(known=True) 24 | opt.batch_size = hyp_dict.get("batch_size") 25 | opt.save_dir = str(increment_path(Path(opt.project) / opt.name, exist_ok=opt.exist_ok or opt.evolve)) 26 | opt.epochs = hyp_dict.get("epochs") 27 | opt.nosave = True 28 | opt.data = hyp_dict.get("data") 29 | device = select_device(opt.device, batch_size=opt.batch_size) 30 | 31 | # train 32 | train(hyp_dict, opt, device, callbacks=Callbacks()) 33 | 34 | 35 | if __name__ == "__main__": 36 | sweep() 37 | -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/loggers/wandb/sweep.yaml: -------------------------------------------------------------------------------- 1 | # Hyperparameters for training 2 | # To set range- 3 | # Provide min and max values as: 4 | # parameter: 5 | # 6 | # min: scalar 7 | # max: scalar 8 | # OR 9 | # 10 | # Set a specific list of search space- 11 | # parameter: 12 | # values: [scalar1, scalar2, scalar3...] 13 | # 14 | # You can use grid, bayesian and hyperopt search strategy 15 | # For more info on configuring sweeps visit - https://docs.wandb.ai/guides/sweeps/configuration 16 | 17 | program: utils/loggers/wandb/sweep.py 18 | method: random 19 | metric: 20 | name: metrics/mAP_0.5 21 | goal: maximize 22 | 23 | parameters: 24 | # hyperparameters: set either min, max range or values list 25 | data: 26 | value: "data/coco128.yaml" 27 | batch_size: 28 | values: [64] 29 | epochs: 30 | values: [10] 31 | 32 | lr0: 33 | distribution: uniform 34 | min: 1e-5 35 | max: 1e-1 36 | lrf: 37 | distribution: uniform 38 | min: 0.01 39 | max: 1.0 40 | momentum: 41 | distribution: uniform 42 | min: 0.6 43 | max: 0.98 44 | weight_decay: 45 | distribution: uniform 46 | min: 0.0 47 | max: 0.001 48 | warmup_epochs: 49 | distribution: uniform 50 | min: 0.0 51 | max: 5.0 52 | warmup_momentum: 53 | distribution: uniform 54 | min: 0.0 55 | max: 0.95 56 | warmup_bias_lr: 57 | distribution: uniform 58 | min: 0.0 59 | max: 0.2 60 | box: 61 | distribution: uniform 62 | min: 0.02 63 | max: 0.2 64 | cls: 65 | distribution: uniform 66 | min: 0.2 67 | max: 4.0 68 | cls_pw: 69 | distribution: uniform 70 | min: 0.5 71 | max: 2.0 72 | obj: 73 | distribution: uniform 74 | min: 0.2 75 | max: 4.0 76 | obj_pw: 77 | distribution: uniform 78 | min: 0.5 79 | max: 2.0 80 | iou_t: 81 | distribution: uniform 82 | min: 0.1 83 | max: 0.7 84 | anchor_t: 85 | distribution: uniform 86 | min: 2.0 87 | max: 8.0 88 | fl_gamma: 89 | distribution: uniform 90 | min: 0.0 91 | max: 0.1 92 | hsv_h: 93 | distribution: uniform 94 | min: 0.0 95 | max: 0.1 96 | hsv_s: 97 | distribution: uniform 98 | min: 0.0 99 | max: 0.9 100 | hsv_v: 101 | distribution: uniform 102 | min: 0.0 103 | max: 0.9 104 | degrees: 105 | distribution: uniform 106 | min: 0.0 107 | max: 45.0 108 | translate: 109 | distribution: uniform 110 | min: 0.0 111 | max: 0.9 112 | scale: 113 | distribution: uniform 114 | min: 0.0 115 | max: 0.9 116 | shear: 117 | distribution: uniform 118 | min: 0.0 119 | max: 10.0 120 | perspective: 121 | distribution: uniform 122 | min: 0.0 123 | max: 0.001 124 | flipud: 125 | distribution: uniform 126 | min: 0.0 127 | max: 1.0 128 | fliplr: 129 | distribution: uniform 130 | min: 0.0 131 | max: 1.0 132 | mosaic: 133 | distribution: uniform 134 | min: 0.0 135 | max: 1.0 136 | mixup: 137 | distribution: uniform 138 | min: 0.0 139 | max: 1.0 140 | copy_paste: 141 | distribution: uniform 142 | min: 0.0 143 | max: 1.0 144 | -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/loss.py: -------------------------------------------------------------------------------- 1 | # YOLOv5 🚀 by Ultralytics, GPL-3.0 license 2 | """ 3 | Loss functions 4 | """ 5 | 6 | import torch 7 | import torch.nn as nn 8 | 9 | from utils.metrics import bbox_iou 10 | from utils.torch_utils import is_parallel 11 | 12 | 13 | def smooth_BCE(eps=0.1): # https://github.com/ultralytics/yolov3/issues/238#issuecomment-598028441 14 | # return positive, negative label smoothing BCE targets 15 | return 1.0 - 0.5 * eps, 0.5 * eps 16 | 17 | 18 | class BCEBlurWithLogitsLoss(nn.Module): 19 | # BCEwithLogitLoss() with reduced missing label effects. 20 | def __init__(self, alpha=0.05): 21 | super(BCEBlurWithLogitsLoss, self).__init__() 22 | self.loss_fcn = nn.BCEWithLogitsLoss(reduction='none') # must be nn.BCEWithLogitsLoss() 23 | self.alpha = alpha 24 | 25 | def forward(self, pred, true): 26 | loss = self.loss_fcn(pred, true) 27 | pred = torch.sigmoid(pred) # prob from logits 28 | dx = pred - true # reduce only missing label effects 29 | # dx = (pred - true).abs() # reduce missing label and false label effects 30 | alpha_factor = 1 - torch.exp((dx - 1) / (self.alpha + 1e-4)) 31 | loss *= alpha_factor 32 | return loss.mean() 33 | 34 | 35 | class FocalLoss(nn.Module): 36 | # Wraps focal loss around existing loss_fcn(), i.e. criteria = FocalLoss(nn.BCEWithLogitsLoss(), gamma=1.5) 37 | def __init__(self, loss_fcn, gamma=1.5, alpha=0.25): 38 | super(FocalLoss, self).__init__() 39 | self.loss_fcn = loss_fcn # must be nn.BCEWithLogitsLoss() 40 | self.gamma = gamma 41 | self.alpha = alpha 42 | self.reduction = loss_fcn.reduction 43 | self.loss_fcn.reduction = 'none' # required to apply FL to each element 44 | 45 | def forward(self, pred, true): 46 | loss = self.loss_fcn(pred, true) 47 | # p_t = torch.exp(-loss) 48 | # loss *= self.alpha * (1.000001 - p_t) ** self.gamma # non-zero power for gradient stability 49 | 50 | # TF implementation https://github.com/tensorflow/addons/blob/v0.7.1/tensorflow_addons/losses/focal_loss.py 51 | pred_prob = torch.sigmoid(pred) # prob from logits 52 | p_t = true * pred_prob + (1 - true) * (1 - pred_prob) 53 | alpha_factor = true * self.alpha + (1 - true) * (1 - self.alpha) 54 | modulating_factor = (1.0 - p_t) ** self.gamma 55 | loss *= alpha_factor * modulating_factor 56 | 57 | if self.reduction == 'mean': 58 | return loss.mean() 59 | elif self.reduction == 'sum': 60 | return loss.sum() 61 | else: # 'none' 62 | return loss 63 | 64 | 65 | class QFocalLoss(nn.Module): 66 | # Wraps Quality focal loss around existing loss_fcn(), i.e. criteria = FocalLoss(nn.BCEWithLogitsLoss(), gamma=1.5) 67 | def __init__(self, loss_fcn, gamma=1.5, alpha=0.25): 68 | super(QFocalLoss, self).__init__() 69 | self.loss_fcn = loss_fcn # must be nn.BCEWithLogitsLoss() 70 | self.gamma = gamma 71 | self.alpha = alpha 72 | self.reduction = loss_fcn.reduction 73 | self.loss_fcn.reduction = 'none' # required to apply FL to each element 74 | 75 | def forward(self, pred, true): 76 | loss = self.loss_fcn(pred, true) 77 | 78 | pred_prob = torch.sigmoid(pred) # prob from logits 79 | alpha_factor = true * self.alpha + (1 - true) * (1 - self.alpha) 80 | modulating_factor = torch.abs(true - pred_prob) ** self.gamma 81 | loss *= alpha_factor * modulating_factor 82 | 83 | if self.reduction == 'mean': 84 | return loss.mean() 85 | elif self.reduction == 'sum': 86 | return loss.sum() 87 | else: # 'none' 88 | return loss 89 | 90 | 91 | class ComputeLoss: 92 | # Compute losses 93 | def __init__(self, model, autobalance=False): 94 | self.sort_obj_iou = False 95 | device = next(model.parameters()).device # get model device 96 | h = model.hyp # hyperparameters 97 | 98 | # Define criteria 99 | BCEcls = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([h['cls_pw']], device=device)) 100 | BCEobj = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([h['obj_pw']], device=device)) 101 | 102 | # Class label smoothing https://arxiv.org/pdf/1902.04103.pdf eqn 3 103 | self.cp, self.cn = smooth_BCE(eps=h.get('label_smoothing', 0.0)) # positive, negative BCE targets 104 | 105 | # Focal loss 106 | g = h['fl_gamma'] # focal loss gamma 107 | if g > 0: 108 | BCEcls, BCEobj = FocalLoss(BCEcls, g), FocalLoss(BCEobj, g) 109 | 110 | det = model.module.model[-1] if is_parallel(model) else model.model[-1] # Detect() module 111 | self.balance = {3: [4.0, 1.0, 0.4]}.get(det.nl, [4.0, 1.0, 0.25, 0.06, .02]) # P3-P7 112 | self.ssi = list(det.stride).index(16) if autobalance else 0 # stride 16 index 113 | self.BCEcls, self.BCEobj, self.gr, self.hyp, self.autobalance = BCEcls, BCEobj, 1.0, h, autobalance 114 | for k in 'na', 'nc', 'nl', 'anchors': 115 | setattr(self, k, getattr(det, k)) 116 | 117 | def __call__(self, p, targets): # predictions, targets, model 118 | device = targets.device 119 | lcls, lbox, lobj = torch.zeros(1, device=device), torch.zeros(1, device=device), torch.zeros(1, device=device) 120 | tcls, tbox, indices, anchors = self.build_targets(p, targets) # targets 121 | 122 | # Losses 123 | for i, pi in enumerate(p): # layer index, layer predictions 124 | b, a, gj, gi = indices[i] # image, anchor, gridy, gridx 125 | tobj = torch.zeros_like(pi[..., 0], device=device) # target obj 126 | 127 | n = b.shape[0] # number of targets 128 | if n: 129 | ps = pi[b, a, gj, gi] # prediction subset corresponding to targets 130 | 131 | # Regression 132 | pxy = ps[:, :2].sigmoid() * 2. - 0.5 133 | pwh = (ps[:, 2:4].sigmoid() * 2) ** 2 * anchors[i] 134 | pbox = torch.cat((pxy, pwh), 1) # predicted box 135 | iou = bbox_iou(pbox.T, tbox[i], x1y1x2y2=False, CIoU=True) # iou(prediction, target) 136 | lbox += (1.0 - iou).mean() # iou loss 137 | 138 | # Objectness 139 | score_iou = iou.detach().clamp(0).type(tobj.dtype) 140 | if self.sort_obj_iou: 141 | sort_id = torch.argsort(score_iou) 142 | b, a, gj, gi, score_iou = b[sort_id], a[sort_id], gj[sort_id], gi[sort_id], score_iou[sort_id] 143 | tobj[b, a, gj, gi] = (1.0 - self.gr) + self.gr * score_iou # iou ratio 144 | 145 | # Classification 146 | if self.nc > 1: # cls loss (only if multiple classes) 147 | t = torch.full_like(ps[:, 5:], self.cn, device=device) # targets 148 | t[range(n), tcls[i]] = self.cp 149 | lcls += self.BCEcls(ps[:, 5:], t) # BCE 150 | 151 | # Append targets to text file 152 | # with open('targets.txt', 'a') as file: 153 | # [file.write('%11.5g ' * 4 % tuple(x) + '\n') for x in torch.cat((txy[i], twh[i]), 1)] 154 | 155 | obji = self.BCEobj(pi[..., 4], tobj) 156 | lobj += obji * self.balance[i] # obj loss 157 | if self.autobalance: 158 | self.balance[i] = self.balance[i] * 0.9999 + 0.0001 / obji.detach().item() 159 | 160 | if self.autobalance: 161 | self.balance = [x / self.balance[self.ssi] for x in self.balance] 162 | lbox *= self.hyp['box'] 163 | lobj *= self.hyp['obj'] 164 | lcls *= self.hyp['cls'] 165 | bs = tobj.shape[0] # batch size 166 | 167 | return (lbox + lobj + lcls) * bs, torch.cat((lbox, lobj, lcls)).detach() 168 | 169 | def build_targets(self, p, targets): 170 | # Build targets for compute_loss(), input targets(image,class,x,y,w,h) 171 | na, nt = self.na, targets.shape[0] # number of anchors, targets 172 | tcls, tbox, indices, anch = [], [], [], [] 173 | gain = torch.ones(7, device=targets.device) # normalized to gridspace gain 174 | ai = torch.arange(na, device=targets.device).float().view(na, 1).repeat(1, nt) # same as .repeat_interleave(nt) 175 | targets = torch.cat((targets.repeat(na, 1, 1), ai[:, :, None]), 2) # append anchor indices 176 | 177 | g = 0.5 # bias 178 | off = torch.tensor([[0, 0], 179 | [1, 0], [0, 1], [-1, 0], [0, -1], # j,k,l,m 180 | # [1, 1], [1, -1], [-1, 1], [-1, -1], # jk,jm,lk,lm 181 | ], device=targets.device).float() * g # offsets 182 | 183 | for i in range(self.nl): 184 | anchors = self.anchors[i] 185 | gain[2:6] = torch.tensor(p[i].shape)[[3, 2, 3, 2]] # xyxy gain 186 | 187 | # Match targets to anchors 188 | t = targets * gain 189 | if nt: 190 | # Matches 191 | r = t[:, :, 4:6] / anchors[:, None] # wh ratio 192 | j = torch.max(r, 1. / r).max(2)[0] < self.hyp['anchor_t'] # compare 193 | # j = wh_iou(anchors, t[:, 4:6]) > model.hyp['iou_t'] # iou(3,n)=wh_iou(anchors(3,2), gwh(n,2)) 194 | t = t[j] # filter 195 | 196 | # Offsets 197 | gxy = t[:, 2:4] # grid xy 198 | gxi = gain[[2, 3]] - gxy # inverse 199 | j, k = ((gxy % 1. < g) & (gxy > 1.)).T 200 | l, m = ((gxi % 1. < g) & (gxi > 1.)).T 201 | j = torch.stack((torch.ones_like(j), j, k, l, m)) 202 | t = t.repeat((5, 1, 1))[j] 203 | offsets = (torch.zeros_like(gxy)[None] + off[:, None])[j] 204 | else: 205 | t = targets[0] 206 | offsets = 0 207 | 208 | # Define 209 | b, c = t[:, :2].long().T # image, class 210 | gxy = t[:, 2:4] # grid xy 211 | gwh = t[:, 4:6] # grid wh 212 | gij = (gxy - offsets).long() 213 | gi, gj = gij.T # grid xy indices 214 | 215 | # Append 216 | a = t[:, 6].long() # anchor indices 217 | indices.append((b, a, gj.clamp_(0, gain[3] - 1), gi.clamp_(0, gain[2] - 1))) # image, anchor, grid indices 218 | tbox.append(torch.cat((gxy - gij, gwh), 1)) # box 219 | anch.append(anchors[a]) # anchors 220 | tcls.append(c) # class 221 | 222 | return tcls, tbox, indices, anch 223 | -------------------------------------------------------------------------------- /YOLOv5-SPD/utils/loss2.py: -------------------------------------------------------------------------------- 1 | 2 | import enum 3 | from yaml.tokens import AnchorToken 4 | import torch 5 | import torch.nn as nn 6 | 7 | from utils.metrics import bbox_iou 8 | from utils.torch_utils import is_parallel 9 | 10 | 11 | def smooth_BCE(eps=0.1): 12 | # return positive, negative label smoothing BCE targets 13 | return 1.0 - 0.5 * eps, 0.5* eps 14 | 15 | class BCEBlurWithLogitsLoss(nn.Module): 16 | # BCEwithLogitLoss() with reduced missing label effects. 17 | def __init__(self, alpha=0.05): 18 | super(BCEBlurWithLogitsLoss, self).__init__() 19 | self.loss_fcn = nn.BCEWithLogitsLoss(reduction='none') 20 | self.alpha = alpha 21 | 22 | def forward(self, pred, true): 23 | loss = self.loss_fcn(pred, true) 24 | pred = torch.sigmoid(pred) 25 | dx = pred - true 26 | alpha_factor = 1 - torch.exp((dx-1) / (self.alpha + 1e-4)) 27 | loss *= alpha_factor 28 | return loss.mean() 29 | 30 | class FocalLoss(nn.Module): 31 | # Wraps focal loss around existinf loss_fcn() i.e i.e. criteria = FocalLoss(nn.BCEWithLogitsLoss(), gamma=1.5) 32 | def __init__(self, loss_fcn, gamma=1.5, alpha = 0.25): 33 | super(FocalLoss, self).__init__() 34 | self.loss_fcn = loss_fcn # must be nn.BCEWithLogitsLoss() 35 | self.gamma = gamma 36 | self.alpha = alpha 37 | self.reduction = loss_fcn.reduction 38 | self.loss_fcn.reduction = 'none' # rewuired to apply FL to each element 39 | 40 | def forward(self, pred, true): 41 | loss = self.loss_fcn(pred, true) 42 | 43 | pred_prob = torch.sigmoid(pred) 44 | p_t = true * pred_prob + (1- true) *(1- pred_prob) 45 | alpha_factor = true * self.alpha + (1- true) *(1-self.alpha) 46 | modulating_factor = (1.0-p_t) ** self.gamma 47 | loss *= alpha_factor * modulating_factor 48 | 49 | if self.reduction =='mean': 50 | return loss.mean() 51 | 52 | elif self.reduction =='sum': 53 | return loss.sum() 54 | 55 | else: 56 | return loss 57 | 58 | 59 | class QFocalLoss(nn.Module): 60 | # Wraps Quality focal loss around existinf loss_fcn() i.e i.e. criteria = FocalLoss(nn.BCEWithLogitsLoss(), gamma=1.5) 61 | def __init__(self, loss_fcn, gamma=1.5, alpha = 0.25): 62 | super(FocalLoss, self).__init__() 63 | self.loss_fcn = loss_fcn # must be nn.BCEWithLogitsLoss() 64 | self.gamma = gamma 65 | self.alpha = alpha 66 | self.reduction = loss_fcn.reduction 67 | self.loss_fcn.reduction = 'none' # rewuired to apply FL to each element 68 | 69 | def forward(self, pred, true): 70 | loss = self.loss_fcn(pred, true) 71 | 72 | pred_prob = torch.sigmoid(pred) 73 | alpha_factor = true * self.alpha + (1- true) *(1-self.alpha) 74 | modulating_factor = torch.abs(true- pred_prob) ** self.gamma 75 | loss *= alpha_factor * modulating_factor 76 | 77 | if self.reduction =='mean': 78 | return loss.mean() 79 | elif self.reduction =='sum': 80 | return loss.sum() 81 | else: 82 | return loss 83 | 84 | 85 | class ComputeLoss: 86 | # Compute losses 87 | def __init__(self, model, autobalance=False): 88 | self.sort_obj_iou = False 89 | device = next(model.parameters()).device # to get the device 90 | h = model.hyp # hyperparameters 91 | 92 | # Define criteria 93 | BCEcls = nn.BCEWithLogitsLoss(pos_weight = torch.tensor([h['cls_pw']],device=device)) 94 | BCEobj = nn.BCEWithLogitsLoss(pos_weight = torch.tensor([h['obj_pw']],device=device)) 95 | 96 | # class label smoothening 97 | self.cp, self.cn = smooth_BCE(eps = h.get('label_smoothing',0.0)) 98 | # positive, negative BCE targets 99 | 100 | # Focal loss 101 | g = h['fl_gamma'] # focal loss gamma 102 | if g > 0: 103 | BCEcls, BCEobj = FocalLoss(BCEcls, g), FocalLoss(BCEobj, g) 104 | 105 | det = model.module.model[-1] if is_parallel(model) else model.model[-1] # Detect() module 106 | self.balance = {3: [4.0, 1.0, 0.4]}.get(det.nl, [4.0, 1.0, 0.25, 0.06, 0.02]) # P3-P7 107 | self.ssi = list(det.stride).index(16) if autobalance else 0 # stride 16 index 108 | self.BCEcls, self.BCEobj, self.gr, self.hyp, self.autobalance = BCEcls, BCEobj, 1.0, h, autobalance 109 | for k in 'na', 'nc', 'nl', 'anchors': 110 | setattr(self, k, getattr(det, k)) 111 | 112 | def __call__(self, p, targets): # predictions, targets 113 | device = targets.device 114 | lcls, lbox, lobj = torch.zeros(1,device=device), torch.zeros(1,device=device), torch.zeros(1,device=device) 115 | tcls, tbox, indices, anchors = self.build_ragets(p,targets) 116 | 117 | # Losses 118 | for i, pi in enumerate(p): # leyer index, layer predictions 119 | b, a, gj, gi = indices[i] # image, anchor, gridy, gridx 120 | tobj = torch.zeros_like(pi[..., 0], device = device) # target obj 121 | 122 | n = b.shape[0] # number of targets 123 | if n: 124 | ps = pi[b, a, gj, gi] # prediction subset correcposing to targets 125 | 126 | # Regression 127 | pxy = ps[:, :2].sigmoid() * 2. - 0.5 128 | pwh = (ps[:, 2:4].sigmoid() * 2) ** 2 * anchors[i] 129 | pbox = torch.cat((pxy, pwh), 1) # predicted box 130 | iou = bbox_iou(pbox.T, tbox[i], x1y1x2y2=False, CIoU=True) 131 | lbox += (1.0 - iou).mean() 132 | 133 | # Objectness 134 | score_iou = iou.detach().clamp(0).type(tobj.dtype) 135 | if self.sort_obj_iou: 136 | sort_id = torch.argsort(score_iou) 137 | b, a, gj, gi, score_iou = b[sort_id], a[sort_id], gj[sort_id], gi[sort_id], score_iou[sort_id] 138 | tobj[b, a, gj, gi] = (1.0 - self.gr) + self.gr * score_iou 139 | 140 | # Classification 141 | 142 | if self.nc > 1: 143 | t = torch.full_like(ps[:, 5:], self.cn, device= device) 144 | t[range(n), tcls[i]] = self.cp 145 | tcls += self.BCEcls(ps[:, 5:], t) # BCE 146 | 147 | obji = self.BCEobj(pi[..., 4], tobj) 148 | lobj += obji * self.balance[i] # obj loss 149 | 150 | if self.autobalance: 151 | self.balance[i] = self.balance[i] * 0.9999 + 0.0001 / obji.detach().item() 152 | 153 | if self.autobalance: 154 | self.balance = [x / self.balance[self.ssi]] for x in self.balance] 155 | lbox *= self.hyp['box'] 156 | lobj *= self.hyp['obj'] 157 | lcls *= self.hyp['cls'] 158 | 159 | bs = tobj.shape[0] # batch size 160 | 161 | return (lbox + lobj + lcls) * bs, torch.cat((lbox, lobj, lcls)).detach() 162 | 163 | 164 | 165 | 166 | 167 | def build_targets(self, p, targets): 168 | # Build targets for computer_loss(), input targets(image, class,x,y,w,h) 169 | na, nt = self.na, targets.shape[0] # number of anchors, targets 170 | tcls, tbox, indices, anch = [], [], [], [] 171 | gain = torch.ones(7, device = targets.device) # normalized to gridspace gain 172 | ai = torch.arange(na, device=targets.device).float().view(na, 1).repeat(1, nt) # same as .repeat_interleave(nt) 173 | targets = torch.cat((targets.repeat(na, 1, 1), ai[:, :, None]), 2) # append anchor indices 174 | 175 | g = 0.5 #bias 176 | off = torch.tensor([[0, 0], 177 | [1, 0], [0, 1], [-1, 0], [0, -1], # j,k,l,m 178 | # [1, 1], [1, -1], [-1, 1], [-1, -1], # jk,jm,lk,lm 179 | ], device=targets.device).float() * g # offsets 180 | 181 | for i in range(self.nl): 182 | anchors = self.anchors[i] 183 | gain[2:6] = torch.tensor(p[i].shape)[[3,2,3,2]] #xyxy gain 184 | 185 | # Match targets to anchors 186 | t = targets * gain 187 | if nt: 188 | # Matches 189 | r = t[:, :, 4:6] / anchors[:, None] # wh ratio 190 | j = torch.max(r, 1. / r).max(2)[0] < self.hyp['anchor_t'] # compare 191 | # j = wh_iou(anchors, t[:, 4:6]) > model.hyp['iou_t'] # iou(3,n)=wh_iou(anchors(3,2), gwh(n,2)) 192 | t = t[j] # filter 193 | 194 | # Offsets 195 | gxy = t[:, 2:4] # grid xy 196 | gxi = gain[[2, 3]] - gxy # inverse 197 | j, k = ((gxy % 1. < g) & (gxy > 1.)).T 198 | l, m = ((gxi % 1. < g) & (gxi > 1.)).T 199 | j = torch.stack((torch.ones_like(j), j, k, l, m)) 200 | t = t.repeat((5, 1, 1))[j] 201 | offsets = (torch.zeros_like(gxy)[None] + off[:, None])[j] 202 | else: 203 | t = targets[0] 204 | offsets = 0 205 | 206 | # Define 207 | b, c = t[:, :2].long().T # image, class 208 | gxy = t[:, 2:4] # grid xy 209 | gwh = t[:, 4:6] # grid wh 210 | gij = (gxy- offsets).long() 211 | gi, gj = gij.T 212 | 213 | # Append 214 | a = t[:, 6].long() # anchor indices 215 | indices.append((b, a, gj.clamp_(0, gain[3] -1), gi.clamp_(0, gain[2] -1))) # image, anchor, grid indices 216 | tbox.append(torch.cat((gxy - gij, gwh), 1)) # box 217 | anch.append(anchors[a]) 218 | tcls.append(c) 219 | 220 | return tcls, tbox, indices, anch 221 | 222 | 223 | 224 | 225 | 226 | 227 | 228 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | # YOLOv5 requirements 2 | # Usage: pip install -r requirements.txt 3 | 4 | # Base -------------------------------------- 5 | matplotlib>=3.2.2 6 | numpy>=1.18.5 7 | opencv-python>=4.1.1 8 | Pillow>=7.1.2 9 | PyYAML>=5.3.1 10 | requests>=2.23.0 11 | scipy>=1.4.1 12 | torch>=1.7.0 13 | torchvision>=0.8.1 14 | tqdm>=4.64.0 15 | protobuf<=3.20.1 # https://github.com/ultralytics/yolov5/issues/8012 16 | 17 | # Logging ------------------------------------- 18 | tensorboard>=2.4.1 19 | # wandb 20 | 21 | # Plotting ------------------------------------ 22 | pandas>=1.1.4 23 | seaborn>=0.11.0 24 | 25 | # Export -------------------------------------- 26 | # coremltools>=5.2 # CoreML export 27 | # onnx>=1.9.0 # ONNX export 28 | # onnx-simplifier>=0.4.1 # ONNX simplifier 29 | # nvidia-pyindex # TensorRT export 30 | # nvidia-tensorrt # TensorRT export 31 | # scikit-learn==0.19.2 # CoreML quantization 32 | # tensorflow>=2.4.1 # TFLite export (or tensorflow-cpu, tensorflow-aarch64) 33 | # tensorflowjs>=3.9.0 # TF.js export 34 | # openvino-dev # OpenVINO export 35 | 36 | # Extras -------------------------------------- 37 | ipython # interactive notebook 38 | psutil # system utilization 39 | thop>=0.1.1 # FLOPs computation 40 | # albumentations>=1.0.3 41 | # pycocotools>=2.0 # COCO mAP 42 | # roboflow# YOLOv5 requirements 43 | # Usage: pip install -r requirements.txt 44 | 45 | # Base ---------------------------------------- 46 | matplotlib>=3.2.2 47 | numpy>=1.18.5 48 | opencv-python>=4.1.1 49 | Pillow>=7.1.2 50 | PyYAML>=5.3.1 51 | requests>=2.23.0 52 | scipy>=1.4.1 53 | torch>=1.7.0 54 | torchvision>=0.8.1 55 | tqdm>=4.64.0 56 | protobuf<=3.20.1 # https://github.com/ultralytics/yolov5/issues/8012 57 | 58 | # Logging ------------------------------------- 59 | tensorboard>=2.4.1 60 | # wandb 61 | 62 | # Plotting ------------------------------------ 63 | pandas>=1.1.4 64 | seaborn>=0.11.0 65 | 66 | # Export -------------------------------------- 67 | # coremltools>=5.2 # CoreML export 68 | # onnx>=1.9.0 # ONNX export 69 | # onnx-simplifier>=0.4.1 # ONNX simplifier 70 | # nvidia-pyindex # TensorRT export 71 | # nvidia-tensorrt # TensorRT export 72 | # scikit-learn==0.19.2 # CoreML quantization 73 | # tensorflow>=2.4.1 # TFLite export (or tensorflow-cpu, tensorflow-aarch64) 74 | # tensorflowjs>=3.9.0 # TF.js export 75 | # openvino-dev # OpenVINO export 76 | 77 | # Extras -------------------------------------- 78 | ipython # interactive notebook 79 | psutil # system utilization 80 | thop>=0.1.1 # FLOPs computation 81 | # albumentations>=1.0.3 82 | # pycocotools>=2.0 # COCO mAP 83 | # roboflow 84 | --------------------------------------------------------------------------------