├── .gitignore ├── LICENSE ├── README.md ├── __pycache__ ├── alex_net.cpython-36.pyc ├── inception3.cpython-36.pyc ├── inceptionv4.cpython-36.pyc ├── net.cpython-36.pyc ├── net2.cpython-36.pyc ├── net_plus1transconv.cpython-36.pyc ├── res_net.cpython-36.pyc ├── res_net_use_this.cpython-36.pyc ├── resnet_generation_upsample.cpython-36.pyc ├── resnet_upsample.cpython-36.pyc ├── resnet_upsample3x3.cpython-36.pyc └── vgg_net.cpython-36.pyc ├── data ├── aug_train_data.mat ├── fall.tar.gz ├── gesture.tar.gz └── test.mat ├── figures └── networks.png ├── model ├── alex_net.py ├── inception3.py ├── inceptionv4.py ├── net.py ├── net2.py ├── net_plus1transconv.py ├── res_net.py ├── res_net_use_this.py ├── resnet_generation_upsample.py ├── resnet_upsample.py ├── resnet_upsample2x2.py ├── resnet_upsample3x3.py ├── solo_task_res_net.py └── vgg_net.py ├── test.py └── train.py /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | *.egg-info/ 24 | .installed.cfg 25 | *.egg 26 | MANIFEST 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *.cover 47 | .hypothesis/ 48 | .pytest_cache/ 49 | 50 | # Translations 51 | *.mo 52 | *.pot 53 | 54 | # Django stuff: 55 | *.log 56 | local_settings.py 57 | db.sqlite3 58 | 59 | # Flask stuff: 60 | instance/ 61 | .webassets-cache 62 | 63 | # Scrapy stuff: 64 | .scrapy 65 | 66 | # Sphinx documentation 67 | docs/_build/ 68 | 69 | # PyBuilder 70 | target/ 71 | 72 | # Jupyter Notebook 73 | .ipynb_checkpoints 74 | 75 | # pyenv 76 | .python-version 77 | 78 | # celery beat schedule file 79 | celerybeat-schedule 80 | 81 | # SageMath parsed files 82 | *.sage.py 83 | 84 | # Environments 85 | .env 86 | .venv 87 | env/ 88 | venv/ 89 | ENV/ 90 | env.bak/ 91 | venv.bak/ 92 | 93 | # Spyder project settings 94 | .spyderproject 95 | .spyproject 96 | 97 | # Rope project settings 98 | .ropeproject 99 | 100 | # mkdocs documentation 101 | /site 102 | 103 | # mypy 104 | .mypy_cache/ 105 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Fei Wang 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # CSI-Net pytorch 2 | ![](figures/networks.png) 3 | 4 | 5 | This code can achieve multi-channel, time-serial signal classification as well as regression by modifying modern backbone networks. In our original paper, we have achieved four human sensing tasks with WiFi signals. 6 | 7 | We also applied this code with little modification to do motor fault diagnosis in engineering industry. Paper has been submitted to the Transactions on Industrial Electronics with title of *Multi-scale Residual Learning for End-to-end Motor Fault Diagnosis Under Nonstationary Conditions* [Code is here](https://github.com/geekfeiw/Multi-Scale-1D-ResNet). We are designing a light-weighted version for sensing tasks with COTS RFID (under progress). We believe this generic deep networks can solve more sensory data tasks. 8 | 9 | ## Tested Environment 10 | 1. python 3.6 11 | 2. pytorch 0.4.1 12 | 3. cuda 8.0/9.0 13 | 4. cudnn 6.0/7.3 14 | 5. Windows7/Ubuntu16.04 15 | 16 | ## Usage 17 | 1. clone repository. 18 | 2. make a directory, name it *weights*, then download pre-trained model into *weights*: *https://drive.google.com/open?id=16HOqFagtigjGry5asx-ErradnWS5rfe2* 19 | 20 | We used model, *model/res_net_use_this.py*, to jointly solve two body characterization tasks. Besides, we provide *model/solo_task_res_net.py* for solo classification tasks. 21 | 22 | For various task/data complexity, we also provide multiple mainstream backbones, including Inception V3, V4, VGG, AlexNet. 23 | 24 | 25 | ## Citation 26 | Please cite this paper in your publications if it helps your research. 27 | 28 | @article{wang2018csi, 29 | title={CSI-Net: Unified Body Characterization and Action Recognition}, 30 | author={Wang, Fei and Han, Jinsong and Zhang, Shiyuan and He, Xu and Huang, Dong}, 31 | journal={arXiv preprint arXiv:1810.03064}, 32 | year={2018} 33 | } 34 | 35 | -------------------------------------------------------------------------------- /__pycache__/alex_net.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekfeiw/CSI-Net/5dcefc2f365167cbe37c004031e0310d60ebd548/__pycache__/alex_net.cpython-36.pyc -------------------------------------------------------------------------------- /__pycache__/inception3.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekfeiw/CSI-Net/5dcefc2f365167cbe37c004031e0310d60ebd548/__pycache__/inception3.cpython-36.pyc -------------------------------------------------------------------------------- /__pycache__/inceptionv4.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekfeiw/CSI-Net/5dcefc2f365167cbe37c004031e0310d60ebd548/__pycache__/inceptionv4.cpython-36.pyc -------------------------------------------------------------------------------- /__pycache__/net.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekfeiw/CSI-Net/5dcefc2f365167cbe37c004031e0310d60ebd548/__pycache__/net.cpython-36.pyc -------------------------------------------------------------------------------- /__pycache__/net2.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekfeiw/CSI-Net/5dcefc2f365167cbe37c004031e0310d60ebd548/__pycache__/net2.cpython-36.pyc -------------------------------------------------------------------------------- /__pycache__/net_plus1transconv.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekfeiw/CSI-Net/5dcefc2f365167cbe37c004031e0310d60ebd548/__pycache__/net_plus1transconv.cpython-36.pyc -------------------------------------------------------------------------------- /__pycache__/res_net.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekfeiw/CSI-Net/5dcefc2f365167cbe37c004031e0310d60ebd548/__pycache__/res_net.cpython-36.pyc -------------------------------------------------------------------------------- /__pycache__/res_net_use_this.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekfeiw/CSI-Net/5dcefc2f365167cbe37c004031e0310d60ebd548/__pycache__/res_net_use_this.cpython-36.pyc -------------------------------------------------------------------------------- /__pycache__/resnet_generation_upsample.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekfeiw/CSI-Net/5dcefc2f365167cbe37c004031e0310d60ebd548/__pycache__/resnet_generation_upsample.cpython-36.pyc -------------------------------------------------------------------------------- /__pycache__/resnet_upsample.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekfeiw/CSI-Net/5dcefc2f365167cbe37c004031e0310d60ebd548/__pycache__/resnet_upsample.cpython-36.pyc -------------------------------------------------------------------------------- /__pycache__/resnet_upsample3x3.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekfeiw/CSI-Net/5dcefc2f365167cbe37c004031e0310d60ebd548/__pycache__/resnet_upsample3x3.cpython-36.pyc -------------------------------------------------------------------------------- /__pycache__/vgg_net.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekfeiw/CSI-Net/5dcefc2f365167cbe37c004031e0310d60ebd548/__pycache__/vgg_net.cpython-36.pyc -------------------------------------------------------------------------------- /data/aug_train_data.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekfeiw/CSI-Net/5dcefc2f365167cbe37c004031e0310d60ebd548/data/aug_train_data.mat -------------------------------------------------------------------------------- /data/fall.tar.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekfeiw/CSI-Net/5dcefc2f365167cbe37c004031e0310d60ebd548/data/fall.tar.gz -------------------------------------------------------------------------------- /data/gesture.tar.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekfeiw/CSI-Net/5dcefc2f365167cbe37c004031e0310d60ebd548/data/gesture.tar.gz -------------------------------------------------------------------------------- /data/test.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekfeiw/CSI-Net/5dcefc2f365167cbe37c004031e0310d60ebd548/data/test.mat -------------------------------------------------------------------------------- /figures/networks.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekfeiw/CSI-Net/5dcefc2f365167cbe37c004031e0310d60ebd548/figures/networks.png -------------------------------------------------------------------------------- /model/alex_net.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import math 3 | 4 | class AlexNet(nn.Module): 5 | 6 | def __init__(self, num_classes=1000): 7 | super(AlexNet, self).__init__() 8 | 9 | self.generation = nn.Sequential( 10 | # 30*1*1 -> 256*2*2 11 | nn.ConvTranspose2d(30, 384, kernel_size=4, stride=2, padding=1, bias=False), 12 | nn.BatchNorm2d(384), 13 | nn.ReLU(), 14 | # 256*2*2 -> 128*4*4 15 | nn.ConvTranspose2d(384, 192, kernel_size=4, stride=2, padding=1, bias=False), 16 | nn.BatchNorm2d(192), 17 | nn.ReLU(), 18 | 19 | # 128*4*4 -> 64*7*7 20 | nn.ConvTranspose2d(192, 96, kernel_size=3, stride=2, padding=1, bias=False), 21 | nn.BatchNorm2d(96), 22 | nn.ReLU(), 23 | 24 | # 7 -> 14 25 | nn.ConvTranspose2d(96, 48, kernel_size=4, stride=2, padding=1, bias=False), 26 | nn.BatchNorm2d(48), 27 | nn.ReLU(), 28 | 29 | # 14 -> 28 30 | nn.ConvTranspose2d(48, 24, kernel_size=4, stride=2, padding=1, bias=False), 31 | nn.BatchNorm2d(24), 32 | nn.ReLU(), 33 | 34 | # 28 -> 56 35 | nn.ConvTranspose2d(24, 12, kernel_size=4, stride=2, padding=1, bias=False), 36 | nn.BatchNorm2d(12), 37 | nn.ReLU(), 38 | 39 | # 56 -> 112 40 | nn.ConvTranspose2d(12, 6, kernel_size=4, stride=2, padding=1, bias=False), 41 | nn.BatchNorm2d(6), 42 | nn.ReLU(), 43 | 44 | # 112 -> 224c 45 | nn.ConvTranspose2d(6, 6, kernel_size=4, stride=2, padding=1, bias=False), 46 | nn.BatchNorm2d(6), 47 | nn.ReLU(), 48 | ) 49 | 50 | 51 | self.features = nn.Sequential( 52 | nn.Conv2d(6, 64, kernel_size=11, stride=4, padding=2), 53 | nn.ReLU(inplace=True), 54 | nn.MaxPool2d(kernel_size=3, stride=2), 55 | nn.Conv2d(64, 192, kernel_size=5, padding=2), 56 | nn.ReLU(inplace=True), 57 | nn.MaxPool2d(kernel_size=3, stride=2), 58 | nn.Conv2d(192, 384, kernel_size=3, padding=1), 59 | nn.ReLU(inplace=True), 60 | nn.Conv2d(384, 256, kernel_size=3, padding=1), 61 | nn.ReLU(inplace=True), 62 | nn.Conv2d(256, 256, kernel_size=3, padding=1), 63 | nn.ReLU(inplace=True), 64 | nn.MaxPool2d(kernel_size=3, stride=2), 65 | ) 66 | self.classifier = nn.Sequential( 67 | nn.Dropout(), 68 | nn.Linear(256 * 6 * 6, 4096), 69 | nn.ReLU(inplace=True), 70 | nn.Dropout(), 71 | nn.Linear(4096, 4096), 72 | nn.ReLU(inplace=True), 73 | nn.Linear(4096, 10), 74 | # nn.Dropout(), 75 | # nn.Linear(256 * 6 * 6, 1024), 76 | # nn.ReLU(inplace=True), 77 | # nn.Dropout(), 78 | # nn.Linear(1024, 512), 79 | # nn.ReLU(inplace=True), 80 | # nn.Linear(512, 2), 81 | ) 82 | 83 | def forward(self, x): 84 | x = self.generation(x) 85 | x = self.features(x) 86 | x = x.view(x.size(0), 256 * 6 * 6) 87 | x = self.classifier(x) 88 | return x -------------------------------------------------------------------------------- /model/inception3.py: -------------------------------------------------------------------------------- 1 | 2 | import torch 3 | import torch.nn as nn 4 | import torch.nn.functional as F 5 | import torch.utils.model_zoo as model_zoo 6 | 7 | 8 | 9 | 10 | class Inception3(nn.Module): 11 | def __init__(self, num_classes=30, aux_logits=False, transform_input=False): 12 | super(Inception3, self).__init__() 13 | self.generation = nn.Sequential( 14 | # 30*1*1 -> 256*2*2 15 | nn.ConvTranspose2d(30, 384, kernel_size=4, stride=2, padding=1, bias=False), 16 | nn.BatchNorm2d(384), 17 | nn.ReLU(), 18 | # 256*2*2 -> 128*4*4 19 | nn.ConvTranspose2d(384, 192, kernel_size=4, stride=2, padding=1, bias=False), 20 | nn.BatchNorm2d(192), 21 | nn.ReLU(), 22 | 23 | # 128*4*4 -> 64*7*7 24 | nn.ConvTranspose2d(192, 96, kernel_size=3, stride=2, padding=1, bias=False), 25 | nn.BatchNorm2d(96), 26 | nn.ReLU(), 27 | 28 | # 7 -> 14 29 | nn.ConvTranspose2d(96, 48, kernel_size=4, stride=2, padding=1, bias=False), 30 | nn.BatchNorm2d(48), 31 | nn.ReLU(), 32 | 33 | # 14 -> 28 34 | nn.ConvTranspose2d(48, 24, kernel_size=4, stride=2, padding=1, bias=False), 35 | nn.BatchNorm2d(24), 36 | nn.ReLU(), 37 | 38 | # 28 -> 56 39 | nn.ConvTranspose2d(24, 12, kernel_size=4, stride=2, padding=1, bias=False), 40 | nn.BatchNorm2d(12), 41 | nn.ReLU(), 42 | 43 | # 56 -> 112 44 | nn.ConvTranspose2d(12, 6, kernel_size=4, stride=2, padding=1, bias=False), 45 | nn.BatchNorm2d(6), 46 | nn.ReLU(), 47 | 48 | # 112 -> 224c 49 | nn.ConvTranspose2d(6, 3, kernel_size=4, stride=2, padding=1, bias=False), 50 | nn.BatchNorm2d(3), 51 | nn.ReLU(), 52 | ) 53 | self.aux_logits = aux_logits 54 | self.transform_input = transform_input 55 | self.Conv2d_1a_3x3 = BasicConv2d(3, 32, kernel_size=3, stride=2) 56 | self.Conv2d_2a_3x3 = BasicConv2d(32, 32, kernel_size=3) 57 | self.Conv2d_2b_3x3 = BasicConv2d(32, 64, kernel_size=3, padding=1) 58 | self.Conv2d_3b_1x1 = BasicConv2d(64, 80, kernel_size=1) 59 | self.Conv2d_4a_3x3 = BasicConv2d(80, 192, kernel_size=3) 60 | self.Mixed_5b = InceptionA(192, pool_features=32) 61 | self.Mixed_5c = InceptionA(256, pool_features=64) 62 | self.Mixed_5d = InceptionA(288, pool_features=64) 63 | self.Mixed_6a = InceptionB(288) 64 | self.Mixed_6b = InceptionC(768, channels_7x7=128) 65 | self.Mixed_6c = InceptionC(768, channels_7x7=160) 66 | self.Mixed_6d = InceptionC(768, channels_7x7=160) 67 | self.Mixed_6e = InceptionC(768, channels_7x7=192) 68 | if aux_logits: 69 | self.AuxLogits = InceptionAux(768, num_classes) 70 | self.Mixed_7a = InceptionD(768) 71 | self.Mixed_7b = InceptionE(1280) 72 | self.Mixed_7c = InceptionE(2048) 73 | 74 | self.fcc = nn.Linear(2048, num_classes) 75 | self.fcr = nn.Linear(2048, 4) 76 | 77 | # for m in self.modules(): 78 | # if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear): 79 | # import scipy.stats as stats 80 | # stddev = m.stddev if hasattr(m, 'stddev') else 0.1 81 | # X = stats.truncnorm(-2, 2, scale=stddev) 82 | # values = torch.Tensor(X.rvs(m.weight.numel())) 83 | # values = values.view(m.weight.size()) 84 | # m.weight.data.copy_(values) 85 | # elif isinstance(m, nn.BatchNorm2d): 86 | # nn.init.constant_(m.weight, 1) 87 | # nn.init.constant_(m.bias, 0) 88 | 89 | def forward(self, x): 90 | x = self.generation(x) 91 | if self.transform_input: 92 | x = x.clone() 93 | x[:, 0] = x[:, 0] * (0.229 / 0.5) + (0.485 - 0.5) / 0.5 94 | x[:, 1] = x[:, 1] * (0.224 / 0.5) + (0.456 - 0.5) / 0.5 95 | x[:, 2] = x[:, 2] * (0.225 / 0.5) + (0.406 - 0.5) / 0.5 96 | # 299 x 299 x 3 97 | x = self.Conv2d_1a_3x3(x) 98 | # 149 x 149 x 32 99 | x = self.Conv2d_2a_3x3(x) 100 | # 147 x 147 x 32 101 | x = self.Conv2d_2b_3x3(x) 102 | # 147 x 147 x 64 103 | x = F.max_pool2d(x, kernel_size=3, stride=2) 104 | # 73 x 73 x 64 105 | x = self.Conv2d_3b_1x1(x) 106 | # 73 x 73 x 80 107 | x = self.Conv2d_4a_3x3(x) 108 | # 71 x 71 x 192 109 | x = F.max_pool2d(x, kernel_size=3, stride=2) 110 | # 35 x 35 x 192 111 | x = self.Mixed_5b(x) 112 | # 35 x 35 x 256 113 | x = self.Mixed_5c(x) 114 | # 35 x 35 x 288 115 | x = self.Mixed_5d(x) 116 | # 35 x 35 x 288 117 | x = self.Mixed_6a(x) 118 | # 17 x 17 x 768 119 | x = self.Mixed_6b(x) 120 | # 17 x 17 x 768 121 | x = self.Mixed_6c(x) 122 | # 17 x 17 x 768 123 | x = self.Mixed_6d(x) 124 | # 17 x 17 x 768 125 | x = self.Mixed_6e(x) 126 | # 17 x 17 x 768 127 | if self.training and self.aux_logits: 128 | aux = self.AuxLogits(x) 129 | # 17 x 17 x 768 130 | x = self.Mixed_7a(x) 131 | # 8 x 8 x 1280 132 | x = self.Mixed_7b(x) 133 | # 8 x 8 x 2048 134 | x = self.Mixed_7c(x) 135 | # 8 x 8 x 2048 136 | x = F.avg_pool2d(x, kernel_size=5) 137 | # 1 x 1 x 2048 138 | x = F.dropout(x, training=self.training) 139 | # 1 x 1 x 2048 140 | x1 = x.view(x.size(0), -1) 141 | # 2048 142 | x = self.fcc(x1) 143 | y = self.fcr(x1) 144 | # 1000 (num_classes) 145 | if self.training and self.aux_logits: 146 | return x, aux 147 | return x, y 148 | 149 | 150 | class InceptionA(nn.Module): 151 | def __init__(self, in_channels, pool_features): 152 | super(InceptionA, self).__init__() 153 | self.branch1x1 = BasicConv2d(in_channels, 64, kernel_size=1) 154 | 155 | self.branch5x5_1 = BasicConv2d(in_channels, 48, kernel_size=1) 156 | self.branch5x5_2 = BasicConv2d(48, 64, kernel_size=5, padding=2) 157 | 158 | self.branch3x3dbl_1 = BasicConv2d(in_channels, 64, kernel_size=1) 159 | self.branch3x3dbl_2 = BasicConv2d(64, 96, kernel_size=3, padding=1) 160 | self.branch3x3dbl_3 = BasicConv2d(96, 96, kernel_size=3, padding=1) 161 | 162 | self.branch_pool = BasicConv2d(in_channels, pool_features, kernel_size=1) 163 | 164 | def forward(self, x): 165 | branch1x1 = self.branch1x1(x) 166 | 167 | branch5x5 = self.branch5x5_1(x) 168 | branch5x5 = self.branch5x5_2(branch5x5) 169 | 170 | branch3x3dbl = self.branch3x3dbl_1(x) 171 | branch3x3dbl = self.branch3x3dbl_2(branch3x3dbl) 172 | branch3x3dbl = self.branch3x3dbl_3(branch3x3dbl) 173 | 174 | branch_pool = F.avg_pool2d(x, kernel_size=3, stride=1, padding=1) 175 | branch_pool = self.branch_pool(branch_pool) 176 | 177 | outputs = [branch1x1, branch5x5, branch3x3dbl, branch_pool] 178 | return torch.cat(outputs, 1) 179 | 180 | 181 | class InceptionB(nn.Module): 182 | def __init__(self, in_channels): 183 | super(InceptionB, self).__init__() 184 | self.branch3x3 = BasicConv2d(in_channels, 384, kernel_size=3, stride=2) 185 | 186 | self.branch3x3dbl_1 = BasicConv2d(in_channels, 64, kernel_size=1) 187 | self.branch3x3dbl_2 = BasicConv2d(64, 96, kernel_size=3, padding=1) 188 | self.branch3x3dbl_3 = BasicConv2d(96, 96, kernel_size=3, stride=2) 189 | 190 | def forward(self, x): 191 | branch3x3 = self.branch3x3(x) 192 | 193 | branch3x3dbl = self.branch3x3dbl_1(x) 194 | branch3x3dbl = self.branch3x3dbl_2(branch3x3dbl) 195 | branch3x3dbl = self.branch3x3dbl_3(branch3x3dbl) 196 | 197 | branch_pool = F.max_pool2d(x, kernel_size=3, stride=2) 198 | 199 | outputs = [branch3x3, branch3x3dbl, branch_pool] 200 | return torch.cat(outputs, 1) 201 | 202 | 203 | class InceptionC(nn.Module): 204 | def __init__(self, in_channels, channels_7x7): 205 | super(InceptionC, self).__init__() 206 | self.branch1x1 = BasicConv2d(in_channels, 192, kernel_size=1) 207 | 208 | c7 = channels_7x7 209 | self.branch7x7_1 = BasicConv2d(in_channels, c7, kernel_size=1) 210 | self.branch7x7_2 = BasicConv2d(c7, c7, kernel_size=(1, 7), padding=(0, 3)) 211 | self.branch7x7_3 = BasicConv2d(c7, 192, kernel_size=(7, 1), padding=(3, 0)) 212 | 213 | self.branch7x7dbl_1 = BasicConv2d(in_channels, c7, kernel_size=1) 214 | self.branch7x7dbl_2 = BasicConv2d(c7, c7, kernel_size=(7, 1), padding=(3, 0)) 215 | self.branch7x7dbl_3 = BasicConv2d(c7, c7, kernel_size=(1, 7), padding=(0, 3)) 216 | self.branch7x7dbl_4 = BasicConv2d(c7, c7, kernel_size=(7, 1), padding=(3, 0)) 217 | self.branch7x7dbl_5 = BasicConv2d(c7, 192, kernel_size=(1, 7), padding=(0, 3)) 218 | 219 | self.branch_pool = BasicConv2d(in_channels, 192, kernel_size=1) 220 | 221 | def forward(self, x): 222 | branch1x1 = self.branch1x1(x) 223 | 224 | branch7x7 = self.branch7x7_1(x) 225 | branch7x7 = self.branch7x7_2(branch7x7) 226 | branch7x7 = self.branch7x7_3(branch7x7) 227 | 228 | branch7x7dbl = self.branch7x7dbl_1(x) 229 | branch7x7dbl = self.branch7x7dbl_2(branch7x7dbl) 230 | branch7x7dbl = self.branch7x7dbl_3(branch7x7dbl) 231 | branch7x7dbl = self.branch7x7dbl_4(branch7x7dbl) 232 | branch7x7dbl = self.branch7x7dbl_5(branch7x7dbl) 233 | 234 | branch_pool = F.avg_pool2d(x, kernel_size=3, stride=1, padding=1) 235 | branch_pool = self.branch_pool(branch_pool) 236 | 237 | outputs = [branch1x1, branch7x7, branch7x7dbl, branch_pool] 238 | return torch.cat(outputs, 1) 239 | 240 | 241 | class InceptionD(nn.Module): 242 | def __init__(self, in_channels): 243 | super(InceptionD, self).__init__() 244 | self.branch3x3_1 = BasicConv2d(in_channels, 192, kernel_size=1) 245 | self.branch3x3_2 = BasicConv2d(192, 320, kernel_size=3, stride=2) 246 | 247 | self.branch7x7x3_1 = BasicConv2d(in_channels, 192, kernel_size=1) 248 | self.branch7x7x3_2 = BasicConv2d(192, 192, kernel_size=(1, 7), padding=(0, 3)) 249 | self.branch7x7x3_3 = BasicConv2d(192, 192, kernel_size=(7, 1), padding=(3, 0)) 250 | self.branch7x7x3_4 = BasicConv2d(192, 192, kernel_size=3, stride=2) 251 | 252 | def forward(self, x): 253 | branch3x3 = self.branch3x3_1(x) 254 | branch3x3 = self.branch3x3_2(branch3x3) 255 | 256 | branch7x7x3 = self.branch7x7x3_1(x) 257 | branch7x7x3 = self.branch7x7x3_2(branch7x7x3) 258 | branch7x7x3 = self.branch7x7x3_3(branch7x7x3) 259 | branch7x7x3 = self.branch7x7x3_4(branch7x7x3) 260 | 261 | branch_pool = F.max_pool2d(x, kernel_size=3, stride=2) 262 | outputs = [branch3x3, branch7x7x3, branch_pool] 263 | return torch.cat(outputs, 1) 264 | 265 | 266 | class InceptionE(nn.Module): 267 | def __init__(self, in_channels): 268 | super(InceptionE, self).__init__() 269 | self.branch1x1 = BasicConv2d(in_channels, 320, kernel_size=1) 270 | 271 | self.branch3x3_1 = BasicConv2d(in_channels, 384, kernel_size=1) 272 | self.branch3x3_2a = BasicConv2d(384, 384, kernel_size=(1, 3), padding=(0, 1)) 273 | self.branch3x3_2b = BasicConv2d(384, 384, kernel_size=(3, 1), padding=(1, 0)) 274 | 275 | self.branch3x3dbl_1 = BasicConv2d(in_channels, 448, kernel_size=1) 276 | self.branch3x3dbl_2 = BasicConv2d(448, 384, kernel_size=3, padding=1) 277 | self.branch3x3dbl_3a = BasicConv2d(384, 384, kernel_size=(1, 3), padding=(0, 1)) 278 | self.branch3x3dbl_3b = BasicConv2d(384, 384, kernel_size=(3, 1), padding=(1, 0)) 279 | 280 | self.branch_pool = BasicConv2d(in_channels, 192, kernel_size=1) 281 | 282 | def forward(self, x): 283 | branch1x1 = self.branch1x1(x) 284 | 285 | branch3x3 = self.branch3x3_1(x) 286 | branch3x3 = [ 287 | self.branch3x3_2a(branch3x3), 288 | self.branch3x3_2b(branch3x3), 289 | ] 290 | branch3x3 = torch.cat(branch3x3, 1) 291 | 292 | branch3x3dbl = self.branch3x3dbl_1(x) 293 | branch3x3dbl = self.branch3x3dbl_2(branch3x3dbl) 294 | branch3x3dbl = [ 295 | self.branch3x3dbl_3a(branch3x3dbl), 296 | self.branch3x3dbl_3b(branch3x3dbl), 297 | ] 298 | branch3x3dbl = torch.cat(branch3x3dbl, 1) 299 | 300 | branch_pool = F.avg_pool2d(x, kernel_size=3, stride=1, padding=1) 301 | branch_pool = self.branch_pool(branch_pool) 302 | 303 | outputs = [branch1x1, branch3x3, branch3x3dbl, branch_pool] 304 | return torch.cat(outputs, 1) 305 | 306 | 307 | class InceptionAux(nn.Module): 308 | def __init__(self, in_channels, num_classes): 309 | super(InceptionAux, self).__init__() 310 | self.conv0 = BasicConv2d(in_channels, 128, kernel_size=1) 311 | self.conv1 = BasicConv2d(128, 768, kernel_size=5) 312 | self.conv1.stddev = 0.01 313 | self.fc = nn.Linear(768, num_classes) 314 | self.fc.stddev = 0.001 315 | 316 | def forward(self, x): 317 | # 17 x 17 x 768 318 | x = F.avg_pool2d(x, kernel_size=5, stride=3) 319 | # 5 x 5 x 768 320 | x = self.conv0(x) 321 | # 5 x 5 x 128 322 | x = self.conv1(x) 323 | # 1 x 1 x 768 324 | x = x.view(x.size(0), -1) 325 | # 768 326 | x = self.fc(x) 327 | # 1000 328 | return x 329 | 330 | 331 | class BasicConv2d(nn.Module): 332 | def __init__(self, in_channels, out_channels, **kwargs): 333 | super(BasicConv2d, self).__init__() 334 | self.conv = nn.Conv2d(in_channels, out_channels, bias=False, **kwargs) 335 | self.bn = nn.BatchNorm2d(out_channels, eps=0.001) 336 | 337 | def forward(self, x): 338 | x = self.conv(x) 339 | x = self.bn(x) 340 | return F.relu(x, inplace=True) -------------------------------------------------------------------------------- /model/inceptionv4.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import torch.utils.model_zoo as model_zoo 4 | import os 5 | import sys 6 | 7 | 8 | class BasicConv2d(nn.Module): 9 | def __init__(self, in_planes, out_planes, kernel_size, stride, padding=0): 10 | super(BasicConv2d, self).__init__() 11 | self.conv = nn.Conv2d(in_planes, out_planes, 12 | kernel_size=kernel_size, stride=stride, 13 | padding=padding, bias=False) # verify bias false 14 | self.bn = nn.BatchNorm2d(out_planes, 15 | eps=0.001, # value found in tensorflow 16 | momentum=0.1, # default pytorch value 17 | affine=True) 18 | self.relu = nn.ReLU(inplace=True) 19 | 20 | def forward(self, x): 21 | x = self.conv(x) 22 | x = self.bn(x) 23 | x = self.relu(x) 24 | return x 25 | 26 | 27 | class Mixed_3a(nn.Module): 28 | def __init__(self): 29 | super(Mixed_3a, self).__init__() 30 | self.maxpool = nn.MaxPool2d(3, stride=2) 31 | self.conv = BasicConv2d(64, 96, kernel_size=3, stride=2) 32 | 33 | def forward(self, x): 34 | x0 = self.maxpool(x) 35 | x1 = self.conv(x) 36 | out = torch.cat((x0, x1), 1) 37 | return out 38 | 39 | 40 | class Mixed_4a(nn.Module): 41 | def __init__(self): 42 | super(Mixed_4a, self).__init__() 43 | 44 | self.branch0 = nn.Sequential( 45 | BasicConv2d(160, 64, kernel_size=1, stride=1), 46 | BasicConv2d(64, 96, kernel_size=3, stride=1) 47 | ) 48 | 49 | self.branch1 = nn.Sequential( 50 | BasicConv2d(160, 64, kernel_size=1, stride=1), 51 | BasicConv2d(64, 64, kernel_size=(1, 7), stride=1, padding=(0, 3)), 52 | BasicConv2d(64, 64, kernel_size=(7, 1), stride=1, padding=(3, 0)), 53 | BasicConv2d(64, 96, kernel_size=(3, 3), stride=1) 54 | ) 55 | 56 | def forward(self, x): 57 | x0 = self.branch0(x) 58 | x1 = self.branch1(x) 59 | out = torch.cat((x0, x1), 1) 60 | return out 61 | 62 | 63 | class Mixed_5a(nn.Module): 64 | def __init__(self): 65 | super(Mixed_5a, self).__init__() 66 | self.conv = BasicConv2d(192, 192, kernel_size=3, stride=2) 67 | self.maxpool = nn.MaxPool2d(3, stride=2) 68 | 69 | def forward(self, x): 70 | x0 = self.conv(x) 71 | x1 = self.maxpool(x) 72 | out = torch.cat((x0, x1), 1) 73 | return out 74 | 75 | 76 | class Inception_A(nn.Module): 77 | def __init__(self): 78 | super(Inception_A, self).__init__() 79 | self.branch0 = BasicConv2d(384, 96, kernel_size=1, stride=1) 80 | 81 | self.branch1 = nn.Sequential( 82 | BasicConv2d(384, 64, kernel_size=1, stride=1), 83 | BasicConv2d(64, 96, kernel_size=3, stride=1, padding=1) 84 | ) 85 | 86 | self.branch2 = nn.Sequential( 87 | BasicConv2d(384, 64, kernel_size=1, stride=1), 88 | BasicConv2d(64, 96, kernel_size=3, stride=1, padding=1), 89 | BasicConv2d(96, 96, kernel_size=3, stride=1, padding=1) 90 | ) 91 | 92 | self.branch3 = nn.Sequential( 93 | nn.AvgPool2d(3, stride=1, padding=1, count_include_pad=False), 94 | BasicConv2d(384, 96, kernel_size=1, stride=1) 95 | ) 96 | 97 | def forward(self, x): 98 | x0 = self.branch0(x) 99 | x1 = self.branch1(x) 100 | x2 = self.branch2(x) 101 | x3 = self.branch3(x) 102 | out = torch.cat((x0, x1, x2, x3), 1) 103 | return out 104 | 105 | 106 | class Reduction_A(nn.Module): 107 | def __init__(self): 108 | super(Reduction_A, self).__init__() 109 | self.branch0 = BasicConv2d(384, 384, kernel_size=3, stride=2) 110 | 111 | self.branch1 = nn.Sequential( 112 | BasicConv2d(384, 192, kernel_size=1, stride=1), 113 | BasicConv2d(192, 224, kernel_size=3, stride=1, padding=1), 114 | BasicConv2d(224, 256, kernel_size=3, stride=2) 115 | ) 116 | 117 | self.branch2 = nn.MaxPool2d(3, stride=2) 118 | 119 | def forward(self, x): 120 | x0 = self.branch0(x) 121 | x1 = self.branch1(x) 122 | x2 = self.branch2(x) 123 | out = torch.cat((x0, x1, x2), 1) 124 | return out 125 | 126 | 127 | class Inception_B(nn.Module): 128 | def __init__(self): 129 | super(Inception_B, self).__init__() 130 | self.branch0 = BasicConv2d(1024, 384, kernel_size=1, stride=1) 131 | 132 | self.branch1 = nn.Sequential( 133 | BasicConv2d(1024, 192, kernel_size=1, stride=1), 134 | BasicConv2d(192, 224, kernel_size=(1, 7), stride=1, padding=(0, 3)), 135 | BasicConv2d(224, 256, kernel_size=(7, 1), stride=1, padding=(3, 0)) 136 | ) 137 | 138 | self.branch2 = nn.Sequential( 139 | BasicConv2d(1024, 192, kernel_size=1, stride=1), 140 | BasicConv2d(192, 192, kernel_size=(7, 1), stride=1, padding=(3, 0)), 141 | BasicConv2d(192, 224, kernel_size=(1, 7), stride=1, padding=(0, 3)), 142 | BasicConv2d(224, 224, kernel_size=(7, 1), stride=1, padding=(3, 0)), 143 | BasicConv2d(224, 256, kernel_size=(1, 7), stride=1, padding=(0, 3)) 144 | ) 145 | 146 | self.branch3 = nn.Sequential( 147 | nn.AvgPool2d(3, stride=1, padding=1, count_include_pad=False), 148 | BasicConv2d(1024, 128, kernel_size=1, stride=1) 149 | ) 150 | 151 | def forward(self, x): 152 | x0 = self.branch0(x) 153 | x1 = self.branch1(x) 154 | x2 = self.branch2(x) 155 | x3 = self.branch3(x) 156 | out = torch.cat((x0, x1, x2, x3), 1) 157 | return out 158 | 159 | 160 | class Reduction_B(nn.Module): 161 | def __init__(self): 162 | super(Reduction_B, self).__init__() 163 | 164 | self.branch0 = nn.Sequential( 165 | BasicConv2d(1024, 192, kernel_size=1, stride=1), 166 | BasicConv2d(192, 192, kernel_size=3, stride=2) 167 | ) 168 | 169 | self.branch1 = nn.Sequential( 170 | BasicConv2d(1024, 256, kernel_size=1, stride=1), 171 | BasicConv2d(256, 256, kernel_size=(1, 7), stride=1, padding=(0, 3)), 172 | BasicConv2d(256, 320, kernel_size=(7, 1), stride=1, padding=(3, 0)), 173 | BasicConv2d(320, 320, kernel_size=3, stride=2) 174 | ) 175 | 176 | self.branch2 = nn.MaxPool2d(3, stride=2) 177 | 178 | def forward(self, x): 179 | x0 = self.branch0(x) 180 | x1 = self.branch1(x) 181 | x2 = self.branch2(x) 182 | out = torch.cat((x0, x1, x2), 1) 183 | return out 184 | 185 | 186 | class Inception_C(nn.Module): 187 | def __init__(self): 188 | super(Inception_C, self).__init__() 189 | 190 | self.branch0 = BasicConv2d(1536, 256, kernel_size=1, stride=1) 191 | 192 | self.branch1_0 = BasicConv2d(1536, 384, kernel_size=1, stride=1) 193 | self.branch1_1a = BasicConv2d(384, 256, kernel_size=(1, 3), stride=1, padding=(0, 1)) 194 | self.branch1_1b = BasicConv2d(384, 256, kernel_size=(3, 1), stride=1, padding=(1, 0)) 195 | 196 | self.branch2_0 = BasicConv2d(1536, 384, kernel_size=1, stride=1) 197 | self.branch2_1 = BasicConv2d(384, 448, kernel_size=(3, 1), stride=1, padding=(1, 0)) 198 | self.branch2_2 = BasicConv2d(448, 512, kernel_size=(1, 3), stride=1, padding=(0, 1)) 199 | self.branch2_3a = BasicConv2d(512, 256, kernel_size=(1, 3), stride=1, padding=(0, 1)) 200 | self.branch2_3b = BasicConv2d(512, 256, kernel_size=(3, 1), stride=1, padding=(1, 0)) 201 | 202 | self.branch3 = nn.Sequential( 203 | nn.AvgPool2d(3, stride=1, padding=1, count_include_pad=False), 204 | BasicConv2d(1536, 256, kernel_size=1, stride=1) 205 | ) 206 | 207 | def forward(self, x): 208 | x0 = self.branch0(x) 209 | 210 | x1_0 = self.branch1_0(x) 211 | x1_1a = self.branch1_1a(x1_0) 212 | x1_1b = self.branch1_1b(x1_0) 213 | x1 = torch.cat((x1_1a, x1_1b), 1) 214 | 215 | x2_0 = self.branch2_0(x) 216 | x2_1 = self.branch2_1(x2_0) 217 | x2_2 = self.branch2_2(x2_1) 218 | x2_3a = self.branch2_3a(x2_2) 219 | x2_3b = self.branch2_3b(x2_2) 220 | x2 = torch.cat((x2_3a, x2_3b), 1) 221 | 222 | x3 = self.branch3(x) 223 | 224 | out = torch.cat((x0, x1, x2, x3), 1) 225 | return out 226 | 227 | 228 | class InceptionV4(nn.Module): 229 | def __init__(self, num_classes=30): 230 | super(InceptionV4, self).__init__() 231 | # Special attributs 232 | self.input_space = None 233 | self.input_size = (224, 224, 6) 234 | self.mean = None 235 | self.std = None 236 | 237 | self.generation = nn.Sequential( 238 | # 30*1*1 -> 256*2*2 239 | nn.ConvTranspose2d(30, 384, kernel_size=4, stride=2, padding=1, bias=False), 240 | nn.BatchNorm2d(384), 241 | nn.ReLU(), 242 | # 256*2*2 -> 128*4*4 243 | nn.ConvTranspose2d(384, 192, kernel_size=4, stride=2, padding=1, bias=False), 244 | nn.BatchNorm2d(192), 245 | nn.ReLU(), 246 | 247 | # 128*4*4 -> 64*7*7 248 | nn.ConvTranspose2d(192, 96, kernel_size=3, stride=2, padding=1, bias=False), 249 | nn.BatchNorm2d(96), 250 | nn.ReLU(), 251 | 252 | # 7 -> 14 253 | nn.ConvTranspose2d(96, 48, kernel_size=4, stride=2, padding=1, bias=False), 254 | nn.BatchNorm2d(48), 255 | nn.ReLU(), 256 | 257 | # 14 -> 28 258 | nn.ConvTranspose2d(48, 24, kernel_size=4, stride=2, padding=1, bias=False), 259 | nn.BatchNorm2d(24), 260 | nn.ReLU(), 261 | 262 | # 28 -> 56 263 | nn.ConvTranspose2d(24, 12, kernel_size=4, stride=2, padding=1, bias=False), 264 | nn.BatchNorm2d(12), 265 | nn.ReLU(), 266 | 267 | # 56 -> 112 268 | nn.ConvTranspose2d(12, 6, kernel_size=4, stride=2, padding=1, bias=False), 269 | nn.BatchNorm2d(6), 270 | nn.ReLU(), 271 | 272 | # 112 -> 224c 273 | nn.ConvTranspose2d(6, 6, kernel_size=4, stride=2, padding=1, bias=False), 274 | nn.BatchNorm2d(6), 275 | nn.ReLU(), 276 | ) 277 | 278 | 279 | # Modules 280 | self.features = nn.Sequential( 281 | BasicConv2d(6, 32, kernel_size=3, stride=2), 282 | BasicConv2d(32, 32, kernel_size=3, stride=1), 283 | BasicConv2d(32, 64, kernel_size=3, stride=1, padding=1), 284 | Mixed_3a(), 285 | Mixed_4a(), 286 | Mixed_5a(), 287 | Inception_A(), 288 | Inception_A(), 289 | Inception_A(), 290 | Inception_A(), 291 | Reduction_A(), # Mixed_6a 292 | Inception_B(), 293 | Inception_B(), 294 | Inception_B(), 295 | Inception_B(), 296 | Inception_B(), 297 | Inception_B(), 298 | Inception_B(), 299 | Reduction_B(), # Mixed_7a 300 | Inception_C(), 301 | Inception_C(), 302 | Inception_C() 303 | ) 304 | self.avg_pool = nn.AvgPool2d(5, count_include_pad=False) 305 | self.id = nn.Linear(1536, num_classes) 306 | self.bio = nn.Linear(1536, 4) 307 | 308 | # def logits(self, features): 309 | # x = self.avg_pool(features) 310 | # x = x.view(x.size(0), -1) 311 | # x = self.last_linear(x) 312 | # return x 313 | 314 | def forward(self, input): 315 | x = self.generation(input) 316 | x = self.features(x) 317 | x = self.avg_pool(x) 318 | x = x.view(x.size(0),-1) 319 | out1 = self.id(x) 320 | out2 = self.bio(x) 321 | return out1, out2 322 | -------------------------------------------------------------------------------- /model/net.py: -------------------------------------------------------------------------------- 1 | import scipy.io as sio 2 | from torch.utils.data import TensorDataset, DataLoader 3 | import numpy as np 4 | # from model import locNN 5 | import torch 6 | import torch.nn as nn 7 | from torch.autograd import Variable 8 | import torch.nn.functional as F 9 | import matplotlib.pyplot as plt 10 | import math 11 | 12 | 13 | # 3x3 Convolution 14 | def conv3x3(in_channels, out_channels, stride=1): 15 | return nn.Conv2d(in_channels, out_channels, kernel_size=3, 16 | stride=stride, padding=1, bias=False) 17 | 18 | 19 | 20 | 21 | # Bottleneck module 22 | class Bottleneck(nn.Module): 23 | expansion = 4 24 | def __init__(self, inplanes, planes, stride=1, downsample=None): 25 | super(Bottleneck, self).__init__() 26 | self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=1, bias=False) 27 | self.bn1 = nn.BatchNorm2d(planes) 28 | self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride, 29 | padding=1, bias=False) 30 | self.bn2 = nn.BatchNorm2d(planes) 31 | self.conv3 = nn.Conv2d(planes, planes * self.expansion, kernel_size=1, bias=False) 32 | self.bn3 = nn.BatchNorm2d(planes * self.expansion) 33 | self.relu = nn.ReLU(inplace=True) 34 | self.downsample = downsample 35 | self.stride = stride 36 | 37 | def forward(self, x): 38 | residual = x 39 | 40 | out = self.conv1(x) 41 | out = self.bn1(out) 42 | out = self.relu(out) 43 | 44 | out = self.conv2(out) 45 | out = self.bn2(out) 46 | out = self.relu(out) 47 | 48 | out = self.conv3(out) 49 | out = self.bn3(out) 50 | 51 | if self.downsample is not None: 52 | residual = self.downsample(x) 53 | 54 | out += residual 55 | out = self.relu(out) 56 | 57 | return out 58 | 59 | 60 | 61 | # Residual Block 62 | class ResidualBlock(nn.Module): 63 | def __init__(self, in_channels, out_channels, stride=1, downsample=None): 64 | super(ResidualBlock, self).__init__() 65 | self.conv1 = conv3x3(in_channels, out_channels, stride) 66 | self.bn1 = nn.BatchNorm2d(out_channels) 67 | self.relu = nn.ReLU(inplace=True) 68 | self.conv2 = conv3x3(out_channels, out_channels) 69 | self.bn2 = nn.BatchNorm2d(out_channels) 70 | self.downsample = downsample 71 | 72 | def forward(self, x): 73 | residual = x 74 | out = self.conv1(x) 75 | out = self.bn1(out) 76 | out = self.relu(out) 77 | out = self.conv2(out) 78 | out = self.bn2(out) 79 | if self.downsample: 80 | residual = self.downsample(x) 81 | out += residual 82 | out = self.relu(out) 83 | return out 84 | 85 | # ResNet Module 86 | class ResNet(nn.Module): 87 | def __init__(self, block, layers): 88 | super(ResNet, self).__init__() 89 | 90 | # self.bn1 = nn.BatchNorm2d(52) 91 | # self.generation = nn.Sequential( 92 | # # 30*1*1 -> 256*2*2 93 | # nn.ConvTranspose2d(30, 256, kernel_size=4, stride=2, padding=1, bias=False), 94 | # nn.BatchNorm2d(256), 95 | # nn.ReLU(), 96 | # # 256*2*2 -> 128*4*4 97 | # nn.ConvTranspose2d(256, 128, kernel_size=4, stride=2, padding=1, bias=False), 98 | # nn.BatchNorm2d(128), 99 | # nn.ReLU(), 100 | # 101 | # # 128*4*4 -> 64*8*8 102 | # nn.ConvTranspose2d(128, 64, kernel_size=4, stride=2, padding=1, bias=False), 103 | # nn.BatchNorm2d(64), 104 | # nn.ReLU(), 105 | # # 64*8*8 -> 32*16*16 106 | # nn.ConvTranspose2d(64, 32, kernel_size=4, stride=2, padding=1, bias=False), 107 | # nn.BatchNorm2d(32), 108 | # nn.ReLU(), 109 | # # 32*16*16 -> 16*32*32 110 | # nn.ConvTranspose2d(32, 16, kernel_size=4, stride=2, padding=1, bias=False), 111 | # nn.BatchNorm2d(16), 112 | # nn.ReLU(), 113 | # 114 | # # 16*32*32 ->64 8*64*64 115 | # nn.ConvTranspose2d(16, 3, kernel_size=4, stride=2, padding=1, bias=False), 116 | # nn.BatchNorm2d(3), 117 | # nn.ReLU(), 118 | # 119 | # ) 120 | 121 | self.generation = nn.Sequential( 122 | # 30*1*1 -> 256*2*2 123 | nn.ConvTranspose2d(30, 384, kernel_size=4, stride=2, padding=1, bias=False), 124 | nn.BatchNorm2d(384), 125 | nn.ReLU(), 126 | # 256*2*2 -> 128*4*4 127 | nn.ConvTranspose2d(384, 192, kernel_size=4, stride=2, padding=1, bias=False), 128 | nn.BatchNorm2d(192), 129 | nn.ReLU(), 130 | 131 | # 128*4*4 -> 64*7*7 132 | nn.ConvTranspose2d(192, 96, kernel_size=3, stride=2, padding=1, bias=False), 133 | nn.BatchNorm2d(96), 134 | nn.ReLU(), 135 | 136 | # 7 -> 14 137 | nn.ConvTranspose2d(96, 48, kernel_size=4, stride=2, padding=1, bias=False), 138 | nn.BatchNorm2d(48), 139 | nn.ReLU(), 140 | 141 | # 14 -> 28 142 | nn.ConvTranspose2d(48, 24, kernel_size=4, stride=2, padding=1, bias=False), 143 | nn.BatchNorm2d(24), 144 | nn.ReLU(), 145 | 146 | # 28 -> 56 147 | nn.ConvTranspose2d(24, 12, kernel_size=4, stride=2, padding=1, bias=False), 148 | nn.BatchNorm2d(12), 149 | nn.ReLU(), 150 | 151 | # 56 -> 112 152 | nn.ConvTranspose2d(12, 6, kernel_size=4, stride=2, padding=1, bias=False), 153 | nn.BatchNorm2d(6), 154 | nn.ReLU(), 155 | 156 | # 112 -> 224c 157 | nn.ConvTranspose2d(6, 6, kernel_size=4, stride=2, padding=1, bias=False), 158 | nn.BatchNorm2d(6), 159 | nn.ReLU(), 160 | ) 161 | 162 | self.conv1 = nn.Sequential( 163 | nn.Conv2d(6, 64, kernel_size=7, stride=2, padding=3, bias=False), 164 | nn.BatchNorm2d(64), 165 | nn.ReLU(inplace=True), 166 | nn.MaxPool2d(kernel_size=3, stride=2, padding=1), 167 | ) 168 | 169 | self.in_channels = 64 170 | self.layer1 = self.make_layer(block, 64, layers[0]) #8*64*64 -> 16*32*32 171 | self.layer2 = self.make_layer(block, 128, layers[1], 2) #16*32*32 -> 32*16*16 172 | self.layer3 = self.make_layer(block, 256, layers[2], 2) #32*16*16 -> 64*8*8 173 | self.layer4 = self.make_layer(block, 512, layers[3], 2) #64*8*8 -> 128*4*4 174 | 175 | 176 | self.humanid = nn.Sequential( 177 | nn.Conv2d(512, 512, kernel_size=3, stride=2, padding=1, bias=False), 178 | nn.BatchNorm2d(512), 179 | nn.ReLU(), 180 | 181 | nn.Conv2d(512, 256, kernel_size=4, stride=2, padding=1, bias=False), 182 | nn.BatchNorm2d(256), 183 | nn.ReLU(), 184 | nn.AvgPool2d(2), 185 | ) 186 | 187 | self.fcc = nn.Linear(256, 30) 188 | 189 | self.biometrics = nn.Sequential( 190 | nn.Conv2d(512, 512, kernel_size=3, stride=2, padding=1, bias=False), 191 | nn.BatchNorm2d(512), 192 | nn.ReLU(), 193 | 194 | nn.Conv2d(512, 256, kernel_size=4, stride=2, padding=1, bias=False), 195 | nn.BatchNorm2d(256), 196 | nn.ReLU(), 197 | nn.AvgPool2d(2), 198 | ) 199 | 200 | self.fcr = nn.Linear(256, 4) 201 | 202 | # self.humanid = nn.Sequential( 203 | # nn.Conv2d(256, 128, kernel_size=3, stride=2, padding=1, bias=False), 204 | # nn.BatchNorm2d(128), 205 | # nn.ReLU(), 206 | # 207 | # nn.Conv2d(128, 64, kernel_size=4, stride=2, padding=1, bias=False), 208 | # nn.BatchNorm2d(64), 209 | # nn.ReLU(), 210 | # nn.AvgPool2d(2), 211 | # ) 212 | # 213 | # self.fcc = nn.Linear(64, 30) 214 | # 215 | # self.biometrics = nn.Sequential( 216 | # nn.Conv2d(256, 128, kernel_size=3, stride=2, padding=1, bias=False), 217 | # nn.BatchNorm2d(128), 218 | # nn.ReLU(), 219 | # 220 | # nn.Conv2d(128, 64, kernel_size=4, stride=2, padding=1, bias=False), 221 | # nn.BatchNorm2d(64), 222 | # nn.ReLU(), 223 | # nn.AvgPool2d(2), 224 | # ) 225 | # 226 | # self.fcr = nn.Linear(64, 4) 227 | 228 | 229 | 230 | # self.avgpool = nn.AvgPool2d(7) #128*4*4 -> 128*1*1 231 | # # self.maxpool = nn.MaxPool2d(4) #128*4*4 -> 128*1*1 232 | # self.dp = nn.Dropout() 233 | # 234 | # #fully-connected layers 235 | # self.fcc1 = nn.Linear(512, 512) #128 -> 64 236 | # self.fcc2 = nn.Linear(512, 128) #64 -> 48 237 | # self.fcc3 = nn.Linear(128, 30) #48 -> 30 238 | # # 239 | # self.fcr1 = nn.Linear(512, 256) 240 | # self.fcr2 = nn.Linear(256, 128) 241 | # self.fcr3 = nn.Linear(128, 32) 242 | # self.fcr4 = nn.Linear(32, 4) 243 | 244 | def make_layer(self, block, out_channels, blocks, stride=1): 245 | downsample = None 246 | if (stride != 1) or (self.in_channels != out_channels): 247 | downsample = nn.Sequential( 248 | conv3x3(self.in_channels, out_channels, stride=stride), 249 | nn.BatchNorm2d(out_channels)) 250 | layers = [] 251 | layers.append(block(self.in_channels, out_channels, stride, downsample)) 252 | self.in_channels = out_channels 253 | for i in range(1, blocks): 254 | layers.append(block(self.in_channels, out_channels)) 255 | return nn.Sequential(*layers) 256 | 257 | def forward(self, x): 258 | out_generation = self.generation(x) 259 | out = self.conv1(out_generation) 260 | out = self.bn1(out) 261 | out = self.relu(out) 262 | out = self.maxpool(out) 263 | # 264 | # 265 | outres1 = self.layer1(out) 266 | outres2 = self.layer2(outres1) 267 | outres3 = self.layer3(outres2) 268 | outres4 = self.layer4(outres3) 269 | # return out 270 | # 271 | out_fc = self.humanid(outres4) 272 | out_fc = out_fc.squeeze() 273 | out1 = self.fcc(out_fc) 274 | # # # # 275 | # # 276 | out2 = self.biometrics(outres4) 277 | out2 = out2.squeeze() 278 | out2 = F.relu(self.fcr(out2)) 279 | 280 | return out1, out2, out_generation, out, outres1, outres2, outres3, outres4, out_fc 281 | # def forward(self, x):c 282 | # out_generation = self.generation(x) 283 | # out = self.conv1(out_generation) 284 | # out = self.bn1(out) 285 | # out = self.relu(out) 286 | # out = self.maxpool(out) 287 | # # 288 | # # 289 | # out = self.layer1(out) 290 | # out = self.layer2(out) 291 | # out = self.layer3(out) 292 | # out = self.layer4(out) 293 | # # return out 294 | # # 295 | # out1 = self.humanid(out) 296 | # out1 = out1.squeeze() 297 | # out1 = self.fcc(out1) 298 | # # # # # 299 | # # # 300 | # out2 = self.biometrics(out) 301 | # out2 = out2.squeeze() 302 | # out2 = F.relu(self.fcr(out2)) 303 | # 304 | # return out1, out2 -------------------------------------------------------------------------------- /model/net2.py: -------------------------------------------------------------------------------- 1 | import scipy.io as sio 2 | from torch.utils.data import TensorDataset, DataLoader 3 | import numpy as np 4 | # from model import locNN 5 | import torch 6 | import torch.nn as nn 7 | from torch.autograd import Variable 8 | import torch.nn.functional as F 9 | import matplotlib.pyplot as plt 10 | import math 11 | 12 | 13 | # 3x3 Convolution 14 | def conv3x3(in_channels, out_channels, stride=1): 15 | return nn.Conv2d(in_channels, out_channels, kernel_size=3, 16 | stride=stride, padding=1, bias=False) 17 | 18 | # Residual Block 19 | class ResidualBlock(nn.Module): 20 | def __init__(self, in_channels, out_channels, stride=1, downsample=None): 21 | super(ResidualBlock, self).__init__() 22 | self.conv1 = conv3x3(in_channels, out_channels, stride) 23 | self.bn1 = nn.BatchNorm2d(out_channels) 24 | self.relu = nn.LeakyReLU(inplace=True) 25 | self.conv2 = conv3x3(out_channels, out_channels) 26 | self.bn2 = nn.BatchNorm2d(out_channels) 27 | self.downsample = downsample 28 | 29 | def forward(self, x): 30 | residual = x 31 | out = self.conv1(x) 32 | out = self.bn1(out) 33 | out = self.relu(out) 34 | out = self.conv2(out) 35 | out = self.bn2(out) 36 | if self.downsample: 37 | residual = self.downsample(x) 38 | out += residual 39 | out = self.relu(out) 40 | return out 41 | 42 | # ResNet Module 43 | class ResNet(nn.Module): 44 | def __init__(self, block, layers): 45 | super(ResNet, self).__init__() 46 | 47 | # self.bn1 = nn.BatchNorm2d(52) 48 | # self.generation = nn.Sequential( 49 | # # 30*1*1 -> 256*2*2 50 | # nn.ConvTranspose2d(30, 256, kernel_size=4, stride=2, padding=1, bias=False), 51 | # nn.BatchNorm2d(256), 52 | # nn.ReLU(), 53 | # # 256*2*2 -> 128*4*4 54 | # nn.ConvTranspose2d(256, 128, kernel_size=4, stride=2, padding=1, bias=False), 55 | # nn.BatchNorm2d(128), 56 | # nn.ReLU(), 57 | # 58 | # # 128*4*4 -> 64*8*8 59 | # nn.ConvTranspose2d(128, 64, kernel_size=4, stride=2, padding=1, bias=False), 60 | # nn.BatchNorm2d(64), 61 | # nn.ReLU(), 62 | # # 64*8*8 -> 32*16*16 63 | # nn.ConvTranspose2d(64, 32, kernel_size=4, stride=2, padding=1, bias=False), 64 | # nn.BatchNorm2d(32), 65 | # nn.ReLU(), 66 | # # 32*16*16 -> 16*32*32 67 | # nn.ConvTranspose2d(32, 16, kernel_size=4, stride=2, padding=1, bias=False), 68 | # nn.BatchNorm2d(16), 69 | # nn.ReLU(), 70 | # 71 | # # 16*32*32 ->64 8*64*64 72 | # nn.ConvTranspose2d(16, 3, kernel_size=4, stride=2, padding=1, bias=False), 73 | # nn.BatchNorm2d(3), 74 | # nn.ReLU(), 75 | # 76 | # ) 77 | 78 | self.generation = nn.Sequential( 79 | # 30*1*1 -> 256*2*2 80 | nn.ConvTranspose2d(30, 384, kernel_size=4, stride=2, padding=1, bias=False), 81 | nn.BatchNorm2d(384), 82 | nn.LeakyReLU(), 83 | # 256*2*2 -> 128*4*4 84 | nn.ConvTranspose2d(384, 192, kernel_size=4, stride=2, padding=1, bias=False), 85 | nn.BatchNorm2d(192), 86 | nn.LeakyReLU(), 87 | 88 | # 128*4*4 -> 64*7*7 89 | nn.ConvTranspose2d(192, 96, kernel_size=3, stride=2, padding=1, bias=False), 90 | nn.BatchNorm2d(96), 91 | nn.LeakyReLU(), 92 | 93 | # 7 -> 14 94 | nn.ConvTranspose2d(96, 48, kernel_size=4, stride=2, padding=1, bias=False), 95 | nn.BatchNorm2d(48), 96 | nn.LeakyReLU(), 97 | 98 | # 14 -> 28 99 | nn.ConvTranspose2d(48, 24, kernel_size=4, stride=2, padding=1, bias=False), 100 | nn.BatchNorm2d(24), 101 | nn.LeakyReLU(), 102 | 103 | # 28 -> 56 104 | nn.ConvTranspose2d(24, 12, kernel_size=4, stride=2, padding=1, bias=False), 105 | nn.BatchNorm2d(12), 106 | nn.LeakyReLU(), 107 | 108 | # 56 -> 112 109 | nn.ConvTranspose2d(12, 6, kernel_size=4, stride=2, padding=1, bias=False), 110 | nn.BatchNorm2d(6), 111 | nn.LeakyReLU(), 112 | 113 | # 112 -> 224c 114 | nn.ConvTranspose2d(6, 3, kernel_size=4, stride=2, padding=1, bias=False), 115 | nn.BatchNorm2d(3), 116 | nn.LeakyReLU(), 117 | ) 118 | 119 | self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=False) 120 | self.bn1 = nn.BatchNorm2d(64) 121 | self.relu = nn.LeakyReLU(inplace=True) 122 | self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1) 123 | 124 | self.in_channels = 64 125 | self.layer1 = self.make_layer(block, 64, layers[0]) #8*64*64 -> 16*32*32 126 | self.layer2 = self.make_layer(block, 128, layers[1], 2) #16*32*32 -> 32*16*16 127 | self.layer3 = self.make_layer(block, 256, layers[2], 2) #32*16*16 -> 64*8*8 128 | self.layer4 = self.make_layer(block, 512, layers[3], 2) #64*8*8 -> 128*4*4 129 | 130 | 131 | self.humanid = nn.Sequential( 132 | nn.Conv2d(512, 512, kernel_size=3, stride=2, padding=1, bias=False), 133 | nn.BatchNorm2d(512), 134 | nn.LeakyReLU(), 135 | 136 | nn.Conv2d(512, 256, kernel_size=4, stride=2, padding=1, bias=False), 137 | nn.BatchNorm2d(256), 138 | nn.LeakyReLU(), 139 | nn.AvgPool2d(2), 140 | ) 141 | 142 | self.fcc = nn.Linear(256, 30) 143 | 144 | self.biometrics = nn.Sequential( 145 | nn.Conv2d(512, 512, kernel_size=3, stride=2, padding=1, bias=False), 146 | nn.BatchNorm2d(512), 147 | nn.LeakyReLU(), 148 | 149 | nn.Conv2d(512, 256, kernel_size=4, stride=2, padding=1, bias=False), 150 | nn.BatchNorm2d(256), 151 | nn.LeakyReLU(), 152 | nn.AvgPool2d(2), 153 | ) 154 | 155 | self.fcr = nn.Linear(256, 4) 156 | 157 | # self.humanid = nn.Sequential( 158 | # nn.Conv2d(256, 128, kernel_size=3, stride=2, padding=1, bias=False), 159 | # nn.BatchNorm2d(128), 160 | # nn.ReLU(), 161 | # 162 | # nn.Conv2d(128, 64, kernel_size=4, stride=2, padding=1, bias=False), 163 | # nn.BatchNorm2d(64), 164 | # nn.ReLU(), 165 | # nn.AvgPool2d(2), 166 | # ) 167 | # 168 | # self.fcc = nn.Linear(64, 30) 169 | # 170 | # self.biometrics = nn.Sequential( 171 | # nn.Conv2d(256, 128, kernel_size=3, stride=2, padding=1, bias=False), 172 | # nn.BatchNorm2d(128), 173 | # nn.ReLU(), 174 | # 175 | # nn.Conv2d(128, 64, kernel_size=4, stride=2, padding=1, bias=False), 176 | # nn.BatchNorm2d(64), 177 | # nn.ReLU(), 178 | # nn.AvgPool2d(2), 179 | # ) 180 | # 181 | # self.fcr = nn.Linear(64, 4) 182 | 183 | 184 | 185 | # self.avgpool = nn.AvgPool2d(7) #128*4*4 -> 128*1*1 186 | # # self.maxpool = nn.MaxPool2d(4) #128*4*4 -> 128*1*1 187 | # self.dp = nn.Dropout() 188 | # 189 | # #fully-connected layers 190 | # self.fcc1 = nn.Linear(512, 512) #128 -> 64 191 | # self.fcc2 = nn.Linear(512, 128) #64 -> 48 192 | # self.fcc3 = nn.Linear(128, 30) #48 -> 30 193 | # # 194 | # self.fcr1 = nn.Linear(512, 256) 195 | # self.fcr2 = nn.Linear(256, 128) 196 | # self.fcr3 = nn.Linear(128, 32) 197 | # self.fcr4 = nn.Linear(32, 4) 198 | 199 | def make_layer(self, block, out_channels, blocks, stride=1): 200 | downsample = None 201 | if (stride != 1) or (self.in_channels != out_channels): 202 | downsample = nn.Sequential( 203 | conv3x3(self.in_channels, out_channels, stride=stride), 204 | nn.BatchNorm2d(out_channels)) 205 | layers = [] 206 | layers.append(block(self.in_channels, out_channels, stride, downsample)) 207 | self.in_channels = out_channels 208 | for i in range(1, blocks): 209 | layers.append(block(self.in_channels, out_channels)) 210 | return nn.Sequential(*layers) 211 | 212 | # def forward(self, x): 213 | # out_generation = self.generation(x) 214 | # out = self.conv1(out_generation) 215 | # out = self.bn1(out) 216 | # out = self.relu(out) 217 | # out = self.maxpool(out) 218 | # # 219 | # # 220 | # outres1 = self.layer1(out) 221 | # outres2 = self.layer2(outres1) 222 | # outres3 = self.layer3(outres2) 223 | # outres4 = self.layer4(outres3) 224 | # # return out 225 | # # 226 | # out_fc = self.humanid(outres4) 227 | # out_fc = out_fc.squeeze() 228 | # out1 = self.fcc(out_fc) 229 | # # # # # 230 | # # # 231 | # out2 = self.biometrics(outres4) 232 | # out2 = out2.squeeze() 233 | # out2 = F.relu(self.fcr(out2)) 234 | # 235 | # return out1, out2, out_generation, out, outres1, outres2, outres3, outres4, out_fc 236 | def forward(self, x): 237 | out_generation = self.generation(x) 238 | out = self.conv1(out_generation) 239 | out = self.bn1(out) 240 | out = self.relu(out) 241 | out = self.maxpool(out) 242 | # 243 | # 244 | out = self.layer1(out) 245 | out = self.layer2(out) 246 | out = self.layer3(out) 247 | out = self.layer4(out) 248 | # return out 249 | # 250 | out1 = self.humanid(out) 251 | out1 = out1.squeeze() 252 | out1 = self.fcc(out1) 253 | # # # # 254 | # # 255 | out2 = self.biometrics(out) 256 | out2 = out2.squeeze() 257 | out2 = F.relu(self.fcr(out2)) 258 | 259 | return out1, out2 -------------------------------------------------------------------------------- /model/net_plus1transconv.py: -------------------------------------------------------------------------------- 1 | import scipy.io as sio 2 | from torch.utils.data import TensorDataset, DataLoader 3 | import numpy as np 4 | # from model import locNN 5 | import torch 6 | import torch.nn as nn 7 | from torch.autograd import Variable 8 | import torch.nn.functional as F 9 | import matplotlib.pyplot as plt 10 | import math 11 | 12 | 13 | # 3x3 Convolution 14 | def conv3x3(in_channels, out_channels, stride=1): 15 | return nn.Conv2d(in_channels, out_channels, kernel_size=3, 16 | stride=stride, padding=1, bias=False) 17 | 18 | # Residual Block 19 | class ResidualBlock(nn.Module): 20 | def __init__(self, in_channels, out_channels, stride=1, downsample=None): 21 | super(ResidualBlock, self).__init__() 22 | self.conv1 = conv3x3(in_channels, out_channels, stride) 23 | self.bn1 = nn.BatchNorm2d(out_channels) 24 | self.relu = nn.ReLU(inplace=True) 25 | self.conv2 = conv3x3(out_channels, out_channels) 26 | self.bn2 = nn.BatchNorm2d(out_channels) 27 | self.downsample = downsample 28 | 29 | def forward(self, x): 30 | residual = x 31 | out = self.conv1(x) 32 | out = self.bn1(out) 33 | out = self.relu(out) 34 | out = self.conv2(out) 35 | out = self.bn2(out) 36 | if self.downsample: 37 | residual = self.downsample(x) 38 | out += residual 39 | out = self.relu(out) 40 | return out 41 | 42 | # ResNet Module 43 | class ResNet(nn.Module): 44 | def __init__(self, block, layers): 45 | super(ResNet, self).__init__() 46 | 47 | 48 | self.generation = nn.Sequential( 49 | # 30*1*1 -> 256*2*2 50 | nn.ConvTranspose2d(30, 384, kernel_size=4, stride=2, padding=1, bias=False), 51 | nn.BatchNorm2d(384), 52 | nn.ReLU(), 53 | # 256*2*2 -> 128*4*4 54 | nn.ConvTranspose2d(384, 192, kernel_size=4, stride=2, padding=1, bias=False), 55 | nn.BatchNorm2d(192), 56 | nn.ReLU(), 57 | 58 | # 128*4*4 -> 64*7*7 59 | nn.ConvTranspose2d(192, 96, kernel_size=3, stride=2, padding=1, bias=False), 60 | nn.BatchNorm2d(96), 61 | nn.ReLU(), 62 | 63 | # 7 -> 14 64 | nn.ConvTranspose2d(96, 48, kernel_size=4, stride=2, padding=1, bias=False), 65 | nn.BatchNorm2d(48), 66 | nn.ReLU(), 67 | 68 | # 14 -> 28 69 | nn.ConvTranspose2d(48, 24, kernel_size=4, stride=2, padding=1, bias=False), 70 | nn.BatchNorm2d(24), 71 | nn.ReLU(), 72 | 73 | # 28 -> 56 74 | nn.ConvTranspose2d(24, 12, kernel_size=4, stride=2, padding=1, bias=False), 75 | nn.BatchNorm2d(12), 76 | nn.ReLU(), 77 | 78 | # 56 -> 112 79 | nn.ConvTranspose2d(12, 6, kernel_size=4, stride=2, padding=1, bias=False), 80 | nn.BatchNorm2d(6), 81 | nn.ReLU(), 82 | 83 | # 112 -> 224c 84 | nn.ConvTranspose2d(6, 6, kernel_size=4, stride=2, padding=1, bias=False), 85 | nn.BatchNorm2d(6), 86 | nn.ReLU(), 87 | 88 | nn.ConvTranspose2d(6, 6, kernel_size=3, stride=1, padding=1, bias=False), 89 | nn.BatchNorm2d(6), 90 | nn.ReLU(), 91 | 92 | ) 93 | 94 | self.conv1 = nn.Conv2d(6, 64, kernel_size=7, stride=2, padding=3, bias=False) 95 | self.bn1 = nn.BatchNorm2d(64) 96 | self.relu = nn.ReLU(inplace=True) 97 | self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1) 98 | 99 | self.in_channels = 64 100 | self.layer1 = self.make_layer(block, 64, layers[0]) #8*64*64 -> 16*32*32 101 | self.layer2 = self.make_layer(block, 128, layers[1], 2) #16*32*32 -> 32*16*16 102 | self.layer3 = self.make_layer(block, 256, layers[2], 2) #32*16*16 -> 64*8*8 103 | self.layer4 = self.make_layer(block, 512, layers[3], 2) #64*8*8 -> 128*4*4 104 | 105 | 106 | self.humanid = nn.Sequential( 107 | nn.Conv2d(512, 256, kernel_size=3, stride=2, padding=1, bias=False), 108 | nn.BatchNorm2d(256), 109 | nn.ReLU(), 110 | 111 | nn.Conv2d(256, 128, kernel_size=4, stride=2, padding=1, bias=False), 112 | nn.BatchNorm2d(128), 113 | nn.ReLU(), 114 | nn.AvgPool2d(2), 115 | ) 116 | 117 | self.fcc = nn.Linear(128, 30) 118 | 119 | self.biometrics = nn.Sequential( 120 | nn.Conv2d(512, 256, kernel_size=3, stride=2, padding=1, bias=False), 121 | nn.BatchNorm2d(256), 122 | nn.ReLU(), 123 | 124 | nn.Conv2d(256, 128, kernel_size=4, stride=2, padding=1, bias=False), 125 | nn.BatchNorm2d(128), 126 | nn.ReLU(), 127 | nn.AvgPool2d(2), 128 | ) 129 | 130 | self.fcr = nn.Linear(128, 4) 131 | 132 | 133 | def make_layer(self, block, out_channels, blocks, stride=1): 134 | downsample = None 135 | if (stride != 1) or (self.in_channels != out_channels): 136 | downsample = nn.Sequential( 137 | conv3x3(self.in_channels, out_channels, stride=stride), 138 | nn.BatchNorm2d(out_channels)) 139 | layers = [] 140 | layers.append(block(self.in_channels, out_channels, stride, downsample)) 141 | self.in_channels = out_channels 142 | for i in range(1, blocks): 143 | layers.append(block(self.in_channels, out_channels)) 144 | return nn.Sequential(*layers) 145 | 146 | # def forward(self, x): 147 | # out_generation = self.generation(x) 148 | # out = self.conv1(out_generation) 149 | # out = self.bn1(out) 150 | # out = self.relu(out) 151 | # out = self.maxpool(out) 152 | # # 153 | # # 154 | # outres1 = self.layer1(out) 155 | # outres2 = self.layer2(outres1) 156 | # outres3 = self.layer3(outres2) 157 | # outres4 = self.layer4(outres3) 158 | # # return out 159 | # # 160 | # out_fc = self.humanid(outres4) 161 | # out_fc = out_fc.squeeze() 162 | # out1 = self.fcc(out_fc) 163 | # # # # # 164 | # # # 165 | # out2 = self.biometrics(outres4) 166 | # out2 = out2.squeeze() 167 | # out2 = F.relu(self.fcr(out2)) 168 | # 169 | # return out1, out2, out_generation, out, outres1, outres2, outres3, outres4, out_fc 170 | def forward(self, x): 171 | out_generation = self.generation(x) 172 | out = self.conv1(out_generation) 173 | out = self.bn1(out) 174 | out = self.relu(out) 175 | out = self.maxpool(out) 176 | # 177 | # 178 | out = self.layer1(out) 179 | out = self.layer2(out) 180 | out = self.layer3(out) 181 | out = self.layer4(out) 182 | # return out 183 | # 184 | out1 = self.humanid(out) 185 | out1 = out1.squeeze() 186 | out1 = self.fcc(out1) 187 | # # # # 188 | # # 189 | out2 = self.biometrics(out) 190 | out2 = out2.squeeze() 191 | out2 = F.relu(self.fcr(out2)) 192 | 193 | return out1, out2 -------------------------------------------------------------------------------- /model/res_net.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import math 3 | import torch.nn.functional as F 4 | import torch.utils.model_zoo as model_zoo 5 | 6 | # __all__ = ['ResNet', 'resnet18', 'resnet34', 'resnet50', 'resnet101', 7 | # 'resnet152'] 8 | # 9 | # model_urls = { 10 | # 'resnet18': 'https://download.pytorch.org/models/resnet18-5c106cde.pth', 11 | # 'resnet34': 'https://download.pytorch.org/models/resnet34-333f7ec4.pth', 12 | # 'resnet50': 'https://download.pytorch.org/models/resnet50-19c8e357.pth', 13 | # 'resnet101': 'https://download.pytorch.org/models/resnet101-5d3b4d8f.pth', 14 | # 'resnet152': 'https://download.pytorch.org/models/resnet152-b121ed2d.pth', 15 | # } 16 | 17 | 18 | def conv3x3(in_planes, out_planes, stride=1): 19 | """3x3 convolution with padding""" 20 | return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride, 21 | padding=1, bias=False) 22 | 23 | 24 | class BasicBlock(nn.Module): 25 | expansion = 1 26 | 27 | def __init__(self, inplanes, planes, stride=1, downsample=None): 28 | super(BasicBlock, self).__init__() 29 | self.conv1 = conv3x3(inplanes, planes, stride) 30 | self.bn1 = nn.BatchNorm2d(planes) 31 | self.relu = nn.ReLU(inplace=True) 32 | self.conv2 = conv3x3(planes, planes) 33 | self.bn2 = nn.BatchNorm2d(planes) 34 | self.downsample = downsample 35 | self.stride = stride 36 | 37 | def forward(self, x): 38 | residual = x 39 | 40 | out = self.conv1(x) 41 | out = self.bn1(out) 42 | out = self.relu(out) 43 | 44 | out = self.conv2(out) 45 | out = self.bn2(out) 46 | 47 | if self.downsample is not None: 48 | residual = self.downsample(x) 49 | 50 | out += residual 51 | out = self.relu(out) 52 | 53 | return out 54 | 55 | 56 | class Bottleneck(nn.Module): 57 | expansion = 4 58 | 59 | def __init__(self, inplanes, planes, stride=1, downsample=None): 60 | super(Bottleneck, self).__init__() 61 | self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=1, bias=False) 62 | self.bn1 = nn.BatchNorm2d(planes) 63 | self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride, 64 | padding=1, bias=False) 65 | self.bn2 = nn.BatchNorm2d(planes) 66 | self.conv3 = nn.Conv2d(planes, planes * 4, kernel_size=1, bias=False) 67 | self.bn3 = nn.BatchNorm2d(planes * 4) 68 | self.relu = nn.ReLU(inplace=True) 69 | self.downsample = downsample 70 | self.stride = stride 71 | 72 | def forward(self, x): 73 | residual = x 74 | 75 | out = self.conv1(x) 76 | out = self.bn1(out) 77 | out = self.relu(out) 78 | 79 | out = self.conv2(out) 80 | out = self.bn2(out) 81 | out = self.relu(out) 82 | 83 | out = self.conv3(out) 84 | out = self.bn3(out) 85 | 86 | if self.downsample is not None: 87 | residual = self.downsample(x) 88 | 89 | out += residual 90 | out = self.relu(out) 91 | 92 | return out 93 | 94 | 95 | class ResNet(nn.Module): 96 | def __init__(self, block, layers, num_classes=1000): 97 | self.inplanes = 8 98 | super(ResNet, self).__init__() 99 | # self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, 100 | # bias=False) 101 | # self.bn1 = nn.BatchNorm2d(64) 102 | # self.relu = nn.ReLU(inplace=True) 103 | # self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1) 104 | # 105 | self.generation = nn.Sequential( 106 | # 30*1*1 -> 256*2*2 107 | nn.ConvTranspose2d(30, 384, kernel_size=4, stride=2, padding=1, bias=False), 108 | nn.BatchNorm2d(384), 109 | nn.ReLU(), 110 | # 256*2*2 -> 128*4*4 111 | nn.ConvTranspose2d(384, 192, kernel_size=4, stride=2, padding=1, bias=False), 112 | nn.BatchNorm2d(192), 113 | nn.ReLU(), 114 | 115 | # 128*4*4 -> 64*7*7 116 | nn.ConvTranspose2d(192, 96, kernel_size=3, stride=2, padding=1, bias=False), 117 | nn.BatchNorm2d(96), 118 | nn.ReLU(), 119 | 120 | # 7 -> 14 121 | nn.ConvTranspose2d(96, 48, kernel_size=4, stride=2, padding=1, bias=False), 122 | nn.BatchNorm2d(48), 123 | nn.ReLU(), 124 | 125 | # 14 -> 28 126 | nn.ConvTranspose2d(48, 24, kernel_size=4, stride=2, padding=1, bias=False), 127 | nn.BatchNorm2d(24), 128 | nn.ReLU(), 129 | 130 | # 28 -> 56 131 | nn.ConvTranspose2d(24, 12, kernel_size=4, stride=2, padding=1, bias=False), 132 | nn.BatchNorm2d(12), 133 | nn.ReLU(), 134 | 135 | # 56 -> 112 136 | nn.ConvTranspose2d(12, 6, kernel_size=4, stride=2, padding=1, bias=False), 137 | nn.BatchNorm2d(6), 138 | nn.ReLU(), 139 | 140 | # 112 -> 224 141 | nn.ConvTranspose2d(6, 3, kernel_size=4, stride=2, padding=1, bias=False), 142 | nn.BatchNorm2d(3), 143 | nn.ReLU(), 144 | ) 145 | 146 | self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, 147 | bias=False) 148 | self.bn1 = nn.BatchNorm2d(64) 149 | self.relu = nn.ReLU(inplace=True) 150 | self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1) 151 | 152 | self.inplanes = 64 153 | 154 | self.layer1 = self._make_layer(block, 64, layers[0]) 155 | self.layer2 = self._make_layer(block, 128, layers[1], stride=2) 156 | self.layer3 = self._make_layer(block, 256, layers[2], stride=2) 157 | self.layer4 = self._make_layer(block, 512, layers[3], stride=2) 158 | 159 | self.humanid = nn.Sequential( 160 | nn.Conv2d(512*4, 512, kernel_size=3, stride=2, padding=1, bias=False), 161 | nn.BatchNorm2d(512), 162 | nn.ReLU(), 163 | 164 | nn.Conv2d(512, 256, kernel_size=4, stride=2, padding=1, bias=False), 165 | nn.BatchNorm2d(256), 166 | nn.ReLU(), 167 | nn.AvgPool2d(2), 168 | ) 169 | 170 | self.fcc = nn.Linear(256, 30) 171 | 172 | self.biometrics = nn.Sequential( 173 | nn.Conv2d(512*4, 512, kernel_size=3, stride=2, padding=1, bias=False), 174 | nn.BatchNorm2d(512), 175 | nn.ReLU(), 176 | 177 | nn.Conv2d(512, 256, kernel_size=4, stride=2, padding=1, bias=False), 178 | nn.BatchNorm2d(256), 179 | nn.ReLU(), 180 | nn.AvgPool2d(2), 181 | ) 182 | 183 | self.fcr = nn.Linear(256, 4) 184 | 185 | 186 | # 187 | for m in self.modules(): 188 | if isinstance(m, nn.Conv2d): 189 | n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels 190 | m.weight.data.normal_(0, math.sqrt(2. / n)) 191 | elif isinstance(m, nn.BatchNorm2d): 192 | m.weight.data.fill_(1) 193 | m.bias.data.zero_() 194 | 195 | def _make_layer(self, block, planes, blocks, stride=1): 196 | downsample = None 197 | if stride != 1 or self.inplanes != planes * block.expansion: 198 | downsample = nn.Sequential( 199 | nn.Conv2d(self.inplanes, planes * block.expansion, 200 | kernel_size=1, stride=stride, bias=False), 201 | nn.BatchNorm2d(planes * block.expansion), 202 | ) 203 | 204 | layers = [] 205 | layers.append(block(self.inplanes, planes, stride, downsample)) 206 | self.inplanes = planes * block.expansion 207 | for i in range(1, blocks): 208 | layers.append(block(self.inplanes, planes)) 209 | 210 | return nn.Sequential(*layers) 211 | 212 | def forward(self, x): 213 | out = self.generation(x) 214 | out = self.conv1(out) 215 | out = self.bn1(out) 216 | out = self.relu(out) 217 | out = self.maxpool(out) 218 | 219 | 220 | out = self.layer1(out) 221 | out = self.layer2(out) 222 | out = self.layer3(out) 223 | out = self.layer4(out) 224 | 225 | out1 = self.humanid(out) 226 | out1 = out1.squeeze() 227 | out1 = self.fcc(out1) 228 | # # # # 229 | # # 230 | out2 = self.biometrics(out) 231 | out2 = out2.squeeze() 232 | out2 = F.relu(self.fcr(out2)) 233 | 234 | return out1, out2 235 | 236 | 237 | # def resnet18(pretrained=False, **kwargs): 238 | # """Constructs a ResNet-18 model. 239 | # Args: 240 | # pretrained (bool): If True, returns a model pre-trained on ImageNet 241 | # """ 242 | # model = ResNet(BasicBlock, [2, 2, 2, 2], **kwargs) 243 | # if pretrained: 244 | # model.load_state_dict(model_zoo.load_url(model_urls['resnet18'])) 245 | # return model 246 | # 247 | # 248 | # def resnet34(pretrained=False, **kwargs): 249 | # """Constructs a ResNet-34 model. 250 | # Args: 251 | # pretrained (bool): If True, returns a model pre-trained on ImageNet 252 | # """ 253 | # model = ResNet(BasicBlock, [3, 4, 6, 3], **kwargs) 254 | # if pretrained: 255 | # model.load_state_dict(model_zoo.load_url(model_urls['resnet34'])) 256 | # return model 257 | # 258 | # 259 | # def resnet50(pretrained=False, **kwargs): 260 | # """Constructs a ResNet-50 model. 261 | # Args: 262 | # pretrained (bool): If True, returns a model pre-trained on ImageNet 263 | # """ 264 | # model = ResNet(Bottleneck, [3, 4, 6, 3], **kwargs) 265 | # if pretrained: 266 | # model.load_state_dict(model_zoo.load_url(model_urls['resnet50'])) 267 | # return model 268 | # 269 | # 270 | # def resnet101(pretrained=False, **kwargs): 271 | # """Constructs a ResNet-101 model. 272 | # Args: 273 | # pretrained (bool): If True, returns a model pre-trained on ImageNet 274 | # """ 275 | # model = ResNet(Bottleneck, [3, 4, 23, 3], **kwargs) 276 | # if pretrained: 277 | # model.load_state_dict(model_zoo.load_url(model_urls['resnet101'])) 278 | # return model 279 | # 280 | # 281 | # def resnet152(pretrained=False, **kwargs): 282 | # """Constructs a ResNet-152 model. 283 | # Args: 284 | # pretrained (bool): If True, returns a model pre-trained on ImageNet 285 | # """ 286 | # model = ResNet(Bottleneck, [3, 8, 36, 3], **kwargs) 287 | # if pretrained: 288 | # model.load_state_dict(model_zoo.load_url(model_urls['resnet152'])) 289 | # return model 290 | 291 | -------------------------------------------------------------------------------- /model/res_net_use_this.py: -------------------------------------------------------------------------------- 1 | import scipy.io as sio 2 | from torch.utils.data import TensorDataset, DataLoader 3 | import numpy as np 4 | # from model import locNN 5 | import torch 6 | import torch.nn as nn 7 | from torch.autograd import Variable 8 | import torch.nn.functional as F 9 | import matplotlib.pyplot as plt 10 | import math 11 | 12 | 13 | # 3x3 Convolution 14 | def conv3x3(in_channels, out_channels, stride=1): 15 | return nn.Conv2d(in_channels, out_channels, kernel_size=3, 16 | stride=stride, padding=1, bias=False) 17 | 18 | # Residual Block 19 | class ResidualBlock(nn.Module): 20 | expansion = 1 21 | def __init__(self, in_channels, out_channels, stride=1, downsample=None): 22 | super(ResidualBlock, self).__init__() 23 | self.conv1 = conv3x3(in_channels, out_channels, stride) 24 | self.bn1 = nn.BatchNorm2d(out_channels) 25 | self.relu = nn.ReLU(inplace=True) 26 | self.conv2 = conv3x3(out_channels, out_channels) 27 | self.bn2 = nn.BatchNorm2d(out_channels) 28 | self.downsample = downsample 29 | 30 | def forward(self, x): 31 | residual = x 32 | out = self.conv1(x) 33 | out = self.bn1(out) 34 | out = self.relu(out) 35 | out = self.conv2(out) 36 | out = self.bn2(out) 37 | if self.downsample: 38 | residual = self.downsample(x) 39 | out += residual 40 | out = self.relu(out) 41 | return out 42 | 43 | class Bottleneck(nn.Module): 44 | expansion = 4 45 | 46 | def __init__(self, in_channels, out_channels, stride=1, downsample=None): 47 | super(Bottleneck, self).__init__() 48 | self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=1, bias=False) 49 | self.bn1 = nn.BatchNorm2d(out_channels) 50 | self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, stride=stride, 51 | padding=1, bias=False) 52 | self.bn2 = nn.BatchNorm2d(out_channels) 53 | self.conv3 = nn.Conv2d(out_channels, out_channels * 4, kernel_size=1, bias=False) 54 | self.bn3 = nn.BatchNorm2d(out_channels * 4) 55 | self.relu = nn.ReLU(inplace=True) 56 | self.downsample = downsample 57 | self.stride = stride 58 | 59 | def forward(self, x): 60 | residual = x 61 | 62 | out = self.conv1(x) 63 | out = self.bn1(out) 64 | out = self.relu(out) 65 | 66 | out = self.conv2(out) 67 | out = self.bn2(out) 68 | out = self.relu(out) 69 | 70 | out = self.conv3(out) 71 | out = self.bn3(out) 72 | 73 | if self.downsample is not None: 74 | residual = self.downsample(x) 75 | 76 | out += residual 77 | out = self.relu(out) 78 | 79 | return out 80 | 81 | 82 | 83 | 84 | 85 | # ResNet Module 86 | class ResNet(nn.Module): 87 | def __init__(self, block, layers, num_classes): 88 | super(ResNet, self).__init__() 89 | 90 | self.generation = nn.Sequential( 91 | # 30*1*1 -> 256*2*2 92 | nn.ConvTranspose2d(30, 384, kernel_size=4, stride=2, padding=1, bias=False), 93 | nn.BatchNorm2d(384), 94 | nn.ReLU(), 95 | # 256*2*2 -> 128*4*4 96 | nn.ConvTranspose2d(384, 192, kernel_size=4, stride=2, padding=1, bias=False), 97 | nn.BatchNorm2d(192), 98 | nn.ReLU(), 99 | 100 | # 128*4*4 -> 64*7*7 101 | nn.ConvTranspose2d(192, 96, kernel_size=3, stride=2, padding=1, bias=False), 102 | nn.BatchNorm2d(96), 103 | nn.ReLU(), 104 | 105 | # 7 -> 14 106 | nn.ConvTranspose2d(96, 48, kernel_size=4, stride=2, padding=1, bias=False), 107 | nn.BatchNorm2d(48), 108 | nn.ReLU(), 109 | 110 | # 14 -> 28 111 | nn.ConvTranspose2d(48, 24, kernel_size=4, stride=2, padding=1, bias=False), 112 | nn.BatchNorm2d(24), 113 | nn.ReLU(), 114 | 115 | # 28 -> 56 116 | nn.ConvTranspose2d(24, 12, kernel_size=4, stride=2, padding=1, bias=False), 117 | nn.BatchNorm2d(12), 118 | nn.ReLU(), 119 | 120 | # 56 -> 112 121 | nn.ConvTranspose2d(12, 6, kernel_size=4, stride=2, padding=1, bias=False), 122 | nn.BatchNorm2d(6), 123 | nn.ReLU(), 124 | 125 | # 112 -> 224c 126 | nn.ConvTranspose2d(6, 6, kernel_size=4, stride=2, padding=1, bias=False), 127 | nn.BatchNorm2d(6), 128 | nn.ReLU(), 129 | ) 130 | 131 | self.conv1 = nn.Sequential( 132 | nn.Conv2d(6, 64, kernel_size=7, stride=2, padding=3, bias=False), 133 | nn.BatchNorm2d(64), 134 | nn.ReLU(inplace=True), 135 | nn.MaxPool2d(kernel_size=3, stride=2, padding=1), 136 | ) 137 | 138 | self.in_channels = 64 139 | self.layer1 = self.make_layer(block, 64, layers[0]) #8*64*64 -> 16*32*32 140 | self.layer2 = self.make_layer(block, 128, layers[1], 2) #16*32*32 -> 32*16*16 141 | self.layer3 = self.make_layer(block, 256, layers[2], 2) #32*16*16 -> 64*8*8 142 | self.layer4 = self.make_layer(block, 512, layers[3], 2) #64*8*8 -> 128*4*4 143 | 144 | # 145 | self.humanid = nn.Sequential( 146 | nn.Conv2d(512*block.expansion, 512*block.expansion, kernel_size=3, stride=2, padding=1, bias=False), 147 | nn.BatchNorm2d(512*block.expansion), 148 | nn.ReLU(), 149 | 150 | nn.Conv2d(512*block.expansion, 256*block.expansion, kernel_size=4, stride=2, padding=1, bias=False), 151 | nn.BatchNorm2d(256*block.expansion), 152 | nn.ReLU(), 153 | nn.AvgPool2d(2), 154 | ) 155 | 156 | self.fcc = nn.Linear(256*block.expansion, num_classes) 157 | 158 | self.biometrics = nn.Sequential( 159 | nn.Conv2d(512*block.expansion, 512*block.expansion, kernel_size=3, stride=2, padding=1, bias=False), 160 | nn.BatchNorm2d(512*block.expansion), 161 | nn.ReLU(), 162 | 163 | nn.Conv2d(512*block.expansion, 256*block.expansion, kernel_size=4, stride=2, padding=1, bias=False), 164 | nn.BatchNorm2d(256*block.expansion), 165 | nn.ReLU(), 166 | nn.AvgPool2d(2), 167 | ) 168 | 169 | self.fcr = nn.Linear(256*block.expansion, 4) 170 | 171 | 172 | 173 | def make_layer(self, block, out_channels, bclocks, stride=1): 174 | downsample = None 175 | if (stride != 1) or (self.in_channels != out_channels*block.expansion): 176 | downsample = nn.Sequential( 177 | conv3x3(self.in_channels, out_channels*block.expansion, stride=stride), 178 | nn.BatchNorm2d(out_channels*block.expansion)) 179 | layers = [] 180 | layers.append(block(self.in_channels, out_channels, stride, downsample)) 181 | self.in_channels = out_channels*block.expansion 182 | for i in range(1, blocks): 183 | layers.append(block(self.in_channels, out_channels)) 184 | return nn.Sequential(*layers) 185 | 186 | 187 | def forward(self, x): 188 | out = self.generation(x) 189 | out = self.conv1(out) 190 | # 191 | out = self.layer1(out) 192 | out = self.layer2(out) 193 | out = self.layer3(out) 194 | out = self.layer4(out) 195 | # 196 | # # 197 | # out = self.avgpool7(out) 198 | out1 = self.humanid(out) 199 | out1 = out1.squeeze() 200 | out1 = self.fcc(out1) 201 | # # # # 202 | # # 203 | out2 = self.biometrics(out) 204 | out2 = out2.squeeze() 205 | out2 = F.relu(self.fcr(out2)) 206 | 207 | return out1, out2 208 | -------------------------------------------------------------------------------- /model/resnet_generation_upsample.py: -------------------------------------------------------------------------------- 1 | import scipy.io as sio 2 | from torch.utils.data import TensorDataset, DataLoader 3 | import numpy as np 4 | # from model import locNN 5 | import torch 6 | import torch.nn as nn 7 | from torch.autograd import Variable 8 | import torch.nn.functional as F 9 | import matplotlib.pyplot as plt 10 | import math 11 | 12 | 13 | # 3x3 Convolution 14 | def conv3x3(in_channels, out_channels, stride=1): 15 | return nn.Conv2d(in_channels, out_channels, kernel_size=3, 16 | stride=stride, padding=1, bias=False) 17 | 18 | # Residual Block 19 | class ResidualBlock(nn.Module): 20 | expansion = 1 21 | def __init__(self, in_channels, out_channels, stride=1, downsample=None): 22 | super(ResidualBlock, self).__init__() 23 | self.conv1 = conv3x3(in_channels, out_channels, stride) 24 | self.bn1 = nn.BatchNorm2d(out_channels) 25 | self.relu = nn.ReLU(inplace=True) 26 | self.conv2 = conv3x3(out_channels, out_channels) 27 | self.bn2 = nn.BatchNorm2d(out_channels) 28 | self.downsample = downsample 29 | 30 | def forward(self, x): 31 | residual = x 32 | out = self.conv1(x) 33 | out = self.bn1(out) 34 | out = self.relu(out) 35 | out = self.conv2(out) 36 | out = self.bn2(out) 37 | if self.downsample: 38 | residual = self.downsample(x) 39 | out += residual 40 | out = self.relu(out) 41 | return out 42 | 43 | class Bottleneck(nn.Module): 44 | expansion = 4 45 | 46 | def __init__(self, in_channels, out_channels, stride=1, downsample=None): 47 | super(Bottleneck, self).__init__() 48 | self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=1, bias=False) 49 | self.bn1 = nn.BatchNorm2d(out_channels) 50 | self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, stride=stride, 51 | padding=1, bias=False) 52 | self.bn2 = nn.BatchNorm2d(out_channels) 53 | self.conv3 = nn.Conv2d(out_channels, out_channels * 4, kernel_size=1, bias=False) 54 | self.bn3 = nn.BatchNorm2d(out_channels * 4) 55 | self.relu = nn.ReLU(inplace=True) 56 | self.downsample = downsample 57 | self.stride = stride 58 | 59 | def forward(self, x): 60 | residual = x 61 | 62 | out = self.conv1(x) 63 | out = self.bn1(out) 64 | out = self.relu(out) 65 | 66 | out = self.conv2(out) 67 | out = self.bn2(out) 68 | out = self.relu(out) 69 | 70 | out = self.conv3(out) 71 | out = self.bn3(out) 72 | 73 | if self.downsample is not None: 74 | residual = self.downsample(x) 75 | 76 | out += residual 77 | out = self.relu(out) 78 | 79 | return out 80 | 81 | 82 | 83 | 84 | 85 | # ResNet Module 86 | class ResNet(nn.Module): 87 | def __init__(self, block, layers, num_classes): 88 | super(ResNet, self).__init__() 89 | self.unit_gen = 16 90 | self.generation = nn.Sequential( 91 | # 30*1*1 -> 256*2*2 92 | nn.ConvTranspose2d(30, 8 * self.unit_gen, kernel_size=4, stride=2, padding=1, bias=False), 93 | nn.BatchNorm2d(8 * self.unit_gen), 94 | nn.ReLU(), 95 | # 256*2*2 -> 4 96 | nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False), 97 | 98 | # 4->7 99 | nn.ConvTranspose2d(8 * self.unit_gen, 4 * self.unit_gen, kernel_size=3, stride=2, padding=1, bias=False), 100 | nn.BatchNorm2d(4 * self.unit_gen), 101 | nn.ReLU(), 102 | # 7->14 103 | nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False), 104 | 105 | # 14->28->56 106 | nn.ConvTranspose2d(4 * self.unit_gen, 2 * self.unit_gen, kernel_size=4, stride=2, padding=1, bias=False), 107 | nn.BatchNorm2d(2 * self.unit_gen), 108 | nn.ReLU(), 109 | # 256*7*7 -> 128*14*14 110 | nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False), 111 | 112 | nn.ConvTranspose2d(2 * self.unit_gen, self.unit_gen, kernel_size=4, stride=2, padding=1, bias=False), 113 | nn.BatchNorm2d(self.unit_gen), 114 | nn.ReLU(), 115 | 116 | nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False), 117 | 118 | ) 119 | 120 | 121 | self.conv1 = nn.Sequential( 122 | nn.Conv2d(self.unit_gen, 64, kernel_size=7, stride=2, padding=3, bias=False), 123 | nn.BatchNorm2d(64), 124 | nn.ReLU(inplace=True), 125 | nn.MaxPool2d(kernel_size=3, stride=2, padding=1), 126 | ) 127 | 128 | self.in_channels = 64 129 | self.layer1 = self.make_layer(block, 64, layers[0]) #8*64*64 -> 16*32*32 130 | self.layer2 = self.make_layer(block, 128, layers[1], 2) #16*32*32 -> 32*16*16 131 | self.layer3 = self.make_layer(block, 256, layers[2], 2) #32*16*16 -> 64*8*8 132 | self.layer4 = self.make_layer(block, 512, layers[3], 2) #64*8*8 -> 128*4*4 133 | 134 | 135 | self.humanid = nn.Sequential( 136 | nn.Conv2d(512*block.expansion, 512*block.expansion, kernel_size=3, stride=2, padding=1, bias=False), 137 | nn.BatchNorm2d(512*block.expansion), 138 | nn.ReLU(), 139 | 140 | nn.Conv2d(512*block.expansion, 256*block.expansion, kernel_size=4, stride=2, padding=1, bias=False), 141 | nn.BatchNorm2d(256*block.expansion), 142 | nn.ReLU(), 143 | nn.AvgPool2d(2), 144 | ) 145 | 146 | self.fcc = nn.Linear(256*block.expansion, num_classes) 147 | 148 | self.biometrics = nn.Sequential( 149 | nn.Conv2d(512*block.expansion, 512*block.expansion, kernel_size=3, stride=2, padding=1, bias=False), 150 | nn.BatchNorm2d(512*block.expansion), 151 | nn.ReLU(), 152 | 153 | nn.Conv2d(512*block.expansion, 256*block.expansion, kernel_size=4, stride=2, padding=1, bias=False), 154 | nn.BatchNorm2d(256*block.expansion), 155 | nn.ReLU(), 156 | nn.AvgPool2d(2), 157 | ) 158 | 159 | self.fcr = nn.Linear(256*block.expansion, 4) 160 | 161 | 162 | 163 | def make_layer(self, block, out_channels, blocks, stride=1): 164 | downsample = None 165 | if (stride != 1) or (self.in_channels != out_channels*block.expansion): 166 | downsample = nn.Sequential( 167 | conv3x3(self.in_channels, out_channels*block.expansion, stride=stride), 168 | nn.BatchNorm2d(out_channels*block.expansion)) 169 | layers = [] 170 | layers.append(block(self.in_channels, out_channels, stride, downsample)) 171 | self.in_channels = out_channels*block.expansion 172 | for i in range(1, blocks): 173 | layers.append(block(self.in_channels, out_channels)) 174 | return nn.Sequential(*layers) 175 | 176 | 177 | def forward(self, x): 178 | out = self.generation(x) 179 | out = self.conv1(out) 180 | # 181 | out = self.layer1(out) 182 | out = self.layer2(out) 183 | out = self.layer3(out) 184 | out = self.layer4(out) 185 | # 186 | # # 187 | # out = self.avgpool7(out) 188 | out1 = self.humanid(out) 189 | out1 = out1.squeeze() 190 | out1 = self.fcc(out1) 191 | # # # # 192 | # # 193 | out2 = self.biometrics(out) 194 | out2 = out2.squeeze() 195 | out2 = F.relu(self.fcr(out2)) 196 | 197 | return out1, out2 198 | -------------------------------------------------------------------------------- /model/resnet_upsample.py: -------------------------------------------------------------------------------- 1 | import scipy.io as sio 2 | from torch.utils.data import TensorDataset, DataLoader 3 | import numpy as np 4 | # from model import locNN 5 | import torch 6 | import torch.nn as nn 7 | from torch.autograd import Variable 8 | import torch.nn.functional as F 9 | import matplotlib.pyplot as plt 10 | import math 11 | 12 | 13 | # 3x3 Convolution 14 | def conv3x3(in_channels, out_channels, stride=1): 15 | return nn.Conv2d(in_channels, out_channels, kernel_size=3, 16 | stride=stride, padding=1, bias=False) 17 | 18 | # Residual Block 19 | class ResidualBlock(nn.Module): 20 | expansion = 1 21 | def __init__(self, in_channels, out_channels, stride=1, downsample=None): 22 | super(ResidualBlock, self).__init__() 23 | self.conv1 = conv3x3(in_channels, out_channels, stride) 24 | self.bn1 = nn.BatchNorm2d(out_channels) 25 | self.relu = nn.ReLU(inplace=True) 26 | self.conv2 = conv3x3(out_channels, out_channels) 27 | self.bn2 = nn.BatchNorm2d(out_channels) 28 | self.downsample = downsample 29 | 30 | def forward(self, x): 31 | residual = x 32 | out = self.conv1(x) 33 | out = self.bn1(out) 34 | out = self.relu(out) 35 | out = self.conv2(out) 36 | out = self.bn2(out) 37 | if self.downsample: 38 | residual = self.downsample(x) 39 | out += residual 40 | out = self.relu(out) 41 | return out 42 | 43 | class Bottleneck(nn.Module): 44 | expansion = 4 45 | 46 | def __init__(self, in_channels, out_channels, stride=1, downsample=None): 47 | super(Bottleneck, self).__init__() 48 | self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=1, bias=False) 49 | self.bn1 = nn.BatchNorm2d(out_channels) 50 | self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, stride=stride, 51 | padding=1, bias=False) 52 | self.bn2 = nn.BatchNorm2d(out_channels) 53 | self.conv3 = nn.Conv2d(out_channels, out_channels * 4, kernel_size=1, bias=False) 54 | self.bn3 = nn.BatchNorm2d(out_channels * 4) 55 | self.relu = nn.ReLU(inplace=True) 56 | self.downsample = downsample 57 | self.stride = stride 58 | 59 | def forward(self, x): 60 | residual = x 61 | 62 | out = self.conv1(x) 63 | out = self.bn1(out) 64 | out = self.relu(out) 65 | 66 | out = self.conv2(out) 67 | out = self.bn2(out) 68 | out = self.relu(out) 69 | 70 | out = self.conv3(out) 71 | out = self.bn3(out) 72 | 73 | if self.downsample is not None: 74 | residual = self.downsample(x) 75 | 76 | out += residual 77 | out = self.relu(out) 78 | 79 | return out 80 | 81 | 82 | 83 | # ResNet Module 84 | class ResNet(nn.Module): 85 | def __init__(self, block, layers, num_classes): 86 | super(ResNet, self).__init__() 87 | 88 | self.upsampling = nn.Upsample(mode='bilinear', scale_factor=224, align_corners=False) 89 | 90 | 91 | self.conv1 = nn.Sequential( 92 | nn.Conv2d(30, 64, kernel_size=7, stride=2, padding=3, bias=False), 93 | nn.BatchNorm2d(64), 94 | nn.ReLU(inplace=True), 95 | nn.MaxPool2d(kernel_size=3, stride=2, padding=1), 96 | ) 97 | 98 | self.in_channels = 64 99 | self.layer1 = self.make_layer(block, 64, layers[0]) #8*64*64 -> 16*32*32 100 | self.layer2 = self.make_layer(block, 128, layers[1], 2) #16*32*32 -> 32*16*16 101 | self.layer3 = self.make_layer(block, 256, layers[2], 2) #32*16*16 -> 64*8*8 102 | self.layer4 = self.make_layer(block, 512, layers[3], 2) #64*8*8 -> 128*4*4 103 | 104 | self.humanid = nn.Sequential( 105 | nn.Conv2d(512 * block.expansion, 512 * block.expansion, kernel_size=3, stride=2, padding=1, bias=False), 106 | nn.BatchNorm2d(512 * block.expansion), 107 | nn.ReLU(), 108 | 109 | nn.Conv2d(512 * block.expansion, 256 * block.expansion, kernel_size=4, stride=2, padding=1, bias=False), 110 | nn.BatchNorm2d(256 * block.expansion), 111 | nn.ReLU(), 112 | nn.AvgPool2d(2), 113 | ) 114 | 115 | self.fcc = nn.Linear(256 * block.expansion, num_classes) 116 | 117 | self.biometrics = nn.Sequential( 118 | nn.Conv2d(512 * block.expansion, 512 * block.expansion, kernel_size=3, stride=2, padding=1, bias=False), 119 | nn.BatchNorm2d(512 * block.expansion), 120 | nn.ReLU(), 121 | 122 | nn.Conv2d(512 * block.expansion, 256 * block.expansion, kernel_size=4, stride=2, padding=1, bias=False), 123 | nn.BatchNorm2d(256 * block.expansion), 124 | nn.ReLU(), 125 | nn.AvgPool2d(2), 126 | ) 127 | 128 | self.fcr = nn.Linear(256 * block.expansion, 4) 129 | 130 | def make_layer(self, block, out_channels, blocks, stride=1): 131 | downsample = None 132 | if (stride != 1) or (self.in_channels != out_channels * block.expansion): 133 | downsample = nn.Sequential( 134 | conv3x3(self.in_channels, out_channels * block.expansion, stride=stride), 135 | nn.BatchNorm2d(out_channels * block.expansion)) 136 | layers = [] 137 | layers.append(block(self.in_channels, out_channels, stride, downsample)) 138 | self.in_channels = out_channels * block.expansion 139 | for i in range(1, blocks): 140 | layers.append(block(self.in_channels, out_channels)) 141 | return nn.Sequential(*layers) 142 | 143 | def forward(self, x): 144 | out = self.upsampling(x) 145 | out = self.conv1(out) 146 | # 147 | out = self.layer1(out) 148 | out = self.layer2(out) 149 | out = self.layer3(out) 150 | out = self.layer4(out) 151 | # 152 | # # 153 | # out = self.avgpool7(out) 154 | out1 = self.humanid(out) 155 | out1 = out1.squeeze() 156 | out1 = self.fcc(out1) 157 | # # # # 158 | # # 159 | out2 = self.biometrics(out) 160 | out2 = out2.squeeze() 161 | out2 = F.relu(self.fcr(out2)) 162 | 163 | return out1, out2 -------------------------------------------------------------------------------- /model/resnet_upsample2x2.py: -------------------------------------------------------------------------------- 1 | import scipy.io as sio 2 | from torch.utils.data import TensorDataset, DataLoader 3 | import numpy as np 4 | # from model import locNN 5 | import torch 6 | import torch.nn as nn 7 | from torch.autograd import Variable 8 | import torch.nn.functional as F 9 | import matplotlib.pyplot as plt 10 | import math 11 | 12 | 13 | # 3x3 Convolution 14 | def conv3x3(in_channels, out_channels, stride=1): 15 | return nn.Conv2d(in_channels, out_channels, kernel_size=3, 16 | stride=stride, padding=1, bias=False) 17 | 18 | # Residual Block 19 | class ResidualBlock(nn.Module): 20 | expansion = 1 21 | def __init__(self, in_channels, out_channels, stride=1, downsample=None): 22 | super(ResidualBlock, self).__init__() 23 | self.conv1 = conv3x3(in_channels, out_channels, stride) 24 | self.bn1 = nn.BatchNorm2d(out_channels) 25 | self.relu = nn.ReLU(inplace=True) 26 | self.conv2 = conv3x3(out_channels, out_channels) 27 | self.bn2 = nn.BatchNorm2d(out_channels) 28 | self.downsample = downsample 29 | 30 | def forward(self, x): 31 | residual = x 32 | out = self.conv1(x) 33 | out = self.bn1(out) 34 | out = self.relu(out) 35 | out = self.conv2(out) 36 | out = self.bn2(out) 37 | if self.downsample: 38 | residual = self.downsample(x) 39 | out += residual 40 | out = self.relu(out) 41 | return out 42 | 43 | class Bottleneck(nn.Module): 44 | expansion = 4 45 | 46 | def __init__(self, in_channels, out_channels, stride=1, downsample=None): 47 | super(Bottleneck, self).__init__() 48 | self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=1, bias=False) 49 | self.bn1 = nn.BatchNorm2d(out_channels) 50 | self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, stride=stride, 51 | padding=1, bias=False) 52 | self.bn2 = nn.BatchNorm2d(out_channels) 53 | self.conv3 = nn.Conv2d(out_channels, out_channels * 4, kernel_size=1, bias=False) 54 | self.bn3 = nn.BatchNorm2d(out_channels * 4) 55 | self.relu = nn.ReLU(inplace=True) 56 | self.downsample = downsample 57 | self.stride = stride 58 | 59 | def forward(self, x): 60 | residual = x 61 | 62 | out = self.conv1(x) 63 | out = self.bn1(out) 64 | out = self.relu(out) 65 | 66 | out = self.conv2(out) 67 | out = self.bn2(out) 68 | out = self.relu(out) 69 | 70 | out = self.conv3(out) 71 | out = self.bn3(out) 72 | 73 | if self.downsample is not None: 74 | residual = self.downsample(x) 75 | 76 | out += residual 77 | out = self.relu(out) 78 | 79 | return out 80 | 81 | 82 | 83 | # ResNet Module 84 | class ResNet(nn.Module): 85 | def __init__(self, block, layers, num_classes): 86 | super(ResNet, self).__init__() 87 | 88 | self.upsampling = nn.Upsample(mode='bilinear', scale_factor=112, align_corners=False) 89 | 90 | 91 | self.conv1 = nn.Sequential( 92 | nn.Conv2d(8, 64, kernel_size=7, stride=2, padding=3, bias=False), 93 | nn.BatchNorm2d(64), 94 | nn.ReLU(inplace=True), 95 | nn.MaxPool2d(kernel_size=3, stride=2, padding=1), 96 | ) 97 | 98 | self.in_channels = 64 99 | self.layer1 = self.make_layer(block, 64, layers[0]) #8*64*64 -> 16*32*32 100 | self.layer2 = self.make_layer(block, 128, layers[1], 2) #16*32*32 -> 32*16*16 101 | self.layer3 = self.make_layer(block, 256, layers[2], 2) #32*16*16 -> 64*8*8 102 | self.layer4 = self.make_layer(block, 512, layers[3], 2) #64*8*8 -> 128*4*4 103 | 104 | self.humanid = nn.Sequential( 105 | nn.Conv2d(512 * block.expansion, 512 * block.expansion, kernel_size=3, stride=2, padding=1, bias=False), 106 | nn.BatchNorm2d(512 * block.expansion), 107 | nn.ReLU(), 108 | 109 | nn.Conv2d(512 * block.expansion, 256 * block.expansion, kernel_size=4, stride=2, padding=1, bias=False), 110 | nn.BatchNorm2d(256 * block.expansion), 111 | nn.ReLU(), 112 | nn.AvgPool2d(2), 113 | ) 114 | 115 | self.fcc = nn.Linear(256 * block.expansion, num_classes) 116 | 117 | self.biometrics = nn.Sequential( 118 | nn.Conv2d(512 * block.expansion, 512 * block.expansion, kernel_size=3, stride=2, padding=1, bias=False), 119 | nn.BatchNorm2d(512 * block.expansion), 120 | nn.ReLU(), 121 | 122 | nn.Conv2d(512 * block.expansion, 256 * block.expansion, kernel_size=4, stride=2, padding=1, bias=False), 123 | nn.BatchNorm2d(256 * block.expansion), 124 | nn.ReLU(), 125 | nn.AvgPool2d(2), 126 | ) 127 | 128 | self.fcr = nn.Linear(256 * block.expansion, 4) 129 | 130 | def make_layer(self, block, out_channels, blocks, stride=1): 131 | downsample = None 132 | if (stride != 1) or (self.in_channels != out_channels * block.expansion): 133 | downsample = nn.Sequential( 134 | conv3x3(self.in_channels, out_channels * block.expansion, stride=stride), 135 | nn.BatchNorm2d(out_channels * block.expansion)) 136 | layers = [] 137 | layers.append(block(self.in_channels, out_channels, stride, downsample)) 138 | self.in_channels = out_channels * block.expansion 139 | for i in range(1, blocks): 140 | layers.append(block(self.in_channels, out_channels)) 141 | return nn.Sequential(*layers) 142 | 143 | def forward(self, x): 144 | out = self.upsampling(x) 145 | out = self.conv1(out) 146 | # 147 | out = self.layer1(out) 148 | out = self.layer2(out) 149 | out = self.layer3(out) 150 | out = self.layer4(out) 151 | # 152 | # # 153 | # out = self.avgpool7(out) 154 | out1 = self.humanid(out) 155 | out1 = out1.squeeze() 156 | out1 = self.fcc(out1) 157 | # # # # 158 | # # 159 | out2 = self.biometrics(out) 160 | out2 = out2.squeeze() 161 | out2 = F.relu(self.fcr(out2)) 162 | 163 | return out1, out2 -------------------------------------------------------------------------------- /model/resnet_upsample3x3.py: -------------------------------------------------------------------------------- 1 | import scipy.io as sio 2 | from torch.utils.data import TensorDataset, DataLoader 3 | import numpy as np 4 | # from model import locNN 5 | import torch 6 | import torch.nn as nn 7 | from torch.autograd import Variable 8 | import torch.nn.functional as F 9 | import matplotlib.pyplot as plt 10 | import math 11 | 12 | 13 | # 3x3 Convolution 14 | def conv3x3(in_channels, out_channels, stride=1): 15 | return nn.Conv2d(in_channels, out_channels, kernel_size=3, 16 | stride=stride, padding=1, bias=False) 17 | 18 | # Residual Block 19 | class ResidualBlock(nn.Module): 20 | expansion = 1 21 | def __init__(self, in_channels, out_channels, stride=1, downsample=None): 22 | super(ResidualBlock, self).__init__() 23 | 24 | self.conv1 = conv3x3(in_channels, out_channels, stride) 25 | self.bn1 = nn.BatchNorm2d(out_channels) 26 | self.relu = nn.ReLU(inplace=True) 27 | self.conv2 = conv3x3(out_channels, out_channels) 28 | self.bn2 = nn.BatchNorm2d(out_channels) 29 | self.downsample = downsample 30 | 31 | def forward(self, x): 32 | residual = x 33 | out = self.conv1(x) 34 | out = self.bn1(out) 35 | out = self.relu(out) 36 | out = self.conv2(out) 37 | out = self.bn2(out) 38 | if self.downsample: 39 | residual = self.downsample(x) 40 | out += residual 41 | out = self.relu(out) 42 | return out 43 | 44 | class Bottleneck(nn.Module): 45 | expansion = 4 46 | 47 | def __init__(self, in_channels, out_channels, stride=1, downsample=None): 48 | super(Bottleneck, self).__init__() 49 | self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=1, bias=False) 50 | self.bn1 = nn.BatchNorm2d(out_channels) 51 | self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, stride=stride, 52 | padding=1, bias=False) 53 | self.bn2 = nn.BatchNorm2d(out_channels) 54 | self.conv3 = nn.Conv2d(out_channels, out_channels * 4, kernel_size=1, bias=False) 55 | self.bn3 = nn.BatchNorm2d(out_channels * 4) 56 | self.relu = nn.ReLU(inplace=True) 57 | self.downsample = downsample 58 | self.stride = stride 59 | 60 | def forward(self, x): 61 | residual = x 62 | 63 | out = self.conv1(x) 64 | out = self.bn1(out) 65 | out = self.relu(out) 66 | 67 | out = self.conv2(out) 68 | out = self.bn2(out) 69 | out = self.relu(out) 70 | 71 | out = self.conv3(out) 72 | out = self.bn3(out) 73 | 74 | if self.downsample is not None: 75 | residual = self.downsample(x) 76 | 77 | out += residual 78 | out = self.relu(out) 79 | 80 | return out 81 | 82 | 83 | 84 | # ResNet Module 85 | class ResNet(nn.Module): 86 | def __init__(self, block, layers, num_classes): 87 | super(ResNet, self).__init__() 88 | self.conv3to4 = nn.ConvTranspose2d(3, 3, kernel_size=4, stride=1, padding=1, bias=False) 89 | self.upsampling = nn.Upsample(mode='bilinear', scale_factor=56, align_corners=False) 90 | self.conv1 = nn.Sequential( 91 | nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=False), 92 | nn.BatchNorm2d(64), 93 | nn.ReLU(inplace=True), 94 | nn.MaxPool2d(kernel_size=3, stride=2, padding=1), 95 | ) 96 | 97 | self.in_channels = 64 98 | self.layer1 = self.make_layer(block, 64, layers[0]) #8*64*64 -> 16*32*32 99 | self.layer2 = self.make_layer(block, 128, layers[1], 2) #16*32*32 -> 32*16*16 100 | self.layer3 = self.make_layer(block, 256, layers[2], 2) #32*16*16 -> 64*8*8 101 | self.layer4 = self.make_layer(block, 512, layers[3], 2) #64*8*8 -> 128*4*4 102 | 103 | self.humanid = nn.Sequential( 104 | nn.Conv2d(512 * block.expansion, 512 * block.expansion, kernel_size=3, stride=2, padding=1, bias=False), 105 | nn.BatchNorm2d(512 * block.expansion), 106 | nn.ReLU(), 107 | 108 | nn.Conv2d(512 * block.expansion, 256 * block.expansion, kernel_size=4, stride=2, padding=1, bias=False), 109 | nn.BatchNorm2d(256 * block.expansion), 110 | nn.ReLU(), 111 | nn.AvgPool2d(2), 112 | ) 113 | 114 | self.fcc = nn.Linear(256 * block.expansion, num_classes) 115 | 116 | self.biometrics = nn.Sequential( 117 | nn.Conv2d(512 * block.expansion, 512 * block.expansion, kernel_size=3, stride=2, padding=1, bias=False), 118 | nn.BatchNorm2d(512 * block.expansion), 119 | nn.ReLU(), 120 | 121 | nn.Conv2d(512 * block.expansion, 256 * block.expansion, kernel_size=4, stride=2, padding=1, bias=False), 122 | nn.BatchNorm2d(256 * block.expansion), 123 | nn.ReLU(), 124 | nn.AvgPool2d(2), 125 | ) 126 | 127 | self.fcr = nn.Linear(256 * block.expansion, 4) 128 | 129 | def make_layer(self, block, out_channels, blocks, stride=1): 130 | downsample = None 131 | if (stride != 1) or (self.in_channels != out_channels * block.expansion): 132 | downsample = nn.Sequential( 133 | conv3x3(self.in_channels, out_channels * block.expansion, stride=stride), 134 | nn.BatchNorm2d(out_channels * block.expansion)) 135 | layers = [] 136 | layers.append(block(self.in_channels, out_channels, stride, downsample)) 137 | self.in_channels = out_channels * block.expansion 138 | for i in range(1, blocks): 139 | layers.append(block(self.in_channels, out_channels)) 140 | return nn.Sequential(*layers) 141 | 142 | def forward(self, x): 143 | out = self.conv3to4(x) 144 | out = self.upsampling(out) 145 | out = self.conv1(out) 146 | # 147 | out = self.layer1(out) 148 | out = self.layer2(out) 149 | out = self.layer3(out) 150 | out = self.layer4(out) 151 | # 152 | # # 153 | # out = self.avgpool7(out) 154 | out1 = self.humanid(out) 155 | out1 = out1.squeeze() 156 | out1 = self.fcc(out1) 157 | # # # # 158 | # # 159 | out2 = self.biometrics(out) 160 | out2 = out2.squeeze() 161 | out2 = F.relu(self.fcr(out2)) 162 | 163 | return out1, out2 -------------------------------------------------------------------------------- /model/solo_task_res_net.py: -------------------------------------------------------------------------------- 1 | import scipy.io as sio 2 | from torch.utils.data import TensorDataset, DataLoader 3 | import numpy as np 4 | # from model import locNN 5 | import torch 6 | import torch.nn as nn 7 | from torch.autograd import Variable 8 | import torch.nn.functional as F 9 | import matplotlib.pyplot as plt 10 | import math 11 | 12 | 13 | # 3x3 Convolution 14 | def conv3x3(in_channels, out_channels, stride=1): 15 | return nn.Conv2d(in_channels, out_channels, kernel_size=3, 16 | stride=stride, padding=1, bias=False) 17 | 18 | # Residual Block 19 | class ResidualBlock(nn.Module): 20 | expansion = 1 21 | def __init__(self, in_channels, out_channels, stride=1, downsample=None): 22 | super(ResidualBlock, self).__init__() 23 | self.conv1 = conv3x3(in_channels, out_channels, stride) 24 | self.bn1 = nn.BatchNorm2d(out_channels) 25 | self.relu = nn.ReLU(inplace=True) 26 | self.conv2 = conv3x3(out_channels, out_channels) 27 | self.bn2 = nn.BatchNorm2d(out_channels) 28 | self.downsample = downsample 29 | 30 | def forward(self, x): 31 | residual = x 32 | out = self.conv1(x) 33 | out = self.bn1(out) 34 | out = self.relu(out) 35 | out = self.conv2(out) 36 | out = self.bn2(out) 37 | if self.downsample: 38 | residual = self.downsample(x) 39 | out += residual 40 | out = self.relu(out) 41 | return out 42 | 43 | class Bottleneck(nn.Module): 44 | expansion = 4 45 | 46 | def __init__(self, in_channels, out_channels, stride=1, downsample=None): 47 | super(Bottleneck, self).__init__() 48 | self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=1, bias=False) 49 | self.bn1 = nn.BatchNorm2d(out_channels) 50 | self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, stride=stride, 51 | padding=1, bias=False) 52 | self.bn2 = nn.BatchNorm2d(out_channels) 53 | self.conv3 = nn.Conv2d(out_channels, out_channels * 4, kernel_size=1, bias=False) 54 | self.bn3 = nn.BatchNorm2d(out_channels * 4) 55 | self.relu = nn.ReLU(inplace=True) 56 | self.downsample = downsample 57 | self.stride = stride 58 | 59 | def forward(self, x): 60 | residual = x 61 | 62 | out = self.conv1(x) 63 | out = self.bn1(out) 64 | out = self.relu(out) 65 | 66 | out = self.conv2(out) 67 | out = self.bn2(out) 68 | out = self.relu(out) 69 | 70 | out = self.conv3(out) 71 | out = self.bn3(out) 72 | 73 | if self.downsample is not None: 74 | residual = self.downsample(x) 75 | 76 | out += residual 77 | out = self.relu(out) 78 | 79 | return out 80 | 81 | 82 | 83 | 84 | 85 | # ResNet Module 86 | class ResNet(nn.Module): 87 | def __init__(self, block, layers, number_class): 88 | super(ResNet, self).__init__() 89 | 90 | self.generation = nn.Sequential( 91 | # 30*1*1 -> 256*2*2 92 | nn.ConvTranspose2d(30, 384, kernel_size=4, stride=2, padding=1, bias=False), 93 | nn.BatchNorm2d(384), 94 | nn.ReLU(), 95 | # 256*2*2 -> 128*4*4 96 | nn.ConvTranspose2d(384, 192, kernel_size=4, stride=2, padding=1, bias=False), 97 | nn.BatchNorm2d(192), 98 | nn.ReLU(), 99 | 100 | # 128*4*4 -> 64*7*7 101 | nn.ConvTranspose2d(192, 96, kernel_size=3, stride=2, padding=1, bias=False), 102 | nn.BatchNorm2d(96), 103 | nn.ReLU(), 104 | 105 | # 7 -> 14 106 | nn.ConvTranspose2d(96, 48, kernel_size=4, stride=2, padding=1, bias=False), 107 | nn.BatchNorm2d(48), 108 | nn.ReLU(), 109 | 110 | # 14 -> 28 111 | nn.ConvTranspose2d(48, 24, kernel_size=4, stride=2, padding=1, bias=False), 112 | nn.BatchNorm2d(24), 113 | nn.ReLU(), 114 | 115 | # 28 -> 56 116 | nn.ConvTranspose2d(24, 12, kernel_size=4, stride=2, padding=1, bias=False), 117 | nn.BatchNorm2d(12), 118 | nn.ReLU(), 119 | 120 | # 56 -> 112 121 | nn.ConvTranspose2d(12, 6, kernel_size=4, stride=2, padding=1, bias=False), 122 | nn.BatchNorm2d(6), 123 | nn.ReLU(), 124 | 125 | # 112 -> 224c 126 | nn.ConvTranspose2d(6, 6, kernel_size=4, stride=2, padding=1, bias=False), 127 | nn.BatchNorm2d(6), 128 | nn.ReLU(), 129 | ) 130 | 131 | self.conv1 = nn.Sequential( 132 | nn.Conv2d(6, 64, kernel_size=7, stride=2, padding=3, bias=False), 133 | nn.BatchNorm2d(64), 134 | nn.ReLU(inplace=True), 135 | nn.MaxPool2d(kernel_size=3, stride=2, padding=1), 136 | ) 137 | 138 | self.in_channels = 64 139 | self.layer1 = self.make_layer(block, 64, layers[0]) #8*64*64 -> 16*32*32 140 | self.layer2 = self.make_layer(block, 128, layers[1], 2) #16*32*32 -> 32*16*16 141 | self.layer3 = self.make_layer(block, 256, layers[2], 2) #32*16*16 -> 64*8*8 142 | self.layer4 = self.make_layer(block, 512, layers[3], 2) #64*8*8 -> 128*4*4 143 | 144 | 145 | self.fallen = nn.Sequential( 146 | nn.Conv2d(512*block.expansion, 256*block.expansion, kernel_size=3, stride=2, padding=1, bias=False), 147 | nn.BatchNorm2d(256*block.expansion), 148 | nn.ReLU(), 149 | 150 | nn.Conv2d(256*block.expansion, 128*block.expansion, kernel_size=4, stride=2, padding=1, bias=False), 151 | nn.BatchNorm2d(128*block.expansion), 152 | nn.ReLU(), 153 | nn.AvgPool2d(2), 154 | ) 155 | 156 | self.fc = nn.Linear(128*block.expansion, number_class) 157 | 158 | def make_layer(self, block, out_channels, blocks, stride=1): 159 | downsample = None 160 | if (stride != 1) or (self.in_channels != out_channels*block.expansion): 161 | downsample = nn.Sequential( 162 | conv3x3(self.in_channels, out_channels*block.expansion, stride=stride), 163 | nn.BatchNorm2d(out_channels*block.expansion)) 164 | layers = [] 165 | layers.append(block(self.in_channels, out_channels, stride, downsample)) 166 | self.in_channels = out_channels*block.expansion 167 | for i in range(1, blocks): 168 | layers.append(block(self.in_channels, out_channels)) 169 | return nn.Sequential(*layers) 170 | 171 | # def forward(self, x): 172 | # out_generation = self.generation(x) 173 | # out = self.conv1(out_generation) 174 | 175 | # outres1 = self.layer1(out) 176 | # outres2 = self.layer2(outres1) 177 | # outres3 = self.layer3(outres2) 178 | # outres4 = self.layer4(outres3) 179 | # # return out 180 | # # 181 | # out_fc = self.humanid(outres4) 182 | # out_fc = out_fc.squeeze() 183 | # out1 = self.fcc(out_fc) 184 | # 185 | # return out1, out_generation, out, outres1, outres2, outres3, outres4, out_fc 186 | def forward(self, x): 187 | out = self.generation(x) 188 | out = self.conv1(out) 189 | # 190 | out = self.layer1(out) 191 | out = self.layer2(out) 192 | out = self.layer3(out) 193 | out = self.layer4(out) 194 | # 195 | # # 196 | # out = self.avgpool7(out) 197 | out = self.fallen(out) 198 | out = out.squeeze() 199 | out = self.fc(out) 200 | # out = self.sm(out) 201 | return out 202 | -------------------------------------------------------------------------------- /model/vgg_net.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import math 3 | 4 | 5 | 6 | class VGG(nn.Module): 7 | 8 | def __init__(self, features, num_classes=1000, init_weights=False): 9 | super(VGG, self).__init__() 10 | 11 | self.generation = nn.Sequential( 12 | # 30*1*1 -> 256*2*2 13 | nn.ConvTranspose2d(30, 384, kernel_size=4, stride=2, padding=1, bias=False), 14 | nn.BatchNorm2d(384), 15 | nn.ReLU(), 16 | # 256*2*2 -> 128*4*4 17 | nn.ConvTranspose2d(384, 192, kernel_size=4, stride=2, padding=1, bias=False), 18 | nn.BatchNorm2d(192), 19 | nn.ReLU(), 20 | 21 | # 128*4*4 -> 64*7*7 22 | nn.ConvTranspose2d(192, 96, kernel_size=3, stride=2, padding=1, bias=False), 23 | nn.BatchNorm2d(96), 24 | nn.ReLU(), 25 | 26 | # 7 -> 14 27 | nn.ConvTranspose2d(96, 48, kernel_size=4, stride=2, padding=1, bias=False), 28 | nn.BatchNorm2d(48), 29 | nn.ReLU(), 30 | 31 | # 14 -> 28 32 | nn.ConvTranspose2d(48, 24, kernel_size=4, stride=2, padding=1, bias=False), 33 | nn.BatchNorm2d(24), 34 | nn.ReLU(), 35 | 36 | # 28 -> 56 37 | nn.ConvTranspose2d(24, 12, kernel_size=4, stride=2, padding=1, bias=False), 38 | nn.BatchNorm2d(12), 39 | nn.ReLU(), 40 | 41 | # 56 -> 112 42 | nn.ConvTranspose2d(12, 6, kernel_size=4, stride=2, padding=1, bias=False), 43 | nn.BatchNorm2d(6), 44 | nn.ReLU(), 45 | 46 | # 112 -> 224c 47 | nn.ConvTranspose2d(6, 3, kernel_size=4, stride=2, padding=1, bias=False), 48 | nn.BatchNorm2d(3), 49 | nn.ReLU(), 50 | ) 51 | 52 | self.features = features 53 | 54 | self.id = nn.Sequential( 55 | nn.Linear(512 * 7 * 7, 4096), 56 | nn.ReLU(True), 57 | nn.Dropout(), 58 | nn.Linear(4096, 4096), 59 | nn.ReLU(True), 60 | nn.Dropout(), 61 | nn.Linear(4096, 30), 62 | ) 63 | 64 | self.bio = nn.Sequential( 65 | nn.Linear(512 * 7 * 7, 4096), 66 | nn.ReLU(True), 67 | nn.Dropout(), 68 | nn.Linear(4096, 4096), 69 | nn.ReLU(True), 70 | nn.Dropout(), 71 | nn.Linear(4096, 4), 72 | ) 73 | if init_weights: 74 | self._initialize_weights() 75 | 76 | def forward(self, x): 77 | x = self.generation(x) 78 | x = self.features(x) 79 | x = x.view(x.size(0), -1) 80 | out1 = self.id(x) 81 | out2 = self.bio(x) 82 | return out1, out2 83 | 84 | def _initialize_weights(self): 85 | for m in self.modules(): 86 | if isinstance(m, nn.Conv2d): 87 | nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu') 88 | if m.bias is not None: 89 | nn.init.constant_(m.bias, 0) 90 | elif isinstance(m, nn.BatchNorm2d): 91 | nn.init.constant_(m.weight, 1) 92 | nn.init.constant_(m.bias, 0) 93 | elif isinstance(m, nn.Linear): 94 | nn.init.normal_(m.weight, 0, 0.01) 95 | nn.init.constant_(m.bias, 0) 96 | 97 | 98 | def make_layers(cfg, batch_norm=False): 99 | layers = [] 100 | in_channels = 3 101 | for v in cfg: 102 | if v == 'M': 103 | layers += [nn.MaxPool2d(kernel_size=2, stride=2)] 104 | else: 105 | conv2d = nn.Conv2d(in_channels, v, kernel_size=3, padding=1) 106 | if batch_norm: 107 | layers += [conv2d, nn.BatchNorm2d(v), nn.ReLU(inplace=True)] 108 | else: 109 | layers += [conv2d, nn.ReLU(inplace=True)] 110 | in_channels = v 111 | return nn.Sequential(*layers) 112 | 113 | 114 | cfg = { 115 | 'A': [64, 'M', 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'], 116 | 'B': [64, 64, 'M', 128, 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'], 117 | 'D': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M'], 118 | 'E': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 256, 'M', 512, 512, 512, 512, 'M', 512, 512, 512, 512, 'M'], 119 | } 120 | 121 | 122 | # def vgg11(pretrained=False, **kwargs): 123 | # """VGG 11-layer model (configuration "A") 124 | # Args: 125 | # pretrained (bool): If True, returns a model pre-trained on ImageNet 126 | # """ 127 | # if pretrained: 128 | # kwargs['init_weights'] = False 129 | # model = VGG(make_layers(cfg['A']), **kwargs) 130 | # if pretrained: 131 | # model.load_state_dict(model_zoo.load_url(model_urls['vgg11'])) 132 | # return model 133 | # 134 | # 135 | # def vgg11_bn(pretrained=False, **kwargs): 136 | # """VGG 11-layer model (configuration "A") with batch normalization 137 | # Args: 138 | # pretrained (bool): If True, returns a model pre-trained on ImageNet 139 | # """ 140 | # if pretrained: 141 | # kwargs['init_weights'] = False 142 | # model = VGG(make_layers(cfg['A'], batch_norm=True), **kwargs) 143 | # if pretrained: 144 | # model.load_state_dict(model_zoo.load_url(model_urls['vgg11_bn'])) 145 | # return model 146 | # 147 | # 148 | # def vgg13(pretrained=False, **kwargs): 149 | # """VGG 13-layer model (configuration "B") 150 | # Args: 151 | # pretrained (bool): If True, returns a model pre-trained on ImageNet 152 | # """ 153 | # if pretrained: 154 | # kwargs['init_weights'] = False 155 | # model = VGG(make_layers(cfg['B']), **kwargs) 156 | # if pretrained: 157 | # model.load_state_dict(model_zoo.load_url(model_urls['vgg13'])) 158 | # return model 159 | # 160 | # 161 | # def vgg13_bn(pretrained=False, **kwargs): 162 | # """VGG 13-layer model (configuration "B") with batch normalization 163 | # Args: 164 | # pretrained (bool): If True, returns a model pre-trained on ImageNet 165 | # """ 166 | # if pretrained: 167 | # kwargs['init_weights'] = False 168 | # model = VGG(make_layers(cfg['B'], batch_norm=True), **kwargs) 169 | # if pretrained: 170 | # model.load_state_dict(model_zoo.load_url(model_urls['vgg13_bn'])) 171 | # return model 172 | # 173 | # 174 | # def vgg16(pretrained=False, **kwargs): 175 | # """VGG 16-layer model (configuration "D") 176 | # Args: 177 | # pretrained (bool): If True, returns a model pre-trained on ImageNet 178 | # """ 179 | # if pretrained: 180 | # kwargs['init_weights'] = False 181 | # model = VGG(make_layers(cfg['D']), **kwargs) 182 | # if pretrained: 183 | # model.load_state_dict(model_zoo.load_url(model_urls['vgg16'])) 184 | # return model 185 | # 186 | # 187 | # def vgg16_bn(pretrained=False, **kwargs): 188 | # """VGG 16-layer model (configuration "D") with batch normalization 189 | # Args: 190 | # pretrained (bool): If True, returns a model pre-trained on ImageNet 191 | # """ 192 | # if pretrained: 193 | # kwargs['init_weights'] = False 194 | # model = VGG(make_layers(cfg['D'], batch_norm=True), **kwargs) 195 | # if pretrained: 196 | # model.load_state_dict(model_zoo.load_url(model_urls['vgg16_bn'])) 197 | # return model 198 | # 199 | # 200 | # def vgg19(pretrained=False, **kwargs): 201 | # """VGG 19-layer model (configuration "E") 202 | # Args: 203 | # pretrained (bool): If True, returns a model pre-trained on ImageNet 204 | # """ 205 | # if pretrained: 206 | # kwargs['init_weights'] = False 207 | # model = VGG(make_layers(cfg['E']), **kwargs) 208 | # if pretrained: 209 | # model.load_state_dict(model_zoo.load_url(model_urls['vgg19'])) 210 | # return model 211 | # 212 | # 213 | # def vgg19_bn(pretrained=False, **kwargs): 214 | # """VGG 19-layer model (configuration 'E') with batch normalization 215 | # Args: 216 | # pretrained (bool): If True, returns a model pre-trained on ImageNet 217 | # """ 218 | # if pretrained: 219 | # kwargs['init_weights'] = False 220 | # model = VGG(make_layers(cfg['E'], batch_norm=True), **kwargs) 221 | # if pretrained: 222 | # model.load_state_dict(model_zoo.load_url(model_urls['vgg19_bn'])) 223 | # return model -------------------------------------------------------------------------------- /test.py: -------------------------------------------------------------------------------- 1 | import scipy.io as sio 2 | from torch.utils.data import TensorDataset, DataLoader 3 | import numpy as np 4 | import torch 5 | import torch.nn as nn 6 | from torch.autograd import Variable 7 | import torch.nn.functional as F 8 | import matplotlib.pyplot as plt 9 | import math 10 | import pandas as pd 11 | from model.res_net_use_this import ResNet, Bottleneck 12 | 13 | # 14 | batch_size = 100 15 | 16 | # prepare data, nSample x nChannel x width x height 17 | data = sio.loadmat('data/test.mat') 18 | test_data = data['test_data'] 19 | test_label = data['test_label'] 20 | test_label[:, 0] = test_label[:, 0] - 1 21 | 22 | num_test_instances = len(test_data) 23 | 24 | test_data = torch.from_numpy(test_data).type(torch.FloatTensor).view(num_test_instances, 30, 1, 1) 25 | test_label = torch.from_numpy(test_label).type(torch.FloatTensor) 26 | test_dataset = TensorDataset(test_data, test_label) 27 | test_data_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=False) 28 | 29 | resnet = torch.load('weights/res18_Train100.0Test93.001.pkl') 30 | resnet = resnet.cuda() 31 | 32 | resnet.eval() 33 | 34 | correct = 0 35 | 36 | for i, (samples, labels) in enumerate(test_data_loader): 37 | with torch.no_grad(): 38 | samplesV = Variable(samples.cuda()) 39 | labelsV = Variable(labels.cuda()) 40 | 41 | predict_label = resnet(samplesV) 42 | 43 | pred = predict_label[0][:, 0:30].data.max(1)[1] 44 | correct += pred.eq(labelsV[:, 0].data.long()).sum() 45 | 46 | if i == 0: 47 | tempC = predict_label[0].data 48 | tempR = predict_label[1].data 49 | temp = np.concatenate((tempC, tempR), axis=1) 50 | human_prediction = temp 51 | 52 | elif i > 0: 53 | tempC = predict_label[0].data 54 | tempR = predict_label[1].data 55 | temp = np.concatenate((tempC, tempR), axis=1) 56 | human_prediction = np.concatenate((human_prediction, temp), axis=0) 57 | 58 | print('Accuracy:', 100 * correct / len(test_data_loader.dataset)) 59 | sio.savemat('results/test_93.mat', {'human_prediction': human_prediction}) 60 | 61 | -------------------------------------------------------------------------------- /train.py: -------------------------------------------------------------------------------- 1 | import scipy.io as sio 2 | from torch.utils.data import TensorDataset, DataLoader 3 | import numpy as np 4 | import torch 5 | import torch.nn as nn 6 | from torch.autograd import Variable 7 | import torch.nn.functional as F 8 | import matplotlib.pyplot as plt 9 | import math 10 | import time 11 | from tqdm import tqdm 12 | 13 | # from resnet_upsample import ResNet, ResidualBlock, Bottleneck 14 | # from resnet_upsample3x3 import ResNet, ResidualBlock, Bottleneck 15 | # from inception3 import * 16 | # from inceptionv4 import * 17 | # from vgg_net import * 18 | # from alex_net import AlexNet 19 | # 20 | # res_net_use_this, csinet1.0 21 | from model.res_net_use_this import ResNet, ResidualBlock, Bottleneck 22 | # resnet_generation_upsample, csinet1.5 23 | # from resnet_generation_upsample import ResNet, ResidualBlock, Bottleneck 24 | 25 | 26 | batch_size = 20 27 | num_epochs = 20 28 | learning_rate = 0.001 29 | 30 | 31 | # load data 32 | data = sio.loadmat('data/aug_train_data.mat') 33 | train_data = data['aug_train_data'] 34 | train_label = data['aug_train_label'] 35 | 36 | # label matrix organized as nSamplex5, where the 1st coloum is the index of personID, the latter 4 are 4 biometrcs 37 | train_label[:, 0] = train_label[:, 0] - 1 # 1--30 -> 0--29 38 | 39 | num_train_instances = len(train_data) 40 | # prepare data, nSample x nChannel x width x height 41 | # reshape train data size to nSample x nSubcarrier x 1 x 1 42 | train_data = torch.from_numpy(train_data).type(torch.FloatTensor).view(num_train_instances, 30, 1, 1) 43 | train_label = torch.from_numpy(train_label).type(torch.FloatTensor) 44 | train_dataset = TensorDataset(train_data, train_label) 45 | train_data_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True) 46 | 47 | 48 | # load test data 49 | data = sio.loadmat('data/test.mat') 50 | test_data = data['test_data'] 51 | test_label = data['test_label'] 52 | test_label[:, 0] = test_label[:, 0] - 1 53 | 54 | num_test_instances = len(test_data) 55 | # prepare data, nSample x nChannel x width x height 56 | # reshape test data size to nSample x nSubcarrier x 1 x 1 57 | test_data = torch.from_numpy(test_data).type(torch.FloatTensor).view(num_test_instances, 30, 1, 1) 58 | test_label = torch.from_numpy(test_label).type(torch.FloatTensor) 59 | test_dataset = TensorDataset(test_data, test_label) 60 | test_data_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=False) 61 | 62 | 63 | resnet = ResNet(ResidualBlock, [2, 2, 2, 2], 30) 64 | # resnet = ResNet(ResidualBlock, [3, 4, 6, 3], 30) 65 | # resnet = ResNet(Bottleneck, [3, 4, 6, 3], 10) 66 | # resnet = ResNet(Bottleneck, [3, 4, 23, 3], 30) 67 | # inception = InceptionV4(30) 68 | # vgg = VGG(make_layers(cfg['E'], batch_norm=True)) 69 | # alexnet = AlexNet().cuda() 70 | # vgg = vgg.cuda() 71 | # alexnet = alexnet.cuda() 72 | # alexnet.eval() 73 | 74 | resnet = resnet.cuda() 75 | 76 | criterion1 = nn.CrossEntropyLoss().cuda() 77 | criterion2 = nn.L1Loss().cuda() 78 | optimizer = torch.optim.Adam(resnet.parameters(), lr=learning_rate) 79 | scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[3, 6, 9, 12, 15, 18], gamma=0.3) 80 | 81 | for epoch in range(num_epochs): 82 | print('Epoch:', epoch) 83 | resnet.train() 84 | 85 | scheduler.step() 86 | # trained_num = 0 87 | for (samples, labels) in tqdm(train_data_loader): 88 | 89 | # sample_len = len(samples) 90 | # trained_num += sample_len 91 | # print('Process', 100*trained_num/num_train_instances) 92 | 93 | samplesV = Variable(samples.cuda()) 94 | labels = labels.squeeze() 95 | labelsV = Variable(labels.cuda()) 96 | 97 | # Forward + Backward + Optimize 98 | optimizer.zero_grad() 99 | predict_label = resnet(samplesV) 100 | 101 | lossC = criterion1(predict_label[0], labelsV[:, 0].type(torch.LongTensor).cuda()) 102 | 103 | lossR1 = criterion2(predict_label[1][:, 0], labelsV[:, 1]) 104 | lossR2 = criterion2(predict_label[1][:, 1], labelsV[:, 2]) 105 | lossR3 = criterion2(predict_label[1][:, 2], labelsV[:, 3]) 106 | lossR4 = criterion2(predict_label[1][:, 3], labelsV[:, 4]) 107 | 108 | loss = lossC + (0.0386*lossR1 + 0.0405*lossR2 + 0.0629*lossR3 + 0.0877*lossR4)/4 109 | # Why 0.0386, 0.0405, 0.06029 and 0.0877: these fours are used to normalize four body biometrics 110 | # fat/muscle/water/bone rates, for example, if looking paper Table 6, where we showed the information of 111 | # 30 recruited subjects. The minimal fat rate is 5, the maximum is 30.9, we decided to normarlize the fat rate 112 | # by dividing (31-5), resulting in 0.0386. 0.0405->[65,90]muscle rate, 0.0629->[49,65]water rate, 0.0877->[1.5 13.0] 113 | # We doing this was spired by Faster RCNN loss, which has a object classfication and a bounding box regression. As paper 114 | # said, they normalized the regression loss. 115 | # 116 | # print(loss.item()) 117 | # loss_every += loss.item() 118 | loss.backward() 119 | optimizer.step() 120 | # # 121 | resnet.eval() 122 | correct_t = 0 123 | for (samples, labels) in tqdm(train_data_loader): 124 | with torch.no_grad(): 125 | samplesV = Variable(samples.cuda()) 126 | labelsV = Variable(labels.cuda()) 127 | # labelsV = labelsV.view(-1) 128 | 129 | predict_label = resnet(samplesV) 130 | prediction = predict_label[0].data.max(1)[1] 131 | correct_t += prediction.eq(labelsV[:, 0].data.long()).sum() 132 | 133 | print("Training accuracy:", (100*float(correct_t)/num_train_instances)) 134 | 135 | trainacc = str(100*float(correct_t)/num_train_instances)[0:6] 136 | 137 | correct_t = 0 138 | for (samples, labels) in tqdm(test_data_loader): 139 | with torch.no_grad(): 140 | samplesV = Variable(samples.cuda()) 141 | labelsV = Variable(labels.cuda()) 142 | # labelsV = labelsV.view(-1) 143 | 144 | predict_label = resnet(samplesV) 145 | prediction = predict_label[0].data.max(1)[1] 146 | correct_t += prediction.eq(labelsV[:, 0].data.long()).sum() 147 | 148 | print("Test accuracy:", (100 * float(correct_t) / num_test_instances)) 149 | 150 | testacc = str(100 * float(correct_t) / num_test_instances)[0:6] 151 | 152 | torch.save(resnet, 'weights/resnet18_Train' + trainacc + 'Test' + testacc + '.pkl') 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | # 163 | 164 | # 165 | # batch_size = 20 166 | # # load data 167 | # data = sio.loadmat('../dataset/test.mat') 168 | # test_data = data['test_data'] 169 | # # test_label = data['test_label_bin'] 170 | # test_label = data['test_label'] 171 | # 172 | # test_label = test_label - 1 173 | # 174 | # # prepare data, nSample x nChannel x width x height 175 | # 176 | # # reshape train data size to nSample x nSubcarrier x 1 x 1 177 | # num_test_instances = len(test_data) 178 | # 179 | # test_data = torch.from_numpy(test_data).type(torch.FloatTensor).view(num_test_instances, 30, 1, 1) 180 | # test_label = torch.from_numpy(test_label).type(torch.FloatTensor) 181 | # test_dataset = TensorDataset(test_data, test_label) 182 | # test_data_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=False) 183 | # 184 | # resnet = torch.load('model/res34.pkl') 185 | # resnet = resnet.cuda() 186 | # 187 | # resnet.eval() 188 | # 189 | # correct = 0 190 | # 191 | # for i, (samples, labels) in enumerate(test_data_loader): 192 | # with torch.no_grad(): 193 | # samplesV = Variable(samples.cuda()) 194 | # labelsV = Variable(labels.cuda()) 195 | # 196 | # predict_label = resnet(samplesV) 197 | # 198 | # pred = predict_label.data.max(1)[1] 199 | # correct += pred.eq(labelsV[:, 0].data.long()).sum() 200 | 201 | # if i == 0: 202 | # temp = predict_label.data 203 | # fallen_pred = temp 204 | # 205 | # elif i > 0: 206 | # temp = predict_label.data 207 | # fallen_pred = np.concatenate((fallen_pred, temp), axis=0) 208 | # 209 | # sio.savemat('results/test_result.mat', {'fallen_pred': fallen_pred}) 210 | 211 | # print('Accuracy:', 100 * correct / num_test_instances) 212 | --------------------------------------------------------------------------------