├── .DS_Store
├── .idea
├── PCRL.iml
├── inspectionProfiles
│ └── profiles_settings.xml
├── misc.xml
├── modules.xml
└── workspace.xml
├── README.md
└── pytorch
├── .DS_Store
├── BalancedDataParallel.py
├── __pycache__
├── losses.cpython-37.pyc
├── memory_c2l.cpython-37.pyc
└── utils.cpython-37.pyc
├── config.py
├── data.py
├── datasets
├── .DS_Store
├── __init__.py
├── __pycache__
│ ├── __init__.cpython-37.pyc
│ ├── chest_fine_tune.cpython-37.pyc
│ ├── chest_pre_task.cpython-37.pyc
│ ├── luna_fine_tune.cpython-37.pyc
│ └── luna_pre_task.cpython-37.pyc
├── luna_pcrl_pretask.py
└── reverse_pcrl_pretask.py
├── initiater.py
├── losses.py
├── main.py
├── memory_c2l.py
├── models
├── __init__.py
├── __pycache__
│ ├── __init__.cpython-37.pyc
│ ├── brats_test_model.cpython-37.pyc
│ ├── resnet_mixup.cpython-37.pyc
│ └── unet3d2.cpython-37.pyc
├── pcrl_model.py
├── pcrl_model_3d.py
├── resnet.py
└── resnet_mixup.py
├── pcrl.py
├── pcrl_3d.py
├── preprocess
└── luna_pre.py
├── train_val_txt
├── __init__.py
├── chest_test.txt
├── chest_train.txt
├── chest_valid.txt
└── luna_train.txt
└── utils.py
/.DS_Store:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Luchixiang/PCRL/197bf91d4872f88742686d159892904c965ebdf6/.DS_Store
--------------------------------------------------------------------------------
/.idea/PCRL.iml:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
11 |
12 |
13 |
14 |
15 |
--------------------------------------------------------------------------------
/.idea/inspectionProfiles/profiles_settings.xml:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
--------------------------------------------------------------------------------
/.idea/misc.xml:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
--------------------------------------------------------------------------------
/.idea/modules.xml:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
--------------------------------------------------------------------------------
/.idea/workspace.xml:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
11 |
12 |
13 |
14 |
15 |
16 |
17 |
18 |
19 |
22 |
23 |
24 |
25 |
26 |
31 |
32 |
33 |
34 |
35 |
36 |
37 |
38 |
39 |
40 |
41 |
42 |
43 |
44 |
45 |
46 |
47 |
48 |
49 |
50 |
51 |
52 |
62 |
63 |
64 |
65 |
66 |
67 |
68 |
69 |
70 |
71 |
72 |
73 |
74 |
75 |
76 |
77 |
78 |
79 |
80 |
81 |
82 |
83 |
84 |
85 |
86 |
87 |
88 |
89 |
90 |
91 |
92 |
93 |
94 |
95 |
96 |
97 |
98 |
99 |
100 |
101 |
102 |
103 |
104 |
105 |
106 |
107 |
108 |
109 |
110 |
113 |
114 |
115 |
116 |
117 |
118 |
119 |
120 |
121 | 1627379380531
122 |
123 |
124 | 1627379380531
125 |
126 |
127 |
128 |
129 |
130 |
131 |
132 |
133 |
134 |
135 |
136 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Preservational Self-supervised Learning
2 | This repo is the official implementation of our ICCV 2021 paper titled "Preservational Learning Improves Self-supervised Medical Image Models by Reconstructing Diverse Contexts". In this repo, we demonstrate how to use PCLR to conduct pre-training on NIH ChestX-ray14 (2D) and LUNA (3D). The employed backbones are ResNet-18 and 3D U-Net, respectively. Note that this repo contains an improved version of our ICCV paper, which means it is possible to achieve higher results using codes in this repo. Also, we made some modifications, such as replacing the outer-product operation in transformation-conditioned attention with channel-wise multiplication, which results in more stable testing results.
3 | ### Dependency
4 | Please install PyTorch (>=1.1) before you run the code. We strongly recommend you to install Anaconda3 where we use Python 3.6. In addition, we use [apex](https://github.com/NVIDIA/apex) for acceleration. We also use [pretrained-models.pytorch](https://github.com/Cadene/pretrained-models.pytorch) and [segmentation_models_pytorch](https://github.com/qubvel/segmentation_models.pytorch) for convenience.
5 |
6 | ### NIH ChestX-ray14 (Chest14)
7 |
8 | #### Step 0
9 |
10 | Please download Chest X-rays from [this link](https://nihcc.app.box.com/v/ChestXray-NIHCC).
11 |
12 | The image folder of Chest14 should look like this:
13 |
14 | ```.python
15 | ./Chest14
16 | images/
17 | 00002639_006.png
18 | 00010571_003.png
19 | ...
20 | ```
21 |
22 | Besides, we also provide the list of training image in ``pytorch/train_val_txt/chest_train.txt``.
23 |
24 | #### Step 1
25 | > git clone https://github.com/Luchixiang/PCRL.git
26 | >
27 | > cd PCRL/pytorch/
28 |
29 | #### Step 2
30 |
31 | > python main.py --data chest14_data_path --phase pretask --model pcrl --b 64 --epochs 240 --lr 1e-3 --output pretrained_model_save_path --optimizer sgd --outchannel 3 --n chest --d 2 --gpus 0,1,2,3 --inchannel 3 --ratio 1.0
32 |
33 | ``--data`` defines the path where you store Chest14.
34 |
35 | ``--d`` defines the type of dataset, ``2`` stands for 2D while ``3`` denotes 3D.
36 |
37 | ``--n`` gives the name of dataset.
38 |
39 | ``--ratio`` determines the percentages of images in the training set for pretraining. Here, ``1`` means using all training images in the training set to for pretraining.
40 |
41 | ### LUNA16
42 |
43 | #### Step 0
44 |
45 | Please download LUNA16 from [this link](https://luna16.grand-challenge.org/Download/)
46 |
47 | The image folder of LUNA16 should looks like this:
48 |
49 | ```python
50 | ./LUNA16
51 | subset0
52 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.979083010707182900091062408058.raw
53 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.979083010707182900091062408058.mhd
54 | ...
55 | subset1
56 | subset2
57 | ...
58 | subset9
59 | ```
60 |
61 | We also provide the list of training image in ``pytorch/train_val_txt/luna_train.txt``.
62 |
63 | #### Step1
64 |
65 | > git clone https://github.com/Luchixiang/PCRL.git
66 | >
67 | > cd PCLR/pytorch
68 |
69 | #### Step 2
70 |
71 | First, you should pre-process the LUNA dataset to get cropped pairs from 3D images.
72 |
73 | > python preprocess/luna_pre.py --input_rows 64 --input_cols 64 --input_deps 32 --data LUNA_dataset_path --save processedLUNA_save_path
74 |
75 | #### Step3
76 |
77 | > python main.py --data processed LUNA_save_path --phase pretask --model pcrl --b 16 --epochs 240 --lr 1e-3 --output pretrained_model_save_path --optimizer sgd --outchannel 1 --n luna --d 3 --gpus 0,1,2,3 --inchannel 1 --ratio 1.0
78 |
79 |
--------------------------------------------------------------------------------
/pytorch/.DS_Store:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Luchixiang/PCRL/197bf91d4872f88742686d159892904c965ebdf6/pytorch/.DS_Store
--------------------------------------------------------------------------------
/pytorch/BalancedDataParallel.py:
--------------------------------------------------------------------------------
1 | import torch
2 | from torch.nn.parallel.data_parallel import DataParallel
3 | from torch.nn.parallel.parallel_apply import parallel_apply
4 | from torch.nn.parallel._functions import Scatter
5 |
6 |
7 | def scatter(inputs, target_gpus, chunk_sizes, dim=0):
8 | r"""
9 | Slices tensors into approximately equal chunks and
10 | distributes them across given GPUs. Duplicates
11 | references to objects that are not tensors.
12 | """
13 |
14 | def scatter_map(obj):
15 | if isinstance(obj, torch.Tensor):
16 | try:
17 | return Scatter.apply(target_gpus, chunk_sizes, dim, obj)
18 | except Exception:
19 | print('obj', obj.size())
20 | print('dim', dim)
21 | print('chunk_sizes', chunk_sizes)
22 | quit()
23 | if isinstance(obj, tuple) and len(obj) > 0:
24 | return list(zip(*map(scatter_map, obj)))
25 | if isinstance(obj, list) and len(obj) > 0:
26 | return list(map(list, zip(*map(scatter_map, obj))))
27 | if isinstance(obj, dict) and len(obj) > 0:
28 | return list(map(type(obj), zip(*map(scatter_map, obj.items()))))
29 | return [obj for targets in target_gpus]
30 |
31 | # After scatter_map is called, a scatter_map cell will exist. This cell
32 | # has a reference to the actual function scatter_map, which has references
33 | # to a closure that has a reference to the scatter_map cell (because the
34 | # fn is recursive). To avoid this reference cycle, we set the function to
35 | # None, clearing the cell
36 | try:
37 | return scatter_map(inputs)
38 | finally:
39 | scatter_map = None
40 |
41 |
42 | def scatter_kwargs(inputs, kwargs, target_gpus, chunk_sizes, dim=0):
43 | """Scatter with support for kwargs dictionary"""
44 | inputs = scatter(inputs, target_gpus, chunk_sizes, dim) if inputs else []
45 | kwargs = scatter(kwargs, target_gpus, chunk_sizes, dim) if kwargs else []
46 | if len(inputs) < len(kwargs):
47 | inputs.extend([() for _ in range(len(kwargs) - len(inputs))])
48 | elif len(kwargs) < len(inputs):
49 | kwargs.extend([{} for _ in range(len(inputs) - len(kwargs))])
50 | inputs = tuple(inputs)
51 | kwargs = tuple(kwargs)
52 | return inputs, kwargs
53 |
54 |
55 | class BalancedDataParallel(DataParallel):
56 |
57 | def __init__(self, gpu0_bsz, *args, **kwargs):
58 | self.gpu0_bsz = gpu0_bsz
59 | super().__init__(*args, **kwargs)
60 |
61 | def forward(self, *inputs, **kwargs):
62 | if not self.device_ids:
63 | return self.module(*inputs, **kwargs)
64 | if self.gpu0_bsz == 0:
65 | device_ids = self.device_ids[1:]
66 | else:
67 | device_ids = self.device_ids
68 | inputs, kwargs = self.scatter(inputs, kwargs, device_ids)
69 | if len(self.device_ids) == 1:
70 | return self.module(*inputs[0], **kwargs[0])
71 | replicas = self.replicate(self.module, self.device_ids)
72 | if self.gpu0_bsz == 0:
73 | replicas = replicas[1:]
74 | outputs = self.parallel_apply(replicas, device_ids, inputs, kwargs)
75 | return self.gather(outputs, self.output_device)
76 |
77 | def parallel_apply(self, replicas, device_ids, inputs, kwargs):
78 | return parallel_apply(replicas, inputs, kwargs, device_ids)
79 |
80 | def scatter(self, inputs, kwargs, device_ids):
81 | bsz = inputs[0].size(self.dim)
82 | num_dev = len(self.device_ids)
83 | gpu0_bsz = self.gpu0_bsz
84 | bsz_unit = (bsz - gpu0_bsz) // (num_dev - 1)
85 | if gpu0_bsz < bsz_unit:
86 | chunk_sizes = [gpu0_bsz] + [bsz_unit] * (num_dev - 1)
87 | delta = bsz - sum(chunk_sizes)
88 | for i in range(delta):
89 | chunk_sizes[i + 1] += 1
90 | if gpu0_bsz == 0:
91 | chunk_sizes = chunk_sizes[1:]
92 | else:
93 | return super().scatter(inputs, kwargs, device_ids)
94 | return scatter_kwargs(inputs, kwargs, device_ids, chunk_sizes, dim=self.dim)
--------------------------------------------------------------------------------
/pytorch/__pycache__/losses.cpython-37.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Luchixiang/PCRL/197bf91d4872f88742686d159892904c965ebdf6/pytorch/__pycache__/losses.cpython-37.pyc
--------------------------------------------------------------------------------
/pytorch/__pycache__/memory_c2l.cpython-37.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Luchixiang/PCRL/197bf91d4872f88742686d159892904c965ebdf6/pytorch/__pycache__/memory_c2l.cpython-37.pyc
--------------------------------------------------------------------------------
/pytorch/__pycache__/utils.cpython-37.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Luchixiang/PCRL/197bf91d4872f88742686d159892904c965ebdf6/pytorch/__pycache__/utils.cpython-37.pyc
--------------------------------------------------------------------------------
/pytorch/config.py:
--------------------------------------------------------------------------------
1 | import os
2 | import shutil
3 |
4 |
5 | class models_genesis_config:
6 | model = "Unet3D"
7 | suffix = "genesis_chest_ct"
8 | exp_name = model + "-" + suffix
9 |
10 | # data
11 | data = "/data1/luchixiang/LUNA16/processed"
12 | distributed = False
13 | train_fold = [0, 1, 2, 3, 4, 5, 6]
14 | valid_fold = [7, 8, 9]
15 | test_fold = [7, 8, 9]
16 | hu_min = -1000.0
17 | hu_max = 1000.0
18 | scale = 32
19 | input_rows = 64
20 | input_cols = 64
21 | input_deps = 32
22 | nb_class = 1
23 | fine_tune_epoch = 200
24 | ratio = 0.8
25 | root = '/data1/luchixiang/LUNA16'
26 | # model pre-training
27 | verbose = 1
28 | weights = None
29 | batch_size = 64
30 | optimizer = "sgd"
31 | workers = 4
32 | max_queue_size = workers * 4
33 | save_samples = "png"
34 | nb_epoch = 10000
35 | patience = 50
36 | lr = 1
37 |
38 | # image deformation
39 | nonlinear_rate = 0.9
40 | paint_rate = 0.9
41 | outpaint_rate = 0.8
42 | inpaint_rate = 1.0 - outpaint_rate
43 | local_rate = 0.5
44 | flip_rate = 0.4
45 | rotate_rate = 0.4
46 | gauss_rate = 0.4
47 |
48 | # logs
49 | model_path = "pretrained_weights"
50 | if not os.path.exists(model_path):
51 | os.makedirs(model_path)
52 | logs_path = os.path.join(model_path, "Logs")
53 | if not os.path.exists(logs_path):
54 | os.makedirs(logs_path)
55 |
56 | def display(self):
57 | """Display Configuration values."""
58 | print("\nConfigurations:")
59 | for a in dir(self):
60 | if not a.startswith("__") and not callable(getattr(self, a)):
61 | print("{:30} {}".format(a, getattr(self, a)))
62 | print("\n")
63 |
--------------------------------------------------------------------------------
/pytorch/data.py:
--------------------------------------------------------------------------------
1 | from torch.utils.data import DataLoader
2 |
3 | from datasets import *
4 | from utils import *
5 | from torchvision import transforms, datasets
6 | import torch
7 | import torchio.transforms
8 |
9 |
10 | class DataGenerator:
11 |
12 | def __init__(self, config):
13 | self.config = config
14 |
15 | def pcrl_luna_pretask(self):
16 | print('using the reverse_aug pretrain on luna')
17 | config = self.config
18 | dataloader = {}
19 | train_fold = [0, 1, 2, 3, 4, 5, 6, ]
20 | valid_fold = [7, 8, 9]
21 | file_list = get_luna_pretrain_list(config.ratio, config.root, train_fold)
22 | x_train, x_valid, _ = get_luna_list(config, train_fold, valid_fold, valid_fold, suffix='_img_',
23 | file_list=file_list)
24 | print(f'total train images {len(x_train)}, valid images {len(x_valid)}')
25 | transforms = [torchio.transforms.RandomFlip(),
26 | torchio.transforms.RandomAffine(),
27 | torchio.transforms.RandomBlur(),
28 | ]
29 | transforms = torchio.transforms.Compose(transforms)
30 | train_ds = PCRLLunaPretask(config, x_train, train=True, transform=transforms)
31 | valid_ds = PCRLLunaPretask(config, x_valid, train=False)
32 | dataloader['train'] = DataLoader(train_ds, batch_size=config.batch_size,
33 | pin_memory=True, shuffle=True, num_workers=config.workers)
34 | dataloader['eval'] = DataLoader(valid_ds, batch_size=config.batch_size,
35 | pin_memory=True, shuffle=False, num_workers=config.workers)
36 | dataloader['test'] = dataloader['eval']
37 | return dataloader
38 |
39 | def pcrl_chest_pretask(self):
40 | print('using reverse aug on chest')
41 | conf = self.config
42 |
43 | image_size = 224
44 | mean = [0.485, 0.456, 0.406]
45 | std = [0.229, 0.224, 0.225]
46 | normalize = transforms.Normalize(mean=mean, std=std)
47 | spatial_transform = transforms.Compose([
48 | transforms.RandomResizedCrop(224, scale=(0.2, 1)),
49 | ])
50 | train_transform = transforms.Compose([
51 | transforms.RandomRotation(10),
52 | transforms.RandomHorizontalFlip(),
53 | transforms.RandomGrayscale(p=0.2),
54 | transforms.RandomApply([GaussianBlur()], p=0.5),
55 | transforms.ColorJitter(0.4, 0.4, 0.4, 0.4),
56 | transforms.ToTensor(),
57 | normalize,
58 | ])
59 | train_transform.transforms.append(Cutout(n_holes=3, length=32))
60 | train_file = './train_val_txt/chest_train.txt'
61 | train_imgs, train_labels = get_chest_list(train_file, conf.data)
62 | train_imgs = train_imgs[:int(len(train_imgs) * conf.ratio)]
63 | train_dataset = PCRLChestPretask(conf, train_imgs, transform=train_transform, train=True,
64 | spatial_transform=spatial_transform)
65 | print(len(train_dataset))
66 | # train_sampler = torch.utils.data.distributed.DistributedSampler(train_dataset)
67 | train_sampler = None
68 | dataloader = {}
69 | train_loader = torch.utils.data.DataLoader(
70 | train_dataset, batch_size=conf.batch_size, shuffle=(train_sampler is None),
71 | num_workers=conf.workers, pin_memory=True, sampler=train_sampler)
72 | dataloader['train'] = train_loader
73 | dataloader['eval'] = train_loader
74 | return dataloader
75 |
--------------------------------------------------------------------------------
/pytorch/datasets/.DS_Store:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Luchixiang/PCRL/197bf91d4872f88742686d159892904c965ebdf6/pytorch/datasets/.DS_Store
--------------------------------------------------------------------------------
/pytorch/datasets/__init__.py:
--------------------------------------------------------------------------------
1 | from .reverse_pcrl_pretask import PCRLChestPretask
2 | from .luna_pcrl_pretask import PCRLLunaPretask
3 |
4 |
--------------------------------------------------------------------------------
/pytorch/datasets/__pycache__/__init__.cpython-37.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Luchixiang/PCRL/197bf91d4872f88742686d159892904c965ebdf6/pytorch/datasets/__pycache__/__init__.cpython-37.pyc
--------------------------------------------------------------------------------
/pytorch/datasets/__pycache__/chest_fine_tune.cpython-37.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Luchixiang/PCRL/197bf91d4872f88742686d159892904c965ebdf6/pytorch/datasets/__pycache__/chest_fine_tune.cpython-37.pyc
--------------------------------------------------------------------------------
/pytorch/datasets/__pycache__/chest_pre_task.cpython-37.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Luchixiang/PCRL/197bf91d4872f88742686d159892904c965ebdf6/pytorch/datasets/__pycache__/chest_pre_task.cpython-37.pyc
--------------------------------------------------------------------------------
/pytorch/datasets/__pycache__/luna_fine_tune.cpython-37.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Luchixiang/PCRL/197bf91d4872f88742686d159892904c965ebdf6/pytorch/datasets/__pycache__/luna_fine_tune.cpython-37.pyc
--------------------------------------------------------------------------------
/pytorch/datasets/__pycache__/luna_pre_task.cpython-37.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Luchixiang/PCRL/197bf91d4872f88742686d159892904c965ebdf6/pytorch/datasets/__pycache__/luna_pre_task.cpython-37.pyc
--------------------------------------------------------------------------------
/pytorch/datasets/luna_pcrl_pretask.py:
--------------------------------------------------------------------------------
1 | import copy
2 | import random
3 | import time
4 |
5 | import numpy as np
6 | import torch
7 | from PIL import Image
8 | from scipy.special import comb
9 | from torch.utils.data import Dataset
10 | import torchio.transforms
11 |
12 |
13 | class PCRLLunaPretask(Dataset):
14 | def __init__(self, config, img_train, train=False, transform=None):
15 | self.config = config
16 | self.imgs = img_train
17 | self.train = train
18 | self.transform = transform
19 | self.norm = torchio.transforms.ZNormalization()
20 |
21 | def __len__(self):
22 | return len(self.imgs)
23 |
24 | def __getitem__(self, index):
25 | image_name = self.imgs[index]
26 | pair = np.load(image_name)
27 | crop1 = pair[0]
28 | crop1 = np.expand_dims(crop1, axis=0)
29 | crop2 = pair[1]
30 | crop2 = np.expand_dims(crop2, axis=0)
31 | gt1 = copy.deepcopy(crop1)
32 | gt2 = copy.deepcopy(crop2)
33 | # gt1 = torch.tensor(gt1, dtype=torch.float)
34 | # gt2 = torch.tensor(gt2, dtype=torch.float)
35 | input1 = self.transform(crop1)
36 | input2 = self.transform(crop2)
37 | input1 = self.local_pixel_shuffling(input1, prob=self.config.local_rate)
38 | input2 = self.local_pixel_shuffling(input2, prob=self.config.local_rate)
39 | # input1 = self.nonlinear_transformation(input1, self.config.nonlinear_rate)
40 | # input2 = self.nonlinear_transformation(input2, self.config.nonlinear_rate)
41 | if random.random() < self.config.paint_rate:
42 | if random.random() < self.config.inpaint_rate:
43 | # Inpainting
44 | input1 = self.image_in_painting(input1)
45 | input2 = self.image_in_painting(input2)
46 | else:
47 | # Outpainting
48 | input1 = self.image_out_painting(input1)
49 | input2 = self.image_out_painting(input2)
50 | mask1 = copy.deepcopy(input1)
51 | mask2 = copy.deepcopy(input2)
52 | mask1, aug_tensor1 = self.spatial_aug(mask1)
53 | mask2, aug_tensor2 = self.spatial_aug(mask2)
54 |
55 | return torch.tensor(input1, dtype=torch.float), torch.tensor(input2, dtype=torch.float), \
56 | torch.tensor(mask1, dtype=torch.float), \
57 | torch.tensor(mask2, dtype=torch.float), \
58 | torch.tensor(gt1, dtype=torch.float), \
59 | torch.tensor(gt2, dtype=torch.float), aug_tensor1, aug_tensor2
60 |
61 | def spatial_aug(self, img):
62 | # img = img.numpy()
63 | aug_tensor = [0 for _ in range(7)]
64 | if random.random() < 0.5:
65 | img = np.flip(img, axis=1)
66 | aug_tensor[0] = 1
67 | if random.random() < 0.5:
68 | img = np.flip(img, axis=2)
69 | aug_tensor[1] = 1
70 | if random.random() < 0.5:
71 | img = np.flip(img, axis=3)
72 | aug_tensor[2] = 1
73 | times = int(random.random() // 0.25)
74 | img = np.rot90(img, k=times, axes=(1, 2))
75 | aug_tensor[times + 3] = 1
76 | return torch.tensor(img.copy(), dtype=torch.float), torch.tensor(aug_tensor)
77 |
78 | def bernstein_poly(self, i, n, t):
79 | """
80 | The Bernstein polynomial of n, i as a function of t
81 | """
82 |
83 | return comb(n, i) * (t ** (n - i)) * (1 - t) ** i
84 |
85 | def bezier_curve(self, points, nTimes=1000):
86 | """
87 | Given a set of control points, return the
88 | bezier curve defined by the control points.
89 |
90 | Control points should be a list of lists, or list of tuples
91 | such as [ [1,1],
92 | [2,3],
93 | [4,5], ..[Xn, Yn] ]
94 | nTimes is the number of time steps, defaults to 1000
95 |
96 | See http://processingjs.nihongoresources.com/bezierinfo/
97 | """
98 |
99 | nPoints = len(points)
100 | xPoints = np.array([p[0] for p in points])
101 | yPoints = np.array([p[1] for p in points])
102 |
103 | t = np.linspace(0.0, 1.0, nTimes)
104 |
105 | polynomial_array = np.array([self.bernstein_poly(i, nPoints - 1, t) for i in range(0, nPoints)])
106 |
107 | xvals = np.dot(xPoints, polynomial_array)
108 | yvals = np.dot(yPoints, polynomial_array)
109 |
110 | return xvals, yvals
111 |
112 | def data_augmentation(self, x, y, prob=0.5):
113 | # augmentation by flipping
114 | cnt = 3
115 | while random.random() < prob and cnt > 0:
116 | degree = random.choice([0, 1, 2])
117 | x = np.flip(x, axis=degree)
118 | y = np.flip(y, axis=degree)
119 | cnt = cnt - 1
120 |
121 | return x, y
122 |
123 | def nonlinear_transformation(self, x, prob=0.5):
124 | if random.random() >= prob:
125 | return x
126 | points = [[0, 0], [random.random(), random.random()], [random.random(), random.random()], [1, 1]]
127 | xpoints = [p[0] for p in points]
128 | ypoints = [p[1] for p in points]
129 | xvals, yvals = self.bezier_curve(points, nTimes=100000)
130 | if random.random() < 0.5:
131 | # Half change to get flip
132 | xvals = np.sort(xvals)
133 | else:
134 | xvals, yvals = np.sort(xvals), np.sort(yvals)
135 | nonlinear_x = np.interp(x, xvals, yvals)
136 | return nonlinear_x
137 |
138 | def local_pixel_shuffling(self, x, prob=0.5):
139 | if random.random() >= prob:
140 | return x
141 | image_temp = copy.deepcopy(x)
142 | orig_image = copy.deepcopy(x)
143 | _, img_rows, img_cols, img_deps = x.shape
144 | num_block = 10000
145 | for _ in range(num_block):
146 | block_noise_size_x = random.randint(1, img_rows // 10)
147 | block_noise_size_y = random.randint(1, img_cols // 10)
148 | block_noise_size_z = random.randint(1, img_deps // 10)
149 | noise_x = random.randint(0, img_rows - block_noise_size_x)
150 | noise_y = random.randint(0, img_cols - block_noise_size_y)
151 | noise_z = random.randint(0, img_deps - block_noise_size_z)
152 | window = orig_image[0, noise_x:noise_x + block_noise_size_x,
153 | noise_y:noise_y + block_noise_size_y,
154 | noise_z:noise_z + block_noise_size_z,
155 | ]
156 | window = window.flatten()
157 | np.random.shuffle(window)
158 | window = window.reshape((block_noise_size_x,
159 | block_noise_size_y,
160 | block_noise_size_z))
161 | image_temp[0, noise_x:noise_x + block_noise_size_x,
162 | noise_y:noise_y + block_noise_size_y,
163 | noise_z:noise_z + block_noise_size_z] = window
164 | local_shuffling_x = image_temp
165 |
166 | return local_shuffling_x
167 |
168 | def image_in_painting(self, x):
169 | _, img_rows, img_cols, img_deps = x.shape
170 | cnt = 5
171 | while cnt > 0 and random.random() < 0.95:
172 | block_noise_size_x = random.randint(img_rows // 6, img_rows // 3)
173 | block_noise_size_y = random.randint(img_cols // 6, img_cols // 3)
174 | block_noise_size_z = random.randint(img_deps // 6, img_deps // 3)
175 | noise_x = random.randint(3, img_rows - block_noise_size_x - 3)
176 | noise_y = random.randint(3, img_cols - block_noise_size_y - 3)
177 | noise_z = random.randint(3, img_deps - block_noise_size_z - 3)
178 | x[:,
179 | noise_x:noise_x + block_noise_size_x,
180 | noise_y:noise_y + block_noise_size_y,
181 | noise_z:noise_z + block_noise_size_z] = np.random.rand(block_noise_size_x,
182 | block_noise_size_y,
183 | block_noise_size_z, ) * 1.0
184 | cnt -= 1
185 | return x
186 |
187 | def image_out_painting(self, x):
188 | _, img_rows, img_cols, img_deps = x.shape
189 | image_temp = copy.deepcopy(x)
190 | x = np.random.rand(x.shape[0], x.shape[1], x.shape[2], x.shape[3], ) * 1.0
191 | block_noise_size_x = img_rows - random.randint(3 * img_rows // 7, 4 * img_rows // 7)
192 | block_noise_size_y = img_cols - random.randint(3 * img_cols // 7, 4 * img_cols // 7)
193 | block_noise_size_z = img_deps - random.randint(3 * img_deps // 7, 4 * img_deps // 7)
194 | noise_x = random.randint(3, img_rows - block_noise_size_x - 3)
195 | noise_y = random.randint(3, img_cols - block_noise_size_y - 3)
196 | noise_z = random.randint(3, img_deps - block_noise_size_z - 3)
197 | x[:,
198 | noise_x:noise_x + block_noise_size_x,
199 | noise_y:noise_y + block_noise_size_y,
200 | noise_z:noise_z + block_noise_size_z] = image_temp[:, noise_x:noise_x + block_noise_size_x,
201 | noise_y:noise_y + block_noise_size_y,
202 | noise_z:noise_z + block_noise_size_z]
203 | cnt = 4
204 | while cnt > 0 and random.random() < 0.95:
205 | block_noise_size_x = img_rows - random.randint(3 * img_rows // 7, 4 * img_rows // 7)
206 | block_noise_size_y = img_cols - random.randint(3 * img_cols // 7, 4 * img_cols // 7)
207 | block_noise_size_z = img_deps - random.randint(3 * img_deps // 7, 4 * img_deps // 7)
208 | noise_x = random.randint(3, img_rows - block_noise_size_x - 3)
209 | noise_y = random.randint(3, img_cols - block_noise_size_y - 3)
210 | noise_z = random.randint(3, img_deps - block_noise_size_z - 3)
211 | x[:,
212 | noise_x:noise_x + block_noise_size_x,
213 | noise_y:noise_y + block_noise_size_y,
214 | noise_z:noise_z + block_noise_size_z] = image_temp[:, noise_x:noise_x + block_noise_size_x,
215 | noise_y:noise_y + block_noise_size_y,
216 | noise_z:noise_z + block_noise_size_z]
217 | cnt -= 1
218 | return x
219 |
--------------------------------------------------------------------------------
/pytorch/datasets/reverse_pcrl_pretask.py:
--------------------------------------------------------------------------------
1 | import copy
2 | import random
3 | import time
4 |
5 | import numpy as np
6 | import torch
7 | from PIL import Image
8 | from scipy.special import comb
9 | from torch.utils.data import Dataset
10 | import torchvision.transforms as transforms
11 | import torchvision.transforms.functional as F
12 |
13 |
14 | class PCRLChestPretask(Dataset):
15 | def __init__(self, config, img_train, train, transform=None, spatial_transform=None):
16 | self.config = config
17 | self.imgs = img_train
18 | self.train = train
19 | self.transform = transform
20 | self.spatial_transform = spatial_transform
21 | mean = [0.485, 0.456, 0.406]
22 | std = [0.229, 0.224, 0.225]
23 | self.normalize_trans = transforms.Compose([transforms.ToTensor(),
24 | transforms.Normalize(mean=mean, std=std)])
25 |
26 | def __len__(self):
27 | return len(self.imgs)
28 |
29 | def __getitem__(self, index):
30 | image_name = self.imgs[index]
31 | y = Image.open(image_name).convert('RGB')
32 | y1 = self.spatial_transform(y)
33 | y2 = self.spatial_transform(y)
34 | gt1 = copy.deepcopy(y1)
35 | gt2 = copy.deepcopy(y2)
36 | gt1 = self.normalize_trans(gt1)
37 | gt2 = self.normalize_trans(gt2)
38 | input1 = self.transform(y1)
39 | input2 = self.transform(y2)
40 | mask1 = copy.deepcopy(input1)
41 | mask2 = copy.deepcopy(input2)
42 | mask1, aug_tensor1 = self.aug(mask1)
43 | mask2, aug_tensor2 = self.aug(mask2)
44 |
45 | return input1, input2, mask1, mask2, gt1, gt2, aug_tensor1, aug_tensor2
46 |
47 | def aug(self, img):
48 | img = img.numpy()
49 | aug_tensor = [0 for _ in range(6)]
50 | if random.random() < 0.5:
51 | img = np.flip(img, axis=1)
52 | aug_tensor[0] = 1
53 | if random.random() < 0.5:
54 | img = np.flip(img, axis=2)
55 | aug_tensor[1] = 1
56 | times = int(random.random() // 0.25)
57 | img = np.rot90(img, k=times, axes=(1, 2))
58 | aug_tensor[times + 2] = 1
59 | return torch.tensor(img.copy(), dtype=torch.float), torch.tensor(aug_tensor)
60 |
--------------------------------------------------------------------------------
/pytorch/initiater.py:
--------------------------------------------------------------------------------
1 | from config import models_genesis_config
2 | from data import DataGenerator
3 | from losses import *
4 | from models import *
5 | import torch.nn as nn
6 | import torch.nn.init as init
7 |
8 |
9 | def get_config(args):
10 | conf = models_genesis_config()
11 | conf.batch_size = args.b
12 | conf.lr = args.lr
13 | conf.weights = args.weight
14 | conf.model_path = args.output
15 | conf.nb_epoch = args.epochs
16 | conf.data = args.data
17 | conf.optimizer = args.optimizer
18 | conf.workers = args.workers
19 | conf.nb_class = args.outchannel
20 | conf.patience = args.patience
21 | conf.ratio = args.ratio
22 | conf.root = args.root
23 | return conf
24 |
25 |
26 | def get_dataloader(args, conf):
27 | generator = DataGenerator(conf)
28 | loader_name = args.model + '_' + args.n + '_' + args.phase
29 | print(loader_name)
30 | dataloader = getattr(generator, loader_name)()
31 | return dataloader
32 |
33 |
34 |
35 | def kaiming_normal(net, a=0, mode='fan_in', nonlinearity='relu'):
36 | for m in net.modules():
37 | if isinstance(m, (nn.modules.conv._ConvNd, nn.Linear)):
38 | init.kaiming_normal_(m.weight, a=a, mode=mode, nonlinearity=nonlinearity)
39 | if m.bias is not None:
40 | init.constant_(m.bias, 0.)
41 | else:
42 | pass
43 | return net
44 |
45 |
46 | def get_loss(args):
47 | if args.loss == 'gan':
48 | print('using gan loss')
49 | criterion = [nn.L1Loss(), nn.BCELoss()]
50 | elif args.loss == 'mse':
51 | print('using mse loss')
52 | criterion = nn.MSELoss()
53 | elif args.loss == 'dice':
54 | print('using dice loss')
55 | criterion = dice_loss
56 | elif args.loss == 'thor_dice':
57 | # weights = torch.FloatTensor([0.5, 1.0, 1.0, 1.0, 1.0]).cuda()
58 | # criterion = nn.CrossEntropyLoss(weight=weights, reduction='mean')
59 | criterion = Multi_Soft_Dice_Loss()
60 | elif args.loss == 'bce':
61 | print('using bce loss')
62 | criterion = nn.BCELoss()
63 | elif args.loss == 'semantic':
64 | print('using semantic loss')
65 | criterion = [nn.MSELoss().cuda(), nn.CrossEntropyLoss().cuda()]
66 | elif args.loss == 'softmax':
67 | print('using softmax loss')
68 | criterion = nn.CrossEntropyLoss()
69 | elif args.loss == 'nce':
70 | print('using nce loss')
71 | criterion = NCECriterion(10)
72 | return criterion
73 |
--------------------------------------------------------------------------------
/pytorch/losses.py:
--------------------------------------------------------------------------------
1 | import torch.nn.functional as F
2 | import torch
3 | import torch.nn as nn
4 |
5 |
6 | class Multi_Soft_Dice_Loss(nn.Module):
7 | def __init__(self):
8 | super(Multi_Soft_Dice_Loss, self).__init__()
9 |
10 | def forward(self, result, target, total_classes=5, train=True):
11 | loss = 0.0
12 | # print(result.shape, target.shape)
13 | target = target[:, 0]
14 | # softmax = nn.Softmax(dim = 1)
15 | # result = softmax(result)
16 | for i in range(result.size(0)):
17 | epsilon = 1e-6
18 | Loss = []
19 | weight = [2, 0.4, 0.9, 0.7]
20 | for j in range(1, total_classes):
21 | result_sum = torch.sum(result[i, j, :, :, :])
22 | target_sum = torch.sum(target[i, :, :, :] == j)
23 | # print(target_sum.data)
24 | intersect = torch.sum(result[i, j, :, :, :] * (target[i, :, :, :] == j).float())
25 | # print(result_sum, target_sum, intersect)
26 | dice = (2. * intersect + epsilon) / (target_sum + result_sum + epsilon)
27 | # print("The {} batch's {} class's dice is {}".format(i, j, dice))
28 | Loss.append(1 - dice)
29 | for m in range(4):
30 | if train:
31 | loss += Loss[m] * weight[m]
32 | else:
33 | loss += Loss[m]
34 | return loss / result.size(0)
35 |
36 |
37 | def bceDiceLoss(input, target, train=True):
38 | bce = F.binary_cross_entropy_with_logits(input, target)
39 | smooth = 1e-5
40 | num = target.size(0)
41 | input = input.view(num, -1)
42 | target = target.view(num, -1)
43 | intersection = (input * target)
44 | dice = (2. * intersection.sum(1) + smooth) / (input.sum(1) + target.sum(1) + smooth)
45 | dice = 1 - dice.sum() / num
46 | if train:
47 | return dice + 0.2 * bce
48 | return dice
49 |
50 |
51 | def thor_dice_loss(input, target, train=True):
52 | # print(input.shape, target.shape)
53 | es_dice = bceDiceLoss(input[:, 0], target[:, 0], train)
54 | tra_dice = bceDiceLoss(input[:, 1], target[:, 1], train)
55 | aor_dice = bceDiceLoss(input[:, 2], target[:, 2], train)
56 | heart_dice = bceDiceLoss(input[:, 3], target[:, 3], train)
57 | print(f'label1 dice {es_dice}, label2 dice {tra_dice}, label3 dice{aor_dice}, label4 dice{heart_dice}')
58 | return es_dice + tra_dice + aor_dice + heart_dice
59 |
60 |
61 | def dice_loss(input, target, train=True):
62 | wt_loss = bceDiceLoss(input[:, 0], target[:, 0], train)
63 | tc_loss = bceDiceLoss(input[:, 1], target[:, 1], train)
64 | et_loss = bceDiceLoss(input[:, 2], target[:, 2], train)
65 | print(f'wt loss: {wt_loss}, tc_loss : {tc_loss}, et_loss: {et_loss}')
66 | return wt_loss + tc_loss + et_loss
67 |
68 |
69 | class NCECriterion(nn.Module):
70 | """
71 | Eq. (12): L_{NCE}
72 | """
73 |
74 | def __init__(self, n_data):
75 | super(NCECriterion, self).__init__()
76 | self.n_data = n_data
77 |
78 | def forward(self, x):
79 | bsz = x.shape[0]
80 | m = x.size(1) - 1
81 | eps = 1e-5
82 |
83 | # noise distribution
84 | Pn = 1 / float(self.n_data)
85 |
86 | # loss for positive pair
87 | P_pos = x.select(1, 0)
88 | log_D1 = torch.div(P_pos, P_pos.add(m * Pn + eps)).log_()
89 |
90 | # loss for K negative pair
91 | P_neg = x.narrow(1, 1, m)
92 | log_D0 = torch.div(P_neg.clone().fill_(m * Pn), P_neg.add(m * Pn + eps)).log_()
93 |
94 | loss = - (log_D1.sum(0) + log_D0.view(-1, 1).sum(0)) / bsz
95 |
96 | return loss
97 |
98 |
99 | class NCESoftmaxLoss(nn.Module):
100 | """Softmax cross-entropy loss (a.k.a., info-NCE loss in CPC paper)"""
101 |
102 | def __init__(self):
103 | super(NCESoftmaxLoss, self).__init__()
104 | self.criterion = nn.CrossEntropyLoss()
105 |
106 | def forward(self, x):
107 | bsz = x.shape[0]
108 | x = x.squeeze()
109 | label = torch.zeros([bsz]).cuda().long()
110 | loss = self.criterion(x, label)
111 | return loss
112 |
113 |
114 | class NCEKLLoss(nn.Module):
115 | """Softmax cross-entropy loss (a.k.a., info-NCE loss in CPC paper)"""
116 |
117 | def __init__(self):
118 | super(NCEKLLoss, self).__init__()
119 | self.criterion = nn.KLDivLoss()
120 |
121 | def forward(self, x):
122 | bsz = x.shape[0]
123 | x = x.squeeze()
124 | x = F.log_softmax(x, dim=1)
125 | label = torch.zeros([bsz]).cuda().long()
126 | loss = self.criterion(x, label)
127 | return loss
128 |
--------------------------------------------------------------------------------
/pytorch/main.py:
--------------------------------------------------------------------------------
1 | import argparse
2 | import os
3 | import warnings
4 |
5 | import torch.backends.cudnn
6 |
7 | from initiater import *
8 |
9 | import torch.multiprocessing as mp
10 |
11 | from pcrl import train_pcrl_2d
12 | import sys
13 | from pcrl_3d import train_pcrl_3d
14 |
15 | warnings.filterwarnings('ignore')
16 | if __name__ == '__main__':
17 | parser = argparse.ArgumentParser(description='Self Training benchmark')
18 | parser.add_argument('--data', metavar='DIR', default='/data1/luchixiang/LUNA16/processed',
19 | help='path to dataset')
20 | parser.add_argument('--model', metavar='MODEL', default='model_genesis', help='choose the model')
21 | parser.add_argument('--phase', default='pretask', type=str, help='pretask or finetune or train from scratch')
22 | parser.add_argument('--b', default=16, type=int, help='batch size')
23 | parser.add_argument('--weight', default=None, type=str, help='weight to load')
24 | parser.add_argument('--epochs', default=100, type=int, help='epochs to train')
25 | parser.add_argument('--lr', default=1e-3, type=float, help='learning rate')
26 | parser.add_argument('--output', default='./model_genesis_pretrain', type=str, help='output path')
27 | parser.add_argument('--optimizer', default='sgd', type=str, help='optimizer to use')
28 | parser.add_argument('--outchannel', default=1, type=int, help='classes num')
29 | parser.add_argument('--n', default='luna', type=str, help='dataset to use')
30 | parser.add_argument('--d', default=3, type=int, help='3d or 2d to run')
31 | parser.add_argument('--workers', default=4, type=int, help='num of workers')
32 | parser.add_argument('--gpus', default='0,1,2,3', type=str, help='gpu indexs')
33 | parser.add_argument('--loss', default='mse', type=str, help='loss to use')
34 | parser.add_argument('--patience', default=50, type=int, help='patience')
35 | parser.add_argument('--inchannel', default=1, type=int, help='input channels')
36 | parser.add_argument('--ratio', default=0.8, type=float, help='ratio of data used for pretraining')
37 | parser.add_argument('--root', default='/data1/luchixiang/LUNA16', help='root path')
38 | parser.add_argument('--step', default=50, type=int)
39 | parser.add_argument('--norm', default='bn')
40 | parser.add_argument('--momentum', default=0.9)
41 | parser.add_argument('--gamma', default=0.5, type=float)
42 | parser.add_argument('--weight_decay', default=1e-4)
43 | parser.add_argument('--final_act', default='sigmoid')
44 | parser.add_argument('--lr_decay_epochs', type=str, default='160,200,240,280,320',
45 | help='where to decay lr, can be a list')
46 | parser.add_argument('--lr_decay_rate', type=float, default=0.1, help='decay rate for learning rate')
47 | parser.add_argument('--moco_t', default=0.2, type=float)
48 |
49 | args = parser.parse_args()
50 | if not os.path.exists(args.output):
51 | os.makedirs(args.output)
52 | print(args)
53 | os.environ["CUDA_VISIBLE_DEVICES"] = args.gpus
54 | # torch.backends.cudnn.benchmark = True
55 | conf = get_config(args)
56 | data_loader = get_dataloader(args, conf)
57 | criterion = get_loss(args)
58 | train_generator = data_loader['train']
59 | valid_generator = data_loader['eval']
60 | device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
61 | print("Total CUDA devices: ", torch.cuda.device_count())
62 | intial_epoch = 0
63 | sys.stdout.flush()
64 | if args.model == 'pcrl' and args.phase == 'pretask' and args.d == 2:
65 | train_pcrl_2d(args, data_loader)
66 | elif args.model == 'pcrl' and args.phase == 'pretask' and args.d == 3:
67 | train_pcrl_3d(args, data_loader)
68 |
--------------------------------------------------------------------------------
/pytorch/memory_c2l.py:
--------------------------------------------------------------------------------
1 | import torch
2 | from torch import nn
3 | import math
4 |
5 |
6 | class MemoryC2L(nn.Module):
7 | """Fixed-size queue with momentum encoder"""
8 |
9 | def __init__(self, inputSize, outputSize, K, T=0.07, use_softmax=False):
10 | super(MemoryC2L, self).__init__()
11 | self.outputSize = outputSize
12 | self.inputSize = inputSize
13 | self.queueSize = K
14 | self.T = T
15 | self.index = 0
16 | self.use_softmax = use_softmax
17 | self.register_buffer('params', torch.tensor([-1]))
18 | stdv = 1. / math.sqrt(inputSize / 3)
19 | self.register_buffer('memory', torch.rand(self.queueSize, inputSize).mul_(2 * stdv).add_(-stdv))
20 | print('using queue shape: ({},{})'.format(self.queueSize, inputSize))
21 |
22 | def forward(self, q, k):
23 | batchSize = q.shape[0]
24 | k = k.detach()
25 |
26 | Z = self.params[0].item()
27 | # pos logit
28 | l_pos = torch.bmm(q.view(batchSize, 1, -1), k.view(batchSize, -1, 1))
29 | l_pos = l_pos.view(batchSize, 1)
30 | # neg logit
31 | queue = self.memory.clone()
32 | l_neg = torch.mm(queue.detach(), q.transpose(1, 0))
33 | l_neg = l_neg.transpose(0, 1)
34 |
35 | out = torch.cat((l_pos, l_neg), dim=1)
36 |
37 | if self.use_softmax:
38 | out = torch.div(out, self.T)
39 | out = out.squeeze().contiguous()
40 | else:
41 | out = torch.exp(torch.div(out, self.T))
42 | if Z < 0:
43 | self.params[0] = out.mean() * self.outputSize
44 | Z = self.params[0].clone().detach().item()
45 | print("normalization constant Z is set to {:.1f}".format(Z))
46 | # compute the out
47 | out = torch.div(out, Z).squeeze().contiguous()
48 |
49 | # # update memory
50 | with torch.no_grad():
51 | out_ids = torch.arange(batchSize).cuda()
52 | out_ids += self.index
53 | out_ids = torch.fmod(out_ids, self.queueSize)
54 | out_ids = out_ids.long()
55 | self.memory.index_copy_(0, out_ids, k)
56 | self.index = (self.index + batchSize) % self.queueSize
57 |
58 | return out
59 |
--------------------------------------------------------------------------------
/pytorch/models/__init__.py:
--------------------------------------------------------------------------------
1 | from .resnet_mixup import InsResNet18, InsResNet101, InsResNet50
2 | from .pcrl_model import PCRLModel
3 | from .pcrl_model_3d import PCRLModel3d
4 |
5 |
--------------------------------------------------------------------------------
/pytorch/models/__pycache__/__init__.cpython-37.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Luchixiang/PCRL/197bf91d4872f88742686d159892904c965ebdf6/pytorch/models/__pycache__/__init__.cpython-37.pyc
--------------------------------------------------------------------------------
/pytorch/models/__pycache__/brats_test_model.cpython-37.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Luchixiang/PCRL/197bf91d4872f88742686d159892904c965ebdf6/pytorch/models/__pycache__/brats_test_model.cpython-37.pyc
--------------------------------------------------------------------------------
/pytorch/models/__pycache__/resnet_mixup.cpython-37.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Luchixiang/PCRL/197bf91d4872f88742686d159892904c965ebdf6/pytorch/models/__pycache__/resnet_mixup.cpython-37.pyc
--------------------------------------------------------------------------------
/pytorch/models/__pycache__/unet3d2.cpython-37.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Luchixiang/PCRL/197bf91d4872f88742686d159892904c965ebdf6/pytorch/models/__pycache__/unet3d2.cpython-37.pyc
--------------------------------------------------------------------------------
/pytorch/models/pcrl_model.py:
--------------------------------------------------------------------------------
1 | import segmentation_models_pytorch as smp
2 | import torch.nn as nn
3 | import torch.nn.functional as F
4 | import torch
5 | from segmentation_models_pytorch.base import modules as md
6 | import numpy as np
7 | from torchvision.models.resnet import ResNet
8 | from torchvision.models.resnet import BasicBlock
9 | from torchvision.models.resnet import Bottleneck
10 | from pretrainedmodels.models.torchvision_models import pretrained_settings
11 | from segmentation_models_pytorch.base.initialization import initialize_decoder, initialize_head
12 | from segmentation_models_pytorch.base import SegmentationHead
13 | from segmentation_models_pytorch.encoders._base import EncoderMixin
14 | import copy
15 | import random
16 |
17 |
18 | def initialize_decoder(module):
19 | for m in module.modules():
20 |
21 | if isinstance(m, nn.Conv2d):
22 | nn.init.kaiming_uniform_(m.weight, mode="fan_in", nonlinearity="relu")
23 | if m.bias is not None:
24 | nn.init.constant_(m.bias, 0)
25 |
26 | elif isinstance(m, nn.BatchNorm2d):
27 | nn.init.constant_(m.weight, 1)
28 | nn.init.constant_(m.bias, 0)
29 |
30 | elif isinstance(m, nn.Linear):
31 | nn.init.xavier_uniform_(m.weight)
32 | if m.bias is not None:
33 | nn.init.constant_(m.bias, 0)
34 |
35 |
36 | def initialize_head(module):
37 | for m in module.modules():
38 | if isinstance(m, (nn.Linear, nn.Conv2d)):
39 | nn.init.xavier_uniform_(m.weight)
40 | if m.bias is not None:
41 | nn.init.constant_(m.bias, 0)
42 |
43 |
44 | class CenterBlock(nn.Sequential):
45 | def __init__(self, in_channels, out_channels, use_batchnorm=True):
46 | conv1 = md.Conv2dReLU(
47 | in_channels,
48 | out_channels,
49 | kernel_size=3,
50 | padding=1,
51 | use_batchnorm=use_batchnorm,
52 | )
53 | conv2 = md.Conv2dReLU(
54 | out_channels,
55 | out_channels,
56 | kernel_size=3,
57 | padding=1,
58 | use_batchnorm=use_batchnorm,
59 | )
60 | super().__init__(conv1, conv2)
61 |
62 |
63 | class DecoderBlock(nn.Module):
64 | def __init__(
65 | self,
66 | in_channels,
67 | skip_channels,
68 | out_channels,
69 | use_batchnorm=True,
70 | attention_type=None,
71 | ):
72 | super().__init__()
73 | self.conv1 = md.Conv2dReLU(
74 | in_channels + skip_channels,
75 | out_channels,
76 | kernel_size=3,
77 | padding=1,
78 | use_batchnorm=use_batchnorm,
79 | )
80 | self.attention1 = md.Attention(attention_type, in_channels=in_channels + skip_channels)
81 | self.conv2 = md.Conv2dReLU(
82 | out_channels,
83 | out_channels,
84 | kernel_size=3,
85 | padding=1,
86 | use_batchnorm=use_batchnorm,
87 | )
88 | self.attention2 = md.Attention(attention_type, in_channels=out_channels)
89 |
90 | def forward(self, x, skip=None):
91 | x = F.interpolate(x, scale_factor=2, mode="nearest")
92 | if skip is not None:
93 | x = torch.cat([x, skip], dim=1)
94 | x = self.attention1(x)
95 | x = self.conv1(x)
96 | x = self.conv2(x)
97 | x = self.attention2(x)
98 | return x
99 |
100 |
101 | class ShuffleUnetDecoder(nn.Module):
102 | def __init__(
103 | self,
104 | # decoder,
105 | encoder_channels=512,
106 | n_class=3,
107 | decoder_channels=(256, 128, 64, 32, 16),
108 | n_blocks=5,
109 | use_batchnorm=True,
110 | center=False,
111 | attention_type=None
112 |
113 | ):
114 | super().__init__()
115 | # self.decoder = decoder
116 | # self.segmentation_head = segmentation_head
117 | if n_blocks != len(decoder_channels):
118 | raise ValueError(
119 | "Model depth is {}, but you provide `decoder_channels` for {} blocks.".format(
120 | n_blocks, len(decoder_channels)
121 | )
122 | )
123 |
124 | encoder_channels = encoder_channels[1:] # remove first skip with same spatial resolution
125 | encoder_channels = encoder_channels[::-1] # reverse channels to start from head of encoder
126 |
127 | # computing blocks input and output channels
128 | head_channels = encoder_channels[0]
129 | in_channels = [head_channels] + list(decoder_channels[:-1])
130 | skip_channels = list(encoder_channels[1:]) + [0]
131 | out_channels = decoder_channels
132 | # self.conv = nn.Conv2d(1024, 512, kernel_size=3, padding=1, stride=1)
133 | if center:
134 | self.center = CenterBlock(
135 | head_channels, head_channels, use_batchnorm=use_batchnorm
136 | )
137 | else:
138 | self.center = nn.Identity()
139 | kwargs = dict(use_batchnorm=use_batchnorm, attention_type=attention_type)
140 | blocks = [
141 | DecoderBlock(in_ch, skip_ch, out_ch, **kwargs)
142 | for in_ch, skip_ch, out_ch in zip(in_channels, skip_channels, out_channels)
143 | ]
144 | self.blocks = nn.ModuleList(blocks)
145 | initialize_decoder(self.blocks)
146 | # self.segmentation_head = SegmentationHead(16, 3)
147 | # initialize_head(self.segmentation_head)
148 | # self.segmentation_head = segmentation_head
149 | #
150 | # # combine decoder keyword arguments
151 |
152 | def forward(self, features1, features2, alpha, aug_tensor1, aug_tensor2, mixup=False):
153 | # x = self.decoder(*features)
154 | # return self.segmentation_head(x)
155 | # def forward(self, features1, features2):
156 | #
157 | features1 = features1[1:] # remove first skip with same spatial resolution
158 | features1 = features1[::-1] # reverse channels to start from head of encoder
159 | features2 = features2[1:]
160 | features2 = features2[::-1]
161 | head1 = features1[0]
162 | skips1 = features1[1:]
163 | head2 = features2[0]
164 | skips2 = features2[1:]
165 | x1 = self.center(head1)
166 | x2 = self.center(head2)
167 | if not mixup:
168 | x1 = x1 * aug_tensor1
169 | x2 = x2 * aug_tensor2
170 | x3 = x1.clone()
171 | x1 = alpha * x1 + (1 - alpha) * x2
172 | for i, decoder_block in enumerate(self.blocks):
173 | # print(i, x1.shape, skips1[i].shape, x2.shape, skips2[i].shape)
174 | skip1 = skips1[i] if i < len(skips1) else None
175 | #skip1_shuffle = self.decoder_shuffle(skip1, shuffle_num + i + 1) if i < len(skips1) else None
176 | x3 = decoder_block(x3, skip1)
177 |
178 | # x1 = decoder_block(x1, skip1)
179 | skip2 = skips2[i] if i < len(skips2) else None
180 | skip = alpha * skip1 + (1 - alpha) * skip2 if i < len(skips2) else None
181 | # skip = self.decoder_shuffle(skip, shuffle_num + i + 1) if i < len(skips2) else None
182 | # x2 = decoder_block(x2, skip2)
183 | x1 = decoder_block(x1, skip)
184 |
185 | # x1 = self.segmentation_head(x1)
186 | return x1, x3
187 |
188 | def decoder_shuffle(self, x, shuffle_num):
189 | w = x.shape[2]
190 | h = x.shape[3]
191 | shuffle_col_index = torch.randperm(w)[:shuffle_num].cuda()
192 | shuffle_row_index = torch.randperm(h)[:shuffle_num].cuda()
193 | col_index = shuffle_col_index[torch.randperm(shuffle_col_index.shape[0])]
194 | row_index = shuffle_row_index[torch.randperm(shuffle_row_index.shape[0])]
195 | # print(col_index, row_index, shuffle_row_index, shuffle_col_index)
196 | # print(shuffle_row_index, x.shape, x[:, :, shuffle_row_index].shape)
197 | x = x.index_copy(2, col_index, x.index_select(2, shuffle_col_index))
198 | x = x.index_copy(3, row_index, x.index_select(3, shuffle_row_index))
199 | return x
200 |
201 |
202 | class PCRLModel(nn.Module):
203 | def __init__(self, n_class=3, low_dim=128, student=False):
204 | super(PCRLModel, self).__init__()
205 | self.model = smp.Unet('resnet18', in_channels=3, classes=n_class, encoder_weights=None)
206 | self.model.decoder = ShuffleUnetDecoder(self.model.encoder.out_channels)
207 | # self.segmentation_head = self.unet.segmentation_head
208 | # self.model = net
209 | self.avg_pool = nn.AdaptiveAvgPool2d((1, 1))
210 | self.fc1 = nn.Linear(512, low_dim)
211 | self.relu = nn.ReLU(inplace=True)
212 | self.student = student
213 | self.fc2 = nn.Linear(low_dim, low_dim)
214 | self.aug_fc1 = nn.Linear(6, 256)
215 | self.aug_fc2 = nn.Linear(256, 512)
216 | self.sigmoid = nn.Sigmoid()
217 |
218 | def forward(self, x, features_ema=None, alpha=None, aug_tensor1=None, aug_tensor2=None, mixup=False):
219 | b = x.shape[0]
220 | features = self.model.encoder(x)
221 | feature = self.avg_pool(features[-1])
222 | feature = feature.view(b, -1)
223 | feature = self.fc1(feature)
224 | feature = self.relu(feature)
225 | feature = self.fc2(feature)
226 | if self.student:
227 | if not mixup:
228 | aug_tensor1 = self.aug_fc1(aug_tensor1)
229 | aug_tensor1 = self.relu(aug_tensor1)
230 | aug_tensor1 = self.aug_fc2(aug_tensor1)
231 | aug_tensor2 = self.aug_fc1(aug_tensor2)
232 | aug_tensor2 = self.relu(aug_tensor2)
233 | aug_tensor2 = self.aug_fc2(aug_tensor2)
234 | aug_tensor1 = self.sigmoid(aug_tensor1)
235 | aug_tensor2 = self.sigmoid(aug_tensor2)
236 | aug_tensor1 = aug_tensor1.view(b, 512, 1, 1)
237 | aug_tensor2 = aug_tensor2.view(b, 512, 1, 1)
238 | # print(aug_tensor2.shape)
239 | decoder_output_alpha, decoder_output = self.model.decoder(features, features_ema, alpha, aug_tensor1,
240 | aug_tensor2, mixup)
241 | masks_alpha = self.model.segmentation_head(decoder_output_alpha)
242 | masks = self.model.segmentation_head(decoder_output)
243 | return feature, masks_alpha, masks
244 | return feature, features
245 |
--------------------------------------------------------------------------------
/pytorch/models/pcrl_model_3d.py:
--------------------------------------------------------------------------------
1 | import torch
2 | import torch.nn as nn
3 |
4 |
5 | class LUConv(nn.Module):
6 | def __init__(self, in_chan, out_chan, act, norm):
7 | super(LUConv, self).__init__()
8 | self.conv1 = nn.Conv3d(in_chan, out_chan, kernel_size=3, padding=1)
9 |
10 | if norm == 'bn':
11 | self.bn1 = nn.BatchNorm3d(num_features=out_chan, momentum=0.1, affine=True)
12 | elif norm == 'gn':
13 | self.bn1 = nn.GroupNorm(num_groups=8, num_channels=out_chan, eps=1e-05, affine=True)
14 | elif norm == 'in':
15 | self.bn1 = nn.InstanceNorm3d(num_features=out_chan, momentum=0.1, affine=True)
16 | else:
17 | raise ValueError('normalization type {} is not supported'.format(norm))
18 |
19 | if act == 'relu':
20 | self.activation = nn.ReLU(inplace=True)
21 | elif act == 'prelu':
22 | self.activation = nn.PReLU(out_chan)
23 | elif act == 'elu':
24 | self.activation = nn.ELU(inplace=True)
25 | else:
26 | raise ValueError('activation type {} is not supported'.format(act))
27 |
28 | def forward(self, x):
29 | out = self.activation(self.bn1(self.conv1(x)))
30 | return out
31 |
32 |
33 | def _make_nConv(in_channel, depth, act, norm, double_chnnel=False):
34 | if double_chnnel:
35 | layer1 = LUConv(in_channel, 32 * (2 ** (depth + 1)), act, norm)
36 | layer2 = LUConv(32 * (2 ** (depth + 1)), 32 * (2 ** (depth + 1)), act, norm)
37 | else:
38 | layer1 = LUConv(in_channel, 32 * (2 ** depth), act, norm)
39 | layer2 = LUConv(32 * (2 ** depth), 32 * (2 ** depth) * 2, act, norm)
40 |
41 | return nn.Sequential(layer1, layer2)
42 |
43 |
44 | class UpTransition(nn.Module):
45 | def __init__(self, inChans, outChans, depth, act, norm):
46 | super(UpTransition, self).__init__()
47 | self.depth = depth
48 | self.up_conv = nn.ConvTranspose3d(inChans, outChans, kernel_size=2, stride=2)
49 | self.ops = _make_nConv(inChans + outChans // 2, depth, act, norm, double_chnnel=True)
50 |
51 | def forward(self, x, skip_x):
52 | out_up_conv = self.up_conv(x)
53 | concat = torch.cat((out_up_conv, skip_x), 1)
54 | return self.ops(concat)
55 |
56 |
57 | class OutputTransition(nn.Module):
58 | def __init__(self, inChans, n_labels):
59 | super(OutputTransition, self).__init__()
60 | self.final_conv = nn.Conv3d(inChans, n_labels, kernel_size=1)
61 | self.sigmoid = nn.Sigmoid()
62 |
63 | def forward(self, x):
64 | out = self.sigmoid(self.final_conv(x))
65 | return out
66 |
67 |
68 | class DownTransition(nn.Module):
69 | def __init__(self, in_channel, depth, act, norm):
70 | super(DownTransition, self).__init__()
71 | self.ops = _make_nConv(in_channel, depth, act, norm)
72 |
73 | def forward(self, x):
74 | return self.ops(x)
75 |
76 |
77 | class PCRLModel3d(nn.Module):
78 | # the number of convolutions in each layer corresponds
79 | # to what is in the actual prototxt, not the intent
80 | def __init__(self, n_class=1, act='relu', norm='bn', in_channels=1, low_dim=128, student=False):
81 | super(PCRLModel3d, self).__init__()
82 | self.maxpool = nn.MaxPool3d(2)
83 | self.down_tr64 = DownTransition(in_channels, 0, act, norm)
84 | self.down_tr128 = DownTransition(64, 1, act, norm)
85 | self.down_tr256 = DownTransition(128, 2, act, norm)
86 | self.down_tr512 = DownTransition(256, 3, act, norm)
87 | self.avg_pool = nn.AdaptiveAvgPool3d((1, 1, 1))
88 | self.fc1 = nn.Linear(512, low_dim)
89 | self.relu = nn.ReLU(inplace=True)
90 | self.student = student
91 | self.fc2 = nn.Linear(low_dim, low_dim)
92 | self.up_tr256 = UpTransition(512, 512, 2, act, norm)
93 | self.up_tr128 = UpTransition(256, 256, 1, act, norm)
94 | self.up_tr64 = UpTransition(128, 128, 0, act, norm)
95 | self.out_tr = OutputTransition(64, n_class)
96 | self.aug_fc1 = nn.Linear(7, 256)
97 | self.aug_fc2 = nn.Linear(256, 512)
98 | self.sigmoid = nn.Sigmoid()
99 |
100 | def forward(self, x, feats_ema=None, alpha=None, aug_tensor1=None, aug_tensor2=None, mixup=False):
101 | b = x.shape[0]
102 | self.skip_out64 = self.down_tr64(x)
103 | self.skip_out128 = self.down_tr128(self.maxpool(self.skip_out64))
104 | self.skip_out256 = self.down_tr256(self.maxpool(self.skip_out128))
105 | self.out512 = self.down_tr512(self.maxpool(self.skip_out256))
106 | feature = self.out512.clone()
107 | feature = self.avg_pool(feature)
108 | feature = feature.view(b, -1)
109 | feature = self.fc1(feature)
110 | feature = self.relu(feature)
111 | feature = self.fc2(feature)
112 | if self.student:
113 | out512_ema, skip_out64_ema, skip_out128_ema, skip_out256_ema = feats_ema
114 | # alpha decoder
115 | if not mixup:
116 | aug_tensor1 = self.aug_fc1(aug_tensor1)
117 | aug_tensor1 = self.relu(aug_tensor1)
118 | aug_tensor1 = self.aug_fc2(aug_tensor1)
119 | aug_tensor2 = self.aug_fc1(aug_tensor2)
120 | aug_tensor2 = self.relu(aug_tensor2)
121 | aug_tensor2 = self.aug_fc2(aug_tensor2)
122 | aug_tensor1 = self.sigmoid(aug_tensor1)
123 | aug_tensor2 = self.sigmoid(aug_tensor2)
124 | aug_tensor1 = aug_tensor1.view(b, 512, 1, 1, 1)
125 | aug_tensor2 = aug_tensor2.view(b, 512, 1, 1, 1)
126 | self.out512 = self.out512 * aug_tensor1
127 | out512_ema = out512_ema * aug_tensor2
128 | out512 = self.out512 * alpha + (1 - alpha) * out512_ema
129 | skip_out256 = self.skip_out256 * alpha + (1 - alpha) * skip_out256_ema
130 | skip_out128 = self.skip_out128 * alpha + (1 - alpha) * skip_out128_ema
131 | skip_out64 = self.skip_out64 * alpha + (1 - alpha) * skip_out64_ema
132 | out_up_256_alpha = self.up_tr256(out512, skip_out256)
133 | out_up_128_alpha = self.up_tr128(out_up_256_alpha, skip_out128)
134 | out_up_64_alpha = self.up_tr64(out_up_128_alpha, skip_out64)
135 | out_alpha = self.out_tr(out_up_64_alpha)
136 | # normal decoder
137 | out_up_256 = self.up_tr256(self.out512, self.skip_out256)
138 | out_up_128 = self.up_tr128(out_up_256, self.skip_out128)
139 | out_up_64 = self.up_tr64(out_up_128, self.skip_out64)
140 | out = self.out_tr(out_up_64)
141 | return feature, out_alpha, out
142 | return feature, [self.out512, self.skip_out64, self.skip_out128, self.skip_out256]
143 |
144 | def decoder_shuffle(self, x, shuffle_num):
145 | w = x.shape[2]
146 | h = x.shape[3]
147 | d = x.shape[4]
148 | shuffle_col_index = torch.randperm(w)[:shuffle_num].cuda()
149 | shuffle_row_index = torch.randperm(h)[:shuffle_num].cuda()
150 | shuffle_d_index = torch.randperm(d)[:shuffle_num].cuda()
151 | col_index = shuffle_col_index[torch.randperm(shuffle_col_index.shape[0])]
152 | row_index = shuffle_row_index[torch.randperm(shuffle_row_index.shape[0])]
153 | d_index = shuffle_d_index[torch.randperm(shuffle_d_index.shape[0])]
154 | # print(col_index, row_index, shuffle_row_index, shuffle_col_index)
155 | # print(shuffle_row_index, x.shape, x[:, :, shuffle_row_index].shape)
156 | x = x.index_copy(2, col_index, x.index_select(2, shuffle_col_index))
157 | x = x.index_copy(3, row_index, x.index_select(3, shuffle_row_index))
158 | x = x.index_copy(4, d_index, x.index_select(4, shuffle_d_index))
159 | return x
160 |
--------------------------------------------------------------------------------
/pytorch/models/resnet.py:
--------------------------------------------------------------------------------
1 | import torch
2 | import torch.nn as nn
3 | from .utils import load_state_dict_from_url
4 |
5 |
6 | __all__ = ['ResNet', 'resnet18', 'resnet34', 'resnet50', 'resnet101',
7 | 'resnet152', 'resnext50_32x4d', 'resnext101_32x8d',
8 | 'wide_resnet50_2', 'wide_resnet101_2']
9 |
10 |
11 | model_urls = {
12 | 'resnet18': 'https://download.pytorch.org/models/resnet18-5c106cde.pth',
13 | 'resnet34': 'https://download.pytorch.org/models/resnet34-333f7ec4.pth',
14 | 'resnet50': 'https://download.pytorch.org/models/resnet50-19c8e357.pth',
15 | 'resnet101': 'https://download.pytorch.org/models/resnet101-5d3b4d8f.pth',
16 | 'resnet152': 'https://download.pytorch.org/models/resnet152-b121ed2d.pth',
17 | 'resnext50_32x4d': 'https://download.pytorch.org/models/resnext50_32x4d-7cdf4587.pth',
18 | 'resnext101_32x8d': 'https://download.pytorch.org/models/resnext101_32x8d-8ba56ff5.pth',
19 | 'wide_resnet50_2': 'https://download.pytorch.org/models/wide_resnet50_2-95faca4d.pth',
20 | 'wide_resnet101_2': 'https://download.pytorch.org/models/wide_resnet101_2-32ee1156.pth',
21 | }
22 |
23 |
24 | def conv3x3(in_planes, out_planes, stride=1, groups=1, dilation=1):
25 | """3x3 convolution with padding"""
26 | return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride,
27 | padding=dilation, groups=groups, bias=False, dilation=dilation)
28 |
29 |
30 | def conv1x1(in_planes, out_planes, stride=1):
31 | """1x1 convolution"""
32 | return nn.Conv2d(in_planes, out_planes, kernel_size=1, stride=stride, bias=False)
33 |
34 |
35 | class BasicBlock(nn.Module):
36 | expansion = 1
37 |
38 | def __init__(self, inplanes, planes, stride=1, downsample=None, groups=1,
39 | base_width=64, dilation=1, norm_layer=None):
40 | super(BasicBlock, self).__init__()
41 | if norm_layer is None:
42 | norm_layer = nn.BatchNorm2d
43 | if groups != 1 or base_width != 64:
44 | raise ValueError('BasicBlock only supports groups=1 and base_width=64')
45 | if dilation > 1:
46 | raise NotImplementedError("Dilation > 1 not supported in BasicBlock")
47 | # Both self.conv1 and self.downsample layers downsample the input when stride != 1
48 | self.conv1 = conv3x3(inplanes, planes, stride)
49 | self.bn1 = norm_layer(planes)
50 | self.relu = nn.ReLU(inplace=True)
51 | self.conv2 = conv3x3(planes, planes)
52 | self.bn2 = norm_layer(planes)
53 | self.downsample = downsample
54 | self.stride = stride
55 |
56 | def forward(self, x):
57 | identity = x
58 |
59 | out = self.conv1(x)
60 | out = self.bn1(out)
61 | out = self.relu(out)
62 |
63 | out = self.conv2(out)
64 | out = self.bn2(out)
65 |
66 | if self.downsample is not None:
67 | identity = self.downsample(x)
68 |
69 | out += identity
70 | out = self.relu(out)
71 |
72 | return out
73 |
74 |
75 | class Bottleneck(nn.Module):
76 | # Bottleneck in torchvision places the stride for downsampling at 3x3 convolution(self.conv2)
77 | # while original implementation places the stride at the first 1x1 convolution(self.conv1)
78 | # according to "Deep residual learning for image recognition"https://arxiv.org/abs/1512.03385.
79 | # This variant is also known as ResNet V1.5 and improves accuracy according to
80 | # https://ngc.nvidia.com/catalog/model-scripts/nvidia:resnet_50_v1_5_for_pytorch.
81 |
82 | expansion = 4
83 |
84 | def __init__(self, inplanes, planes, stride=1, downsample=None, groups=1,
85 | base_width=64, dilation=1, norm_layer=None):
86 | super(Bottleneck, self).__init__()
87 | if norm_layer is None:
88 | norm_layer = nn.BatchNorm2d
89 | width = int(planes * (base_width / 64.)) * groups
90 | # Both self.conv2 and self.downsample layers downsample the input when stride != 1
91 | self.conv1 = conv1x1(inplanes, width)
92 | self.bn1 = norm_layer(width)
93 | self.conv2 = conv3x3(width, width, stride, groups, dilation)
94 | self.bn2 = norm_layer(width)
95 | self.conv3 = conv1x1(width, planes * self.expansion)
96 | self.bn3 = norm_layer(planes * self.expansion)
97 | self.relu = nn.ReLU(inplace=True)
98 | self.downsample = downsample
99 | self.stride = stride
100 |
101 | def forward(self, x):
102 | identity = x
103 |
104 | out = self.conv1(x)
105 | out = self.bn1(out)
106 | out = self.relu(out)
107 |
108 | out = self.conv2(out)
109 | out = self.bn2(out)
110 | out = self.relu(out)
111 |
112 | out = self.conv3(out)
113 | out = self.bn3(out)
114 |
115 | if self.downsample is not None:
116 | identity = self.downsample(x)
117 |
118 | out += identity
119 | out = self.relu(out)
120 |
121 | return out
122 |
123 |
124 | class ResNet(nn.Module):
125 |
126 | def __init__(self, block, layers, num_classes=1000, zero_init_residual=False,
127 | groups=1, width_per_group=64, replace_stride_with_dilation=None,
128 | norm_layer=None):
129 | super(ResNet, self).__init__()
130 | if norm_layer is None:
131 | norm_layer = nn.BatchNorm2d
132 | self._norm_layer = norm_layer
133 |
134 | self.inplanes = 64
135 | self.dilation = 1
136 | if replace_stride_with_dilation is None:
137 | # each element in the tuple indicates if we should replace
138 | # the 2x2 stride with a dilated convolution instead
139 | replace_stride_with_dilation = [False, False, False]
140 | if len(replace_stride_with_dilation) != 3:
141 | raise ValueError("replace_stride_with_dilation should be None "
142 | "or a 3-element tuple, got {}".format(replace_stride_with_dilation))
143 | self.groups = groups
144 | self.base_width = width_per_group
145 | self.conv1 = nn.Conv2d(3, self.inplanes, kernel_size=7, stride=2, padding=3,
146 | bias=False)
147 | self.bn1 = norm_layer(self.inplanes)
148 | self.relu = nn.ReLU(inplace=True)
149 | self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
150 | self.layer1 = self._make_layer(block, 64, layers[0])
151 | self.layer2 = self._make_layer(block, 128, layers[1], stride=2,
152 | dilate=replace_stride_with_dilation[0])
153 | self.layer3 = self._make_layer(block, 256, layers[2], stride=2,
154 | dilate=replace_stride_with_dilation[1])
155 | self.layer4 = self._make_layer(block, 512, layers[3], stride=2,
156 | dilate=replace_stride_with_dilation[2])
157 | self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
158 | self.fc = nn.Linear(512 * block.expansion, num_classes)
159 |
160 | for m in self.modules():
161 | if isinstance(m, nn.Conv2d):
162 | nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
163 | elif isinstance(m, (nn.BatchNorm2d, nn.GroupNorm)):
164 | nn.init.constant_(m.weight, 1)
165 | nn.init.constant_(m.bias, 0)
166 |
167 | # Zero-initialize the last BN in each residual branch,
168 | # so that the residual branch starts with zeros, and each residual block behaves like an identity.
169 | # This improves the model by 0.2~0.3% according to https://arxiv.org/abs/1706.02677
170 | if zero_init_residual:
171 | for m in self.modules():
172 | if isinstance(m, Bottleneck):
173 | nn.init.constant_(m.bn3.weight, 0)
174 | elif isinstance(m, BasicBlock):
175 | nn.init.constant_(m.bn2.weight, 0)
176 |
177 | def _make_layer(self, block, planes, blocks, stride=1, dilate=False):
178 | norm_layer = self._norm_layer
179 | downsample = None
180 | previous_dilation = self.dilation
181 | if dilate:
182 | self.dilation *= stride
183 | stride = 1
184 | if stride != 1 or self.inplanes != planes * block.expansion:
185 | downsample = nn.Sequential(
186 | conv1x1(self.inplanes, planes * block.expansion, stride),
187 | norm_layer(planes * block.expansion),
188 | )
189 |
190 | layers = []
191 | layers.append(block(self.inplanes, planes, stride, downsample, self.groups,
192 | self.base_width, previous_dilation, norm_layer))
193 | self.inplanes = planes * block.expansion
194 | for _ in range(1, blocks):
195 | layers.append(block(self.inplanes, planes, groups=self.groups,
196 | base_width=self.base_width, dilation=self.dilation,
197 | norm_layer=norm_layer))
198 |
199 | return nn.Sequential(*layers)
200 |
201 | def _forward_impl(self, x):
202 | # See note [TorchScript super()]
203 | x = self.conv1(x)
204 | x = self.bn1(x)
205 | x = self.relu(x)
206 | x = self.maxpool(x)
207 |
208 | x = self.layer1(x)
209 | x = self.layer2(x)
210 | x = self.layer3(x)
211 | self.feature = self.layer4(x)
212 |
213 | x = self.avgpool(self.feature)
214 | x = torch.flatten(x, 1)
215 | x = self.fc(x)
216 |
217 | return x
218 |
219 | def forward(self, x):
220 | return self._forward_impl(x)
221 |
222 |
223 | def _resnet(arch, block, layers, pretrained, progress, **kwargs):
224 | model = ResNet(block, layers, **kwargs)
225 | if pretrained:
226 | state_dict = load_state_dict_from_url(model_urls[arch],
227 | progress=progress)
228 | model.load_state_dict(state_dict)
229 | return model
230 |
231 |
232 | def resnet18(pretrained=False, progress=True, **kwargs):
233 | r"""ResNet-18 model from
234 | `"Deep Residual Learning for Image Recognition" `_
235 | Args:
236 | pretrained (bool): If True, returns a model pre-trained on ImageNet
237 | progress (bool): If True, displays a progress bar of the download to stderr
238 | """
239 | return _resnet('resnet18', BasicBlock, [2, 2, 2, 2], pretrained, progress,
240 | **kwargs)
241 |
242 |
243 | def resnet34(pretrained=False, progress=True, **kwargs):
244 | r"""ResNet-34 model from
245 | `"Deep Residual Learning for Image Recognition" `_
246 | Args:
247 | pretrained (bool): If True, returns a model pre-trained on ImageNet
248 | progress (bool): If True, displays a progress bar of the download to stderr
249 | """
250 | return _resnet('resnet34', BasicBlock, [3, 4, 6, 3], pretrained, progress,
251 | **kwargs)
252 |
253 |
254 | def resnet50(pretrained=False, progress=True, **kwargs):
255 | r"""ResNet-50 model from
256 | `"Deep Residual Learning for Image Recognition" `_
257 | Args:
258 | pretrained (bool): If True, returns a model pre-trained on ImageNet
259 | progress (bool): If True, displays a progress bar of the download to stderr
260 | """
261 | return _resnet('resnet50', Bottleneck, [3, 4, 6, 3], pretrained, progress,
262 | **kwargs)
263 |
264 |
265 | def resnet101(pretrained=False, progress=True, **kwargs):
266 | r"""ResNet-101 model from
267 | `"Deep Residual Learning for Image Recognition" `_
268 | Args:
269 | pretrained (bool): If True, returns a model pre-trained on ImageNet
270 | progress (bool): If True, displays a progress bar of the download to stderr
271 | """
272 | return _resnet('resnet101', Bottleneck, [3, 4, 23, 3], pretrained, progress,
273 | **kwargs)
274 |
275 |
276 | def resnet152(pretrained=False, progress=True, **kwargs):
277 | r"""ResNet-152 model from
278 | `"Deep Residual Learning for Image Recognition" `_
279 | Args:
280 | pretrained (bool): If True, returns a model pre-trained on ImageNet
281 | progress (bool): If True, displays a progress bar of the download to stderr
282 | """
283 | return _resnet('resnet152', Bottleneck, [3, 8, 36, 3], pretrained, progress,
284 | **kwargs)
285 |
286 |
287 | def resnext50_32x4d(pretrained=False, progress=True, **kwargs):
288 | r"""ResNeXt-50 32x4d model from
289 | `"Aggregated Residual Transformation for Deep Neural Networks" `_
290 | Args:
291 | pretrained (bool): If True, returns a model pre-trained on ImageNet
292 | progress (bool): If True, displays a progress bar of the download to stderr
293 | """
294 | kwargs['groups'] = 32
295 | kwargs['width_per_group'] = 4
296 | return _resnet('resnext50_32x4d', Bottleneck, [3, 4, 6, 3],
297 | pretrained, progress, **kwargs)
298 |
299 |
300 | def resnext101_32x8d(pretrained=False, progress=True, **kwargs):
301 | r"""ResNeXt-101 32x8d model from
302 | `"Aggregated Residual Transformation for Deep Neural Networks" `_
303 | Args:
304 | pretrained (bool): If True, returns a model pre-trained on ImageNet
305 | progress (bool): If True, displays a progress bar of the download to stderr
306 | """
307 | kwargs['groups'] = 32
308 | kwargs['width_per_group'] = 8
309 | return _resnet('resnext101_32x8d', Bottleneck, [3, 4, 23, 3],
310 | pretrained, progress, **kwargs)
311 |
312 |
313 | def wide_resnet50_2(pretrained=False, progress=True, **kwargs):
314 | r"""Wide ResNet-50-2 model from
315 | `"Wide Residual Networks" `_
316 | The model is the same as ResNet except for the bottleneck number of channels
317 | which is twice larger in every block. The number of channels in outer 1x1
318 | convolutions is the same, e.g. last block in ResNet-50 has 2048-512-2048
319 | channels, and in Wide ResNet-50-2 has 2048-1024-2048.
320 | Args:
321 | pretrained (bool): If True, returns a model pre-trained on ImageNet
322 | progress (bool): If True, displays a progress bar of the download to stderr
323 | """
324 | kwargs['width_per_group'] = 64 * 2
325 | return _resnet('wide_resnet50_2', Bottleneck, [3, 4, 6, 3],
326 | pretrained, progress, **kwargs)
327 |
328 |
329 | def wide_resnet101_2(pretrained=False, progress=True, **kwargs):
330 | r"""Wide ResNet-101-2 model from
331 | `"Wide Residual Networks" `_
332 | The model is the same as ResNet except for the bottleneck number of channels
333 | which is twice larger in every block. The number of channels in outer 1x1
334 | convolutions is the same, e.g. last block in ResNet-50 has 2048-512-2048
335 | channels, and in Wide ResNet-50-2 has 2048-1024-2048.
336 | Args:
337 | pretrained (bool): If True, returns a model pre-trained on ImageNet
338 | progress (bool): If True, displays a progress bar of the download to stderr
339 | """
340 | kwargs['width_per_group'] = 64 * 2
341 | return _resnet('wide_resnet101_2', Bottleneck, [3, 4, 23, 3],
342 | pretrained, progress, **kwargs)
--------------------------------------------------------------------------------
/pytorch/models/resnet_mixup.py:
--------------------------------------------------------------------------------
1 | import torch
2 | import torch.nn as nn
3 | import math
4 | import numpy as np
5 | import torch.utils.model_zoo as model_zoo
6 |
7 | __all__ = ['ResNet', 'resnet18', 'resnet34', 'resnet50', 'resnet101',
8 | 'resnet152']
9 |
10 | model_urls = {
11 | 'resnet18': 'https://download.pytorch.org/models/resnet18-5c106cde.pth',
12 | 'resnet34': 'https://download.pytorch.org/models/resnet34-333f7ec4.pth',
13 | 'resnet50': 'https://download.pytorch.org/models/resnet50-19c8e357.pth',
14 | 'resnet101': 'https://download.pytorch.org/models/resnet101-5d3b4d8f.pth',
15 | 'resnet152': 'https://download.pytorch.org/models/resnet152-b121ed2d.pth',
16 | }
17 |
18 |
19 | def conv3x3(in_planes, out_planes, stride=1):
20 | """3x3 convolution with padding"""
21 | return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride,
22 | padding=1, bias=False)
23 |
24 |
25 | class Normalize(nn.Module):
26 |
27 | def __init__(self, power=2):
28 | super(Normalize, self).__init__()
29 | self.power = power
30 |
31 | def forward(self, x):
32 | norm = x.pow(self.power).sum(1, keepdim=True).pow(1. / self.power)
33 | out = x.div(norm)
34 | return out
35 |
36 |
37 | class BasicBlock(nn.Module):
38 | expansion = 1
39 |
40 | def __init__(self, inplanes, planes, stride=1, downsample=None):
41 | super(BasicBlock, self).__init__()
42 | self.conv1 = conv3x3(inplanes, planes, stride)
43 | self.bn1 = nn.BatchNorm2d(planes)
44 | self.relu = nn.ReLU(inplace=True)
45 | self.conv2 = conv3x3(planes, planes)
46 | self.bn2 = nn.BatchNorm2d(planes)
47 | self.downsample = downsample
48 | self.stride = stride
49 |
50 | def forward(self, x):
51 | residual = x
52 |
53 | out = self.conv1(x)
54 | out = self.bn1(out)
55 | out = self.relu(out)
56 |
57 | out = self.conv2(out)
58 | out = self.bn2(out)
59 |
60 | if self.downsample is not None:
61 | residual = self.downsample(x)
62 |
63 | out += residual
64 | out = self.relu(out)
65 |
66 | return out
67 |
68 |
69 | class Bottleneck(nn.Module):
70 | expansion = 4
71 |
72 | def __init__(self, inplanes, planes, stride=1, downsample=None):
73 | super(Bottleneck, self).__init__()
74 | self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=1, bias=False)
75 | self.bn1 = nn.BatchNorm2d(planes)
76 | self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride,
77 | padding=1, bias=False)
78 | self.bn2 = nn.BatchNorm2d(planes)
79 | self.conv3 = nn.Conv2d(planes, planes * 4, kernel_size=1, bias=False)
80 | self.bn3 = nn.BatchNorm2d(planes * 4)
81 | self.relu = nn.ReLU(inplace=True)
82 | self.downsample = downsample
83 | self.stride = stride
84 |
85 | def forward(self, x):
86 | residual = x
87 |
88 | out = self.conv1(x)
89 | out = self.bn1(out)
90 | out = self.relu(out)
91 |
92 | out = self.conv2(out)
93 | out = self.bn2(out)
94 | out = self.relu(out)
95 |
96 | out = self.conv3(out)
97 | out = self.bn3(out)
98 |
99 | if self.downsample is not None:
100 | residual = self.downsample(x)
101 |
102 | out += residual
103 | out = self.relu(out)
104 |
105 | return out
106 |
107 |
108 | class ResNet(nn.Module):
109 |
110 | def __init__(self, block, layers, low_dim=128, in_channel=3, width=1):
111 | self.inplanes = 64
112 | super(ResNet, self).__init__()
113 | self.conv1 = nn.Conv2d(in_channel, 64, kernel_size=7, stride=2, padding=3,
114 | bias=False)
115 | self.bn1 = nn.BatchNorm2d(64)
116 | self.relu = nn.ReLU(inplace=True)
117 |
118 | self.base = int(64 * width)
119 |
120 | self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
121 | self.layer1 = self._make_layer(block, self.base, layers[0])
122 | self.layer2 = self._make_layer(block, self.base * 2, layers[1], stride=2)
123 | self.layer3 = self._make_layer(block, self.base * 4, layers[2], stride=2)
124 | self.layer4 = self._make_layer(block, self.base * 8, layers[3], stride=2)
125 | # self.avgpool = nn.AdaptiveAvgPool2d((1,1))
126 | self.avgpool = nn.AvgPool2d(7, stride=1)
127 | self.fc1 = nn.Linear(self.base * 8 * block.expansion, self.base * 8 * block.expansion)
128 | self.fc2 = nn.Linear(self.base * 8 * block.expansion, low_dim)
129 | #self.fc = nn.Linear(self.base * 8 * block.expansion, low_dim)
130 | # self.dropout = nn.Dropout()
131 | self.l2norm = Normalize(2)
132 |
133 | for m in self.modules():
134 | if isinstance(m, nn.Conv2d):
135 | n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
136 | m.weight.data.normal_(0, math.sqrt(2. / n))
137 | elif isinstance(m, nn.BatchNorm2d):
138 | m.weight.data.fill_(1)
139 | m.bias.data.zero_()
140 |
141 | def _make_layer(self, block, planes, blocks, stride=1):
142 | downsample = None
143 | if stride != 1 or self.inplanes != planes * block.expansion:
144 | downsample = nn.Sequential(
145 | nn.Conv2d(self.inplanes, planes * block.expansion,
146 | kernel_size=1, stride=stride, bias=False),
147 | nn.BatchNorm2d(planes * block.expansion),
148 | )
149 |
150 | layers = []
151 | layers.append(block(self.inplanes, planes, stride, downsample))
152 | self.inplanes = planes * block.expansion
153 | for i in range(1, blocks):
154 | layers.append(block(self.inplanes, planes))
155 |
156 | return nn.Sequential(*layers)
157 |
158 | def forward(self, x, layer=7):
159 | if layer <= 0:
160 | return x
161 | x = self.conv1(x)
162 | x = self.bn1(x)
163 | x = self.relu(x)
164 | x = self.maxpool(x)
165 | if layer == 1:
166 | return x
167 | x = self.layer1(x)
168 | if layer == 2:
169 | return x
170 | x = self.layer2(x)
171 | if layer == 3:
172 | return x
173 | x = self.layer3(x)
174 | if layer == 4:
175 | return x
176 | x = self.layer4(x)
177 | if layer == 5:
178 | return x
179 | x = self.avgpool(x)
180 | x = x.view(x.size(0), -1)
181 | if layer == 6:
182 | return x
183 | x = self.fc1(x)
184 | x = self.relu(x)
185 | x = self.fc2(x)
186 | #x = self.fc(x)
187 | # x = self.dropout(x)
188 | # x = self.l2norm(x)
189 |
190 | return x
191 |
192 |
193 | def resnet18(pretrained=False, **kwargs):
194 | """Constructs a ResNet-18 model.
195 | Args:
196 | pretrained (bool): If True, returns a model pre-trained on ImageNet
197 | """
198 | model = ResNet(BasicBlock, [2, 2, 2, 2], **kwargs)
199 | if pretrained:
200 | # model.load_state_dict(model_zoo.load_url(model_urls['resnet18']))
201 | state_dict = model_zoo.load_url(model_urls['resnet18'])
202 | encoder_dict = model.state_dict()
203 | unParalled_state_dict = {k: v for k, v in state_dict.items() if k in encoder_dict and 'fc' not in k}
204 | encoder_dict.update(unParalled_state_dict)
205 | print(unParalled_state_dict.keys())
206 | model.load_state_dict(encoder_dict)
207 | return model
208 |
209 |
210 | def resnet34(pretrained=False, **kwargs):
211 | """Constructs a ResNet-34 model.
212 | Args:
213 | pretrained (bool): If True, returns a model pre-trained on ImageNet
214 | """
215 | model = ResNet(BasicBlock, [3, 4, 6, 3], **kwargs)
216 | if pretrained:
217 | model.load_state_dict(model_zoo.load_url(model_urls['resnet34']))
218 | return model
219 |
220 |
221 | def resnet50(pretrained=False, **kwargs):
222 | """Constructs a ResNet-50 model.
223 | Args:
224 | pretrained (bool): If True, returns a model pre-trained on ImageNet
225 | """
226 | model = ResNet(Bottleneck, [3, 4, 6, 3], **kwargs)
227 | if pretrained:
228 | model.load_state_dict(model_zoo.load_url(model_urls['resnet50']))
229 | return model
230 |
231 |
232 | def resnet101(pretrained=False, **kwargs):
233 | """Constructs a ResNet-101 model.
234 | Args:
235 | pretrained (bool): If True, returns a model pre-trained on ImageNet
236 | """
237 | model = ResNet(Bottleneck, [3, 4, 23, 3], **kwargs)
238 | if pretrained:
239 | model.load_state_dict(model_zoo.load_url(model_urls['resnet101']))
240 | return model
241 |
242 |
243 | def resnet152(pretrained=False, **kwargs):
244 | """Constructs a ResNet-152 model.
245 | Args:
246 | pretrained (bool): If True, returns a model pre-trained on ImageNet
247 | """
248 | model = ResNet(Bottleneck, [3, 8, 36, 3], **kwargs)
249 | if pretrained:
250 | model.load_state_dict(model_zoo.load_url(model_urls['resnet152']))
251 | return model
252 |
253 |
254 | class InsResNet18(nn.Module):
255 | """Encoder for instance discrimination and MoCo"""
256 |
257 | def __init__(self, width=1):
258 | super(InsResNet18, self).__init__()
259 | self.encoder = resnet18(width=width)
260 | # self.encoder = nn.DataParallel(self.encoder)
261 |
262 | def forward(self, x, layer=7):
263 | return self.encoder(x, layer)
264 |
265 |
266 | class InsResNet50(nn.Module):
267 | """Encoder for instance discrimination and MoCo"""
268 |
269 | def __init__(self, width=1):
270 | super(InsResNet50, self).__init__()
271 | self.encoder = resnet50(width=width)
272 | # self.encoder = nn.DataParallel(self.encoder)
273 |
274 | def forward(self, x, layer=7):
275 | return self.encoder(x, layer)
276 |
277 |
278 | class InsResNet101(nn.Module):
279 | """Encoder for instance discrimination and MoCo"""
280 |
281 | def __init__(self, width=1):
282 | super(InsResNet101, self).__init__()
283 | self.encoder = resnet101(width=width)
284 | # self.encoder = nn.DataParallel(self.encoder)
285 |
286 | def forward(self, x, layer=7):
287 | return self.encoder(x, layer)
288 |
289 |
290 | class ResNetV1(nn.Module):
291 | def __init__(self, name='resnet50'):
292 | super(ResNetV1, self).__init__()
293 | if name == 'resnet50':
294 | self.l_to_ab = resnet50(in_channel=1, width=0.5)
295 | self.ab_to_l = resnet50(in_channel=2, width=0.5)
296 | elif name == 'resnet18':
297 | self.l_to_ab = resnet18(in_channel=1, width=0.5)
298 | self.ab_to_l = resnet18(in_channel=2, width=0.5)
299 | elif name == 'resnet101':
300 | self.l_to_ab = resnet101(in_channel=1, width=0.5)
301 | self.ab_to_l = resnet101(in_channel=2, width=0.5)
302 | else:
303 | raise NotImplementedError('model {} is not implemented'.format(name))
304 |
305 | def forward(self, x, layer=7):
306 | l, ab = torch.split(x, [1, 2], dim=1)
307 | feat_l = self.l_to_ab(l, layer)
308 | feat_ab = self.ab_to_l(ab, layer)
309 | return feat_l, feat_ab
310 |
311 |
312 | class ResNetV2(nn.Module):
313 | def __init__(self, name='resnet50'):
314 | super(ResNetV2, self).__init__()
315 | if name == 'resnet50':
316 | self.l_to_ab = resnet50(in_channel=1, width=1)
317 | self.ab_to_l = resnet50(in_channel=2, width=1)
318 | elif name == 'resnet18':
319 | self.l_to_ab = resnet18(in_channel=1, width=1)
320 | self.ab_to_l = resnet18(in_channel=2, width=1)
321 | elif name == 'resnet101':
322 | self.l_to_ab = resnet101(in_channel=1, width=1)
323 | self.ab_to_l = resnet101(in_channel=2, width=1)
324 | else:
325 | raise NotImplementedError('model {} is not implemented'.format(name))
326 |
327 | def forward(self, x, layer=7):
328 | l, ab = torch.split(x, [1, 2], dim=1)
329 | feat_l = self.l_to_ab(l, layer)
330 | feat_ab = self.ab_to_l(ab, layer)
331 | return feat_l, feat_ab
332 |
333 |
334 | class ResNetV3(nn.Module):
335 | def __init__(self, name='resnet50'):
336 | super(ResNetV3, self).__init__()
337 | if name == 'resnet50':
338 | self.l_to_ab = resnet50(in_channel=1, width=2)
339 | self.ab_to_l = resnet50(in_channel=2, width=2)
340 | elif name == 'resnet18':
341 | self.l_to_ab = resnet18(in_channel=1, width=2)
342 | self.ab_to_l = resnet18(in_channel=2, width=2)
343 | elif name == 'resnet101':
344 | self.l_to_ab = resnet101(in_channel=1, width=2)
345 | self.ab_to_l = resnet101(in_channel=2, width=2)
346 | else:
347 | raise NotImplementedError('model {} is not implemented'.format(name))
348 |
349 | def forward(self, x, layer=7):
350 | l, ab = torch.split(x, [1, 2], dim=1)
351 | feat_l = self.l_to_ab(l, layer)
352 | feat_ab = self.ab_to_l(ab, layer)
353 | return feat_l, feat_ab
354 |
355 |
356 | class MyResNetsCMC(nn.Module):
357 | def __init__(self, name='resnet50v1'):
358 | super(MyResNetsCMC, self).__init__()
359 | if name.endswith('v1'):
360 | self.encoder = ResNetV1(name[:-2])
361 | elif name.endswith('v2'):
362 | self.encoder = ResNetV2(name[:-2])
363 | elif name.endswith('v3'):
364 | self.encoder = ResNetV3(name[:-2])
365 | else:
366 | raise NotImplementedError('model not support: {}'.format(name))
367 |
368 | self.encoder = nn.DataParallel(self.encoder)
369 |
370 | def forward(self, x, layer=7):
371 | return self.encoder(x, layer)
372 |
--------------------------------------------------------------------------------
/pytorch/pcrl.py:
--------------------------------------------------------------------------------
1 | """
2 | Training code for C2L
3 | """
4 | from __future__ import print_function
5 |
6 | import os
7 | import sys
8 | import time
9 |
10 | import numpy as np
11 | import torch
12 | import torch.backends.cudnn as cudnn
13 | import torch.nn as nn
14 | from models.pcrl_model import PCRLModel
15 | from losses import NCECriterion
16 | from memory_c2l import MemoryC2L
17 | from models import InsResNet18
18 | from utils import adjust_learning_rate, AverageMeter
19 | import segmentation_models_pytorch as smp
20 |
21 | try:
22 | from apex import amp, optimizers
23 | except ImportError:
24 | pass
25 |
26 |
27 | def Normalize(x):
28 | norm_x = x.pow(2).sum(1, keepdim=True).pow(1. / 2.)
29 | x = x.div(norm_x)
30 | return x
31 |
32 |
33 | def moment_update(model, model_ema, m):
34 | """ model_ema = m * model_ema + (1 - m) model """
35 | for p1, p2 in zip(model.parameters(), model_ema.parameters()):
36 | p2.data.mul_(m).add_(1 - m, p1.detach().data)
37 |
38 |
39 | def get_shuffle_ids(bsz):
40 | """generate shuffle ids for ShuffleBN"""
41 | forward_inds = torch.randperm(bsz).long().cuda()
42 | backward_inds = torch.zeros(bsz).long().cuda()
43 | value = torch.arange(bsz).long().cuda()
44 | backward_inds.index_copy_(0, forward_inds, value)
45 | return forward_inds, backward_inds
46 |
47 |
48 | def mixup_data(x, y, alpha=1.0, index=None, lam=None, use_cuda=True):
49 | '''Returns mixed inputs, pairs of targets, and lambda'''
50 | if lam is None:
51 | lam = np.random.beta(alpha, alpha)
52 | else:
53 | lam = lam
54 |
55 | lam = max(lam, 1 - lam)
56 | batch_size = x.size()[0]
57 | if index is None:
58 | index = torch.randperm(batch_size).cuda()
59 | else:
60 | index = index
61 |
62 | mixed_x = lam * x + (1 - lam) * x[index, :]
63 | mixed_y = lam * y + (1 - lam) * y[index]
64 | return mixed_x, mixed_y, lam, index
65 |
66 |
67 | def train_pcrl_2d(args, data_loader, out_channel=3):
68 | torch.autograd.set_detect_anomaly(True)
69 | nce_k = 16384
70 | nce_t = args.moco_t
71 | nce_m = 0.5
72 | train_loader = data_loader['train']
73 | # create model and optimizer
74 | n_data = len(train_loader)
75 | model = PCRLModel(student=True)
76 | model_ema = PCRLModel(student=False)
77 | model = model.cuda()
78 | model_ema = model_ema.cuda()
79 | optimizer = torch.optim.SGD(model.parameters(),
80 | lr=args.lr,
81 | momentum=args.momentum,
82 | weight_decay=args.weight_decay)
83 |
84 | cudnn.benchmark = True
85 |
86 | optimizer_ema = torch.optim.SGD(model_ema.parameters(),
87 | lr=0,
88 | momentum=0,
89 | weight_decay=0)
90 | model, optimizer = amp.initialize(model, optimizer, opt_level='O1')
91 | model_ema, optimizer_ema = amp.initialize(model_ema, optimizer_ema, opt_level='O1')
92 | # set the contrast memory and
93 | contrast = MemoryC2L(128, n_data, nce_k, nce_t, False).cuda()
94 | criterion = NCECriterion(n_data).cuda()
95 | criterion2 = nn.MSELoss().cuda()
96 |
97 | model = nn.DataParallel(model)
98 | model_ema = nn.DataParallel(model_ema)
99 |
100 | moment_update(model, model_ema, 0)
101 |
102 | for epoch in range(0, args.epochs + 1):
103 |
104 | adjust_learning_rate(epoch, args, optimizer)
105 | print("==> training...")
106 |
107 | time1 = time.time()
108 |
109 | loss, prob = train_rep_C2L(epoch, train_loader, model, model_ema, contrast, criterion, optimizer, criterion2,
110 | )
111 | time2 = time.time()
112 | print('epoch {}, total time {:.2f}'.format(epoch, time2 - time1))
113 | # save model
114 | if epoch % 60 == 0:
115 | # saving the model
116 | print('==> Saving...')
117 | state = {'opt': args, 'state_dict': model.module.model.encoder.state_dict(),
118 | 'decoder_state_dict': model.module.model.decoder.state_dict(),
119 | 'contrast': contrast.state_dict(),
120 | 'optimizer': optimizer.state_dict(), 'epoch': epoch, 'model_ema': model_ema.state_dict()}
121 |
122 | save_file = os.path.join(args.output,
123 | args.model + "_" + args.n + '_' + args.phase + '_' + str(
124 | args.ratio) + '_' + str(epoch) + '.pt')
125 | torch.save(state, save_file)
126 | # help release GPU memory
127 | del state
128 | if epoch == 242:
129 | break
130 | torch.cuda.empty_cache()
131 |
132 |
133 | def train_rep_C2L(epoch, train_loader, model, model_ema, contrast, criterion, optimizer, criterion2):
134 | """
135 | one epoch training for instance discrimination
136 | """
137 |
138 | model.train()
139 | model_ema.eval()
140 |
141 | def set_bn_train(m):
142 | classname = m.__class__.__name__
143 | if classname.find('BatchNorm') != -1:
144 | m.train()
145 |
146 | model_ema.apply(set_bn_train)
147 |
148 | batch_time = AverageMeter()
149 | data_time = AverageMeter()
150 | c2l_loss_meter = AverageMeter()
151 | mg_loss_meter = AverageMeter()
152 | prob_meter = AverageMeter()
153 |
154 | end = time.time()
155 | for idx, (input1, input2, mask1, mask2, gt1, gt2, aug_tensor1, aug_tensor2) in enumerate(train_loader):
156 | data_time.update(time.time() - end)
157 |
158 | bsz = input1.size(0)
159 | x1 = input1.float().cuda()
160 | x2 = input2.float().cuda()
161 | mask1 = mask1.float().cuda()
162 | mask2 = mask2.float().cuda()
163 | aug_tensor1 = aug_tensor1.float().cuda()
164 | aug_tensor2 = aug_tensor2.float().cuda()
165 | gt1 = gt1.float().cuda()
166 | gt2 = gt2.float().cuda()
167 | # ===================forward=====================
168 | # ids for ShuffleBN
169 | shuffle_ids, reverse_ids = get_shuffle_ids(bsz)
170 | alpha1 = np.random.beta(1., 1.)
171 | alpha1 = max(alpha1, 1 - alpha1)
172 | with torch.no_grad():
173 | x2 = x2[shuffle_ids]
174 | feat_k, feats_k = model_ema(x2)
175 | feats_k = [tmp[reverse_ids] for tmp in feats_k]
176 | feat_k = feat_k[reverse_ids]
177 | x2 = x2[reverse_ids]
178 | feat_q, unet_out_alpha, unet_out = model(x1, feats_k, alpha1, aug_tensor1, aug_tensor2)
179 | out = contrast(Normalize(feat_q), Normalize(feat_k))
180 | mixed_x1, mixed_feat1, lam1, index = mixup_data(x1.clone(),
181 | feat_q.clone())
182 | mixed_x2, mixed_feat2, _, _ = mixup_data(x2.clone(), feat_k.clone(),
183 | index=index, lam=lam1)
184 | mixed_gt1, _, _, _ = mixup_data(gt1.clone(), feat_q.clone(), index=index, lam=lam1)
185 | mixed_gt2, _, _, _ = mixup_data(gt2.clone(), feat_q.clone(), index=index, lam=lam1)
186 | alpha2 = np.random.beta(1., 1.)
187 | alpha2 = max(alpha2, 1 - alpha2)
188 | with torch.no_grad():
189 | mixed_x2 = mixed_x2[shuffle_ids]
190 | mixed_feat_k, mixed_feats_k = model_ema(mixed_x2)
191 | mixed_feats_k = [tmp[reverse_ids] for tmp in mixed_feats_k]
192 | mixed_feat_k = mixed_feat_k[reverse_ids]
193 | mixed_x2 = mixed_x2[reverse_ids]
194 |
195 | mixed_feat_q, mixed_unet_out_alpha, mixed_unet_out = model(mixed_x1, mixed_feats_k, alpha2, aug_tensor1,
196 | aug_tensor2, mixup=True)
197 | mixed_feat_q_norm = Normalize(mixed_feat_q)
198 | mixed_feat_k_norm = Normalize(mixed_feat_k)
199 | mixed_feat1_norm = Normalize(mixed_feat1)
200 | mixed_feat2_norm = Normalize(mixed_feat2)
201 |
202 | out2 = contrast(mixed_feat_q_norm, mixed_feat_k_norm)
203 | out3 = contrast(mixed_feat_q_norm, mixed_feat2_norm)
204 | c2l_loss = (criterion(out) + criterion(out2) + criterion(out3)) / 3.
205 | # c2l_loss = criterion(out)
206 | c2l_loss_meter.update(c2l_loss.item(), bsz)
207 | mg_loss = (criterion2(unet_out, mask1) + criterion2(mixed_unet_out, mixed_gt1)
208 | + criterion2(unet_out_alpha, alpha1 * mask1 + (1 - alpha1) * mask2) +
209 | criterion2(mixed_unet_out_alpha, alpha2 * mixed_gt1 + (1 - alpha2) * mixed_gt2)) / 4.
210 | loss = c2l_loss + mg_loss
211 | prob = out[:, 0].mean()
212 |
213 | # ===================backward=====================
214 | optimizer.zero_grad()
215 | #loss.backward()
216 | with amp.scale_loss(loss, optimizer) as scaled_loss:
217 | scaled_loss.backward()
218 | optimizer.step()
219 |
220 | # ===================meters=====================
221 | mg_loss_meter.update(mg_loss.item(), bsz)
222 | prob_meter.update(prob.item(), bsz)
223 |
224 | moment_update(model, model_ema, 0.999)
225 |
226 | torch.cuda.synchronize()
227 | batch_time.update(time.time() - end)
228 | end = time.time()
229 |
230 | # print info
231 | if (idx + 1) % 5 == 0:
232 | print('Train: [{0}][{1}/{2}]\t'
233 | 'BT {batch_time.val:.3f} ({batch_time.avg:.3f})\t'
234 | 'DT {data_time.val:.3f} ({data_time.avg:.3f})\t'
235 | 'c2l loss {c2l_loss.val:.3f} ({c2l_loss.avg:.3f})\t'
236 | 'mg loss {mg_loss.val:.3f} ({mg_loss.avg:.3f})\t'
237 | 'prob {prob.val:.3f} ({prob.avg:.3f})'.format(
238 | epoch, idx + 1, len(train_loader), batch_time=batch_time,
239 | data_time=data_time, c2l_loss=c2l_loss_meter, mg_loss=mg_loss_meter, prob=prob_meter))
240 | # print(out.shape)
241 | sys.stdout.flush()
242 |
243 | return mg_loss_meter.avg, prob_meter.avg
244 |
--------------------------------------------------------------------------------
/pytorch/pcrl_3d.py:
--------------------------------------------------------------------------------
1 | """
2 | Training code for C2L
3 | """
4 | from __future__ import print_function
5 |
6 | import os
7 | import sys
8 | import time
9 |
10 | import numpy as np
11 | import torch
12 | import torch.backends.cudnn as cudnn
13 | import torch.nn as nn
14 | from models.pcrl_model_3d import PCRLModel3d
15 | from losses import NCECriterion
16 | from memory_c2l import MemoryC2L
17 | from models import InsResNet18
18 | from utils import adjust_learning_rate, AverageMeter
19 | import segmentation_models_pytorch as smp
20 |
21 | try:
22 | from apex import amp, optimizers
23 | except ImportError:
24 | pass
25 |
26 |
27 | def Normalize(x):
28 | norm_x = x.pow(2).sum(1, keepdim=True).pow(1. / 2.)
29 | x = x.div(norm_x)
30 | return x
31 |
32 |
33 | def moment_update(model, model_ema, m):
34 | """ model_ema = m * model_ema + (1 - m) model """
35 | for p1, p2 in zip(model.parameters(), model_ema.parameters()):
36 | p2.data.mul_(m).add_(1 - m, p1.detach().data)
37 |
38 |
39 | def get_shuffle_ids(bsz):
40 | """generate shuffle ids for ShuffleBN"""
41 | forward_inds = torch.randperm(bsz).long().cuda()
42 | backward_inds = torch.zeros(bsz).long().cuda()
43 | value = torch.arange(bsz).long().cuda()
44 | backward_inds.index_copy_(0, forward_inds, value)
45 | return forward_inds, backward_inds
46 |
47 |
48 | def mixup_data(x, y, alpha=1.0, index=None, lam=None, use_cuda=True):
49 | '''Returns mixed inputs, pairs of targets, and lambda'''
50 | if lam is None:
51 | lam = np.random.beta(alpha, alpha)
52 | else:
53 | lam = lam
54 |
55 | lam = max(lam, 1 - lam)
56 | batch_size = x.size()[0]
57 | if index is None:
58 | index = torch.randperm(batch_size).cuda()
59 | else:
60 | index = index
61 |
62 | mixed_x = lam * x + (1 - lam) * x[index, :]
63 | mixed_y = lam * y + (1 - lam) * y[index]
64 | return mixed_x, mixed_y, lam, index
65 |
66 |
67 | def train_pcrl_3d(args, data_loader, out_channel=3):
68 | torch.autograd.set_detect_anomaly(True)
69 | nce_k = 16384
70 | nce_t = args.moco_t
71 | nce_m = 0.5
72 | train_loader = data_loader['train']
73 | # create model and optimizer
74 | n_data = len(train_loader)
75 | model = PCRLModel3d(in_channels=1, n_class=1, student=True, norm=args.norm)
76 | model_ema = PCRLModel3d(in_channels=1, n_class=1, student=False, norm=args.norm)
77 | model = model.cuda()
78 | model_ema = model_ema.cuda()
79 | optimizer = torch.optim.SGD(model.parameters(),
80 | lr=args.lr,
81 | momentum=args.momentum,
82 | weight_decay=args.weight_decay)
83 |
84 | cudnn.benchmark = True
85 |
86 | optimizer_ema = torch.optim.SGD(model_ema.parameters(),
87 | lr=0,
88 | momentum=0,
89 | weight_decay=0)
90 | model, optimizer = amp.initialize(model, optimizer, opt_level='O1')
91 | model_ema, optimizer_ema = amp.initialize(model_ema, optimizer_ema, opt_level='O1')
92 | # set the contrast memory and
93 | contrast = MemoryC2L(128, n_data, nce_k, nce_t, False).cuda()
94 | criterion = NCECriterion(n_data).cuda()
95 | criterion2 = nn.MSELoss().cuda()
96 |
97 | model = nn.DataParallel(model)
98 | model_ema = nn.DataParallel(model_ema)
99 |
100 | moment_update(model, model_ema, 0)
101 |
102 | for epoch in range(0, args.epochs + 1):
103 |
104 | adjust_learning_rate(epoch, args, optimizer)
105 | print("==> training...")
106 |
107 | time1 = time.time()
108 |
109 | loss, prob = train_rep_C2L(epoch, train_loader, model, model_ema, contrast, criterion, optimizer, criterion2,
110 | )
111 | time2 = time.time()
112 | print('epoch {}, total time {:.2f}'.format(epoch, time2 - time1))
113 | # save model
114 | if epoch % 60 == 0:
115 | # saving the model
116 | print('==> Saving...')
117 | state = {'opt': args, 'state_dict': model.module.state_dict(),
118 | 'contrast': contrast.state_dict(),
119 | 'optimizer': optimizer.state_dict(), 'epoch': epoch, 'model_ema': model_ema.state_dict()}
120 |
121 | save_file = os.path.join(args.output,
122 | args.model + "_" + args.n + '_' + args.phase + '_' + str(
123 | args.ratio) + '_' + str(epoch) + '.pt')
124 | torch.save(state, save_file)
125 | # help release GPU memory
126 | del state
127 | if epoch == 242:
128 | break
129 |
130 | torch.cuda.empty_cache()
131 |
132 |
133 | def train_rep_C2L(epoch, train_loader, model, model_ema, contrast, criterion, optimizer, criterion2):
134 | """
135 | one epoch training for instance discrimination
136 | """
137 |
138 | model.train()
139 | model_ema.eval()
140 |
141 | def set_bn_train(m):
142 | classname = m.__class__.__name__
143 | if classname.find('BatchNorm') != -1:
144 | m.train()
145 |
146 | model_ema.apply(set_bn_train)
147 |
148 | batch_time = AverageMeter()
149 | data_time = AverageMeter()
150 | c2l_loss_meter = AverageMeter()
151 | mg_loss_meter = AverageMeter()
152 | prob_meter = AverageMeter()
153 |
154 | end = time.time()
155 | for idx, (input1, input2, mask1, mask2, gt1, gt2, aug_tensor1, aug_tensor2) in enumerate(train_loader):
156 | data_time.update(time.time() - end)
157 |
158 | bsz = input1.size(0)
159 | x1 = input1.float().cuda()
160 | x2 = input2.float().cuda()
161 | mask1 = mask1.float().cuda()
162 | mask2 = mask2.float().cuda()
163 | aug_tensor1 = aug_tensor1.float().cuda()
164 | aug_tensor2 = aug_tensor2.float().cuda()
165 | gt1 = gt1.float().cuda()
166 | gt2 = gt2.float().cuda()
167 | # ===================forward=====================
168 | # ids for ShuffleBN
169 | shuffle_ids, reverse_ids = get_shuffle_ids(bsz)
170 | alpha1 = np.random.beta(1., 1.)
171 | alpha1 = max(alpha1, 1 - alpha1)
172 | with torch.no_grad():
173 | x2 = x2[shuffle_ids]
174 | feat_k, feats_k = model_ema(x2)
175 | feats_k = [tmp[reverse_ids] for tmp in feats_k]
176 | feat_k = feat_k[reverse_ids]
177 | x2 = x2[reverse_ids]
178 | feat_q, unet_out_alpha, unet_out = model(x1, feats_k, alpha1, aug_tensor1, aug_tensor2)
179 | out = contrast(Normalize(feat_q), Normalize(feat_k))
180 | mixed_x1, mixed_feat1, lam1, index = mixup_data(x1.clone(),
181 | feat_q.clone())
182 | mixed_x2, mixed_feat2, _, _ = mixup_data(x2.clone(), feat_k.clone(),
183 | index=index, lam=lam1)
184 | mixed_gt1, _, _, _ = mixup_data(gt1.clone(), feat_q.clone(), index=index, lam=lam1)
185 | mixed_gt2, _, _, _ = mixup_data(gt2.clone(), feat_q.clone(), index=index, lam=lam1)
186 | alpha2 = np.random.beta(1., 1.)
187 | alpha2 = max(alpha2, 1 - alpha2)
188 | with torch.no_grad():
189 | mixed_x2 = mixed_x2[shuffle_ids]
190 | mixed_feat_k, mixed_feats_k = model_ema(mixed_x2)
191 | mixed_feats_k = [tmp[reverse_ids] for tmp in mixed_feats_k]
192 | mixed_feat_k = mixed_feat_k[reverse_ids]
193 | mixed_x2 = mixed_x2[reverse_ids]
194 |
195 | mixed_feat_q, mixed_unet_out_alpha, mixed_unet_out = model(mixed_x1, mixed_feats_k, alpha2, aug_tensor1,
196 | aug_tensor2, mixup=True)
197 | mixed_feat_q_norm = Normalize(mixed_feat_q)
198 | mixed_feat_k_norm = Normalize(mixed_feat_k)
199 | mixed_feat1_norm = Normalize(mixed_feat1)
200 | mixed_feat2_norm = Normalize(mixed_feat2)
201 |
202 | out2 = contrast(mixed_feat_q_norm, mixed_feat_k_norm)
203 | out3 = contrast(mixed_feat_q_norm, mixed_feat2_norm)
204 | c2l_loss = (criterion(out) + criterion(out2) + criterion(out3)) / 3.
205 | # c2l_loss = criterion(out)
206 | c2l_loss_meter.update(c2l_loss.item(), bsz)
207 | mg_loss = (criterion2(unet_out, mask1) + criterion2(mixed_unet_out, mixed_gt1)
208 | + criterion2(unet_out_alpha, alpha1 * mask1 + (1 - alpha1) * mask2) +
209 | criterion2(mixed_unet_out_alpha, alpha2 * mixed_gt1 + (1 - alpha2) * mixed_gt2)) / 4.
210 | # mg_loss = (criterion2(unet_out, mask1) + criterion2(unet_out_alpha,
211 | # alpha1 * mask1 + (1 - alpha1) * mask2)) / 2.
212 | loss = c2l_loss + mg_loss
213 | prob = out[:, 0].mean()
214 |
215 | # ===================backward=====================
216 | optimizer.zero_grad()
217 | # loss.backward()
218 | with amp.scale_loss(loss, optimizer) as scaled_loss:
219 | scaled_loss.backward()
220 | optimizer.step()
221 |
222 | # ===================meters=====================
223 | mg_loss_meter.update(mg_loss.item(), bsz)
224 | prob_meter.update(prob.item(), bsz)
225 |
226 | moment_update(model, model_ema, 0.999)
227 |
228 | torch.cuda.synchronize()
229 | batch_time.update(time.time() - end)
230 | end = time.time()
231 |
232 | # print info
233 | if (idx + 1) % 5 == 0:
234 | print('Train: [{0}][{1}/{2}]\t'
235 | 'BT {batch_time.val:.3f} ({batch_time.avg:.3f})\t'
236 | 'DT {data_time.val:.3f} ({data_time.avg:.3f})\t'
237 | 'c2l loss {c2l_loss.val:.3f} ({c2l_loss.avg:.3f})\t'
238 | 'mg loss {mg_loss.val:.3f} ({mg_loss.avg:.3f})\t'
239 | 'prob {prob.val:.3f} ({prob.avg:.3f})'.format(
240 | epoch, idx + 1, len(train_loader), batch_time=batch_time,
241 | data_time=data_time, c2l_loss=c2l_loss_meter, mg_loss=mg_loss_meter, prob=prob_meter))
242 | print(out.shape)
243 | sys.stdout.flush()
244 |
245 | return mg_loss_meter.avg, prob_meter.avg
246 |
--------------------------------------------------------------------------------
/pytorch/preprocess/luna_pre.py:
--------------------------------------------------------------------------------
1 | # coding: utf-8
2 |
3 | """
4 | for subset in `seq 0 9`
5 | do
6 | python -W ignore infinite_generator_3D.py \
7 | --fold $subset \
8 | --scale 32 \
9 | --data /mnt/dataset/shared/zongwei/LUNA16 \
10 | --save generated_cubes
11 | done
12 | """
13 |
14 | # In[1]:
15 |
16 | import warnings
17 | from skimage.transform import resize
18 |
19 | warnings.filterwarnings('ignore')
20 | import os
21 |
22 | os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' # or any {'0', '1', '2'}
23 |
24 | import sys
25 | import random
26 |
27 | import numpy as np
28 | import SimpleITK as sitk
29 |
30 | from tqdm import tqdm
31 | from optparse import OptionParser
32 | from glob import glob
33 | from multiprocessing import Pool
34 |
35 | sys.setrecursionlimit(40000)
36 |
37 | parser = OptionParser()
38 | parser.add_option("--fold", dest="fold", help="fold of subset", default=None, type="int")
39 | parser.add_option("--input_rows", dest="input_rows", help="input rows", default=64, type="int")
40 | parser.add_option("--input_cols", dest="input_cols", help="input cols", default=64, type="int")
41 | parser.add_option("--input_deps", dest="input_deps", help="input deps", default=32, type="int")
42 | parser.add_option("--crop_rows", dest="crop_rows", help="crop rows", default=64, type="int")
43 | parser.add_option("--crop_cols", dest="crop_cols", help="crop cols", default=64, type="int")
44 | parser.add_option("--data", dest="data", help="the directory of LUNA16 dataset", default='/data1/luchixiang/LUNA16',
45 | type="string")
46 | parser.add_option("--save", dest="save", help="the directory of processed 3D cubes",
47 | default='/data1/luchixiang/LUNA16/finetune', type="string")
48 | parser.add_option("--scale", dest="scale", help="scale of the generator", default=16, type="int")
49 | (options, args) = parser.parse_args()
50 | fold = options.fold
51 |
52 | seed = 1
53 | random.seed(seed)
54 |
55 | assert options.data is not None
56 | assert options.save is not None
57 | # assert options.fold >= 0 and options.fold <= 9
58 |
59 | if not os.path.exists(options.save):
60 | os.makedirs(options.save)
61 |
62 |
63 | class setup_config():
64 | hu_max = 1000.0
65 | hu_min = -1000.0
66 | HU_thred = (-150.0 - hu_min) / (hu_max - hu_min)
67 |
68 | def __init__(self,
69 | input_rows=None,
70 | input_cols=None,
71 | input_deps=None,
72 | crop_rows=None,
73 | crop_cols=None,
74 | len_border=None,
75 | len_border_z=None,
76 | scale=None,
77 | DATA_DIR=None,
78 | train_fold=[0, 1, 2, 3, 4],
79 | valid_fold=[5, 6],
80 | test_fold=[7, 8, 9],
81 | len_depth=None,
82 | lung_min=0.7,
83 | lung_max=1.0,
84 | ):
85 | self.input_rows = input_rows
86 | self.input_cols = input_cols
87 | self.input_deps = input_deps
88 | self.crop_rows = crop_rows
89 | self.crop_cols = crop_cols
90 | self.len_border = len_border
91 | self.len_border_z = len_border_z
92 | self.scale = scale
93 | self.DATA_DIR = DATA_DIR
94 | self.train_fold = train_fold
95 | self.valid_fold = valid_fold
96 | self.test_fold = test_fold
97 | self.len_depth = len_depth
98 | self.lung_min = lung_min
99 | self.lung_max = lung_max
100 |
101 | def display(self):
102 | """Display Configuration values."""
103 | print("\nConfigurations:")
104 | for a in dir(self):
105 | if not a.startswith("__") and not callable(getattr(self, a)):
106 | print("{:30} {}".format(a, getattr(self, a)))
107 | print("\n")
108 |
109 |
110 | config = setup_config(input_rows=options.input_rows,
111 | input_cols=options.input_cols,
112 | input_deps=options.input_deps,
113 | crop_rows=options.crop_rows,
114 | crop_cols=options.crop_cols,
115 | scale=options.scale,
116 | len_border=70,
117 | len_border_z=15,
118 | len_depth=3,
119 | lung_min=0.7,
120 | lung_max=0.15,
121 | DATA_DIR=options.data,
122 | )
123 | config.display()
124 |
125 | col_size = [(64, 64, 32), (96, 96, 48), (96, 96, 96), (32, 32, 16), (112, 112, 64)]
126 | input_rows, input_cols, input_depth = (64, 64, 32)
127 |
128 |
129 | def infinite_generator_from_one_volume(img_array, save_dir, name):
130 | img_array[img_array < config.hu_min] = config.hu_min
131 | img_array[img_array > config.hu_max] = config.hu_max
132 | img_array = 1.0 * (img_array - config.hu_min) / (config.hu_max - config.hu_min)
133 | num_pair = 0
134 | while True:
135 | crop_window1, crop_window2 = crop_pair(img_array)
136 | crop_window = np.stack((crop_window1, crop_window2), axis=0)
137 | # print(crop_window.shape)
138 | np.save(os.path.join(save_dir, name + '_img_' + str(num_pair) + '.npy'), crop_window)
139 | num_pair += 1
140 | if num_pair == config.scale:
141 | break
142 |
143 |
144 | def crop_pair(img_array):
145 | while True:
146 | size_x, size_y, size_z = img_array.shape
147 | # print(img_array.shape)
148 | img_array1 = img_array.copy()
149 | img_array2 = img_array.copy()
150 | if size_z - 64 - config.len_depth - 1 - config.len_border_z < config.len_border_z:
151 | pad = size_z - 64 - config.len_depth - 1 - config.len_border_z - config.len_border_z
152 | padding = [0, 0, -pad + 1]
153 | img_array1 = np.pad(img_array1, padding, mode='constant', constant_values=0)
154 |
155 | if size_z - 64 - config.len_depth - 1 - config.len_border_z < config.len_border_z:
156 | pad = size_z - 64 - config.len_depth - 1 - config.len_border_z - config.len_border_z
157 | padding = [0, 0, -pad + 1]
158 | img_array2 = np.pad(img_array2, padding, mode='constant', constant_values=0)
159 | size_z += -pad + 1
160 | while True:
161 | size_index1 = np.random.randint(0, len(col_size))
162 | crop_rows1, crop_cols1, crop_deps1 = col_size[size_index1]
163 | size_index2 = np.random.randint(0, len(col_size))
164 | crop_rows2, crop_cols2, crop_deps2 = col_size[size_index2]
165 | if size_x - crop_rows1 - 1 - config.len_border <= config.len_border:
166 | crop_rows1 -= 32
167 | crop_cols1 -= 32
168 | if size_x - crop_rows2 - 1 - config.len_border <= config.len_border:
169 | crop_rows2 -= 32
170 | crop_cols2 -= 32
171 | start_x1 = random.randint(0 + config.len_border, size_x - crop_rows1 - 1 - config.len_border)
172 | start_y1 = random.randint(0 + config.len_border, size_y - crop_cols1 - 1 - config.len_border)
173 | start_z1 = random.randint(0 + config.len_border_z,
174 | size_z - crop_deps1 - config.len_depth - 1 - config.len_border_z)
175 | start_x2 = random.randint(0 + config.len_border, size_x - crop_rows2 - 1 - config.len_border)
176 | start_y2 = random.randint(0 + config.len_border, size_y - crop_cols2 - 1 - config.len_border)
177 | start_z2 = random.randint(0 + config.len_border_z,
178 | size_z - crop_deps2 - config.len_depth - 1 - config.len_border_z)
179 |
180 | iou = cal_iou(
181 | (start_x1, start_x1 + crop_rows1, start_y1, start_y1 + crop_cols1, start_z1, start_z1 + crop_deps1),
182 | (start_x2, start_x2 + crop_rows2, start_y2, start_y2 + crop_cols2, start_z2,
183 | start_z2 + crop_deps2))
184 | # print(iou, start_x1, start_y1, start_z1, start_x2, start_y2, start_z2)
185 | if iou > 0.25:
186 | break
187 |
188 | crop_window1 = img_array1[start_x1: start_x1 + crop_rows1,
189 | start_y1: start_y1 + crop_cols1,
190 | start_z1: start_z1 + crop_deps1 + config.len_depth,
191 | ]
192 |
193 | crop_window2 = img_array2[start_x2: start_x2 + crop_rows2,
194 | start_y2: start_y2 + crop_cols2,
195 | start_z2: start_z2 + crop_deps2 + config.len_depth,
196 | ]
197 |
198 | if crop_rows1 != input_rows or crop_cols1 != input_cols or crop_deps1 != input_depth:
199 | crop_window1 = resize(crop_window1,
200 | (input_rows, input_cols, input_depth + config.len_depth),
201 | preserve_range=True,
202 | )
203 | if crop_rows2 != input_rows or crop_cols2 != input_cols or crop_deps2 != input_depth:
204 | crop_window2 = resize(crop_window2,
205 | (input_rows, input_cols, input_depth + config.len_depth),
206 | preserve_range=True,
207 | )
208 | t_img1 = np.zeros((input_rows, input_cols, input_depth), dtype=float)
209 | d_img1 = np.zeros((input_rows, input_cols, input_depth), dtype=float)
210 | t_img2 = np.zeros((input_rows, input_cols, input_depth), dtype=float)
211 | d_img2 = np.zeros((input_rows, input_cols, input_depth), dtype=float)
212 | for d in range(input_depth):
213 | for i in range(input_rows):
214 | for j in range(input_cols):
215 | for k in range(config.len_depth):
216 | if crop_window1[i, j, d + k] >= config.HU_thred:
217 | t_img1[i, j, d] = crop_window1[i, j, d + k]
218 | d_img1[i, j, d] = k
219 | break
220 | if k == config.len_depth - 1:
221 | d_img1[i, j, d] = k
222 | for d in range(input_depth):
223 | for i in range(input_rows):
224 | for j in range(input_cols):
225 | for k in range(config.len_depth):
226 | if crop_window2[i, j, d + k] >= config.HU_thred:
227 | t_img2[i, j, d] = crop_window2[i, j, d + k]
228 | d_img2[i, j, d] = k
229 | break
230 | if k == config.len_depth - 1:
231 | d_img2[i, j, d] = k
232 |
233 | d_img1 = d_img1.astype('float32')
234 | d_img1 /= (config.len_depth - 1)
235 | d_img1 = 1.0 - d_img1
236 | d_img2 = d_img2.astype('float32')
237 | d_img2 /= (config.len_depth - 1)
238 | d_img2 = 1.0 - d_img2
239 |
240 | if np.sum(d_img1) > config.lung_max * crop_deps1 * crop_rows1 * crop_cols1:
241 | continue
242 | # print(np.sum(d_img1))
243 | if np.sum(d_img2) > config.lung_max * crop_deps2 * crop_rows2 * crop_cols2:
244 | continue
245 | return crop_window1[:, :, :input_depth], crop_window2[:, :, :input_depth]
246 |
247 |
248 | def get_self_learning_data(fold):
249 | save_path = '/data1/luchixiang/LUNA16/shuffle_0.25'
250 | for index_subset in fold:
251 | print(">> Fold {}".format(index_subset))
252 | luna_subset_path = os.path.join(config.DATA_DIR, "subset" + str(index_subset))
253 | file_list = glob(os.path.join(luna_subset_path, "*.mhd"))
254 | save_dir = os.path.join(save_path, 'subset' + str(index_subset))
255 | os.makedirs(save_dir, exist_ok=True)
256 | for i, img_file in enumerate(tqdm(file_list)):
257 | img_name = os.path.split(img_file)[-1]
258 | img_array = load_sitk_with_resample(img_file)
259 | img_array = sitk.GetArrayFromImage(img_array)
260 | img_array = img_array.transpose(2, 1, 0)
261 | # print(img_array.shape)
262 | infinite_generator_from_one_volume(img_array, save_dir, img_name[:-4])
263 |
264 |
265 | def cal_iou(box1, box2):
266 | """
267 | :param box1: = [xmin1, ymin1, xmax1, ymax1]
268 | :param box2: = [xmin2, ymin2, xmax2, ymax2]
269 | :return:
270 | """
271 | xmin1, xmax1, ymin1, ymax1, zmin1, zmax1 = box1
272 | xmin2, xmax2, ymin2, ymax2, zmin2, zmax2 = box2
273 | # 计算每个矩形的面积
274 | s1 = (xmax1 - xmin1) * (ymax1 - ymin1) * (zmax1 - zmin1) # C的面积
275 | s2 = (xmax2 - xmin2) * (ymax2 - ymin2) * (zmax2 - zmin2) # G的面积
276 |
277 | # 计算相交矩形
278 | xmin = max(xmin1, xmin2)
279 | ymin = max(ymin1, ymin2)
280 | xmax = min(xmax1, xmax2)
281 | ymax = min(ymax1, ymax2)
282 | zmin = max(zmin1, zmin2)
283 | zmax = min(zmax1, zmax2)
284 | w = max(0, xmax - xmin)
285 | h = max(0, ymax - ymin)
286 | d = max(0, zmax - zmin)
287 | area = w * h * d # C∩G的面积
288 | iou = area / (s1 + s2 - area)
289 | return iou
290 |
291 |
292 | def load_sitk_with_resample(img_path):
293 | outsize = [0, 0, 0]
294 | outspacing = [1, 1, 1]
295 |
296 | # 读取文件的size和spacing信息
297 | vol = sitk.ReadImage(img_path)
298 | tmp = sitk.GetArrayFromImage(vol)
299 | inputsize = vol.GetSize()
300 | inputspacing = vol.GetSpacing()
301 |
302 | transform = sitk.Transform()
303 | transform.SetIdentity()
304 | # 计算改变spacing后的size，用物理尺寸/体素的大小
305 | outsize[0] = int(inputsize[0] * inputspacing[0] / outspacing[0] + 0.5)
306 | outsize[1] = int(inputsize[1] * inputspacing[1] / outspacing[1] + 0.5)
307 | outsize[2] = int(inputsize[2] * inputspacing[2] / outspacing[2] + 0.5)
308 |
309 | # 设定重采样的一些参数
310 | resampler = sitk.ResampleImageFilter()
311 | resampler.SetTransform(transform)
312 | resampler.SetInterpolator(sitk.sitkLinear)
313 | resampler.SetOutputOrigin(vol.GetOrigin())
314 | resampler.SetOutputSpacing(outspacing)
315 | resampler.SetOutputDirection(vol.GetDirection())
316 | resampler.SetSize(outsize)
317 | newvol = resampler.Execute(vol)
318 | return newvol
319 |
320 |
321 | with Pool(2) as p:
322 | p.map(get_self_learning_data, [[0, 1, 2, 3, 4], [5, 6, 7, 8, 9]])
323 | # get_self_learning_data([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
324 | # get_self_learning_data([fold])
325 |
--------------------------------------------------------------------------------
/pytorch/train_val_txt/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Luchixiang/PCRL/197bf91d4872f88742686d159892904c965ebdf6/pytorch/train_val_txt/__init__.py
--------------------------------------------------------------------------------
/pytorch/train_val_txt/luna_train.txt:
--------------------------------------------------------------------------------
1 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.334517907433161353885866806005
2 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.141069661700670042960678408762
3 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.210837812047373739447725050963
4 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.564534197011295112247542153557
5 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.231645134739451754302647733304
6 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.250438451287314206124484591986
7 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.250863365157630276148828903732
8 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.310626494937915759224334597176
9 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.278660284797073139172446973682
10 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.227962600322799211676960828223
11 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.657775098760536289051744981056
12 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.272042302501586336192628818865
13 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.188209889686363159853715266493
14 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.313835996725364342034830119490
15 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.868211851413924881662621747734
16 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.404364125369979066736354549484
17 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.898642529028521482602829374444
18 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.219909753224298157409438012179
19 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.238522526736091851696274044574
20 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.832260670372728970918746541371
21 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.979083010707182900091062408058
22 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.124154461048929153767743874565
23 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.108197895896446896160048741492
24 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.137763212752154081977261297097
25 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.323302986710576400812869264321
26 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.122763913896761494371822656720
27 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.216882370221919561230873289517
28 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.323859712968543712594665815359
29 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.154677396354641150280013275227
30 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.395623571499047043765181005112
31 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.128023902651233986592378348912
32 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.138080888843357047811238713686
33 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.534006575256943390479252771547
34 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.194465340552956447447896167830
35 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.302134342469412607966016057827
36 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.280972147860943609388015648430
37 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.305858704835252413616501469037
38 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.295420274214095686326263147663
39 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.311981398931043315779172047718
40 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.333145094436144085379032922488
41 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.313605260055394498989743099991
42 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.640729228179368154416184318668
43 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.303421828981831854739626597495
44 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.219087313261026510628926082729
45 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.213140617640021803112060161074
46 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.313334055029671473836954456733
47 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.293757615532132808762625441831
48 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.202811684116768680758082619196
49 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.332453873575389860371315979768
50 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.109002525524522225658609808059
51 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.566816709786169715745131047975
52 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.126264578931778258890371755354
53 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.716498695101447665580610403574
54 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.139713436241461669335487719526
55 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.975254950136384517744116790879
56 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.281489753704424911132261151767
57 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.525937963993475482158828421281
58 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.241570579760883349458693655367
59 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.310548927038333190233889983845
60 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.269689294231892620436462818860
61 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.752756872840730509471096155114
62 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.194440094986948071643661798326
63 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.397062004302272014259317520874
64 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.295298571102631191572192562523
65 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.805925269324902055566754756843
66 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.126121460017257137098781143514
67 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.188376349804761988217597754952
68 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.187451715205085403623595258748
69 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.277445975068759205899107114231
70 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.130438550890816550994739120843
71 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.249530219848512542668813996730
72 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.430109407146633213496148200410
73 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.417815314896088956784723476543
74 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.144438612068946916340281098509
75 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.724251104254976962355686318345
76 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.905371958588660410240398317235
77 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.129055977637338639741695800950
78 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.450501966058662668272378865145
79 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.534083630500464995109143618896
80 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.317087518531899043292346860596
81 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.511347030803753100045216493273
82 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.105756658031515062000744821260
83 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.621916089407825046337959219998
84 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.134996872583497382954024478441
85 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.826812708000318290301835871780
86 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.139258777898746693365877042411
87 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.294188507421106424248264912111
88 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.111172165674661221381920536987
89 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.146429221666426688999739595820
90 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.275766318636944297772360944907
91 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.663019255629770796363333877035
92 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.309672797925724868457151381131
93 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.282512043257574309474415322775
94 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.336894364358709782463716339027
95 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.243094273518213382155770295147
96 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.140527383975300992150799777603
97 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.146603910507557786636779705509
98 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.616033753016904899083676284739
99 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.314789075871001236641548593165
100 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.479402560265137632920333093071
101 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.247769845138587733933485039556
102 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.169128136262002764211589185953
103 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.259543921154154401875872845498
104 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.272961322147784625028175033640
105 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.193808128386712859512130599234
106 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.503980049263254396021509831276
107 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.145759169833745025756371695397
108 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.128881800399702510818644205032
109 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.163994693532965040247348251579
110 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.183184435049555024219115904825
111 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.162901839201654862079549658100
112 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.128059192202504367870633619224
113 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.306948744223170422945185006551
114 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.171919524048654494439256263785
115 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.179162671133894061547290922949
116 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.226456162308124493341905600418
117 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.287966244644280690737019247886
118 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.216652640878960522552873394709
119 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.161073793312426102774780216551
120 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.200558451375970945040979397866
121 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.335866409407244673864352309754
122 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.308183340111270052562662456038
123 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.121824995088859376862458155637
124 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.111017101339429664883879536171
125 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.310395752124284049604069960014
126 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.888291896309937415860209787179
127 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.113697708991260454310623082679
128 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.193408384740507320589857096592
129 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.326057189095429101398977448288
130 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.222087811960706096424718056430
131 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.756684168227383088294595834066
132 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.206539885154775002929031534291
133 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.114218724025049818743426522343
134 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.162718361851587451505896742103
135 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.186021279664749879526003668137
136 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.197987940182806628828566429132
137 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.910607280658963002048724648683
138 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.286647622786041008124419915089
139 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.331211682377519763144559212009
140 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.134370886216012873213579659366
141 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.152684536713461901635595118048
142 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.315214756157389122376518747372
143 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.935683764293840351008008793409
144 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.208737629504245244513001631764
145 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.179049373636438705059720603192
146 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.952265563663939823135367733681
147 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.655242448149322898770987310561
148 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.270390050141765094612147226290
149 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.250397690690072950000431855143
150 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.173106154739244262091404659845
151 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.561458563853929400124470098603
152 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.690929968028676628605553365896
153 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.231834776365874788440767645596
154 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.861997885565255340442123234170
155 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.768276876111112560631432843476
156 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.162207236104936931957809623059
157 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.334105754605642100456249422350
158 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.231002159523969307155990628066
159 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.184019785706727365023450012318
160 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.300271604576987336866436407488
161 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.161002239822118346732951898613
162 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.100684836163890911914061745866
163 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.168605638657404145360275453085
164 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.144943344795414353192059796098
165 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.168037818448885856452592057286
166 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.458525794434429386945463560826
167 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.183843376225716802567192412456
168 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.802595762867498341201607992711
169 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.970264865033574190975654369557
170 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.106719103982792863757268101375
171 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.108231420525711026834210228428
172 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.801945620899034889998809817499
173 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.892375496445736188832556446335
174 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.104562737760173137525888934217
175 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.139595277234735528205899724196
176 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.259018373683540453277752706262
177 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.325164338773720548739146851679
178 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.674809958213117379592437424616
179 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.922852847124879997825997808179
180 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.102133688497886810253331438797
181 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.227796349777753378641347819780
182 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.259227883564429312164962953756
183 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.139444426690868429919252698606
184 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.283733738239331719775105586296
185 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.235364978775280910367690540811
186 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.144883090372691745980459537053
187 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.199171741859530285887752432478
188 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.190298296009658115773239776160
189 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.692598144815688523679745963696
190 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.803808126682275425758092691689
191 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.240969450540588211676803094518
192 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.221945191226273284587353530424
193 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.272259794130271010519952623746
194 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.302557165094691896097534021075
195 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.461155505515403114280165935891
196 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.212346425055214308006918165305
197 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.230078008964732806419498631442
198 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.253283426904813468115158375647
199 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.964952370561266624992539111877
200 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.116492508532884962903000261147
201 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.223098610241551815995595311693
202 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.133378195429627807109985347209
203 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.267519732763035023633235877753
204 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.232071262560365924176679652948
205 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.281967919138248195763602360723
206 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.183924380327950237519832859527
207 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.743969234977916254223533321294
208 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.265453131727473342790950829556
209 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.191301539558980174217770205256
210 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.939152384493874708850321969356
211 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.299767339686526858593516834230
212 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.283569726884265181140892667131
213 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.100621383016233746780170740405
214 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.187108608022306504546286626125
215 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.707218743153927597786179232739
216 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.267957701183569638795986183786
217 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.504324996863016748259361352296
218 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.253322967203074795232627653819
219 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.213022585153512920098588556742
220 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.226383054119800793308721198594
221 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.117383608379722740629083782428
222 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.306788423710427765311352901943
223 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.142485715518010940961688015191
224 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.159996104466052855396410079250
225 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.147250707071097813243473865421
226 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.199220738144407033276946096708
227 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.227885601428639043345478571594
228 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.323753921818102744511069914832
229 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.325580698241281352835338693869
230 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.220205300714852483483213840572
231 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.280072876841890439628529365478
232 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.225154811831720426832024114593
233 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.121993590721161347818774929286
234 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.153536305742006952753134773630
235 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.216526102138308489357443843021
236 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.252634638822000832774167856951
237 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.143412474064515942785157561636
238 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.443400977949406454649939526179
239 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.466284753932369813717081722101
240 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.113586291551175790743673929831
241 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.217589936421986638139451480826
242 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.172845185165807139298420209778
243 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.217697417596902141600884006982
244 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.311236942972970815890902714604
245 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.156322145453198768801776721493
246 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.184412674007117333405073397832
247 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.156579001330474859527530187095
248 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.199975006921901879512837687266
249 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.126704785377921920210612476953
250 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.885292267869246639232975687131
251 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.270152671889301412052226973069
252 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.187966156856911682643615997798
253 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.176362912420491262783064585333
254 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.557875302364105947813979213632
255 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.137375498893536422914241295628
256 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.124663713663969377020085460568
257 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.217955041973656886482758642958
258 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.470912100568074901744259213968
259 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.776429308535398795601496131524
260 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.163901773171373940247829492387
261 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.943403138251347598519939390311
262 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.669518152156802508672627785405
263 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.159521777966998275980367008904
264 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.624425075947752229712087113746
265 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.265133389948279331857097127422
266 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.192256506776434538421891524301
267 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.296863826932699509516219450076
268 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.842317928015463083368074520378
269 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.822128649427327893802314908658
270 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.614147706162329660656328811671
271 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.127965161564033605177803085629
272 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.106164978370116976238911317774
273 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.199670099218798685977406484591
274 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.710845873679853791427022019413
275 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.319066480138812986026181758474
276 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.126631670596873065041988320084
277 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.324290109423920971676288828329
278 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.336225579776978874775723463327
279 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.162845309248822193437735868939
280 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.149463915556499304732434215056
281 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.314519596680450457855054746285
282 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.225227615446398900698431118292
283 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.272190966764020277652079081128
284 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.177685820605315926524514718990
285 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.275007193025729362844652516689
286 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.203741923654363010377298352671
287 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.293593766328917170359373773080
288 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.254254303842550572473665729969
289 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.321465552859463184018938648244
290 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.278010349511857248000260557753
291 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.969607480572818589276327766720
292 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.277662902666135640561346462196
293 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.975426625618184773401026809852
294 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.121391737347333465796214915391
295 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.204303454658845815034433453512
296 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.172573195301625265149778785969
297 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.319009811633846643966578282371
298 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.269075535958871753309238331179
299 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.300246184547502297539521283806
300 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.191266041369462391833537519639
301 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.276556509002726404418399209377
302 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.191617711875409989053242965150
303 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.244204120220889433826451158706
304 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.487745546557477250336016826588
305 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.202187810895588720702176009630
306 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.194488534645348916700259325236
307 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.202476538079060560282495099956
308 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.323541312620128092852212458228
309 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.199069398344356765037879821616
310 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.965620538050807352935663552285
311 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.145474881373882284343459153872
312 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.160216916075817913953530562493
313 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.202464973819273687476049035824
314 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.125356649712550043958727288500
315 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.100953483028192176989979435275
316 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.292057261351416339496913597985
317 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.324567010179873305471925391582
318 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.274052674198758621258447180130
319 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.264090899378396711987322794314
320 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.210426531621179400035178209430
321 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.237428977311365557972720635401
322 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.970428941353693253759289796610
323 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.215086589927307766627151367533
324 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.135657246677982059395844827629
325 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.334022941831199910030220864961
326 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.244447966386688625240438849169
327 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.619372068417051974713149104919
328 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.100620385482151095585000946543
329 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.603166427542096384265514998412
330 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.367204840301639918160517361062
331 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.313283554967554803238484128406
332 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.154703816225841204080664115280
333 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.205852555362702089950453265567
334 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.244681063194071446501270815660
335 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.961063442349005937536597225349
336 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.398955972049286139436103068984
337 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.252697338970999211181671881792
338 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.177785764461425908755977367558
339 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.148447286464082095534651426689
340 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.219349715895470349269596532320
341 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.404768898286087278137462774930
342 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.123697637451437522065941162930
343 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.171667800241622018839592854574
344 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.197063290812663596858124411210
345 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.625270601160880745954773142570
346 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.192419869605596446455526220766
347 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.850739282072340578344345230132
348 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.339142594937666268384335506819
349 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.208511362832825683639135205368
350 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.481278873893653517789960724156
351 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.268589491017129166376960414534
352 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.268838889380981659524993261082
353 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.179730018513720561213088132029
354 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.229664630348267553620068691756
355 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.177985905159808659201278495182
356 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.167919147233131417984739058859
357 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.173101104804533997398137418032
358 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.339546614783708685476232944897
359 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.741709061958490690246385302477
360 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.146987333806092287055399155268
361 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.316393351033132458296975008261
362 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.232058316950007760548968840196
363 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.141511313712034597336182402384
364 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.103115201714075993579787468219
365 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.104780906131535625872840889059
366 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.248357157975955379661896491341
367 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.337845202462615014431060697507
368 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.261678072503577216586082745513
369 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.584871944187559733312703328980
370 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.287560874054243719452635194040
371 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.826829446346820089862659555750
372 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.185226274332527104841463955058
373 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.100530488926682752765845212286
374 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.218476624578721885561483687176
375 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.198698492013538481395497694975
376 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.164988920331211858091402361989
377 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.145283812746259413053188838096
378 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.797637294244261543517154417124
379 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.204802250386343794613980417281
380 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.390513733720659266816639651938
381 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.200837896655745926888305239398
382 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.320967206808467952819309001585
383 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.178680586845223339579041794709
384 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.114195693932194925962391697338
385 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.228511122591230092662900221600
386 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.142154819868944114554521645782
387 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.428038562098395445838061018440
388 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.156821379677057223126714881626
389 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.230416590143922549745658357505
390 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.385151742584074711135621089321
391 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.202643836890896697853521610450
392 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.303865116731361029078599241306
393 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.148229375703208214308676934766
394 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.670107649586205629860363487713
395 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.330425234131526435132846006585
396 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.329326052298830421573852261436
397 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.211051626197585058967163339846
398 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.333319057944372470283038483725
399 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.268992195564407418480563388746
400 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.631047517458234322522264161877
401 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.304676828064484590312919543151
402 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.463214953282361219537913355115
403 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.205993750485568250373835565680
404 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.115386642382564804180764325545
405 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.244590453955380448651329424024
406 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.320111824803959660037459294083
407 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.910435939545691201820711078950
408 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.449254134266555649028108149727
409 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.107351566259572521472765997306
410 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.885168397833922082085837240429
411 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.259123825760999546551970425757
412 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.419601611032172899567156073142
413 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.799582546798528864710752164515
414 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.401389720232123950202941034290
415 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.214800939017429618305208626314
416 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.163931625580639955914619627409
417 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.119806527488108718706404165837
418 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.211956804948320236390242845468
419 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.312704771348460502013249647868
420 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.177252583002664900748714851615
421 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.203179378754043776171267611064
422 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.307835307280028057486413359377
423 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.141430002307216644912805017227
424 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.897684031374557757145405000951
425 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.276710697414087561012670296643
426 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.209269973797560820442292189762
427 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.211071908915618528829547301883
428 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.270215889102603268207599305185
429 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.304700823314998198591652152637
430 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.125067060506283419853742462394
431 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.122914038048856168343065566972
432 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.216252660192313507027754194207
433 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.780558315515979171413904604168
434 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.779493719385047675154892222907
435 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.161855583909753609742728521805
436 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.300146276266881736689307479986
437 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.232011770495640253949434620907
438 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.189483585244687808087477024767
439 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.811825890493256320617655474043
440 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.242761658169703141430370511586
441 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.438308540025607517017949816111
442 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.265775376735520890308424143898
443 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.390009458146468860187238398197
444 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.265960756233787099041040311282
445 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.252358625003143649770119512644
446 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.750792629100457382099842515038
447 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.194246472548954252250399902051
448 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.296738183013079390785739615169
449 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.188265424231150847356515802868
450 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.290135156874098366424871975734
451 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.338114620394879648539943280992
452 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.238042459915048190592571019348
453 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.275755514659958628040305922764
454 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.143782059748737055784173697516
455 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.301582691063019848479942618641
456 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.132817748896065918417924920957
457 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.174907798609768549012640380786
458 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.178391668569567816549737454720
459 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.100332161840553388986847034053
460 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.138904664700896606480369521124
461 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.129567032250534530765928856531
462 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.174935793360491516757154875981
463 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.986011151772797848993829243183
464 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.234400932423244218697302970157
465 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.201890795870532056891161597218
466 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.200841000324240313648595016964
467 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.140239815496047437552471323962
468 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.745109871503276594185453478952
469 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.190144948425835566841437565646
470 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.627998298349675613581885874395
471 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.338104567770715523699587505022
472 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.910757789941076242457816491305
473 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.842980983137518332429408284002
474 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.133132722052053001903031735878
475 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.118140393257625250121502185026
476 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.339882192295517122002429068974
477 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.205615524269596458818376243313
478 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.219428004988664846407984058588
479 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.257840703452266097926250569223
480 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.205523326998654833765855998037
481 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.154837327827713479309898027966
482 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.188484197846284733942365679565
483 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.120196332569034738680965284519
484 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.725023183844147505748475581290
485 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.553241901808946577644850294647
486 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.238855414831158993232534884296
487 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.219281726101239572270900838145
488 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.245391706475696258069508046497
489 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.309955814083231537823157605135
490 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.413896555982844732694353377538
491 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.176638348958425792989125209419
492 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.152706273988004688708784163325
493 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.101228986346984399347858840086
494 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.229171189693734694696158152904
495 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.111258527162678142285870245028
496 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.100398138793540579077826395208
497 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.162351539386551708034407968929
498 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.308655308958459380153492314021
499 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.275986221854423197884953496664
500 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.224465398054769500989828256685
501 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.338875090785618956575597613546
502 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.504845428620607044098514803031
503 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.693480911433291675609148051914
504 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.273525289046256012743471155680
505 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.291539125579672469833850180824
506 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.596908385953413160131451426904
507 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.328695385904874796172316226975
508 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.258220324170977900491673635112
509 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.323899724653546164058849558431
510 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.138894439026794145866157853158
511 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.323408652979949774528873200770
512 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.394470743585708729682444806008
513 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.160124400349792614505500125883
514 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.299476369290630280560355838785
515 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.334184846571549530235084187602
516 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.134638281277099121660656324702
517 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.111780708132595903430640048766
518 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.183056151780567460322586876100
519 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.246225645401227472829175288633
520 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.244442540088515471945035689377
521 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.112740418331256326754121315800
522 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.866845763956586959109892274084
523 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.246589849815292078281051154201
524 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.285926554490515269336267972830
525 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.262736997975960398949912434623
526 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.143410010885830403003179808334
527 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.439153572396640163898529626096
528 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.255999614855292116767517149228
529 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.110678335949765929063942738609
530 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.160586340600816116143631200450
531 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.397202838387416555106806022938
532 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.245349763807614756148761326488
533 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.980362852713685276785310240144
534 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.106419850406056634877579573537
535 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.255409701134762680010928250229
536 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.569096986145782511000054443951
537 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.948414623428298219623354433437
538 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.171682845383273105440297561095
539 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.803987517543436570820681016103
540 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.315770913282450940389971401304
541 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.149893110752986700464921264055
542 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.190937805243443708408459490152
543 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.955688628308192728558382581802
544 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.252814707117018427472206147014
545 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.170052181746004939527661217512
546 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.174168737938619557573021395302
547 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.123654356399290048011621921476
548 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.669435869708883155232318480131
549 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.292194861362266467652267941663
550 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.282779922503707013097174625409
551 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.275849601663847251574860892603
552 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.251215764736737018371915284679
553 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.177888806135892723698313903329
554 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.156016499715048493339281864474
555 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.117040183261056772902616195387
556 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.200725988589959521302320481687
557 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.173556680294801532247454313511
558 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.119304665257760307862874140576
559 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.414288023902112119945238126594
560 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.106630482085576298661469304872
561 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.119209873306155771318545953948
562 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.266009527139315622265711325223
563 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.309955999522338651429118207446
564 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.306558074682524259000586270818
565 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.240630002689062442926543993263
566 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.129007566048223160327836686225
567 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.168737928729363683423228050295
568 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.315187221221054114974341475212
569 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.286217539434358186648717203667
570 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.164790817284381538042494285101
571 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.475325201787910087416720919680
572 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.463588161905537526756964393219
573 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.329404588567903628160652715124
574 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.308153138776443962077214577161
575 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.120842785645314664964010792308
576 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.341557859428950960906150406596
577 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.221017801605543296514746423389
578 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.130765375502800983459674173881
579 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.167237290696350215427953159586
580 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.652347820272212119124022644822
581 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.245248446973732759194067808002
582 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.330544495001617450666819906758
583 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.321935195060268166151738328001
584 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.182798854785392200340436516930
585 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.249450003033735700817635168066
586 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.270951128717816232360812849541
587 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.241717018262666382493757419144
588 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.404457313935200882843898832756
589 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.316900421002460665752357657094
590 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.297251044869095073091780740645
591 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.233652865358649579816568545171
592 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.277452631455527999380186898011
593 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.338447145504282422142824032832
594 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.233433352108264931671753343044
595 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.147325126373007278009743173696
596 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.161067514225109999586362698069
597 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.725236073737175770730904408416
598 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.247816269490470394602288565775
599 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.561423049201987049884663740668
600 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.307921770358136677021532761235
601 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.168985655485163461062675655739
602 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.176030616406569931557298712518
603 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.618434772073433276874225174904
604 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.658611160253017715059194304729
605 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.513023675145166449943177283490
606 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.170921541362033046216100409521
607 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.877026508860018521147620598474
608 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.119515474430718803379832249911
609 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.313756547848086902190878548835
610 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.235217371152464582553341729176
611 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.151764021165118974848436095034
612 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.136830368929967292376608088362
613 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.290410217650314119074833254861
614 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.107109359065300889765026303943
615 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.416701701108520592702405866796
616 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.329624439086643515259182406526
617 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.153646219551578201092527860224
618 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.150684298696437181894923266019
619 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.254473943359963613733707320244
620 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.316911475886263032009840828684
621 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.226564372605239604660221582288
622 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.465032801496479029639448332481
623 | 1.3.6.1.4.1.14519.5.2.1.6279.6001.254138388912084634057282064266
624 |
--------------------------------------------------------------------------------
/pytorch/utils.py:
--------------------------------------------------------------------------------
1 | import os
2 | import torch
3 | import numpy as np
4 | from PIL import ImageFilter
5 | import random
6 | import math
7 |
8 |
9 | def get_chest_list(txt_path, data_dir):
10 | image_names = []
11 | labels = []
12 | with open(txt_path, "r") as f:
13 | for line in f:
14 | items = line.split()
15 | image_name = items[0]
16 | label = items[1:]
17 | label = [int(i) for i in label]
18 | image_name = os.path.join(data_dir, image_name)
19 | image_names.append(image_name)
20 | labels.append(label)
21 | return image_names, labels
22 |
23 |
24 | def get_luna_list(config, train_fold, valid_fold, test_fold, suffix, file_list):
25 | x_train = []
26 | x_valid = []
27 | x_test = []
28 | for i in train_fold:
29 | for file in os.listdir(os.path.join(config.data, 'subset' + str(i))):
30 | if suffix in file:
31 | if file_list is not None and file.split('_')[0] in file_list:
32 | x_train.append(os.path.join(config.data, 'subset' + str(i), file))
33 | elif file_list is None:
34 | x_train.append(os.path.join(config.data, 'subset' + str(i), file))
35 | for i in valid_fold:
36 | for file in os.listdir(os.path.join(config.data, 'subset' + str(i))):
37 | if suffix in file:
38 | x_valid.append(os.path.join(config.data, 'subset' + str(i), file))
39 | for i in test_fold:
40 | for file in os.listdir(os.path.join(config.data, 'subset' + str(i))):
41 | if suffix in file:
42 | x_test.append(os.path.join(config.data, 'subset' + str(i), file))
43 | return x_train, x_valid, x_test
44 |
45 |
46 | def get_luna_pretrain_list(ratio, path, train_fold):
47 | x_train = []
48 | # for i in train_fold:
49 | # for file in os.listdir(os.path.join(path, 'subset' + str(i))):
50 | # if 'mhd' in file:
51 | # x_train.append(file[:-4])
52 | with open('train_val_txt/luna_train.txt', 'r') as f:
53 | for line in f:
54 | x_train.append(line.strip('\n'))
55 | return x_train[:int(len(x_train) * ratio)]
56 |
57 |
58 | def get_luna_finetune_list(ratio, path, train_fold):
59 | x_train = []
60 | with open('train_val_txt/luna_train.txt', 'r') as f:
61 | for line in f:
62 | x_train.append(line.strip('\n'))
63 | return x_train[int(len(x_train) * ratio):]
64 |
65 |
66 | class Cutout(object):
67 | """Randomly mask out one or more patches from an image.
68 | Args:
69 | n_holes (int): Number of patches to cut out of each image.
70 | length (int): The length (in pixels) of each square patch.
71 | """
72 |
73 | def __init__(self, n_holes, length):
74 | self.n_holes = n_holes
75 | self.length = length
76 |
77 | def __call__(self, img):
78 | """
79 | Args:
80 | img (Tensor): Tensor image of size (C, H, W).
81 | Returns:
82 | Tensor: Image with n_holes of dimension length x length cut out of it.
83 | """
84 | h = img.size(1)
85 | w = img.size(2)
86 |
87 | mask = np.ones((h, w), np.float32)
88 |
89 | for n in range(self.n_holes):
90 | y = np.random.randint(h)
91 | x = np.random.randint(w)
92 |
93 | y1 = np.clip(y - self.length // 2, 0, h)
94 | y2 = np.clip(y + self.length // 2, 0, h)
95 | x1 = np.clip(x - self.length // 2, 0, w)
96 | x2 = np.clip(x + self.length // 2, 0, w)
97 |
98 | mask[y1: y2, x1: x2] = 0.
99 |
100 | mask = torch.from_numpy(mask)
101 | mask = mask.expand_as(img)
102 | img = img * mask
103 |
104 | return img
105 |
106 |
107 | def adjust_learning_rate(epoch, args, optimizer):
108 | """Sets the learning rate to the initial LR decayed by 0.2 every steep step"""
109 | # iterations = opt.lr_decay_epochs.split(',')
110 | # opt.lr_decay_epochs_list = list([])
111 | # for it in iterations:
112 | # opt.lr_decay_epochs_list.append(int(it))
113 | # steps = np.sum(epoch > np.asarray(opt.lr_decay_epochs_list))
114 | # if steps > 0:
115 | # new_lr = opt.lr * (opt.lr_decay_rate ** steps)
116 | # for param_group in optimizer.param_groups:
117 | # param_group['lr'] = new_lr
118 | lr = args.lr
119 | lr *= 0.5 * (1. + math.cos(math.pi * epoch / args.epochs))
120 | for param_group in optimizer.param_groups:
121 | param_group['lr'] = lr
122 |
123 |
124 | class AverageMeter(object):
125 | """Computes and stores the average and current value"""
126 |
127 | def __init__(self):
128 | self.val = 0
129 | self.avg = 0
130 | self.sum = 0
131 | self.count = 0
132 | self.reset()
133 |
134 | def reset(self):
135 | self.val = 0
136 | self.avg = 0
137 | self.sum = 0
138 | self.count = 0
139 |
140 | def update(self, val, n=1):
141 | self.val = val
142 | self.sum += val * n
143 | self.count += n
144 | self.avg = self.sum / self.count
145 |
146 |
147 | def accuracy(output, target, topk=(1,)):
148 | """Computes the accuracy over the k top predictions for the specified values of k"""
149 | with torch.no_grad():
150 | maxk = max(topk)
151 | batch_size = target.size(0)
152 |
153 | _, pred = output.topk(maxk, 1, True, True)
154 | pred = pred.t()
155 | correct = pred.eq(target.view(1, -1).expand_as(pred))
156 |
157 | res = []
158 | for k in topk:
159 | correct_k = correct[:k].view(-1).float().sum(0, keepdim=True)
160 | res.append(correct_k.mul_(100.0 / batch_size))
161 | return res
162 |
163 |
164 | class GaussianBlur(object):
165 | """Gaussian blur augmentation in SimCLR https://arxiv.org/abs/2002.05709"""
166 |
167 | def __init__(self, sigma=[.1, 2.]):
168 | self.sigma = sigma
169 |
170 | def __call__(self, x):
171 | sigma = random.uniform(self.sigma[0], self.sigma[1])
172 | x = x.filter(ImageFilter.GaussianBlur(radius=sigma))
173 | return x
174 |
175 |
176 | def get_params(output_size, img_size=(224, 224)):
177 | h, w = img_size
178 | th, tw = output_size
179 | i = random.randint(0, h - th + 1)
180 | j = random.randint(0, w - tw + 1)
181 | return i, j, th, tw
182 |
183 |
184 | def get_random_2dboxs(box_num, batch_size):
185 | """
186 | get the random box on batch images
187 | Arguments:
188 | box_num: the number of boxes on every image
189 | batch_size: batch size
190 | Returns:
191 | a set of boxes : [K, 5]
192 | K is batch_size * box_num
193 | every box correspond to [batch_index, x1,y1,x2,y2]
194 | batch_index is the index of image in the batch
195 | """
196 | boxes = []
197 | for bid in range(batch_size):
198 | bboxes = []
199 | while len(bboxes) < box_num:
200 | h = random.randint(50, 223) # 下限调高一点
201 | w = random.randint(50, 223)
202 | if 10000 <= h * w <= 30000:
203 | i, j, h, w = get_params((h, w))
204 | bboxes.append((i, j, i + h, j + w))
205 | boxes.append((bid, i, j, i + h, w + j))
206 | return np.array(boxes)
207 |
208 |
209 | def get_batch_random_crop(box_num=50, batch_size=128):
210 | boxes = []
211 | all_boxes = []
212 | while len(boxes) < box_num:
213 | h = random.randint(50, 223) # 下限调高一点
214 | w = random.randint(50, 223)
215 | if 10000 <= h * w <= 30000:
216 | i, j, h, w = get_params((h, w))
217 | # bboxes.append((i, j, i + h, j + w))
218 | boxes.append((i, j, i + h, w + j))
219 | for bid in range(batch_size):
220 | for box in boxes:
221 | i, j, m, n = box
222 | all_boxes.append((bid, i, j, m, n))
223 | return np.array(boxes), np.array(all_boxes)
224 |
225 |
226 | def box_area(boxes):
227 | """
228 | Computes the area of a set of bounding boxes, which are specified by its
229 | (x1, y1, x2, y2) coordinates.
230 | Arguments:
231 | boxes (Tensor[N, 4]): boxes for which the area will be computed. They
232 | are expected to be in (x1, y1, x2, y2) format
233 | Returns:
234 | area (Tensor[N]): area for each box
235 | """
236 | return (boxes[:, 2] - boxes[:, 0]) * (boxes[:, 3] - boxes[:, 1])
237 |
238 |
239 | # implementation from https://github.com/kuangliu/torchcv/blob/master/torchcv/utils/box.py
240 | # with slight modifications
241 | def box_iou(boxes1, boxes2):
242 | """
243 | Return intersection-over-union (Jaccard index) of boxes.
244 | Both sets of boxes are expected to be in (x1, y1, x2, y2) format.
245 | Arguments:
246 | boxes1 (Tensor[N, 4])
247 | boxes2 (Tensor[M, 4])
248 | Returns:
249 | iou (Tensor[N, M]): the NxM matrix containing the pairwise
250 | IoU values for every element in boxes1 and boxes2
251 | """
252 | area1 = box_area(boxes1)
253 | area2 = box_area(boxes2)
254 |
255 | lt = torch.max(boxes1[:, None, :2], boxes2[:, :2]) # [N,M,2]
256 | rb = torch.min(boxes1[:, None, 2:], boxes2[:, 2:]) # [N,M,2]
257 | # pdb.set_trace()
258 | wh = (rb - lt).clamp(min=0) # [N,M,2]
259 |
260 | inter = wh[:, :, 0] * wh[:, :, 1] # [N,M]
261 |
262 | iou = inter / (area1[:, None] + area2 - inter)
263 | return iou
264 |
265 |
266 | def same(boxes):
267 | """
268 | return a matrix whose element denotes whether the element
269 | in the boxes is in the same image
270 | Arguments:
271 | boxes (Tensor[N]): batch_index of boxes
272 | Returns:
273 | area (Tensor[N, N])
274 | """
275 | boxes = boxes.unsqueeze(dim=1)
276 | boxes2 = boxes.t()
277 | return torch.eq(boxes, boxes2)
278 |
--------------------------------------------------------------------------------