├── LICENSE ├── README.md ├── cam2ir.py ├── data ├── cls_labels_voc.npy ├── data_coco.py ├── data_voc.py ├── labels_coco.txt ├── make_cls_labels.py ├── test.txt ├── train_coco.txt ├── train_voc.txt ├── trainaug_voc.txt ├── val_coco.txt └── val_voc.txt ├── eval_cam.py ├── eval_sem_seg.py ├── fig └── framework1.png ├── log ├── sipe_coco.log ├── sipe_coco_rep.log ├── sipe_voc.log └── sipe_voc_rep.log ├── make_cam.py ├── make_seg_labels.py ├── network ├── resnet50.py ├── resnet50_SIPE.py └── resnet50_irn.py ├── requirements.txt ├── run_coco.sh ├── run_voc.sh ├── tool ├── imutils.py ├── indexing.py ├── inferutils.py ├── iouutils.py ├── pyutils.py ├── torchutils.py └── visualization.py ├── train_irn.py └── train_resnet50_SIPE.py /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 chenqi1126 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Self-supervised Image-specific Prototype Exploration for Weakly Supervised Semantic Segmentation (SIPE) 2 | 3 | framework 4 | 5 | The implementation of [**Self-supervised Image-specific Prototype Exploration for Weakly Supervised Semantic Segmentation**](https://openaccess.thecvf.com/content/CVPR2022/papers/Chen_Self-Supervised_Image-Specific_Prototype_Exploration_for_Weakly_Supervised_Semantic_Segmentation_CVPR_2022_paper.pdf), Qi Chen, Lingxiao Yang, Jianhuang Lai, and Xiaohua Xie, CVPR 2022. 6 | 7 | ## Abstract 8 | Weakly Supervised Semantic Segmentation (WSSS) based on image-level labels has attracted much attention due to low annotation costs. Existing methods often rely on Class Activation Mapping (CAM) that measures the correlation between image pixels and classifier weight. However, the classifier focuses only on the discriminative regions while ignoring other useful information in each image, resulting in incomplete localization maps. To address this issue, we propose a Self-supervised Image-specific Prototype Exploration (SIPE) that consists of an Image-specific Prototype Exploration (IPE) and a General-Specific Consistency (GSC) loss. Specifically, IPE tailors prototypes for every image to capture complete regions, formed our Image-Specific CAM (IS-CAM), which is realized by two sequential steps. In addition, GSC is proposed to construct the consistency of general CAM and our specific IS-CAM, which further optimizes the feature representation and empowers a self-correction ability of prototype exploration. Extensive experiments are conducted on PASCAL VOC 2012 and MS COCO 2014 segmentation benchmark and results show our SIPE achieves new state-of-the-art performance using only image-level labels. 9 | 10 | ## Environment 11 | 12 | - Python >= 3.6.6 13 | - Pytorch >= 1.6.0 14 | - Torchvision 15 | 16 | ## Usage 17 | 18 | #### Step 1. Prepare Dataset 19 | 20 | - PASCAL VOC 2012: [Download](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/). 21 | - MS COCO 2014: [Image](https://cocodataset.org/#home) and [Label](https://drive.google.com/file/d/1Pm_OH8an5MzZh56QKTcdlXNI3RNmZB9d/view?usp=sharing). 22 | 23 | #### Step 2. Train SIPE 24 | 25 | ```bash 26 | # PASCAL VOC 2012 27 | bash run_voc.sh 28 | 29 | # MS COCO 2014 30 | bash run_coco.sh 31 | ``` 32 | 33 | #### Step 3. Train Fully Supervised Segmentation Models 34 | 35 | To train fully supervised segmentation models, we refer to [deeplab-pytorch](https://github.com/kazuto1011/deeplab-pytorch) and [seamv1](https://github.com/YudeWang/semantic-segmentation-codebase/tree/main/experiment/seamv1-pseudovoc). 36 | 37 | ## Results 38 | 39 | #### Localization maps 40 | 41 | | Dataset | Model | mIoU (Train) | Weight | Training log | 42 | | --------------- | ----------- | ------------ | ------------------------------------------------------------ | -------------------------------- | 43 | | PASCAL VOC 2012 | CVPR submit | 58.65 | [Download](https://drive.google.com/file/d/1-_GXZq-1gxcbR7FdY1888tnxBAE39R-P/view?usp=sharing) | [Logfile](log/sipe_voc.log) | 44 | | PASCAL VOC 2012 | This repo | 58.88 | [Download](https://drive.google.com/file/d/1YYYYXleRperCUrhcU4pT1eXybhlUQedW/view?usp=sharing) | [Logfile](log/sipe_voc_rep.log) | 45 | | MS COCO 2014 | CVPR submit | 34.41 | [Download](https://drive.google.com/file/d/1qWLvgjyd9eunyWJPyj02HcDQciiMKMu0/view?usp=sharing) | [Logfile](log/sipe_coco.log) | 46 | | MS COCO 2014 | This repo | 35.05 | [Download](https://drive.google.com/file/d/103gU8AmTDXSnebh2q9xihOSxw4yoPGZb/view?usp=sharing) | [Logfile](log/sipe_coco_rep.log) | 47 | 48 | #### Segmentation maps 49 | 50 | | Dataset | Model | mIoU (Val) | mIoU (Test) | Weight | 51 | | --------------- | ----------- | ------------ | ------------------------------------------------------------ | -------------------------------- | 52 | | PASCAL VOC 2012 | WideResNet38 | 68.2 | [69.5](http://host.robots.ox.ac.uk:8080/anonymous/NGICBM.html) | [Download](https://drive.google.com/file/d/1V2h-5znTXWQNvOq2cH1nACF9Ym6wd02-/view?usp=sharing) | 53 | | PASCAL VOC 2012 | ResNet101 | 68.8 | [69.7](http://host.robots.ox.ac.uk:8080/anonymous/UU6VNX.html) | [Download](https://drive.google.com/file/d/1wN7-O-aXNtPgHPIvzr14tfaLssOc2FzI/view?usp=sharing) | 54 | | MS COCO 2014 | WideResNet38 | 43.6 | - | [Download](https://drive.google.com/file/d/1w9jyHbcR8GzHMNo0QIelWQJFxC33wit4/view?usp=sharing)| 55 | | MS COCO 2014 | ResNet101 | 40.6 | - | [Download](https://drive.google.com/file/d/1WQUzL8wwRnu_9kWh-t-mItyyrMTsH8Kn/view?usp=sharing)| 56 | 57 | ## Citation 58 | ``` 59 | @InProceedings{Chen_2022_CVPR_SIPE, 60 | author = {Chen, Qi and Yang, Lingxiao and Lai, Jian-Huang and Xie, Xiaohua}, 61 | title = {Self-Supervised Image-Specific Prototype Exploration for Weakly Supervised Semantic Segmentation}, 62 | booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, 63 | month = {June}, 64 | year = {2022}, 65 | pages = {4288-4298} 66 | } 67 | ``` 68 | -------------------------------------------------------------------------------- /cam2ir.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | import imageio 4 | from torch import multiprocessing 5 | from torch.utils.data import DataLoader 6 | from data import data_voc, data_coco 7 | from tool import torchutils, imutils 8 | import argparse 9 | 10 | def _work(process_id, infer_dataset, args): 11 | 12 | databin = infer_dataset[process_id] 13 | infer_data_loader = DataLoader(databin, shuffle=False, num_workers=0, pin_memory=False) 14 | cam_out_dir = os.path.join(args.session_name, 'npy') 15 | ir_label_out_dir = os.path.join(args.session_name, 'ir_label') 16 | 17 | for iter, pack in enumerate(infer_data_loader): 18 | img_name = pack['name'][0] 19 | img = pack['img'][0].numpy() 20 | 21 | cam_dict = np.load(os.path.join(cam_out_dir, img_name + '.npy'), allow_pickle=True).item() 22 | cams = cam_dict['IS_CAM'] 23 | keys = cam_dict['keys'] 24 | 25 | # 1. find confident fg & bg 26 | fg_conf_cam = cams 27 | fg_conf_cam = np.argmax(fg_conf_cam, axis=0) 28 | pred = imutils.crf_inference_label(img, fg_conf_cam, n_labels=keys.shape[0]) 29 | fg_conf = keys[pred] 30 | 31 | bg_conf_cam = cams 32 | bg_conf_cam[0] = bg_conf_cam[0]*0.5 33 | bg_conf_cam = np.argmax(bg_conf_cam, axis=0) 34 | pred = imutils.crf_inference_label(img, bg_conf_cam, n_labels=keys.shape[0]) 35 | bg_conf = keys[pred] 36 | 37 | # 2. combine confident fg & bg 38 | conf = fg_conf.copy() 39 | conf[fg_conf == 0] = 255 40 | conf[bg_conf + fg_conf == 0] = 0 41 | 42 | imageio.imwrite(os.path.join(ir_label_out_dir, img_name + '.png'), conf.astype(np.uint8)) 43 | 44 | if process_id == args.num_workers - 1 and iter % (len(databin) // 20) == 0: 45 | print("%d " % ((5 * iter + 1) // (len(databin) // 20)), end='') 46 | 47 | 48 | if __name__ == '__main__': 49 | 50 | parser = argparse.ArgumentParser() 51 | parser.add_argument("--dataset", default="voc", type=str) 52 | parser.add_argument("--num_workers", default=os.cpu_count()//2, type=int) 53 | parser.add_argument("--session_name", default='exp', type=str) 54 | args = parser.parse_args() 55 | 56 | assert args.dataset in ['voc', 'coco'], 'Dataset must be voc or coco in this project.' 57 | 58 | if args.dataset == 'voc': 59 | dataset_root = '../PascalVOC2012/VOCdevkit/VOC2012' 60 | dataset = data_voc.VOC12ImageDataset('data/train_' + args.dataset + '.txt', voc12_root=dataset_root, img_normal=None, to_torch=False) 61 | 62 | elif args.dataset == 'coco': 63 | dataset_root = "../../dataset/ms_coco_14&15/images" 64 | dataset = data_coco.COCOImageDataset('data/train_' + args.dataset + '.txt', coco_root=dataset_root, img_normal=None, to_torch=False) 65 | 66 | dataset = torchutils.split_dataset(dataset, args.num_workers) 67 | os.makedirs(os.path.join(args.session_name, 'ir_label'), exist_ok=True) 68 | 69 | print('[ ', end='') 70 | multiprocessing.spawn(_work, nprocs=args.num_workers, args=(dataset, args), join=True) 71 | print(']') 72 | -------------------------------------------------------------------------------- /data/cls_labels_voc.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chenqi1126/SIPE/e0c6a3bc578a6bade2cf85fe34a1793dce71170d/data/cls_labels_voc.npy -------------------------------------------------------------------------------- /data/data_coco.py: -------------------------------------------------------------------------------- 1 | 2 | import numpy as np 3 | import torch 4 | from torch.utils.data import Dataset 5 | import PIL.Image 6 | import os.path 7 | import scipy.misc 8 | import imageio 9 | from torchvision import transforms 10 | from tool import pyutils, imutils, torchutils 11 | import pdb 12 | import cv2 13 | import random 14 | 15 | IMG_FOLDER_NAME = "JPEGImages" 16 | MASK_FOLDER_NAME = "SegmentationClass" 17 | 18 | 19 | def load_image_label_list_from_npy(img_name_list): 20 | 21 | cls_labels_dict = np.load('data/cls_labels_coco.npy', allow_pickle=True).item() 22 | 23 | return [cls_labels_dict[img_name] for img_name in img_name_list] 24 | 25 | 26 | def get_img_path(img_name, coco_root): 27 | return os.path.join(coco_root, 'train2014', img_name + '.jpg') 28 | 29 | def get_mask_path(mask_name, coco_root): 30 | return os.path.join(coco_root, MASK_FOLDER_NAME, mask_name + '.png') 31 | 32 | def load_img_name_list(dataset_path): 33 | 34 | img_gt_name_list = open(dataset_path).read().splitlines() 35 | img_name_list = [img_gt_name for img_gt_name in img_gt_name_list] 36 | 37 | return img_name_list 38 | 39 | 40 | class TorchvisionNormalize(): 41 | def __init__(self, mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)): 42 | self.mean = mean 43 | self.std = std 44 | 45 | def __call__(self, img): 46 | imgarr = np.asarray(img) 47 | proc_img = np.empty_like(imgarr, np.float32) 48 | 49 | proc_img[..., 0] = (imgarr[..., 0] / 255. - self.mean[0]) / self.std[0] 50 | proc_img[..., 1] = (imgarr[..., 1] / 255. - self.mean[1]) / self.std[1] 51 | proc_img[..., 2] = (imgarr[..., 2] / 255. - self.mean[2]) / self.std[2] 52 | 53 | return proc_img 54 | 55 | class COCOImageDataset(Dataset): 56 | def __init__(self, img_name_list_path, coco_root, resize=None, 57 | resize_long=None, rescale=None, img_normal=TorchvisionNormalize(), hor_flip=False, 58 | crop_size=None, crop_method=None, to_torch=True): 59 | self.img_name_list = load_img_name_list(img_name_list_path) 60 | self.coco_root = coco_root 61 | 62 | self.resize = resize 63 | self.resize_long = resize_long 64 | self.rescale = rescale 65 | self.crop_size = crop_size 66 | self.img_normal = img_normal 67 | self.hor_flip = hor_flip 68 | self.crop_method = crop_method 69 | self.to_torch = to_torch 70 | 71 | def __len__(self): 72 | return len(self.img_name_list) 73 | 74 | 75 | def __getitem__(self, idx): 76 | name = self.img_name_list[idx] 77 | 78 | img = np.array(PIL.Image.open(get_img_path(name, self.coco_root)).convert('RGB')) 79 | 80 | if self.resize: 81 | img = imutils.pil_resize(img, size=self.resize, order=3) 82 | 83 | if self.resize_long: 84 | img = imutils.random_resize_long(img, self.resize_long[0], self.resize_long[1]) 85 | 86 | if self.rescale: 87 | img = imutils.random_scale(img, scale_range=self.rescale, order=3) 88 | 89 | if self.img_normal: 90 | img = self.img_normal(img) 91 | 92 | if self.hor_flip: 93 | img = imutils.random_lr_flip(img) 94 | 95 | if self.crop_size: 96 | valid_mask = torch.zeros((81, self.crop_size, self.crop_size)) 97 | if self.crop_method == "random": 98 | img, box = imutils.random_crop(img, self.crop_size, 0) 99 | valid_mask[:,box[0]:box[1],box[2]:box[3]] = 1 100 | else: 101 | img = imutils.top_left_crop(img, self.crop_size, 0) 102 | valid_mask = torch.ones((81, self.crop_size, self.crop_size)) 103 | 104 | else: 105 | valid_mask = torch.ones((81, img.shape[0], img.shape[1])) 106 | 107 | if self.to_torch: 108 | img = np.ascontiguousarray(imutils.HWC_to_CHW(img)) 109 | 110 | return {'name': name, 'img': img, 'valid_mask': valid_mask} 111 | 112 | class COCOClsDataset(COCOImageDataset): 113 | 114 | def __init__(self, img_name_list_path, coco_root, resize=None, 115 | resize_long=None, rescale=None, img_normal=TorchvisionNormalize(), hor_flip=False, 116 | crop_size=None, crop_method=None): 117 | super().__init__(img_name_list_path, coco_root, resize, 118 | resize_long, rescale, img_normal, hor_flip, 119 | crop_size, crop_method) 120 | self.label_list = load_image_label_list_from_npy(self.img_name_list) 121 | 122 | def __getitem__(self, idx): 123 | out = super().__getitem__(idx) 124 | 125 | out['label'] = torch.from_numpy(self.label_list[idx]) 126 | 127 | return out 128 | 129 | class COCOClsDatasetMSF(COCOClsDataset): 130 | 131 | def __init__(self, img_name_list_path, coco_root, img_normal=TorchvisionNormalize(), scales=(1.0,)): 132 | super().__init__(img_name_list_path, coco_root) 133 | self.scales = scales 134 | 135 | def __getitem__(self, idx): 136 | name = self.img_name_list[idx] 137 | img = np.asarray(PIL.Image.open(get_img_path(name, self.coco_root)).convert('RGB')) 138 | # img = imageio.imread(get_img_path(name, self.coco_root)).convert('RGB') 139 | 140 | ms_img_list = [] 141 | for s in self.scales: 142 | if s == 1: 143 | s_img = img 144 | else: 145 | s_img = imutils.pil_rescale(img, s, order=3) 146 | s_img = self.img_normal(s_img) 147 | s_img = imutils.HWC_to_CHW(s_img) 148 | ms_img_list.append(np.stack([s_img, np.flip(s_img, -1)], axis=0)) 149 | 150 | if len(self.scales) == 1: 151 | ms_img_list = ms_img_list[0] 152 | 153 | # mask = PIL.Image.open(get_mask_path(name, self.coco_root)) 154 | 155 | out = {"name": name, "img": ms_img_list, "size": (img.shape[0], img.shape[1]), 156 | "label": torch.from_numpy(self.label_list[idx]), "img_path": get_img_path(name, self.coco_root)} 157 | 158 | 159 | return out 160 | 161 | class COCOSegmentationDataset(Dataset): 162 | 163 | def __init__(self, img_name_list_path, label_dir, crop_size, coco_root, 164 | rescale=None, img_normal=TorchvisionNormalize(), hor_flip=False, 165 | crop_method = 'random'): 166 | 167 | self.img_name_list = load_img_name_list(img_name_list_path) 168 | self.coco_root = coco_root 169 | 170 | self.label_dir = label_dir 171 | 172 | self.rescale = rescale 173 | self.crop_size = crop_size 174 | self.img_normal = img_normal 175 | self.hor_flip = hor_flip 176 | self.crop_method = crop_method 177 | 178 | def __len__(self): 179 | return len(self.img_name_list) 180 | 181 | def __getitem__(self, idx): 182 | name = self.img_name_list[idx] 183 | 184 | img = PIL.Image.open(get_img_path(name, self.coco_root)).convert('RGB') 185 | label = imageio.imread(os.path.join(self.label_dir, name + '.png')) 186 | 187 | img = np.asarray(img) 188 | 189 | if self.rescale: 190 | img, label = imutils.random_scale((img, label), scale_range=self.rescale, order=(3, 0)) 191 | 192 | if self.img_normal: 193 | img = self.img_normal(img) 194 | 195 | if self.hor_flip: 196 | img, label = imutils.random_lr_flip((img, label)) 197 | 198 | 199 | if self.crop_method == "random": 200 | img, label = imutils.random_crop((img, label), self.crop_size, (0, 255)) 201 | else: 202 | img = imutils.top_left_crop(img, self.crop_size, 0) 203 | label = imutils.top_left_crop(label, self.crop_size, 255) 204 | 205 | label = label.astype(np.long) 206 | img = imutils.HWC_to_CHW(img) 207 | return {'name': name, 'img': img, 'label': label} 208 | 209 | class GetAffinityLabelFromIndices(): 210 | 211 | def __init__(self, indices_from, indices_to): 212 | 213 | self.indices_from = indices_from 214 | self.indices_to = indices_to 215 | 216 | def __call__(self, segm_map): 217 | 218 | segm_map_flat = np.reshape(segm_map, -1) 219 | 220 | segm_label_from = np.expand_dims(segm_map_flat[self.indices_from], axis=0) 221 | segm_label_to = segm_map_flat[self.indices_to] 222 | 223 | valid_label = np.logical_and(np.less(segm_label_from, 81), np.less(segm_label_to, 81)) 224 | 225 | equal_label = np.equal(segm_label_from, segm_label_to) 226 | 227 | pos_affinity_label = np.logical_and(equal_label, valid_label) 228 | 229 | bg_pos_affinity_label = np.logical_and(pos_affinity_label, np.equal(segm_label_from, 0)).astype(np.float32) 230 | fg_pos_affinity_label = np.logical_and(pos_affinity_label, np.greater(segm_label_from, 0)).astype(np.float32) 231 | 232 | neg_affinity_label = np.logical_and(np.logical_not(equal_label), valid_label).astype(np.float32) 233 | 234 | return torch.from_numpy(bg_pos_affinity_label), torch.from_numpy(fg_pos_affinity_label), \ 235 | torch.from_numpy(neg_affinity_label) 236 | 237 | class COCOAffinityDataset(COCOSegmentationDataset): 238 | def __init__(self, img_name_list_path, label_dir, crop_size, coco_root, 239 | indices_from, indices_to, 240 | rescale=None, img_normal=TorchvisionNormalize(), hor_flip=False, crop_method=None): 241 | super().__init__(img_name_list_path, label_dir, crop_size, coco_root, rescale, img_normal, hor_flip, crop_method=crop_method) 242 | 243 | self.extract_aff_lab_func = GetAffinityLabelFromIndices(indices_from, indices_to) 244 | 245 | def __len__(self): 246 | return len(self.img_name_list) 247 | 248 | def __getitem__(self, idx): 249 | out = super().__getitem__(idx) 250 | 251 | reduced_label = imutils.pil_rescale(out['label'].astype(np.uint8), 0.25, 0) 252 | 253 | out['aff_bg_pos_label'], out['aff_fg_pos_label'], out['aff_neg_label'] = self.extract_aff_lab_func(reduced_label) 254 | 255 | return out 256 | -------------------------------------------------------------------------------- /data/data_voc.py: -------------------------------------------------------------------------------- 1 | 2 | import numpy as np 3 | import torch 4 | from torch.utils.data import Dataset 5 | import PIL.Image 6 | import os.path 7 | import scipy.misc 8 | import imageio 9 | from torchvision import transforms 10 | from tool import pyutils, imutils, torchutils 11 | import pdb 12 | import cv2 13 | import random 14 | 15 | IMG_FOLDER_NAME = "JPEGImages" 16 | MASK_FOLDER_NAME = "SegmentationClass" 17 | 18 | 19 | def load_image_label_list_from_npy(img_name_list): 20 | 21 | cls_labels_dict = np.load('data/cls_labels_voc.npy', allow_pickle=True).item() 22 | 23 | return [cls_labels_dict[img_name] for img_name in img_name_list] 24 | 25 | 26 | def load_image_label_pair_list_from_npy(img_name_pair_list): 27 | 28 | cls_labels_dict = np.load('data/cls_labels_voc.npy', allow_pickle=True).item() 29 | 30 | return [(cls_labels_dict[img_name_pair[0]], cls_labels_dict[img_name_pair[1]]) for img_name_pair in img_name_pair_list] 31 | 32 | 33 | def get_img_path(img_name, voc12_root): 34 | return os.path.join(voc12_root, IMG_FOLDER_NAME, img_name + '.jpg') 35 | 36 | def get_mask_path(mask_name, voc12_root): 37 | return os.path.join(voc12_root, MASK_FOLDER_NAME, mask_name + '.png') 38 | 39 | def load_img_name_list(dataset_path): 40 | 41 | img_gt_name_list = open(dataset_path).read().splitlines() 42 | img_name_list = [img_gt_name.split(' ')[0][-15:-4] for img_gt_name in img_gt_name_list] 43 | 44 | return img_name_list 45 | 46 | 47 | def load_img_name_pair_list(dataset_path): 48 | 49 | img_gt_name_list = open(dataset_path).read().splitlines() 50 | # img_name_pair_list = [(img_gt_name.split(' ')[0][-15:-4], img_gt_name.split(' ')[1][-15:-4]) for img_gt_name in img_gt_name_list] 51 | # common_label_list = [int(img_gt_name.split(' ')[2]) for img_gt_name in img_gt_name_list] 52 | img_name_pair_list = [(img_gt_name.split(' ')[0][-15:-4], img_gt_name.split(' ')[1][-15:-4]) for img_gt_name in 53 | img_gt_name_list] 54 | common_label_list = [int(img_gt_name.split(' ')[2]) for img_gt_name in img_gt_name_list] 55 | 56 | return img_name_pair_list, common_label_list 57 | 58 | 59 | class TorchvisionNormalize(): 60 | def __init__(self, mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)): 61 | self.mean = mean 62 | self.std = std 63 | 64 | def __call__(self, img): 65 | imgarr = np.asarray(img) 66 | proc_img = np.empty_like(imgarr, np.float32) 67 | 68 | proc_img[..., 0] = (imgarr[..., 0] / 255. - self.mean[0]) / self.std[0] 69 | proc_img[..., 1] = (imgarr[..., 1] / 255. - self.mean[1]) / self.std[1] 70 | proc_img[..., 2] = (imgarr[..., 2] / 255. - self.mean[2]) / self.std[2] 71 | 72 | return proc_img 73 | 74 | class VOC12ImageDataset(Dataset): 75 | def __init__(self, img_name_list_path, voc12_root, resize=None, 76 | resize_long=None, rescale=None, img_normal=TorchvisionNormalize(), hor_flip=False, 77 | crop_size=None, crop_method=None, to_torch=True): 78 | self.img_name_list = load_img_name_list(img_name_list_path) 79 | self.voc12_root = voc12_root 80 | 81 | self.resize = resize 82 | self.resize_long = resize_long 83 | self.rescale = rescale 84 | self.crop_size = crop_size 85 | self.img_normal = img_normal 86 | self.hor_flip = hor_flip 87 | self.crop_method = crop_method 88 | self.to_torch = to_torch 89 | 90 | def __len__(self): 91 | return len(self.img_name_list) 92 | 93 | def __getitem__(self, idx): 94 | name = self.img_name_list[idx] 95 | 96 | img = np.asarray(imageio.imread(get_img_path(name, self.voc12_root))) 97 | 98 | if self.resize: 99 | img = imutils.pil_resize(img, size=self.resize, order=3) 100 | 101 | if self.resize_long: 102 | img = imutils.random_resize_long(img, self.resize_long[0], self.resize_long[1]) 103 | 104 | if self.rescale: 105 | img = imutils.random_scale(img, scale_range=self.rescale, order=3) 106 | 107 | if self.img_normal: 108 | img = self.img_normal(img) 109 | 110 | if self.hor_flip: 111 | img = imutils.random_lr_flip(img) 112 | 113 | if self.crop_size: 114 | valid_mask = torch.zeros((21, self.crop_size, self.crop_size)) 115 | if self.crop_method == "random": 116 | img, box = imutils.random_crop(img, self.crop_size, 0) 117 | valid_mask[:,box[0]:box[1],box[2]:box[3]] = 1 118 | else: 119 | img = imutils.top_left_crop(img, self.crop_size, 0) 120 | valid_mask = torch.ones((21, self.crop_size, self.crop_size)) 121 | 122 | else: 123 | valid_mask = torch.ones((21, img.shape[0], img.shape[1])) 124 | 125 | if self.to_torch: 126 | img = np.ascontiguousarray(imutils.HWC_to_CHW(img)) 127 | 128 | return {'name': name, 'img': img, 'valid_mask': valid_mask} 129 | 130 | class VOC12ClsDataset(VOC12ImageDataset): 131 | 132 | def __init__(self, img_name_list_path, voc12_root, resize=None, 133 | resize_long=None, rescale=None, img_normal=TorchvisionNormalize(), hor_flip=False, 134 | crop_size=None, crop_method=None): 135 | super().__init__(img_name_list_path, voc12_root, resize, 136 | resize_long, rescale, img_normal, hor_flip, 137 | crop_size, crop_method) 138 | self.label_list = load_image_label_list_from_npy(self.img_name_list) 139 | 140 | def __getitem__(self, idx): 141 | out = super().__getitem__(idx) 142 | 143 | out['label'] = torch.from_numpy(self.label_list[idx]) 144 | 145 | return out 146 | 147 | class VOC12ClsDatasetMSF(VOC12ClsDataset): 148 | 149 | def __init__(self, img_name_list_path, voc12_root, img_normal=TorchvisionNormalize(), scales=(1.0,)): 150 | super().__init__(img_name_list_path, voc12_root) 151 | self.scales = scales 152 | 153 | def __getitem__(self, idx): 154 | name = self.img_name_list[idx] 155 | img = imageio.imread(get_img_path(name, self.voc12_root)) 156 | 157 | ms_img_list = [] 158 | for s in self.scales: 159 | if s == 1: 160 | s_img = img 161 | else: 162 | s_img = imutils.pil_rescale(img, s, order=3) 163 | s_img = self.img_normal(s_img) 164 | s_img = imutils.HWC_to_CHW(s_img) 165 | ms_img_list.append(np.stack([s_img, np.flip(s_img, -1)], axis=0)) 166 | 167 | if len(self.scales) == 1: 168 | ms_img_list = ms_img_list[0] 169 | 170 | # mask = PIL.Image.open(get_mask_path(name, self.voc12_root)) 171 | 172 | out = {"name": name, "img": ms_img_list, "size": (img.shape[0], img.shape[1]), 173 | "label": torch.from_numpy(self.label_list[idx]), "img_path": get_img_path(name, self.voc12_root)} 174 | 175 | 176 | return out 177 | 178 | class VOC12SegmentationDataset(Dataset): 179 | 180 | def __init__(self, img_name_list_path, label_dir, crop_size, voc12_root, 181 | rescale=None, img_normal=TorchvisionNormalize(), hor_flip=False, 182 | crop_method = 'random'): 183 | 184 | self.img_name_list = load_img_name_list(img_name_list_path) 185 | self.voc12_root = voc12_root 186 | 187 | self.label_dir = label_dir 188 | 189 | self.rescale = rescale 190 | self.crop_size = crop_size 191 | self.img_normal = img_normal 192 | self.hor_flip = hor_flip 193 | self.crop_method = crop_method 194 | 195 | def __len__(self): 196 | return len(self.img_name_list) 197 | 198 | def __getitem__(self, idx): 199 | name = self.img_name_list[idx] 200 | 201 | img = imageio.imread(get_img_path(name, self.voc12_root)) 202 | label = imageio.imread(os.path.join(self.label_dir, name + '.png')) 203 | 204 | img = np.asarray(img) 205 | 206 | if self.rescale: 207 | img, label = imutils.random_scale((img, label), scale_range=self.rescale, order=(3, 0)) 208 | 209 | if self.img_normal: 210 | img = self.img_normal(img) 211 | 212 | if self.hor_flip: 213 | img, label = imutils.random_lr_flip((img, label)) 214 | 215 | 216 | if self.crop_method == "random": 217 | (img, label), _ = imutils.random_crop((img, label), self.crop_size, (0, 255)) 218 | else: 219 | img = imutils.top_left_crop(img, self.crop_size, 0) 220 | label = imutils.top_left_crop(label, self.crop_size, 255) 221 | 222 | label = label.astype(np.uint8) 223 | img = imutils.HWC_to_CHW(img) 224 | return {'name': name, 'img': img, 'label': label} 225 | 226 | class GetAffinityLabelFromIndices(): 227 | 228 | def __init__(self, indices_from, indices_to): 229 | 230 | self.indices_from = indices_from 231 | self.indices_to = indices_to 232 | 233 | def __call__(self, segm_map): 234 | 235 | segm_map_flat = np.reshape(segm_map, -1) 236 | 237 | segm_label_from = np.expand_dims(segm_map_flat[self.indices_from], axis=0) 238 | segm_label_to = segm_map_flat[self.indices_to] 239 | 240 | valid_label = np.logical_and(np.less(segm_label_from, 21), np.less(segm_label_to, 21)) 241 | 242 | equal_label = np.equal(segm_label_from, segm_label_to) 243 | 244 | pos_affinity_label = np.logical_and(equal_label, valid_label) 245 | 246 | bg_pos_affinity_label = np.logical_and(pos_affinity_label, np.equal(segm_label_from, 0)).astype(np.float32) 247 | fg_pos_affinity_label = np.logical_and(pos_affinity_label, np.greater(segm_label_from, 0)).astype(np.float32) 248 | 249 | neg_affinity_label = np.logical_and(np.logical_not(equal_label), valid_label).astype(np.float32) 250 | 251 | return torch.from_numpy(bg_pos_affinity_label), torch.from_numpy(fg_pos_affinity_label), \ 252 | torch.from_numpy(neg_affinity_label) 253 | 254 | class VOC12AffinityDataset(VOC12SegmentationDataset): 255 | def __init__(self, img_name_list_path, label_dir, crop_size, voc12_root, 256 | indices_from, indices_to, 257 | rescale=None, img_normal=TorchvisionNormalize(), hor_flip=False, crop_method=None): 258 | super().__init__(img_name_list_path, label_dir, crop_size, voc12_root, rescale, img_normal, hor_flip, crop_method=crop_method) 259 | 260 | self.extract_aff_lab_func = GetAffinityLabelFromIndices(indices_from, indices_to) 261 | 262 | def __len__(self): 263 | return len(self.img_name_list) 264 | 265 | def __getitem__(self, idx): 266 | out = super().__getitem__(idx) 267 | reduced_label = imutils.pil_rescale(out['label'], 0.25, 0) 268 | 269 | out['aff_bg_pos_label'], out['aff_fg_pos_label'], out['aff_neg_label'] = self.extract_aff_lab_func(reduced_label) 270 | 271 | return out 272 | -------------------------------------------------------------------------------- /data/labels_coco.txt: -------------------------------------------------------------------------------- 1 | 0 background 2 | 1 person 3 | 2 bicycle 4 | 3 car 5 | 4 motorcycle 6 | 5 airplane 7 | 6 bus 8 | 7 train 9 | 8 truck 10 | 9 boat 11 | 10 traffic_light 12 | 11 fire_hydrant 13 | 12 stop_sign 14 | 13 parking_meter 15 | 14 bench 16 | 15 bird 17 | 16 cat 18 | 17 dog 19 | 18 horse 20 | 19 sheep 21 | 20 cow 22 | 21 elephant 23 | 22 bear 24 | 23 zebra 25 | 24 giraffe 26 | 25 backpack 27 | 26 umbrella 28 | 27 handbag 29 | 28 tie 30 | 29 suitcase 31 | 30 frisbee 32 | 31 skis 33 | 32 snowboard 34 | 33 sports_ball 35 | 34 kite 36 | 35 baseball_bat 37 | 36 baseball_glove 38 | 37 skateboard 39 | 38 surfboard 40 | 39 tennis_racket 41 | 40 bottle 42 | 41 wine_glass 43 | 42 cup 44 | 43 fork 45 | 44 knife 46 | 45 spoon 47 | 46 bowl 48 | 47 banana 49 | 48 apple 50 | 49 sandwich 51 | 50 orange 52 | 51 broccoli 53 | 52 carrot 54 | 53 hot_dog 55 | 54 pizza 56 | 55 donut 57 | 56 cake 58 | 57 chair 59 | 58 couch 60 | 59 potted_plant 61 | 60 bed 62 | 61 dining_table 63 | 62 toilet 64 | 63 tv 65 | 64 laptop 66 | 65 mouse 67 | 66 remote 68 | 67 keyboard 69 | 68 cell_phone 70 | 69 microwave 71 | 70 oven 72 | 71 toaster 73 | 72 sink 74 | 73 refrigerator 75 | 74 book 76 | 75 clock 77 | 76 vase 78 | 77 scissors 79 | 78 teddy_bear 80 | 79 hair_drier 81 | 80 toothbrush 82 | -------------------------------------------------------------------------------- /data/make_cls_labels.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import voc12.data 3 | import numpy as np 4 | 5 | if __name__ == '__main__': 6 | 7 | parser = argparse.ArgumentParser() 8 | parser.add_argument("--train_list", default='train_aug.txt', type=str) 9 | parser.add_argument("--val_list", default='val.txt', type=str) 10 | parser.add_argument("--out", default="cls_labels.npy", type=str) 11 | parser.add_argument("--voc12_root", required=True, type=str) 12 | args = parser.parse_args() 13 | 14 | img_name_list = voc12.data.load_img_name_list(args.train_list) 15 | img_name_list.extend(voc12.data.load_img_name_list(args.val_list)) 16 | label_list = voc12.data.load_image_label_list_from_xml(img_name_list, args.voc12_root) 17 | 18 | d = dict() 19 | for img_name, label in zip(img_name_list, label_list): 20 | d[img_name] = label 21 | 22 | np.save(args.out, d) -------------------------------------------------------------------------------- /eval_cam.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import os 3 | import argparse 4 | from PIL import Image 5 | from tool import pyutils 6 | from data import data_voc, data_coco 7 | from tqdm import tqdm 8 | from torch.utils.data import DataLoader 9 | from chainercv.evaluations import calc_semantic_segmentation_confusion 10 | 11 | def run(args, predict_dir, num_cls): 12 | preds = [] 13 | masks = [] 14 | n_images = 0 15 | for iter, pack in tqdm(enumerate(dataloader)): 16 | n_images += 1 17 | cam_dict = np.load(os.path.join(predict_dir, pack['name'][0] + '.npy'), allow_pickle=True).item() 18 | cams = cam_dict['IS_CAM'] 19 | keys = cam_dict['keys'] 20 | cls_labels = np.argmax(cams, axis=0) 21 | cls_labels = keys[cls_labels] 22 | preds.append(cls_labels.copy()) 23 | 24 | mask = np.array(Image.open(os.path.join(args.gt_path, pack['name'][0] + '.png'))) 25 | masks.append(mask.copy()) 26 | 27 | confusion = calc_semantic_segmentation_confusion(preds, masks)[:num_cls, :num_cls] 28 | 29 | gtj = confusion.sum(axis=1) 30 | resj = confusion.sum(axis=0) 31 | gtjresj = np.diag(confusion) 32 | denominator = gtj + resj - gtjresj 33 | iou = gtjresj / denominator 34 | print({'iou': iou, 'miou': np.nanmean(iou)}) 35 | 36 | if __name__ == '__main__': 37 | 38 | parser = argparse.ArgumentParser() 39 | parser.add_argument("--dataset", default="voc", type=str) 40 | parser.add_argument("--gt_path", default='../PascalVOC2012/VOCdevkit/VOC2012/SegmentationClass', type=str) 41 | parser.add_argument('--session_name', default="exp", type=str) 42 | args = parser.parse_args() 43 | 44 | assert args.dataset in ['voc', 'coco'], 'Dataset must be voc or coco in this project.' 45 | 46 | if args.dataset == 'voc': 47 | dataset_root = '../PascalVOC2012/VOCdevkit/VOC2012' 48 | num_cls = 21 49 | dataset = data_voc.VOC12ImageDataset('data/train_' + args.dataset + '.txt', voc12_root=dataset_root, img_normal=None, to_torch=False) 50 | 51 | elif args.dataset == 'coco': 52 | args.gt_path = "../ms_coco_14&15/SegmentationClass/train2014/" 53 | dataset_root = "../ms_coco_14&15/images" 54 | num_cls = 81 55 | dataset = data_coco.COCOImageDataset('data/train_' + args.dataset + '.txt', coco_root=dataset_root, img_normal=None, to_torch=False) 56 | 57 | dataloader = DataLoader(dataset, batch_size=1, shuffle=False, num_workers=0, pin_memory=True, drop_last=True) 58 | pyutils.Logger(os.path.join(args.session_name, 'eval_' + args.session_name + '.log')) 59 | run(args, args.session_name + "/npy", num_cls) -------------------------------------------------------------------------------- /eval_sem_seg.py: -------------------------------------------------------------------------------- 1 | 2 | import numpy as np 3 | import os 4 | from chainercv.datasets import VOCSemanticSegmentationDataset 5 | from chainercv.evaluations import calc_semantic_segmentation_confusion 6 | import imageio 7 | import argparse 8 | 9 | def run(args): 10 | dataset = VOCSemanticSegmentationDataset(split=args.chainer_eval_set, data_dir=args.voc12_root) 11 | 12 | preds = [] 13 | labels = [] 14 | n_img = 0 15 | for i, id in enumerate(dataset.ids): 16 | cls_labels = imageio.imread(os.path.join(args.sem_seg_out_dir, id + '.png')).astype(np.uint8) 17 | preds.append(cls_labels.copy()) 18 | labels.append(dataset.get_example_by_keys(i, (1,))[0]) 19 | n_img += 1 20 | 21 | confusion = calc_semantic_segmentation_confusion(preds, labels)[:21, :21] 22 | 23 | gtj = confusion.sum(axis=1) 24 | resj = confusion.sum(axis=0) 25 | gtjresj = np.diag(confusion) 26 | denominator = gtj + resj - gtjresj 27 | fp = 1. - gtj / denominator 28 | fn = 1. - resj / denominator 29 | iou = gtjresj / denominator 30 | print("total images", n_img) 31 | print(fp[0], fn[0]) 32 | print(np.mean(fp[1:]), np.mean(fn[1:])) 33 | 34 | print({'iou': iou, 'miou': np.nanmean(iou)}) 35 | 36 | if __name__ == '__main__': 37 | 38 | parser = argparse.ArgumentParser() 39 | parser.add_argument("--voc12_root", default='../PascalVOC2012/VOCdevkit/VOC2012', type=str) 40 | parser.add_argument("--gt_dir", default='../PascalVOC2012/VOCdevkit/VOC2012/SegmentationClass', type=str) 41 | parser.add_argument("--chainer_eval_set", default="train", type=str) 42 | parser.add_argument("--sem_seg_out_dir", default="exp/pseudo_label", type=str) 43 | args = parser.parse_args() 44 | 45 | run(args) -------------------------------------------------------------------------------- /fig/framework1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chenqi1126/SIPE/e0c6a3bc578a6bade2cf85fe34a1793dce71170d/fig/framework1.png -------------------------------------------------------------------------------- /log/sipe_voc.log: -------------------------------------------------------------------------------- 1 | Epoch 1/5 2 | step: 0/ 3305 lossC:0.7002 lossL1:0.0394 imps:1.6 lr: 0.0500 etc:Wed Jul 7 00:36:29 2021 3 | step: 10/ 3305 lossC:0.2708 lossL1:0.0205 imps:13.3 lr: 0.0499 etc:Tue Jul 6 16:48:07 2021 4 | step: 20/ 3305 lossC:0.2432 lossL1:0.0154 imps:20.7 lr: 0.0497 etc:Tue Jul 6 16:24:20 2021 5 | step: 30/ 3305 lossC:0.2294 lossL1:0.0113 imps:26.1 lr: 0.0496 etc:Tue Jul 6 16:15:36 2021 6 | step: 40/ 3305 lossC:0.2096 lossL1:0.0141 imps:30.1 lr: 0.0495 etc:Tue Jul 6 16:11:06 2021 7 | step: 50/ 3305 lossC:0.2108 lossL1:0.0140 imps:33.1 lr: 0.0493 etc:Tue Jul 6 16:08:27 2021 8 | step: 60/ 3305 lossC:0.1820 lossL1:0.0171 imps:35.5 lr: 0.0492 etc:Tue Jul 6 16:06:35 2021 9 | step: 70/ 3305 lossC:0.1670 lossL1:0.0165 imps:37.4 lr: 0.0490 etc:Tue Jul 6 16:05:21 2021 10 | step: 80/ 3305 lossC:0.1625 lossL1:0.0172 imps:39.0 lr: 0.0489 etc:Tue Jul 6 16:04:24 2021 11 | step: 90/ 3305 lossC:0.1491 lossL1:0.0184 imps:40.3 lr: 0.0488 etc:Tue Jul 6 16:03:40 2021 12 | step: 100/ 3305 lossC:0.1635 lossL1:0.0170 imps:41.4 lr: 0.0486 etc:Tue Jul 6 16:03:04 2021 13 | step: 110/ 3305 lossC:0.1343 lossL1:0.0174 imps:41.2 lr: 0.0485 etc:Tue Jul 6 16:03:10 2021 14 | step: 120/ 3305 lossC:0.1257 lossL1:0.0170 imps:42.2 lr: 0.0484 etc:Tue Jul 6 16:02:39 2021 15 | step: 130/ 3305 lossC:0.1180 lossL1:0.0164 imps:43.0 lr: 0.0482 etc:Tue Jul 6 16:02:17 2021 16 | step: 140/ 3305 lossC:0.1201 lossL1:0.0161 imps:43.7 lr: 0.0481 etc:Tue Jul 6 16:01:59 2021 17 | step: 150/ 3305 lossC:0.1185 lossL1:0.0166 imps:44.3 lr: 0.0480 etc:Tue Jul 6 16:01:40 2021 18 | step: 160/ 3305 lossC:0.1243 lossL1:0.0167 imps:44.8 lr: 0.0478 etc:Tue Jul 6 16:01:26 2021 19 | step: 170/ 3305 lossC:0.1105 lossL1:0.0158 imps:45.4 lr: 0.0477 etc:Tue Jul 6 16:01:12 2021 20 | step: 180/ 3305 lossC:0.0960 lossL1:0.0163 imps:45.9 lr: 0.0475 etc:Tue Jul 6 16:00:59 2021 21 | step: 190/ 3305 lossC:0.0915 lossL1:0.0165 imps:46.3 lr: 0.0474 etc:Tue Jul 6 16:00:49 2021 22 | step: 200/ 3305 lossC:0.1200 lossL1:0.0161 imps:46.8 lr: 0.0473 etc:Tue Jul 6 16:00:38 2021 23 | step: 210/ 3305 lossC:0.0887 lossL1:0.0159 imps:46.5 lr: 0.0471 etc:Tue Jul 6 16:00:44 2021 24 | step: 220/ 3305 lossC:0.1096 lossL1:0.0162 imps:46.8 lr: 0.0470 etc:Tue Jul 6 16:00:37 2021 25 | step: 230/ 3305 lossC:0.0927 lossL1:0.0161 imps:47.1 lr: 0.0469 etc:Tue Jul 6 16:00:29 2021 26 | step: 240/ 3305 lossC:0.0815 lossL1:0.0158 imps:47.5 lr: 0.0467 etc:Tue Jul 6 16:00:21 2021 27 | step: 250/ 3305 lossC:0.0951 lossL1:0.0161 imps:47.7 lr: 0.0466 etc:Tue Jul 6 16:00:16 2021 28 | step: 260/ 3305 lossC:0.0952 lossL1:0.0162 imps:48.0 lr: 0.0464 etc:Tue Jul 6 16:00:09 2021 29 | step: 270/ 3305 lossC:0.0926 lossL1:0.0161 imps:48.3 lr: 0.0463 etc:Tue Jul 6 16:00:03 2021 30 | step: 280/ 3305 lossC:0.0837 lossL1:0.0159 imps:48.5 lr: 0.0462 etc:Tue Jul 6 15:59:58 2021 31 | step: 290/ 3305 lossC:0.1186 lossL1:0.0164 imps:48.7 lr: 0.0460 etc:Tue Jul 6 15:59:52 2021 32 | step: 300/ 3305 lossC:0.0841 lossL1:0.0161 imps:49.0 lr: 0.0459 etc:Tue Jul 6 15:59:47 2021 33 | step: 310/ 3305 lossC:0.0818 lossL1:0.0165 imps:48.6 lr: 0.0458 etc:Tue Jul 6 15:59:56 2021 34 | step: 320/ 3305 lossC:0.0877 lossL1:0.0168 imps:48.8 lr: 0.0456 etc:Tue Jul 6 15:59:51 2021 35 | step: 330/ 3305 lossC:0.0775 lossL1:0.0157 imps:49.0 lr: 0.0455 etc:Tue Jul 6 15:59:47 2021 36 | step: 340/ 3305 lossC:0.0783 lossL1:0.0160 imps:49.2 lr: 0.0453 etc:Tue Jul 6 15:59:42 2021 37 | step: 350/ 3305 lossC:0.0956 lossL1:0.0162 imps:49.4 lr: 0.0452 etc:Tue Jul 6 15:59:38 2021 38 | step: 360/ 3305 lossC:0.1001 lossL1:0.0166 imps:49.6 lr: 0.0451 etc:Tue Jul 6 15:59:34 2021 39 | step: 370/ 3305 lossC:0.0816 lossL1:0.0168 imps:49.7 lr: 0.0449 etc:Tue Jul 6 15:59:32 2021 40 | step: 380/ 3305 lossC:0.0862 lossL1:0.0157 imps:49.8 lr: 0.0448 etc:Tue Jul 6 15:59:29 2021 41 | step: 390/ 3305 lossC:0.0713 lossL1:0.0161 imps:50.0 lr: 0.0447 etc:Tue Jul 6 15:59:25 2021 42 | step: 400/ 3305 lossC:0.0860 lossL1:0.0160 imps:50.1 lr: 0.0445 etc:Tue Jul 6 15:59:22 2021 43 | step: 410/ 3305 lossC:0.0770 lossL1:0.0157 imps:49.8 lr: 0.0444 etc:Tue Jul 6 15:59:30 2021 44 | step: 420/ 3305 lossC:0.0735 lossL1:0.0161 imps:49.9 lr: 0.0442 etc:Tue Jul 6 15:59:26 2021 45 | step: 430/ 3305 lossC:0.0720 lossL1:0.0155 imps:50.1 lr: 0.0441 etc:Tue Jul 6 15:59:23 2021 46 | step: 440/ 3305 lossC:0.0816 lossL1:0.0162 imps:50.2 lr: 0.0440 etc:Tue Jul 6 15:59:20 2021 47 | step: 450/ 3305 lossC:0.0767 lossL1:0.0153 imps:50.3 lr: 0.0438 etc:Tue Jul 6 15:59:19 2021 48 | step: 460/ 3305 lossC:0.0865 lossL1:0.0152 imps:50.4 lr: 0.0437 etc:Tue Jul 6 15:59:16 2021 49 | step: 470/ 3305 lossC:0.0728 lossL1:0.0163 imps:50.5 lr: 0.0436 etc:Tue Jul 6 15:59:14 2021 50 | step: 480/ 3305 lossC:0.0900 lossL1:0.0151 imps:50.6 lr: 0.0434 etc:Tue Jul 6 15:59:12 2021 51 | step: 490/ 3305 lossC:0.0780 lossL1:0.0154 imps:50.7 lr: 0.0433 etc:Tue Jul 6 15:59:09 2021 52 | step: 500/ 3305 lossC:0.0716 lossL1:0.0157 imps:50.8 lr: 0.0431 etc:Tue Jul 6 15:59:08 2021 53 | validating ... {'iou_bkg': array([0.78516055, 0.44993605, 0.28712452, 0.49026533, 0.38245563, 54 | 0.47301526, 0.67605444, 0.53691007, 0.58495375, 0.21490392, 55 | 0.54415626, 0.31798507, 0.64676658, 0.59992422, 0.63913759, 56 | 0.59175731, 0.45247726, 0.49515567, 0.34611537, 0.46119832, 57 | 0.38272116]), 'miou_bkg': 0.49324639599118814} 58 | step: 510/ 3305 lossC:0.0994 lossL1:0.0159 imps:36.6 lr: 0.0430 etc:Tue Jul 6 16:05:54 2021 59 | step: 520/ 3305 lossC:0.0576 lossL1:0.0156 imps:36.8 lr: 0.0429 etc:Tue Jul 6 16:05:43 2021 60 | step: 530/ 3305 lossC:0.0854 lossL1:0.0149 imps:37.1 lr: 0.0427 etc:Tue Jul 6 16:05:33 2021 61 | step: 540/ 3305 lossC:0.0750 lossL1:0.0150 imps:37.3 lr: 0.0426 etc:Tue Jul 6 16:05:25 2021 62 | step: 550/ 3305 lossC:0.0713 lossL1:0.0155 imps:37.5 lr: 0.0424 etc:Tue Jul 6 16:05:16 2021 63 | step: 560/ 3305 lossC:0.0890 lossL1:0.0149 imps:37.8 lr: 0.0423 etc:Tue Jul 6 16:05:07 2021 64 | step: 570/ 3305 lossC:0.0857 lossL1:0.0154 imps:38.0 lr: 0.0422 etc:Tue Jul 6 16:04:59 2021 65 | step: 580/ 3305 lossC:0.0932 lossL1:0.0169 imps:38.2 lr: 0.0420 etc:Tue Jul 6 16:04:51 2021 66 | step: 590/ 3305 lossC:0.0709 lossL1:0.0161 imps:38.4 lr: 0.0419 etc:Tue Jul 6 16:04:43 2021 67 | step: 600/ 3305 lossC:0.0864 lossL1:0.0150 imps:38.6 lr: 0.0418 etc:Tue Jul 6 16:04:36 2021 68 | step: 610/ 3305 lossC:0.0941 lossL1:0.0144 imps:38.7 lr: 0.0416 etc:Tue Jul 6 16:04:35 2021 69 | step: 620/ 3305 lossC:0.0862 lossL1:0.0159 imps:38.9 lr: 0.0415 etc:Tue Jul 6 16:04:28 2021 70 | step: 630/ 3305 lossC:0.0715 lossL1:0.0155 imps:39.1 lr: 0.0413 etc:Tue Jul 6 16:04:21 2021 71 | step: 640/ 3305 lossC:0.0698 lossL1:0.0154 imps:39.3 lr: 0.0412 etc:Tue Jul 6 16:04:15 2021 72 | step: 650/ 3305 lossC:0.0902 lossL1:0.0150 imps:39.5 lr: 0.0411 etc:Tue Jul 6 16:04:07 2021 73 | step: 660/ 3305 lossC:0.0731 lossL1:0.0160 imps:39.7 lr: 0.0409 etc:Tue Jul 6 16:03:59 2021 74 | Epoch 2/5 75 | step: 670/ 3305 lossC:0.0618 lossL1:0.0158 imps:25.7 lr: 0.0408 etc:Tue Jul 6 16:04:11 2021 76 | step: 680/ 3305 lossC:0.0786 lossL1:0.0152 imps:34.1 lr: 0.0406 etc:Tue Jul 6 16:04:07 2021 77 | step: 690/ 3305 lossC:0.0754 lossL1:0.0144 imps:39.4 lr: 0.0405 etc:Tue Jul 6 16:04:01 2021 78 | step: 700/ 3305 lossC:0.0840 lossL1:0.0165 imps:42.6 lr: 0.0404 etc:Tue Jul 6 16:03:55 2021 79 | step: 710/ 3305 lossC:0.0749 lossL1:0.0154 imps:41.6 lr: 0.0402 etc:Tue Jul 6 16:03:56 2021 80 | step: 720/ 3305 lossC:0.0648 lossL1:0.0159 imps:43.2 lr: 0.0401 etc:Tue Jul 6 16:03:51 2021 81 | step: 730/ 3305 lossC:0.0711 lossL1:0.0169 imps:44.5 lr: 0.0399 etc:Tue Jul 6 16:03:46 2021 82 | step: 740/ 3305 lossC:0.0580 lossL1:0.0151 imps:45.6 lr: 0.0398 etc:Tue Jul 6 16:03:42 2021 83 | step: 750/ 3305 lossC:0.0803 lossL1:0.0152 imps:46.6 lr: 0.0397 etc:Tue Jul 6 16:03:37 2021 84 | step: 760/ 3305 lossC:0.0815 lossL1:0.0151 imps:47.4 lr: 0.0395 etc:Tue Jul 6 16:03:32 2021 85 | step: 770/ 3305 lossC:0.0661 lossL1:0.0168 imps:48.3 lr: 0.0394 etc:Tue Jul 6 16:03:26 2021 86 | step: 780/ 3305 lossC:0.0691 lossL1:0.0156 imps:48.9 lr: 0.0392 etc:Tue Jul 6 16:03:22 2021 87 | step: 790/ 3305 lossC:0.0479 lossL1:0.0155 imps:49.4 lr: 0.0391 etc:Tue Jul 6 16:03:17 2021 88 | step: 800/ 3305 lossC:0.0658 lossL1:0.0164 imps:49.9 lr: 0.0390 etc:Tue Jul 6 16:03:13 2021 89 | step: 810/ 3305 lossC:0.0826 lossL1:0.0154 imps:49.1 lr: 0.0388 etc:Tue Jul 6 16:03:13 2021 90 | step: 820/ 3305 lossC:0.0568 lossL1:0.0163 imps:49.5 lr: 0.0387 etc:Tue Jul 6 16:03:09 2021 91 | step: 830/ 3305 lossC:0.0666 lossL1:0.0147 imps:49.8 lr: 0.0385 etc:Tue Jul 6 16:03:05 2021 92 | step: 840/ 3305 lossC:0.0829 lossL1:0.0151 imps:50.1 lr: 0.0384 etc:Tue Jul 6 16:03:01 2021 93 | step: 850/ 3305 lossC:0.0685 lossL1:0.0155 imps:50.5 lr: 0.0383 etc:Tue Jul 6 16:02:57 2021 94 | step: 860/ 3305 lossC:0.0753 lossL1:0.0161 imps:50.7 lr: 0.0381 etc:Tue Jul 6 16:02:53 2021 95 | step: 870/ 3305 lossC:0.0724 lossL1:0.0156 imps:50.9 lr: 0.0380 etc:Tue Jul 6 16:02:49 2021 96 | step: 880/ 3305 lossC:0.0680 lossL1:0.0161 imps:51.2 lr: 0.0378 etc:Tue Jul 6 16:02:46 2021 97 | step: 890/ 3305 lossC:0.0755 lossL1:0.0153 imps:51.4 lr: 0.0377 etc:Tue Jul 6 16:02:42 2021 98 | step: 900/ 3305 lossC:0.0632 lossL1:0.0165 imps:51.5 lr: 0.0376 etc:Tue Jul 6 16:02:39 2021 99 | step: 910/ 3305 lossC:0.0544 lossL1:0.0149 imps:50.8 lr: 0.0374 etc:Tue Jul 6 16:02:40 2021 100 | step: 920/ 3305 lossC:0.0614 lossL1:0.0147 imps:51.0 lr: 0.0373 etc:Tue Jul 6 16:02:37 2021 101 | step: 930/ 3305 lossC:0.0679 lossL1:0.0150 imps:51.1 lr: 0.0371 etc:Tue Jul 6 16:02:34 2021 102 | step: 940/ 3305 lossC:0.0715 lossL1:0.0152 imps:51.3 lr: 0.0370 etc:Tue Jul 6 16:02:30 2021 103 | step: 950/ 3305 lossC:0.0723 lossL1:0.0157 imps:51.5 lr: 0.0369 etc:Tue Jul 6 16:02:27 2021 104 | step: 960/ 3305 lossC:0.0568 lossL1:0.0148 imps:51.6 lr: 0.0367 etc:Tue Jul 6 16:02:24 2021 105 | step: 970/ 3305 lossC:0.0610 lossL1:0.0150 imps:51.7 lr: 0.0366 etc:Tue Jul 6 16:02:21 2021 106 | step: 980/ 3305 lossC:0.0658 lossL1:0.0149 imps:51.9 lr: 0.0364 etc:Tue Jul 6 16:02:18 2021 107 | step: 990/ 3305 lossC:0.0804 lossL1:0.0160 imps:52.0 lr: 0.0363 etc:Tue Jul 6 16:02:15 2021 108 | step: 1000/ 3305 lossC:0.0717 lossL1:0.0143 imps:52.2 lr: 0.0362 etc:Tue Jul 6 16:02:12 2021 109 | validating ... {'iou_bkg': array([0.77425301, 0.4463232 , 0.28064056, 0.54189112, 0.40272627, 110 | 0.45669421, 0.72001691, 0.52486195, 0.63269662, 0.22910405, 111 | 0.56984009, 0.41296709, 0.66126449, 0.58536916, 0.62955448, 112 | 0.53865592, 0.40882911, 0.56712101, 0.40460345, 0.47156603, 113 | 0.36740199]), 'miou_bkg': 0.5060181286190909} 114 | step: 1010/ 3305 lossC:0.0808 lossL1:0.0160 imps:32.7 lr: 0.0360 etc:Tue Jul 6 16:05:38 2021 115 | step: 1020/ 3305 lossC:0.0789 lossL1:0.0149 imps:33.1 lr: 0.0359 etc:Tue Jul 6 16:05:33 2021 116 | step: 1030/ 3305 lossC:0.0580 lossL1:0.0150 imps:33.5 lr: 0.0357 etc:Tue Jul 6 16:05:28 2021 117 | step: 1040/ 3305 lossC:0.0603 lossL1:0.0150 imps:33.9 lr: 0.0356 etc:Tue Jul 6 16:05:23 2021 118 | step: 1050/ 3305 lossC:0.0572 lossL1:0.0149 imps:34.3 lr: 0.0354 etc:Tue Jul 6 16:05:18 2021 119 | step: 1060/ 3305 lossC:0.0619 lossL1:0.0150 imps:34.6 lr: 0.0353 etc:Tue Jul 6 16:05:13 2021 120 | step: 1070/ 3305 lossC:0.0737 lossL1:0.0154 imps:35.0 lr: 0.0352 etc:Tue Jul 6 16:05:09 2021 121 | step: 1080/ 3305 lossC:0.0613 lossL1:0.0158 imps:35.3 lr: 0.0350 etc:Tue Jul 6 16:05:04 2021 122 | step: 1090/ 3305 lossC:0.0685 lossL1:0.0155 imps:35.6 lr: 0.0349 etc:Tue Jul 6 16:05:00 2021 123 | step: 1100/ 3305 lossC:0.0692 lossL1:0.0161 imps:36.0 lr: 0.0347 etc:Tue Jul 6 16:04:55 2021 124 | step: 1110/ 3305 lossC:0.0771 lossL1:0.0156 imps:36.0 lr: 0.0346 etc:Tue Jul 6 16:04:55 2021 125 | step: 1120/ 3305 lossC:0.0594 lossL1:0.0155 imps:36.3 lr: 0.0345 etc:Tue Jul 6 16:04:51 2021 126 | step: 1130/ 3305 lossC:0.0975 lossL1:0.0156 imps:36.6 lr: 0.0343 etc:Tue Jul 6 16:04:46 2021 127 | step: 1140/ 3305 lossC:0.0739 lossL1:0.0154 imps:36.9 lr: 0.0342 etc:Tue Jul 6 16:04:43 2021 128 | step: 1150/ 3305 lossC:0.0727 lossL1:0.0152 imps:37.2 lr: 0.0340 etc:Tue Jul 6 16:04:38 2021 129 | step: 1160/ 3305 lossC:0.0747 lossL1:0.0151 imps:37.4 lr: 0.0339 etc:Tue Jul 6 16:04:35 2021 130 | step: 1170/ 3305 lossC:0.0514 lossL1:0.0145 imps:37.7 lr: 0.0337 etc:Tue Jul 6 16:04:31 2021 131 | step: 1180/ 3305 lossC:0.0806 lossL1:0.0153 imps:37.9 lr: 0.0336 etc:Tue Jul 6 16:04:27 2021 132 | step: 1190/ 3305 lossC:0.0695 lossL1:0.0146 imps:38.2 lr: 0.0335 etc:Tue Jul 6 16:04:24 2021 133 | step: 1200/ 3305 lossC:0.0645 lossL1:0.0141 imps:38.4 lr: 0.0333 etc:Tue Jul 6 16:04:20 2021 134 | step: 1210/ 3305 lossC:0.0529 lossL1:0.0148 imps:38.4 lr: 0.0332 etc:Tue Jul 6 16:04:20 2021 135 | step: 1220/ 3305 lossC:0.0639 lossL1:0.0146 imps:38.6 lr: 0.0330 etc:Tue Jul 6 16:04:17 2021 136 | step: 1230/ 3305 lossC:0.0667 lossL1:0.0161 imps:38.8 lr: 0.0329 etc:Tue Jul 6 16:04:13 2021 137 | step: 1240/ 3305 lossC:0.0676 lossL1:0.0150 imps:39.1 lr: 0.0327 etc:Tue Jul 6 16:04:10 2021 138 | step: 1250/ 3305 lossC:0.0603 lossL1:0.0159 imps:39.3 lr: 0.0326 etc:Tue Jul 6 16:04:07 2021 139 | step: 1260/ 3305 lossC:0.0596 lossL1:0.0148 imps:39.5 lr: 0.0325 etc:Tue Jul 6 16:04:04 2021 140 | step: 1270/ 3305 lossC:0.0589 lossL1:0.0147 imps:39.7 lr: 0.0323 etc:Tue Jul 6 16:04:00 2021 141 | step: 1280/ 3305 lossC:0.0574 lossL1:0.0146 imps:39.8 lr: 0.0322 etc:Tue Jul 6 16:03:58 2021 142 | step: 1290/ 3305 lossC:0.0664 lossL1:0.0157 imps:40.0 lr: 0.0320 etc:Tue Jul 6 16:03:54 2021 143 | step: 1300/ 3305 lossC:0.0585 lossL1:0.0149 imps:40.2 lr: 0.0319 etc:Tue Jul 6 16:03:51 2021 144 | step: 1310/ 3305 lossC:0.0726 lossL1:0.0158 imps:40.3 lr: 0.0317 etc:Tue Jul 6 16:03:51 2021 145 | step: 1320/ 3305 lossC:0.0632 lossL1:0.0150 imps:40.5 lr: 0.0316 etc:Tue Jul 6 16:03:47 2021 146 | Epoch 3/5 147 | step: 1330/ 3305 lossC:0.0517 lossL1:0.0146 imps:23.2 lr: 0.0315 etc:Tue Jul 6 16:03:54 2021 148 | step: 1340/ 3305 lossC:0.0505 lossL1:0.0154 imps:32.5 lr: 0.0313 etc:Tue Jul 6 16:03:52 2021 149 | step: 1350/ 3305 lossC:0.0492 lossL1:0.0145 imps:37.6 lr: 0.0312 etc:Tue Jul 6 16:03:49 2021 150 | step: 1360/ 3305 lossC:0.0570 lossL1:0.0157 imps:40.8 lr: 0.0310 etc:Tue Jul 6 16:03:47 2021 151 | step: 1370/ 3305 lossC:0.0564 lossL1:0.0151 imps:42.9 lr: 0.0309 etc:Tue Jul 6 16:03:44 2021 152 | step: 1380/ 3305 lossC:0.0622 lossL1:0.0147 imps:44.7 lr: 0.0307 etc:Tue Jul 6 16:03:41 2021 153 | step: 1390/ 3305 lossC:0.0582 lossL1:0.0159 imps:46.1 lr: 0.0306 etc:Tue Jul 6 16:03:39 2021 154 | step: 1400/ 3305 lossC:0.0630 lossL1:0.0155 imps:47.2 lr: 0.0305 etc:Tue Jul 6 16:03:36 2021 155 | step: 1410/ 3305 lossC:0.0640 lossL1:0.0151 imps:46.2 lr: 0.0303 etc:Tue Jul 6 16:03:36 2021 156 | step: 1420/ 3305 lossC:0.0570 lossL1:0.0154 imps:47.0 lr: 0.0302 etc:Tue Jul 6 16:03:34 2021 157 | step: 1430/ 3305 lossC:0.0793 lossL1:0.0152 imps:47.7 lr: 0.0300 etc:Tue Jul 6 16:03:31 2021 158 | step: 1440/ 3305 lossC:0.0612 lossL1:0.0158 imps:48.3 lr: 0.0299 etc:Tue Jul 6 16:03:29 2021 159 | step: 1450/ 3305 lossC:0.0581 lossL1:0.0152 imps:48.7 lr: 0.0297 etc:Tue Jul 6 16:03:26 2021 160 | step: 1460/ 3305 lossC:0.0779 lossL1:0.0153 imps:49.2 lr: 0.0296 etc:Tue Jul 6 16:03:24 2021 161 | step: 1470/ 3305 lossC:0.0661 lossL1:0.0143 imps:49.6 lr: 0.0294 etc:Tue Jul 6 16:03:22 2021 162 | step: 1480/ 3305 lossC:0.0489 lossL1:0.0144 imps:49.8 lr: 0.0293 etc:Tue Jul 6 16:03:20 2021 163 | step: 1490/ 3305 lossC:0.0514 lossL1:0.0149 imps:50.1 lr: 0.0292 etc:Tue Jul 6 16:03:17 2021 164 | step: 1500/ 3305 lossC:0.0550 lossL1:0.0147 imps:50.4 lr: 0.0290 etc:Tue Jul 6 16:03:15 2021 165 | validating ... {'iou_bkg': array([0.77606756, 0.42219262, 0.27585971, 0.54401912, 0.37409399, 166 | 0.45688989, 0.70295877, 0.52678437, 0.66064882, 0.23183555, 167 | 0.57850477, 0.47550253, 0.65101585, 0.59649907, 0.63632155, 168 | 0.5635885 , 0.44116175, 0.61391331, 0.41872937, 0.52473245, 169 | 0.39792098]), 'miou_bkg': 0.5175828823567498} 170 | step: 1510/ 3305 lossC:0.0574 lossL1:0.0142 imps:25.0 lr: 0.0289 etc:Tue Jul 6 16:05:27 2021 171 | step: 1520/ 3305 lossC:0.0661 lossL1:0.0155 imps:25.7 lr: 0.0287 etc:Tue Jul 6 16:05:23 2021 172 | step: 1530/ 3305 lossC:0.0635 lossL1:0.0156 imps:26.4 lr: 0.0286 etc:Tue Jul 6 16:05:20 2021 173 | step: 1540/ 3305 lossC:0.0729 lossL1:0.0161 imps:27.1 lr: 0.0284 etc:Tue Jul 6 16:05:17 2021 174 | step: 1550/ 3305 lossC:0.0612 lossL1:0.0152 imps:27.7 lr: 0.0283 etc:Tue Jul 6 16:05:14 2021 175 | step: 1560/ 3305 lossC:0.0537 lossL1:0.0156 imps:28.4 lr: 0.0281 etc:Tue Jul 6 16:05:11 2021 176 | step: 1570/ 3305 lossC:0.0582 lossL1:0.0156 imps:28.9 lr: 0.0280 etc:Tue Jul 6 16:05:08 2021 177 | step: 1580/ 3305 lossC:0.0601 lossL1:0.0141 imps:29.5 lr: 0.0279 etc:Tue Jul 6 16:05:05 2021 178 | step: 1590/ 3305 lossC:0.0492 lossL1:0.0141 imps:30.0 lr: 0.0277 etc:Tue Jul 6 16:05:02 2021 179 | step: 1600/ 3305 lossC:0.0526 lossL1:0.0143 imps:30.6 lr: 0.0276 etc:Tue Jul 6 16:04:59 2021 180 | step: 1610/ 3305 lossC:0.0493 lossL1:0.0145 imps:30.8 lr: 0.0274 etc:Tue Jul 6 16:04:58 2021 181 | step: 1620/ 3305 lossC:0.0449 lossL1:0.0152 imps:31.3 lr: 0.0273 etc:Tue Jul 6 16:04:55 2021 182 | step: 1630/ 3305 lossC:0.0612 lossL1:0.0161 imps:31.8 lr: 0.0271 etc:Tue Jul 6 16:04:53 2021 183 | step: 1640/ 3305 lossC:0.0561 lossL1:0.0149 imps:32.2 lr: 0.0270 etc:Tue Jul 6 16:04:50 2021 184 | step: 1650/ 3305 lossC:0.0570 lossL1:0.0145 imps:32.7 lr: 0.0268 etc:Tue Jul 6 16:04:47 2021 185 | step: 1660/ 3305 lossC:0.0580 lossL1:0.0161 imps:33.1 lr: 0.0267 etc:Tue Jul 6 16:04:44 2021 186 | step: 1670/ 3305 lossC:0.0648 lossL1:0.0144 imps:33.5 lr: 0.0265 etc:Tue Jul 6 16:04:41 2021 187 | step: 1680/ 3305 lossC:0.0639 lossL1:0.0148 imps:33.9 lr: 0.0264 etc:Tue Jul 6 16:04:39 2021 188 | step: 1690/ 3305 lossC:0.0520 lossL1:0.0147 imps:34.2 lr: 0.0262 etc:Tue Jul 6 16:04:36 2021 189 | step: 1700/ 3305 lossC:0.0587 lossL1:0.0142 imps:34.6 lr: 0.0261 etc:Tue Jul 6 16:04:34 2021 190 | step: 1710/ 3305 lossC:0.0579 lossL1:0.0152 imps:34.8 lr: 0.0260 etc:Tue Jul 6 16:04:33 2021 191 | step: 1720/ 3305 lossC:0.0629 lossL1:0.0147 imps:35.1 lr: 0.0258 etc:Tue Jul 6 16:04:30 2021 192 | step: 1730/ 3305 lossC:0.0630 lossL1:0.0149 imps:35.5 lr: 0.0257 etc:Tue Jul 6 16:04:28 2021 193 | step: 1740/ 3305 lossC:0.0513 lossL1:0.0141 imps:35.8 lr: 0.0255 etc:Tue Jul 6 16:04:25 2021 194 | step: 1750/ 3305 lossC:0.0741 lossL1:0.0147 imps:36.1 lr: 0.0254 etc:Tue Jul 6 16:04:23 2021 195 | step: 1760/ 3305 lossC:0.0536 lossL1:0.0146 imps:36.4 lr: 0.0252 etc:Tue Jul 6 16:04:20 2021 196 | step: 1770/ 3305 lossC:0.0492 lossL1:0.0159 imps:36.7 lr: 0.0251 etc:Tue Jul 6 16:04:18 2021 197 | step: 1780/ 3305 lossC:0.0550 lossL1:0.0152 imps:37.0 lr: 0.0249 etc:Tue Jul 6 16:04:15 2021 198 | step: 1790/ 3305 lossC:0.0647 lossL1:0.0156 imps:37.3 lr: 0.0248 etc:Tue Jul 6 16:04:13 2021 199 | step: 1800/ 3305 lossC:0.0618 lossL1:0.0142 imps:37.5 lr: 0.0246 etc:Tue Jul 6 16:04:11 2021 200 | step: 1810/ 3305 lossC:0.0666 lossL1:0.0144 imps:37.7 lr: 0.0245 etc:Tue Jul 6 16:04:10 2021 201 | step: 1820/ 3305 lossC:0.0471 lossL1:0.0144 imps:37.9 lr: 0.0243 etc:Tue Jul 6 16:04:08 2021 202 | step: 1830/ 3305 lossC:0.0651 lossL1:0.0160 imps:38.2 lr: 0.0242 etc:Tue Jul 6 16:04:06 2021 203 | step: 1840/ 3305 lossC:0.0545 lossL1:0.0148 imps:38.4 lr: 0.0240 etc:Tue Jul 6 16:04:03 2021 204 | step: 1850/ 3305 lossC:0.0617 lossL1:0.0147 imps:38.6 lr: 0.0239 etc:Tue Jul 6 16:04:01 2021 205 | step: 1860/ 3305 lossC:0.0506 lossL1:0.0144 imps:38.8 lr: 0.0237 etc:Tue Jul 6 16:03:59 2021 206 | step: 1870/ 3305 lossC:0.0442 lossL1:0.0146 imps:39.1 lr: 0.0236 etc:Tue Jul 6 16:03:57 2021 207 | step: 1880/ 3305 lossC:0.0453 lossL1:0.0145 imps:39.3 lr: 0.0235 etc:Tue Jul 6 16:03:55 2021 208 | step: 1890/ 3305 lossC:0.0380 lossL1:0.0138 imps:39.5 lr: 0.0233 etc:Tue Jul 6 16:03:53 2021 209 | step: 1900/ 3305 lossC:0.0528 lossL1:0.0148 imps:39.7 lr: 0.0232 etc:Tue Jul 6 16:03:51 2021 210 | step: 1910/ 3305 lossC:0.0494 lossL1:0.0147 imps:39.7 lr: 0.0230 etc:Tue Jul 6 16:03:51 2021 211 | step: 1920/ 3305 lossC:0.0621 lossL1:0.0151 imps:39.9 lr: 0.0229 etc:Tue Jul 6 16:03:49 2021 212 | step: 1930/ 3305 lossC:0.0443 lossL1:0.0146 imps:40.1 lr: 0.0227 etc:Tue Jul 6 16:03:46 2021 213 | step: 1940/ 3305 lossC:0.0587 lossL1:0.0142 imps:40.3 lr: 0.0226 etc:Tue Jul 6 16:03:44 2021 214 | step: 1950/ 3305 lossC:0.0578 lossL1:0.0145 imps:40.5 lr: 0.0224 etc:Tue Jul 6 16:03:43 2021 215 | step: 1960/ 3305 lossC:0.0481 lossL1:0.0145 imps:40.7 lr: 0.0223 etc:Tue Jul 6 16:03:41 2021 216 | step: 1970/ 3305 lossC:0.0542 lossL1:0.0143 imps:40.9 lr: 0.0221 etc:Tue Jul 6 16:03:38 2021 217 | step: 1980/ 3305 lossC:0.0593 lossL1:0.0144 imps:41.1 lr: 0.0220 etc:Tue Jul 6 16:03:36 2021 218 | Epoch 4/5 219 | step: 1990/ 3305 lossC:0.0559 lossL1:0.0146 imps:22.1 lr: 0.0218 etc:Tue Jul 6 16:03:40 2021 220 | step: 2000/ 3305 lossC:0.0504 lossL1:0.0145 imps:32.6 lr: 0.0217 etc:Tue Jul 6 16:03:39 2021 221 | validating ... {'iou_bkg': array([0.77791907, 0.42053454, 0.27607498, 0.57091107, 0.39701417, 222 | 0.45651733, 0.73026371, 0.53010228, 0.66553399, 0.24069007, 223 | 0.56222503, 0.45824862, 0.65332889, 0.58331313, 0.60938952, 224 | 0.55573553, 0.39488914, 0.61396203, 0.43392172, 0.52161598, 225 | 0.40302996]), 'miou_bkg': 0.5169152744618255} 226 | step: 2010/ 3305 lossC:0.0558 lossL1:0.0144 imps:6.1 lr: 0.0215 etc:Tue Jul 6 16:05:19 2021 227 | step: 2020/ 3305 lossC:0.0497 lossL1:0.0159 imps:7.9 lr: 0.0214 etc:Tue Jul 6 16:05:17 2021 228 | step: 2030/ 3305 lossC:0.0522 lossL1:0.0140 imps:9.7 lr: 0.0212 etc:Tue Jul 6 16:05:14 2021 229 | step: 2040/ 3305 lossC:0.0514 lossL1:0.0149 imps:11.3 lr: 0.0211 etc:Tue Jul 6 16:05:12 2021 230 | step: 2050/ 3305 lossC:0.0538 lossL1:0.0148 imps:12.8 lr: 0.0209 etc:Tue Jul 6 16:05:10 2021 231 | step: 2060/ 3305 lossC:0.0478 lossL1:0.0143 imps:14.2 lr: 0.0208 etc:Tue Jul 6 16:05:08 2021 232 | step: 2070/ 3305 lossC:0.0482 lossL1:0.0145 imps:15.5 lr: 0.0206 etc:Tue Jul 6 16:05:05 2021 233 | step: 2080/ 3305 lossC:0.0451 lossL1:0.0145 imps:16.8 lr: 0.0205 etc:Tue Jul 6 16:05:03 2021 234 | step: 2090/ 3305 lossC:0.0482 lossL1:0.0149 imps:17.9 lr: 0.0203 etc:Tue Jul 6 16:05:01 2021 235 | step: 2100/ 3305 lossC:0.0631 lossL1:0.0149 imps:19.0 lr: 0.0202 etc:Tue Jul 6 16:04:59 2021 236 | step: 2110/ 3305 lossC:0.0480 lossL1:0.0149 imps:19.9 lr: 0.0200 etc:Tue Jul 6 16:04:58 2021 237 | step: 2120/ 3305 lossC:0.0562 lossL1:0.0145 imps:20.9 lr: 0.0199 etc:Tue Jul 6 16:04:56 2021 238 | step: 2130/ 3305 lossC:0.0488 lossL1:0.0145 imps:21.8 lr: 0.0197 etc:Tue Jul 6 16:04:54 2021 239 | step: 2140/ 3305 lossC:0.0375 lossL1:0.0140 imps:22.7 lr: 0.0196 etc:Tue Jul 6 16:04:51 2021 240 | step: 2150/ 3305 lossC:0.0493 lossL1:0.0153 imps:23.6 lr: 0.0194 etc:Tue Jul 6 16:04:49 2021 241 | step: 2160/ 3305 lossC:0.0461 lossL1:0.0153 imps:24.3 lr: 0.0193 etc:Tue Jul 6 16:04:47 2021 242 | step: 2170/ 3305 lossC:0.0535 lossL1:0.0157 imps:25.1 lr: 0.0191 etc:Tue Jul 6 16:04:45 2021 243 | step: 2180/ 3305 lossC:0.0465 lossL1:0.0136 imps:25.8 lr: 0.0190 etc:Tue Jul 6 16:04:43 2021 244 | step: 2190/ 3305 lossC:0.0555 lossL1:0.0142 imps:26.5 lr: 0.0188 etc:Tue Jul 6 16:04:41 2021 245 | step: 2200/ 3305 lossC:0.0463 lossL1:0.0150 imps:27.2 lr: 0.0187 etc:Tue Jul 6 16:04:39 2021 246 | step: 2210/ 3305 lossC:0.0498 lossL1:0.0151 imps:27.5 lr: 0.0185 etc:Tue Jul 6 16:04:39 2021 247 | step: 2220/ 3305 lossC:0.0577 lossL1:0.0147 imps:28.1 lr: 0.0183 etc:Tue Jul 6 16:04:37 2021 248 | step: 2230/ 3305 lossC:0.0441 lossL1:0.0147 imps:28.7 lr: 0.0182 etc:Tue Jul 6 16:04:35 2021 249 | step: 2240/ 3305 lossC:0.0455 lossL1:0.0150 imps:29.3 lr: 0.0180 etc:Tue Jul 6 16:04:33 2021 250 | step: 2250/ 3305 lossC:0.0493 lossL1:0.0151 imps:29.8 lr: 0.0179 etc:Tue Jul 6 16:04:32 2021 251 | step: 2260/ 3305 lossC:0.0540 lossL1:0.0146 imps:30.3 lr: 0.0177 etc:Tue Jul 6 16:04:30 2021 252 | step: 2270/ 3305 lossC:0.0466 lossL1:0.0147 imps:30.8 lr: 0.0176 etc:Tue Jul 6 16:04:28 2021 253 | step: 2280/ 3305 lossC:0.0594 lossL1:0.0151 imps:31.3 lr: 0.0174 etc:Tue Jul 6 16:04:26 2021 254 | step: 2290/ 3305 lossC:0.0519 lossL1:0.0140 imps:31.8 lr: 0.0173 etc:Tue Jul 6 16:04:24 2021 255 | step: 2300/ 3305 lossC:0.0436 lossL1:0.0145 imps:32.2 lr: 0.0171 etc:Tue Jul 6 16:04:22 2021 256 | step: 2310/ 3305 lossC:0.0428 lossL1:0.0146 imps:32.5 lr: 0.0170 etc:Tue Jul 6 16:04:21 2021 257 | step: 2320/ 3305 lossC:0.0461 lossL1:0.0140 imps:32.9 lr: 0.0168 etc:Tue Jul 6 16:04:20 2021 258 | step: 2330/ 3305 lossC:0.0371 lossL1:0.0146 imps:33.3 lr: 0.0167 etc:Tue Jul 6 16:04:18 2021 259 | step: 2340/ 3305 lossC:0.0546 lossL1:0.0149 imps:33.7 lr: 0.0165 etc:Tue Jul 6 16:04:16 2021 260 | step: 2350/ 3305 lossC:0.0578 lossL1:0.0150 imps:34.1 lr: 0.0164 etc:Tue Jul 6 16:04:14 2021 261 | step: 2360/ 3305 lossC:0.0531 lossL1:0.0141 imps:34.4 lr: 0.0162 etc:Tue Jul 6 16:04:12 2021 262 | step: 2370/ 3305 lossC:0.0441 lossL1:0.0146 imps:34.8 lr: 0.0160 etc:Tue Jul 6 16:04:11 2021 263 | step: 2380/ 3305 lossC:0.0553 lossL1:0.0155 imps:35.2 lr: 0.0159 etc:Tue Jul 6 16:04:09 2021 264 | step: 2390/ 3305 lossC:0.0539 lossL1:0.0147 imps:35.5 lr: 0.0157 etc:Tue Jul 6 16:04:07 2021 265 | step: 2400/ 3305 lossC:0.0455 lossL1:0.0137 imps:35.8 lr: 0.0156 etc:Tue Jul 6 16:04:05 2021 266 | step: 2410/ 3305 lossC:0.0506 lossL1:0.0141 imps:36.0 lr: 0.0154 etc:Tue Jul 6 16:04:05 2021 267 | step: 2420/ 3305 lossC:0.0492 lossL1:0.0156 imps:36.3 lr: 0.0153 etc:Tue Jul 6 16:04:03 2021 268 | step: 2430/ 3305 lossC:0.0407 lossL1:0.0144 imps:36.6 lr: 0.0151 etc:Tue Jul 6 16:04:01 2021 269 | step: 2440/ 3305 lossC:0.0390 lossL1:0.0152 imps:36.9 lr: 0.0150 etc:Tue Jul 6 16:04:00 2021 270 | step: 2450/ 3305 lossC:0.0526 lossL1:0.0140 imps:37.2 lr: 0.0148 etc:Tue Jul 6 16:03:58 2021 271 | step: 2460/ 3305 lossC:0.0494 lossL1:0.0139 imps:37.4 lr: 0.0147 etc:Tue Jul 6 16:03:56 2021 272 | step: 2470/ 3305 lossC:0.0373 lossL1:0.0143 imps:37.7 lr: 0.0145 etc:Tue Jul 6 16:03:55 2021 273 | step: 2480/ 3305 lossC:0.0439 lossL1:0.0140 imps:38.0 lr: 0.0143 etc:Tue Jul 6 16:03:53 2021 274 | step: 2490/ 3305 lossC:0.0456 lossL1:0.0147 imps:38.2 lr: 0.0142 etc:Tue Jul 6 16:03:51 2021 275 | step: 2500/ 3305 lossC:0.0612 lossL1:0.0149 imps:38.5 lr: 0.0140 etc:Tue Jul 6 16:03:50 2021 276 | validating ... {'iou_bkg': array([0.77802502, 0.42497813, 0.28280943, 0.55857459, 0.3960244 , 277 | 0.47201243, 0.73486968, 0.54453976, 0.68416337, 0.25791521, 278 | 0.58844509, 0.45894063, 0.6651823 , 0.58904204, 0.62447471, 279 | 0.55412892, 0.39186552, 0.63654976, 0.44332001, 0.52120326, 280 | 0.38943611]), 'miou_bkg': 0.5236428741090146} 281 | step: 2510/ 3305 lossC:0.0534 lossL1:0.0143 imps:29.9 lr: 0.0139 etc:Tue Jul 6 16:05:12 2021 282 | step: 2520/ 3305 lossC:0.0544 lossL1:0.0148 imps:30.2 lr: 0.0137 etc:Tue Jul 6 16:05:11 2021 283 | step: 2530/ 3305 lossC:0.0473 lossL1:0.0149 imps:30.4 lr: 0.0136 etc:Tue Jul 6 16:05:09 2021 284 | step: 2540/ 3305 lossC:0.0446 lossL1:0.0142 imps:30.7 lr: 0.0134 etc:Tue Jul 6 16:05:07 2021 285 | step: 2550/ 3305 lossC:0.0325 lossL1:0.0147 imps:31.0 lr: 0.0132 etc:Tue Jul 6 16:05:05 2021 286 | step: 2560/ 3305 lossC:0.0528 lossL1:0.0154 imps:31.2 lr: 0.0131 etc:Tue Jul 6 16:05:03 2021 287 | step: 2570/ 3305 lossC:0.0380 lossL1:0.0146 imps:31.4 lr: 0.0129 etc:Tue Jul 6 16:05:01 2021 288 | step: 2580/ 3305 lossC:0.0521 lossL1:0.0154 imps:31.7 lr: 0.0128 etc:Tue Jul 6 16:04:59 2021 289 | step: 2590/ 3305 lossC:0.0529 lossL1:0.0144 imps:31.9 lr: 0.0126 etc:Tue Jul 6 16:04:58 2021 290 | step: 2600/ 3305 lossC:0.0545 lossL1:0.0138 imps:32.2 lr: 0.0124 etc:Tue Jul 6 16:04:56 2021 291 | step: 2610/ 3305 lossC:0.0475 lossL1:0.0139 imps:32.2 lr: 0.0123 etc:Tue Jul 6 16:04:56 2021 292 | step: 2620/ 3305 lossC:0.0554 lossL1:0.0141 imps:32.4 lr: 0.0121 etc:Tue Jul 6 16:04:54 2021 293 | step: 2630/ 3305 lossC:0.0429 lossL1:0.0150 imps:32.7 lr: 0.0120 etc:Tue Jul 6 16:04:52 2021 294 | step: 2640/ 3305 lossC:0.0513 lossL1:0.0148 imps:32.9 lr: 0.0118 etc:Tue Jul 6 16:04:50 2021 295 | Epoch 5/5 296 | step: 2650/ 3305 lossC:0.0489 lossL1:0.0142 imps:21.2 lr: 0.0117 etc:Tue Jul 6 16:04:53 2021 297 | step: 2660/ 3305 lossC:0.0429 lossL1:0.0144 imps:32.3 lr: 0.0115 etc:Tue Jul 6 16:04:52 2021 298 | step: 2670/ 3305 lossC:0.0445 lossL1:0.0138 imps:38.0 lr: 0.0113 etc:Tue Jul 6 16:04:50 2021 299 | step: 2680/ 3305 lossC:0.0414 lossL1:0.0144 imps:41.4 lr: 0.0112 etc:Tue Jul 6 16:04:49 2021 300 | step: 2690/ 3305 lossC:0.0468 lossL1:0.0146 imps:44.0 lr: 0.0110 etc:Tue Jul 6 16:04:47 2021 301 | step: 2700/ 3305 lossC:0.0474 lossL1:0.0148 imps:45.3 lr: 0.0108 etc:Tue Jul 6 16:04:46 2021 302 | step: 2710/ 3305 lossC:0.0559 lossL1:0.0146 imps:44.2 lr: 0.0107 etc:Tue Jul 6 16:04:46 2021 303 | step: 2720/ 3305 lossC:0.0519 lossL1:0.0148 imps:45.4 lr: 0.0105 etc:Tue Jul 6 16:04:44 2021 304 | step: 2730/ 3305 lossC:0.0491 lossL1:0.0145 imps:46.2 lr: 0.0104 etc:Tue Jul 6 16:04:43 2021 305 | step: 2740/ 3305 lossC:0.0508 lossL1:0.0143 imps:47.0 lr: 0.0102 etc:Tue Jul 6 16:04:41 2021 306 | step: 2750/ 3305 lossC:0.0402 lossL1:0.0144 imps:47.8 lr: 0.0100 etc:Tue Jul 6 16:04:39 2021 307 | step: 2760/ 3305 lossC:0.0350 lossL1:0.0144 imps:48.2 lr: 0.0099 etc:Tue Jul 6 16:04:38 2021 308 | step: 2770/ 3305 lossC:0.0345 lossL1:0.0152 imps:48.9 lr: 0.0097 etc:Tue Jul 6 16:04:36 2021 309 | step: 2780/ 3305 lossC:0.0443 lossL1:0.0142 imps:49.5 lr: 0.0095 etc:Tue Jul 6 16:04:35 2021 310 | step: 2790/ 3305 lossC:0.0432 lossL1:0.0142 imps:49.7 lr: 0.0094 etc:Tue Jul 6 16:04:33 2021 311 | step: 2800/ 3305 lossC:0.0521 lossL1:0.0148 imps:50.2 lr: 0.0092 etc:Tue Jul 6 16:04:32 2021 312 | step: 2810/ 3305 lossC:0.0446 lossL1:0.0143 imps:49.7 lr: 0.0091 etc:Tue Jul 6 16:04:31 2021 313 | step: 2820/ 3305 lossC:0.0435 lossL1:0.0138 imps:50.0 lr: 0.0089 etc:Tue Jul 6 16:04:30 2021 314 | step: 2830/ 3305 lossC:0.0481 lossL1:0.0156 imps:50.2 lr: 0.0087 etc:Tue Jul 6 16:04:28 2021 315 | step: 2840/ 3305 lossC:0.0497 lossL1:0.0145 imps:50.6 lr: 0.0086 etc:Tue Jul 6 16:04:27 2021 316 | step: 2850/ 3305 lossC:0.0445 lossL1:0.0145 imps:50.8 lr: 0.0084 etc:Tue Jul 6 16:04:25 2021 317 | step: 2860/ 3305 lossC:0.0446 lossL1:0.0145 imps:51.0 lr: 0.0082 etc:Tue Jul 6 16:04:24 2021 318 | step: 2870/ 3305 lossC:0.0470 lossL1:0.0132 imps:51.3 lr: 0.0081 etc:Tue Jul 6 16:04:22 2021 319 | step: 2880/ 3305 lossC:0.0383 lossL1:0.0147 imps:51.4 lr: 0.0079 etc:Tue Jul 6 16:04:21 2021 320 | step: 2890/ 3305 lossC:0.0462 lossL1:0.0145 imps:51.6 lr: 0.0077 etc:Tue Jul 6 16:04:20 2021 321 | step: 2900/ 3305 lossC:0.0556 lossL1:0.0151 imps:51.8 lr: 0.0076 etc:Tue Jul 6 16:04:18 2021 322 | step: 2910/ 3305 lossC:0.0469 lossL1:0.0153 imps:51.4 lr: 0.0074 etc:Tue Jul 6 16:04:18 2021 323 | step: 2920/ 3305 lossC:0.0421 lossL1:0.0149 imps:51.4 lr: 0.0072 etc:Tue Jul 6 16:04:17 2021 324 | step: 2930/ 3305 lossC:0.0451 lossL1:0.0145 imps:51.6 lr: 0.0071 etc:Tue Jul 6 16:04:15 2021 325 | step: 2940/ 3305 lossC:0.0437 lossL1:0.0142 imps:51.8 lr: 0.0069 etc:Tue Jul 6 16:04:14 2021 326 | step: 2950/ 3305 lossC:0.0299 lossL1:0.0141 imps:51.9 lr: 0.0067 etc:Tue Jul 6 16:04:12 2021 327 | step: 2960/ 3305 lossC:0.0405 lossL1:0.0149 imps:52.0 lr: 0.0065 etc:Tue Jul 6 16:04:11 2021 328 | step: 2970/ 3305 lossC:0.0284 lossL1:0.0143 imps:52.1 lr: 0.0064 etc:Tue Jul 6 16:04:10 2021 329 | step: 2980/ 3305 lossC:0.0405 lossL1:0.0143 imps:52.2 lr: 0.0062 etc:Tue Jul 6 16:04:08 2021 330 | step: 2990/ 3305 lossC:0.0487 lossL1:0.0149 imps:52.3 lr: 0.0060 etc:Tue Jul 6 16:04:07 2021 331 | step: 3000/ 3305 lossC:0.0562 lossL1:0.0148 imps:52.4 lr: 0.0059 etc:Tue Jul 6 16:04:06 2021 332 | validating ... {'iou_bkg': array([0.76802149, 0.40001662, 0.26745541, 0.55049392, 0.37765112, 333 | 0.48111135, 0.73401677, 0.52836657, 0.67561716, 0.2568666 , 334 | 0.58328802, 0.46601148, 0.65700628, 0.57356224, 0.61704192, 335 | 0.54915853, 0.38335854, 0.62096559, 0.44201802, 0.53031113, 336 | 0.40538269]), 'miou_bkg': 0.5175105459016939} 337 | step: 3010/ 3305 lossC:0.0418 lossL1:0.0145 imps:33.9 lr: 0.0057 etc:Tue Jul 6 16:05:12 2021 338 | step: 3020/ 3305 lossC:0.0424 lossL1:0.0140 imps:34.3 lr: 0.0055 etc:Tue Jul 6 16:05:10 2021 339 | step: 3030/ 3305 lossC:0.0414 lossL1:0.0142 imps:34.6 lr: 0.0053 etc:Tue Jul 6 16:05:09 2021 340 | step: 3040/ 3305 lossC:0.0422 lossL1:0.0146 imps:35.0 lr: 0.0052 etc:Tue Jul 6 16:05:07 2021 341 | step: 3050/ 3305 lossC:0.0400 lossL1:0.0142 imps:35.3 lr: 0.0050 etc:Tue Jul 6 16:05:06 2021 342 | step: 3060/ 3305 lossC:0.0374 lossL1:0.0148 imps:35.6 lr: 0.0048 etc:Tue Jul 6 16:05:04 2021 343 | step: 3070/ 3305 lossC:0.0436 lossL1:0.0150 imps:35.9 lr: 0.0046 etc:Tue Jul 6 16:05:03 2021 344 | step: 3080/ 3305 lossC:0.0425 lossL1:0.0151 imps:36.2 lr: 0.0045 etc:Tue Jul 6 16:05:01 2021 345 | step: 3090/ 3305 lossC:0.0457 lossL1:0.0142 imps:36.5 lr: 0.0043 etc:Tue Jul 6 16:05:00 2021 346 | step: 3100/ 3305 lossC:0.0525 lossL1:0.0149 imps:36.8 lr: 0.0041 etc:Tue Jul 6 16:04:58 2021 347 | step: 3110/ 3305 lossC:0.0385 lossL1:0.0143 imps:36.9 lr: 0.0039 etc:Tue Jul 6 16:04:58 2021 348 | step: 3120/ 3305 lossC:0.0498 lossL1:0.0146 imps:37.2 lr: 0.0037 etc:Tue Jul 6 16:04:56 2021 349 | step: 3130/ 3305 lossC:0.0430 lossL1:0.0154 imps:37.5 lr: 0.0036 etc:Tue Jul 6 16:04:55 2021 350 | step: 3140/ 3305 lossC:0.0476 lossL1:0.0147 imps:37.7 lr: 0.0034 etc:Tue Jul 6 16:04:53 2021 351 | step: 3150/ 3305 lossC:0.0452 lossL1:0.0142 imps:37.9 lr: 0.0032 etc:Tue Jul 6 16:04:52 2021 352 | step: 3160/ 3305 lossC:0.0395 lossL1:0.0148 imps:38.2 lr: 0.0030 etc:Tue Jul 6 16:04:50 2021 353 | step: 3170/ 3305 lossC:0.0383 lossL1:0.0138 imps:38.4 lr: 0.0028 etc:Tue Jul 6 16:04:49 2021 354 | step: 3180/ 3305 lossC:0.0394 lossL1:0.0139 imps:38.6 lr: 0.0026 etc:Tue Jul 6 16:04:48 2021 355 | step: 3190/ 3305 lossC:0.0429 lossL1:0.0133 imps:38.9 lr: 0.0024 etc:Tue Jul 6 16:04:46 2021 356 | step: 3200/ 3305 lossC:0.0384 lossL1:0.0135 imps:39.1 lr: 0.0022 etc:Tue Jul 6 16:04:45 2021 357 | step: 3210/ 3305 lossC:0.0372 lossL1:0.0144 imps:39.1 lr: 0.0020 etc:Tue Jul 6 16:04:45 2021 358 | step: 3220/ 3305 lossC:0.0323 lossL1:0.0147 imps:39.4 lr: 0.0019 etc:Tue Jul 6 16:04:43 2021 359 | step: 3230/ 3305 lossC:0.0396 lossL1:0.0145 imps:39.5 lr: 0.0017 etc:Tue Jul 6 16:04:42 2021 360 | step: 3240/ 3305 lossC:0.0490 lossL1:0.0140 imps:39.7 lr: 0.0015 etc:Tue Jul 6 16:04:41 2021 361 | step: 3250/ 3305 lossC:0.0441 lossL1:0.0155 imps:39.9 lr: 0.0013 etc:Tue Jul 6 16:04:39 2021 362 | step: 3260/ 3305 lossC:0.0516 lossL1:0.0145 imps:40.1 lr: 0.0010 etc:Tue Jul 6 16:04:38 2021 363 | step: 3270/ 3305 lossC:0.0386 lossL1:0.0146 imps:40.3 lr: 0.0008 etc:Tue Jul 6 16:04:37 2021 364 | step: 3280/ 3305 lossC:0.0344 lossL1:0.0136 imps:40.5 lr: 0.0006 etc:Tue Jul 6 16:04:35 2021 365 | step: 3290/ 3305 lossC:0.0370 lossL1:0.0141 imps:40.7 lr: 0.0004 etc:Tue Jul 6 16:04:34 2021 366 | step: 3300/ 3305 lossC:0.0373 lossL1:0.0150 imps:40.9 lr: 0.0001 etc:Tue Jul 6 16:04:32 2021 367 | -------------------------------------------------------------------------------- /log/sipe_voc_rep.log: -------------------------------------------------------------------------------- 1 | Epoch 1/5 2 | step: 0/ 3305 lossCLS:0.6907 lossGSC:0.0358 imps:1.7 lr: 0.1000 etc:Thu Mar 10 22:12:15 2022 3 | step: 10/ 3305 lossCLS:0.2611 lossGSC:0.0162 imps:11.8 lr: 0.0997 etc:Thu Mar 10 14:46:40 2022 4 | step: 20/ 3305 lossCLS:0.2304 lossGSC:0.0092 imps:16.8 lr: 0.0995 etc:Thu Mar 10 14:24:22 2022 5 | step: 30/ 3305 lossCLS:0.2266 lossGSC:0.0116 imps:20.6 lr: 0.0992 etc:Thu Mar 10 14:14:37 2022 6 | step: 40/ 3305 lossCLS:0.2136 lossGSC:0.0138 imps:22.4 lr: 0.0989 etc:Thu Mar 10 14:11:10 2022 7 | step: 50/ 3305 lossCLS:0.1818 lossGSC:0.0165 imps:23.7 lr: 0.0986 etc:Thu Mar 10 14:09:04 2022 8 | step: 60/ 3305 lossCLS:0.1654 lossGSC:0.0163 imps:24.6 lr: 0.0984 etc:Thu Mar 10 14:07:40 2022 9 | step: 70/ 3305 lossCLS:0.1665 lossGSC:0.0178 imps:25.2 lr: 0.0981 etc:Thu Mar 10 14:06:51 2022 10 | step: 80/ 3305 lossCLS:0.1367 lossGSC:0.0175 imps:25.7 lr: 0.0978 etc:Thu Mar 10 14:06:09 2022 11 | step: 90/ 3305 lossCLS:0.1408 lossGSC:0.0172 imps:26.2 lr: 0.0975 etc:Thu Mar 10 14:05:33 2022 12 | step: 100/ 3305 lossCLS:0.1283 lossGSC:0.0168 imps:26.6 lr: 0.0973 etc:Thu Mar 10 14:05:04 2022 13 | step: 110/ 3305 lossCLS:0.1092 lossGSC:0.0170 imps:26.9 lr: 0.0970 etc:Thu Mar 10 14:04:39 2022 14 | step: 120/ 3305 lossCLS:0.1202 lossGSC:0.0170 imps:27.2 lr: 0.0967 etc:Thu Mar 10 14:04:14 2022 15 | step: 130/ 3305 lossCLS:0.1055 lossGSC:0.0171 imps:27.5 lr: 0.0965 etc:Thu Mar 10 14:03:59 2022 16 | step: 140/ 3305 lossCLS:0.1041 lossGSC:0.0159 imps:27.5 lr: 0.0962 etc:Thu Mar 10 14:03:57 2022 17 | step: 150/ 3305 lossCLS:0.1116 lossGSC:0.0153 imps:27.7 lr: 0.0959 etc:Thu Mar 10 14:03:43 2022 18 | step: 160/ 3305 lossCLS:0.1127 lossGSC:0.0166 imps:27.8 lr: 0.0956 etc:Thu Mar 10 14:03:32 2022 19 | step: 170/ 3305 lossCLS:0.0992 lossGSC:0.0162 imps:28.0 lr: 0.0954 etc:Thu Mar 10 14:03:21 2022 20 | step: 180/ 3305 lossCLS:0.1060 lossGSC:0.0168 imps:28.2 lr: 0.0951 etc:Thu Mar 10 14:03:11 2022 21 | step: 190/ 3305 lossCLS:0.1082 lossGSC:0.0165 imps:28.3 lr: 0.0948 etc:Thu Mar 10 14:03:01 2022 22 | step: 200/ 3305 lossCLS:0.0987 lossGSC:0.0165 imps:28.5 lr: 0.0945 etc:Thu Mar 10 14:02:50 2022 23 | step: 210/ 3305 lossCLS:0.1058 lossGSC:0.0158 imps:28.6 lr: 0.0943 etc:Thu Mar 10 14:02:44 2022 24 | step: 220/ 3305 lossCLS:0.0933 lossGSC:0.0161 imps:28.7 lr: 0.0940 etc:Thu Mar 10 14:02:38 2022 25 | step: 230/ 3305 lossCLS:0.0878 lossGSC:0.0156 imps:28.8 lr: 0.0937 etc:Thu Mar 10 14:02:32 2022 26 | step: 240/ 3305 lossCLS:0.1112 lossGSC:0.0172 imps:28.9 lr: 0.0934 etc:Thu Mar 10 14:02:25 2022 27 | step: 250/ 3305 lossCLS:0.0940 lossGSC:0.0165 imps:29.0 lr: 0.0932 etc:Thu Mar 10 14:02:19 2022 28 | step: 260/ 3305 lossCLS:0.0904 lossGSC:0.0157 imps:29.1 lr: 0.0929 etc:Thu Mar 10 14:02:12 2022 29 | step: 270/ 3305 lossCLS:0.0908 lossGSC:0.0163 imps:29.1 lr: 0.0926 etc:Thu Mar 10 14:02:08 2022 30 | step: 280/ 3305 lossCLS:0.0828 lossGSC:0.0158 imps:29.2 lr: 0.0923 etc:Thu Mar 10 14:02:07 2022 31 | step: 290/ 3305 lossCLS:0.0821 lossGSC:0.0160 imps:29.1 lr: 0.0921 etc:Thu Mar 10 14:02:08 2022 32 | step: 300/ 3305 lossCLS:0.0880 lossGSC:0.0163 imps:29.2 lr: 0.0918 etc:Thu Mar 10 14:02:05 2022 33 | step: 310/ 3305 lossCLS:0.0797 lossGSC:0.0164 imps:29.2 lr: 0.0915 etc:Thu Mar 10 14:02:02 2022 34 | step: 320/ 3305 lossCLS:0.0868 lossGSC:0.0163 imps:29.2 lr: 0.0912 etc:Thu Mar 10 14:02:04 2022 35 | step: 330/ 3305 lossCLS:0.0937 lossGSC:0.0164 imps:29.2 lr: 0.0910 etc:Thu Mar 10 14:02:07 2022 36 | step: 340/ 3305 lossCLS:0.0984 lossGSC:0.0165 imps:29.1 lr: 0.0907 etc:Thu Mar 10 14:02:08 2022 37 | step: 350/ 3305 lossCLS:0.0881 lossGSC:0.0163 imps:29.2 lr: 0.0904 etc:Thu Mar 10 14:02:05 2022 38 | step: 360/ 3305 lossCLS:0.1022 lossGSC:0.0162 imps:29.2 lr: 0.0901 etc:Thu Mar 10 14:02:03 2022 39 | step: 370/ 3305 lossCLS:0.0918 lossGSC:0.0161 imps:29.3 lr: 0.0899 etc:Thu Mar 10 14:01:59 2022 40 | step: 380/ 3305 lossCLS:0.0804 lossGSC:0.0154 imps:29.3 lr: 0.0896 etc:Thu Mar 10 14:01:56 2022 41 | step: 390/ 3305 lossCLS:0.0791 lossGSC:0.0150 imps:29.4 lr: 0.0893 etc:Thu Mar 10 14:01:54 2022 42 | step: 400/ 3305 lossCLS:0.0784 lossGSC:0.0156 imps:29.4 lr: 0.0890 etc:Thu Mar 10 14:01:52 2022 43 | step: 410/ 3305 lossCLS:0.0972 lossGSC:0.0163 imps:29.4 lr: 0.0888 etc:Thu Mar 10 14:01:51 2022 44 | step: 420/ 3305 lossCLS:0.0776 lossGSC:0.0159 imps:29.4 lr: 0.0885 etc:Thu Mar 10 14:01:49 2022 45 | step: 430/ 3305 lossCLS:0.0876 lossGSC:0.0162 imps:29.5 lr: 0.0882 etc:Thu Mar 10 14:01:47 2022 46 | step: 440/ 3305 lossCLS:0.0900 lossGSC:0.0173 imps:29.5 lr: 0.0879 etc:Thu Mar 10 14:01:46 2022 47 | step: 450/ 3305 lossCLS:0.0883 lossGSC:0.0163 imps:29.5 lr: 0.0877 etc:Thu Mar 10 14:01:44 2022 48 | step: 460/ 3305 lossCLS:0.0834 lossGSC:0.0156 imps:29.6 lr: 0.0874 etc:Thu Mar 10 14:01:42 2022 49 | step: 470/ 3305 lossCLS:0.0925 lossGSC:0.0164 imps:29.6 lr: 0.0871 etc:Thu Mar 10 14:01:41 2022 50 | step: 480/ 3305 lossCLS:0.0756 lossGSC:0.0152 imps:29.7 lr: 0.0868 etc:Thu Mar 10 14:01:34 2022 51 | step: 490/ 3305 lossCLS:0.0881 lossGSC:0.0150 imps:29.7 lr: 0.0866 etc:Thu Mar 10 14:01:31 2022 52 | step: 500/ 3305 lossCLS:0.0785 lossGSC:0.0151 imps:29.8 lr: 0.0863 etc:Thu Mar 10 14:01:30 2022 53 | validating ... {'iou': array([0.79288635, 0.49043208, 0.29905373, 0.54197695, 0.39179012, 54 | 0.49763909, 0.7461943 , 0.60308012, 0.64424431, 0.20224494, 55 | 0.51302857, 0.44051402, 0.61325274, 0.6151208 , 0.67692742, 56 | 0.54998199, 0.46103338, 0.58297177, 0.43481994, 0.51470415, 57 | 0.40958095]), 'miou': 0.5248322724354337} 58 | step: 510/ 3305 lossCLS:0.0844 lossGSC:0.0147 imps:25.2 lr: 0.0860 etc:Thu Mar 10 14:06:51 2022 59 | step: 520/ 3305 lossCLS:0.0841 lossGSC:0.0157 imps:25.3 lr: 0.0857 etc:Thu Mar 10 14:06:44 2022 60 | step: 530/ 3305 lossCLS:0.0822 lossGSC:0.0146 imps:25.4 lr: 0.0854 etc:Thu Mar 10 14:06:37 2022 61 | step: 540/ 3305 lossCLS:0.0898 lossGSC:0.0152 imps:25.5 lr: 0.0852 etc:Thu Mar 10 14:06:31 2022 62 | step: 550/ 3305 lossCLS:0.0839 lossGSC:0.0158 imps:25.5 lr: 0.0849 etc:Thu Mar 10 14:06:24 2022 63 | step: 560/ 3305 lossCLS:0.0941 lossGSC:0.0151 imps:25.6 lr: 0.0846 etc:Thu Mar 10 14:06:18 2022 64 | step: 570/ 3305 lossCLS:0.0791 lossGSC:0.0150 imps:25.7 lr: 0.0843 etc:Thu Mar 10 14:06:12 2022 65 | step: 580/ 3305 lossCLS:0.0922 lossGSC:0.0148 imps:25.8 lr: 0.0841 etc:Thu Mar 10 14:06:05 2022 66 | step: 590/ 3305 lossCLS:0.0729 lossGSC:0.0151 imps:25.8 lr: 0.0838 etc:Thu Mar 10 14:06:01 2022 67 | step: 600/ 3305 lossCLS:0.0800 lossGSC:0.0153 imps:25.9 lr: 0.0835 etc:Thu Mar 10 14:05:58 2022 68 | step: 610/ 3305 lossCLS:0.0857 lossGSC:0.0160 imps:25.9 lr: 0.0832 etc:Thu Mar 10 14:05:53 2022 69 | step: 620/ 3305 lossCLS:0.0737 lossGSC:0.0159 imps:26.0 lr: 0.0829 etc:Thu Mar 10 14:05:50 2022 70 | step: 630/ 3305 lossCLS:0.0769 lossGSC:0.0159 imps:26.0 lr: 0.0827 etc:Thu Mar 10 14:05:46 2022 71 | step: 640/ 3305 lossCLS:0.0696 lossGSC:0.0158 imps:26.1 lr: 0.0824 etc:Thu Mar 10 14:05:41 2022 72 | step: 650/ 3305 lossCLS:0.0793 lossGSC:0.0149 imps:26.1 lr: 0.0821 etc:Thu Mar 10 14:05:37 2022 73 | step: 660/ 3305 lossCLS:0.0786 lossGSC:0.0152 imps:26.2 lr: 0.0818 etc:Thu Mar 10 14:05:30 2022 74 | Epoch 2/5 75 | step: 670/ 3305 lossCLS:0.0735 lossGSC:0.0154 imps:21.1 lr: 0.0816 etc:Thu Mar 10 14:05:38 2022 76 | step: 680/ 3305 lossCLS:0.0843 lossGSC:0.0153 imps:25.7 lr: 0.0813 etc:Thu Mar 10 14:05:32 2022 77 | step: 690/ 3305 lossCLS:0.0664 lossGSC:0.0151 imps:27.6 lr: 0.0810 etc:Thu Mar 10 14:05:27 2022 78 | step: 700/ 3305 lossCLS:0.0785 lossGSC:0.0166 imps:28.2 lr: 0.0807 etc:Thu Mar 10 14:05:23 2022 79 | step: 710/ 3305 lossCLS:0.0677 lossGSC:0.0149 imps:28.1 lr: 0.0804 etc:Thu Mar 10 14:05:21 2022 80 | step: 720/ 3305 lossCLS:0.0718 lossGSC:0.0155 imps:28.3 lr: 0.0802 etc:Thu Mar 10 14:05:19 2022 81 | step: 730/ 3305 lossCLS:0.0693 lossGSC:0.0146 imps:28.6 lr: 0.0799 etc:Thu Mar 10 14:05:15 2022 82 | step: 740/ 3305 lossCLS:0.0616 lossGSC:0.0157 imps:28.9 lr: 0.0796 etc:Thu Mar 10 14:05:11 2022 83 | step: 750/ 3305 lossCLS:0.0701 lossGSC:0.0148 imps:29.7 lr: 0.0793 etc:Thu Mar 10 14:05:03 2022 84 | step: 760/ 3305 lossCLS:0.0803 lossGSC:0.0145 imps:29.9 lr: 0.0790 etc:Thu Mar 10 14:04:58 2022 85 | step: 770/ 3305 lossCLS:0.0704 lossGSC:0.0151 imps:30.1 lr: 0.0788 etc:Thu Mar 10 14:04:54 2022 86 | step: 780/ 3305 lossCLS:0.0633 lossGSC:0.0164 imps:30.3 lr: 0.0785 etc:Thu Mar 10 14:04:49 2022 87 | step: 790/ 3305 lossCLS:0.0699 lossGSC:0.0152 imps:30.5 lr: 0.0782 etc:Thu Mar 10 14:04:45 2022 88 | step: 800/ 3305 lossCLS:0.0727 lossGSC:0.0156 imps:30.6 lr: 0.0779 etc:Thu Mar 10 14:04:41 2022 89 | step: 810/ 3305 lossCLS:0.0707 lossGSC:0.0153 imps:30.4 lr: 0.0776 etc:Thu Mar 10 14:04:39 2022 90 | step: 820/ 3305 lossCLS:0.0641 lossGSC:0.0140 imps:30.4 lr: 0.0774 etc:Thu Mar 10 14:04:37 2022 91 | step: 830/ 3305 lossCLS:0.0817 lossGSC:0.0153 imps:30.3 lr: 0.0771 etc:Thu Mar 10 14:04:35 2022 92 | step: 840/ 3305 lossCLS:0.0656 lossGSC:0.0154 imps:30.4 lr: 0.0768 etc:Thu Mar 10 14:04:31 2022 93 | step: 850/ 3305 lossCLS:0.0585 lossGSC:0.0141 imps:30.5 lr: 0.0765 etc:Thu Mar 10 14:04:28 2022 94 | step: 860/ 3305 lossCLS:0.0656 lossGSC:0.0146 imps:30.6 lr: 0.0762 etc:Thu Mar 10 14:04:24 2022 95 | step: 870/ 3305 lossCLS:0.0713 lossGSC:0.0157 imps:30.7 lr: 0.0760 etc:Thu Mar 10 14:04:20 2022 96 | step: 880/ 3305 lossCLS:0.0715 lossGSC:0.0151 imps:30.8 lr: 0.0757 etc:Thu Mar 10 14:04:16 2022 97 | step: 890/ 3305 lossCLS:0.0683 lossGSC:0.0156 imps:30.8 lr: 0.0754 etc:Thu Mar 10 14:04:13 2022 98 | step: 900/ 3305 lossCLS:0.0783 lossGSC:0.0151 imps:30.9 lr: 0.0751 etc:Thu Mar 10 14:04:10 2022 99 | step: 910/ 3305 lossCLS:0.0729 lossGSC:0.0153 imps:30.8 lr: 0.0748 etc:Thu Mar 10 14:04:09 2022 100 | step: 920/ 3305 lossCLS:0.0805 lossGSC:0.0142 imps:30.7 lr: 0.0746 etc:Thu Mar 10 14:04:08 2022 101 | step: 930/ 3305 lossCLS:0.0629 lossGSC:0.0146 imps:30.7 lr: 0.0743 etc:Thu Mar 10 14:04:06 2022 102 | step: 940/ 3305 lossCLS:0.0578 lossGSC:0.0145 imps:30.8 lr: 0.0740 etc:Thu Mar 10 14:04:02 2022 103 | step: 950/ 3305 lossCLS:0.0671 lossGSC:0.0148 imps:30.8 lr: 0.0737 etc:Thu Mar 10 14:03:59 2022 104 | step: 960/ 3305 lossCLS:0.0632 lossGSC:0.0148 imps:30.9 lr: 0.0734 etc:Thu Mar 10 14:03:56 2022 105 | step: 970/ 3305 lossCLS:0.0829 lossGSC:0.0140 imps:30.9 lr: 0.0731 etc:Thu Mar 10 14:03:53 2022 106 | step: 980/ 3305 lossCLS:0.0654 lossGSC:0.0151 imps:31.0 lr: 0.0729 etc:Thu Mar 10 14:03:49 2022 107 | step: 990/ 3305 lossCLS:0.0707 lossGSC:0.0151 imps:31.0 lr: 0.0726 etc:Thu Mar 10 14:03:46 2022 108 | step: 1000/ 3305 lossCLS:0.0694 lossGSC:0.0152 imps:31.0 lr: 0.0723 etc:Thu Mar 10 14:03:44 2022 109 | validating ... {'iou': array([0.79087287, 0.43485958, 0.26171275, 0.55395219, 0.36973715, 110 | 0.4485016 , 0.72867764, 0.58084356, 0.69368716, 0.2348285 , 111 | 0.61418616, 0.40272933, 0.65253774, 0.59791612, 0.66172302, 112 | 0.56139111, 0.40216813, 0.64341218, 0.47655521, 0.49347382, 113 | 0.41969351]), 'miou': 0.5249266341872264} 114 | step: 1010/ 3305 lossCLS:0.0679 lossGSC:0.0155 imps:23.5 lr: 0.0720 etc:Thu Mar 10 14:06:51 2022 115 | step: 1020/ 3305 lossCLS:0.0739 lossGSC:0.0154 imps:23.7 lr: 0.0717 etc:Thu Mar 10 14:06:47 2022 116 | step: 1030/ 3305 lossCLS:0.0720 lossGSC:0.0152 imps:23.8 lr: 0.0715 etc:Thu Mar 10 14:06:43 2022 117 | step: 1040/ 3305 lossCLS:0.0619 lossGSC:0.0147 imps:24.0 lr: 0.0712 etc:Thu Mar 10 14:06:38 2022 118 | step: 1050/ 3305 lossCLS:0.0603 lossGSC:0.0161 imps:24.2 lr: 0.0709 etc:Thu Mar 10 14:06:34 2022 119 | step: 1060/ 3305 lossCLS:0.0738 lossGSC:0.0143 imps:24.3 lr: 0.0706 etc:Thu Mar 10 14:06:30 2022 120 | step: 1070/ 3305 lossCLS:0.0682 lossGSC:0.0143 imps:24.5 lr: 0.0703 etc:Thu Mar 10 14:06:25 2022 121 | step: 1080/ 3305 lossCLS:0.0657 lossGSC:0.0153 imps:24.6 lr: 0.0700 etc:Thu Mar 10 14:06:21 2022 122 | step: 1090/ 3305 lossCLS:0.0658 lossGSC:0.0147 imps:24.8 lr: 0.0698 etc:Thu Mar 10 14:06:17 2022 123 | step: 1100/ 3305 lossCLS:0.0672 lossGSC:0.0147 imps:24.9 lr: 0.0695 etc:Thu Mar 10 14:06:14 2022 124 | step: 1110/ 3305 lossCLS:0.0687 lossGSC:0.0146 imps:24.9 lr: 0.0692 etc:Thu Mar 10 14:06:12 2022 125 | step: 1120/ 3305 lossCLS:0.0762 lossGSC:0.0146 imps:25.0 lr: 0.0689 etc:Thu Mar 10 14:06:10 2022 126 | step: 1130/ 3305 lossCLS:0.0729 lossGSC:0.0143 imps:25.2 lr: 0.0686 etc:Thu Mar 10 14:06:06 2022 127 | step: 1140/ 3305 lossCLS:0.0665 lossGSC:0.0150 imps:25.3 lr: 0.0683 etc:Thu Mar 10 14:06:03 2022 128 | step: 1150/ 3305 lossCLS:0.0651 lossGSC:0.0146 imps:25.4 lr: 0.0681 etc:Thu Mar 10 14:05:59 2022 129 | step: 1160/ 3305 lossCLS:0.0587 lossGSC:0.0145 imps:25.5 lr: 0.0678 etc:Thu Mar 10 14:05:55 2022 130 | step: 1170/ 3305 lossCLS:0.0759 lossGSC:0.0143 imps:25.6 lr: 0.0675 etc:Thu Mar 10 14:05:51 2022 131 | step: 1180/ 3305 lossCLS:0.0690 lossGSC:0.0146 imps:25.7 lr: 0.0672 etc:Thu Mar 10 14:05:48 2022 132 | step: 1190/ 3305 lossCLS:0.0692 lossGSC:0.0148 imps:25.8 lr: 0.0669 etc:Thu Mar 10 14:05:45 2022 133 | step: 1200/ 3305 lossCLS:0.0607 lossGSC:0.0148 imps:25.9 lr: 0.0666 etc:Thu Mar 10 14:05:43 2022 134 | step: 1210/ 3305 lossCLS:0.0662 lossGSC:0.0138 imps:25.9 lr: 0.0663 etc:Thu Mar 10 14:05:42 2022 135 | step: 1220/ 3305 lossCLS:0.0691 lossGSC:0.0130 imps:26.0 lr: 0.0661 etc:Thu Mar 10 14:05:39 2022 136 | step: 1230/ 3305 lossCLS:0.0647 lossGSC:0.0148 imps:26.1 lr: 0.0658 etc:Thu Mar 10 14:05:36 2022 137 | step: 1240/ 3305 lossCLS:0.0693 lossGSC:0.0143 imps:26.2 lr: 0.0655 etc:Thu Mar 10 14:05:33 2022 138 | step: 1250/ 3305 lossCLS:0.0653 lossGSC:0.0146 imps:26.3 lr: 0.0652 etc:Thu Mar 10 14:05:29 2022 139 | step: 1260/ 3305 lossCLS:0.0555 lossGSC:0.0142 imps:26.3 lr: 0.0649 etc:Thu Mar 10 14:05:26 2022 140 | step: 1270/ 3305 lossCLS:0.0734 lossGSC:0.0147 imps:26.4 lr: 0.0646 etc:Thu Mar 10 14:05:23 2022 141 | step: 1280/ 3305 lossCLS:0.0591 lossGSC:0.0144 imps:26.5 lr: 0.0643 etc:Thu Mar 10 14:05:20 2022 142 | step: 1290/ 3305 lossCLS:0.0610 lossGSC:0.0145 imps:26.6 lr: 0.0641 etc:Thu Mar 10 14:05:18 2022 143 | step: 1300/ 3305 lossCLS:0.0618 lossGSC:0.0141 imps:26.6 lr: 0.0638 etc:Thu Mar 10 14:05:16 2022 144 | step: 1310/ 3305 lossCLS:0.0634 lossGSC:0.0132 imps:26.7 lr: 0.0635 etc:Thu Mar 10 14:05:14 2022 145 | step: 1320/ 3305 lossCLS:0.0608 lossGSC:0.0145 imps:26.7 lr: 0.0632 etc:Thu Mar 10 14:05:12 2022 146 | Epoch 3/5 147 | step: 1330/ 3305 lossCLS:0.0520 lossGSC:0.0154 imps:14.6 lr: 0.0629 etc:Thu Mar 10 14:05:23 2022 148 | step: 1340/ 3305 lossCLS:0.0625 lossGSC:0.0142 imps:20.4 lr: 0.0626 etc:Thu Mar 10 14:05:21 2022 149 | step: 1350/ 3305 lossCLS:0.0649 lossGSC:0.0144 imps:23.3 lr: 0.0623 etc:Thu Mar 10 14:05:18 2022 150 | step: 1360/ 3305 lossCLS:0.0697 lossGSC:0.0137 imps:25.1 lr: 0.0621 etc:Thu Mar 10 14:05:15 2022 151 | step: 1370/ 3305 lossCLS:0.0742 lossGSC:0.0147 imps:26.3 lr: 0.0618 etc:Thu Mar 10 14:05:13 2022 152 | step: 1380/ 3305 lossCLS:0.0617 lossGSC:0.0144 imps:27.1 lr: 0.0615 etc:Thu Mar 10 14:05:10 2022 153 | step: 1390/ 3305 lossCLS:0.0690 lossGSC:0.0143 imps:27.8 lr: 0.0612 etc:Thu Mar 10 14:05:07 2022 154 | step: 1400/ 3305 lossCLS:0.0658 lossGSC:0.0138 imps:28.2 lr: 0.0609 etc:Thu Mar 10 14:05:05 2022 155 | step: 1410/ 3305 lossCLS:0.0517 lossGSC:0.0141 imps:28.4 lr: 0.0606 etc:Thu Mar 10 14:05:03 2022 156 | step: 1420/ 3305 lossCLS:0.0566 lossGSC:0.0141 imps:28.5 lr: 0.0603 etc:Thu Mar 10 14:05:02 2022 157 | step: 1430/ 3305 lossCLS:0.0564 lossGSC:0.0143 imps:28.5 lr: 0.0600 etc:Thu Mar 10 14:05:01 2022 158 | step: 1440/ 3305 lossCLS:0.0606 lossGSC:0.0142 imps:28.8 lr: 0.0598 etc:Thu Mar 10 14:04:58 2022 159 | step: 1450/ 3305 lossCLS:0.0454 lossGSC:0.0145 imps:29.1 lr: 0.0595 etc:Thu Mar 10 14:04:56 2022 160 | step: 1460/ 3305 lossCLS:0.0470 lossGSC:0.0145 imps:29.4 lr: 0.0592 etc:Thu Mar 10 14:04:53 2022 161 | step: 1470/ 3305 lossCLS:0.0578 lossGSC:0.0146 imps:29.8 lr: 0.0589 etc:Thu Mar 10 14:04:50 2022 162 | step: 1480/ 3305 lossCLS:0.0603 lossGSC:0.0146 imps:29.9 lr: 0.0586 etc:Thu Mar 10 14:04:47 2022 163 | step: 1490/ 3305 lossCLS:0.0519 lossGSC:0.0148 imps:30.0 lr: 0.0583 etc:Thu Mar 10 14:04:45 2022 164 | step: 1500/ 3305 lossCLS:0.0582 lossGSC:0.0152 imps:30.1 lr: 0.0580 etc:Thu Mar 10 14:04:43 2022 165 | validating ... {'iou': array([0.79253155, 0.45073266, 0.28492017, 0.57591874, 0.43450175, 166 | 0.45996473, 0.73058337, 0.58066205, 0.60378022, 0.25047197, 167 | 0.65223107, 0.42218174, 0.67664039, 0.60482871, 0.65377192, 168 | 0.57647781, 0.42009622, 0.63612262, 0.44180773, 0.55818888, 169 | 0.40613419]), 'miou': 0.5339308794334802} 170 | step: 1510/ 3305 lossCLS:0.0669 lossGSC:0.0143 imps:19.2 lr: 0.0577 etc:Thu Mar 10 14:06:47 2022 171 | step: 1520/ 3305 lossCLS:0.0573 lossGSC:0.0135 imps:19.6 lr: 0.0574 etc:Thu Mar 10 14:06:44 2022 172 | step: 1530/ 3305 lossCLS:0.0481 lossGSC:0.0146 imps:19.9 lr: 0.0572 etc:Thu Mar 10 14:06:41 2022 173 | step: 1540/ 3305 lossCLS:0.0653 lossGSC:0.0141 imps:20.3 lr: 0.0569 etc:Thu Mar 10 14:06:38 2022 174 | step: 1550/ 3305 lossCLS:0.0732 lossGSC:0.0149 imps:20.6 lr: 0.0566 etc:Thu Mar 10 14:06:35 2022 175 | step: 1560/ 3305 lossCLS:0.0653 lossGSC:0.0139 imps:21.0 lr: 0.0563 etc:Thu Mar 10 14:06:33 2022 176 | step: 1570/ 3305 lossCLS:0.0606 lossGSC:0.0139 imps:21.2 lr: 0.0560 etc:Thu Mar 10 14:06:31 2022 177 | step: 1580/ 3305 lossCLS:0.0564 lossGSC:0.0134 imps:21.4 lr: 0.0557 etc:Thu Mar 10 14:06:29 2022 178 | step: 1590/ 3305 lossCLS:0.0443 lossGSC:0.0136 imps:21.7 lr: 0.0554 etc:Thu Mar 10 14:06:27 2022 179 | step: 1600/ 3305 lossCLS:0.0613 lossGSC:0.0148 imps:21.9 lr: 0.0551 etc:Thu Mar 10 14:06:25 2022 180 | step: 1610/ 3305 lossCLS:0.0590 lossGSC:0.0141 imps:22.1 lr: 0.0548 etc:Thu Mar 10 14:06:24 2022 181 | step: 1620/ 3305 lossCLS:0.0671 lossGSC:0.0135 imps:22.3 lr: 0.0545 etc:Thu Mar 10 14:06:22 2022 182 | step: 1630/ 3305 lossCLS:0.0499 lossGSC:0.0148 imps:22.4 lr: 0.0542 etc:Thu Mar 10 14:06:20 2022 183 | step: 1640/ 3305 lossCLS:0.0659 lossGSC:0.0145 imps:22.6 lr: 0.0540 etc:Thu Mar 10 14:06:17 2022 184 | step: 1650/ 3305 lossCLS:0.0564 lossGSC:0.0135 imps:22.9 lr: 0.0537 etc:Thu Mar 10 14:06:15 2022 185 | step: 1660/ 3305 lossCLS:0.0597 lossGSC:0.0146 imps:23.1 lr: 0.0534 etc:Thu Mar 10 14:06:12 2022 186 | step: 1670/ 3305 lossCLS:0.0551 lossGSC:0.0143 imps:23.2 lr: 0.0531 etc:Thu Mar 10 14:06:10 2022 187 | step: 1680/ 3305 lossCLS:0.0630 lossGSC:0.0150 imps:23.4 lr: 0.0528 etc:Thu Mar 10 14:06:08 2022 188 | step: 1690/ 3305 lossCLS:0.0592 lossGSC:0.0145 imps:23.6 lr: 0.0525 etc:Thu Mar 10 14:06:05 2022 189 | step: 1700/ 3305 lossCLS:0.0551 lossGSC:0.0144 imps:23.7 lr: 0.0522 etc:Thu Mar 10 14:06:04 2022 190 | step: 1710/ 3305 lossCLS:0.0502 lossGSC:0.0134 imps:23.8 lr: 0.0519 etc:Thu Mar 10 14:06:02 2022 191 | step: 1720/ 3305 lossCLS:0.0551 lossGSC:0.0143 imps:24.0 lr: 0.0516 etc:Thu Mar 10 14:06:00 2022 192 | step: 1730/ 3305 lossCLS:0.0633 lossGSC:0.0146 imps:24.2 lr: 0.0513 etc:Thu Mar 10 14:05:57 2022 193 | step: 1740/ 3305 lossCLS:0.0543 lossGSC:0.0144 imps:24.3 lr: 0.0510 etc:Thu Mar 10 14:05:54 2022 194 | step: 1750/ 3305 lossCLS:0.0667 lossGSC:0.0142 imps:24.5 lr: 0.0507 etc:Thu Mar 10 14:05:52 2022 195 | step: 1760/ 3305 lossCLS:0.0470 lossGSC:0.0144 imps:24.6 lr: 0.0504 etc:Thu Mar 10 14:05:50 2022 196 | step: 1770/ 3305 lossCLS:0.0528 lossGSC:0.0135 imps:24.7 lr: 0.0501 etc:Thu Mar 10 14:05:48 2022 197 | step: 1780/ 3305 lossCLS:0.0556 lossGSC:0.0141 imps:24.8 lr: 0.0499 etc:Thu Mar 10 14:05:45 2022 198 | step: 1790/ 3305 lossCLS:0.0540 lossGSC:0.0139 imps:25.0 lr: 0.0496 etc:Thu Mar 10 14:05:44 2022 199 | step: 1800/ 3305 lossCLS:0.0517 lossGSC:0.0149 imps:25.0 lr: 0.0493 etc:Thu Mar 10 14:05:43 2022 200 | step: 1810/ 3305 lossCLS:0.0431 lossGSC:0.0140 imps:25.1 lr: 0.0490 etc:Thu Mar 10 14:05:41 2022 201 | step: 1820/ 3305 lossCLS:0.0679 lossGSC:0.0141 imps:25.2 lr: 0.0487 etc:Thu Mar 10 14:05:39 2022 202 | step: 1830/ 3305 lossCLS:0.0548 lossGSC:0.0142 imps:25.3 lr: 0.0484 etc:Thu Mar 10 14:05:37 2022 203 | step: 1840/ 3305 lossCLS:0.0575 lossGSC:0.0147 imps:25.4 lr: 0.0481 etc:Thu Mar 10 14:05:35 2022 204 | step: 1850/ 3305 lossCLS:0.0475 lossGSC:0.0139 imps:25.5 lr: 0.0478 etc:Thu Mar 10 14:05:33 2022 205 | step: 1860/ 3305 lossCLS:0.0510 lossGSC:0.0140 imps:25.6 lr: 0.0475 etc:Thu Mar 10 14:05:32 2022 206 | step: 1870/ 3305 lossCLS:0.0596 lossGSC:0.0149 imps:25.7 lr: 0.0472 etc:Thu Mar 10 14:05:30 2022 207 | step: 1880/ 3305 lossCLS:0.0481 lossGSC:0.0150 imps:25.8 lr: 0.0469 etc:Thu Mar 10 14:05:28 2022 208 | step: 1890/ 3305 lossCLS:0.0464 lossGSC:0.0136 imps:25.8 lr: 0.0466 etc:Thu Mar 10 14:05:27 2022 209 | step: 1900/ 3305 lossCLS:0.0507 lossGSC:0.0130 imps:25.9 lr: 0.0463 etc:Thu Mar 10 14:05:26 2022 210 | step: 1910/ 3305 lossCLS:0.0433 lossGSC:0.0138 imps:26.0 lr: 0.0460 etc:Thu Mar 10 14:05:24 2022 211 | step: 1920/ 3305 lossCLS:0.0506 lossGSC:0.0137 imps:26.1 lr: 0.0457 etc:Thu Mar 10 14:05:22 2022 212 | step: 1930/ 3305 lossCLS:0.0568 lossGSC:0.0141 imps:26.1 lr: 0.0454 etc:Thu Mar 10 14:05:20 2022 213 | step: 1940/ 3305 lossCLS:0.0610 lossGSC:0.0140 imps:26.2 lr: 0.0451 etc:Thu Mar 10 14:05:18 2022 214 | step: 1950/ 3305 lossCLS:0.0609 lossGSC:0.0145 imps:26.3 lr: 0.0448 etc:Thu Mar 10 14:05:16 2022 215 | step: 1960/ 3305 lossCLS:0.0448 lossGSC:0.0136 imps:26.4 lr: 0.0445 etc:Thu Mar 10 14:05:14 2022 216 | step: 1970/ 3305 lossCLS:0.0532 lossGSC:0.0142 imps:26.4 lr: 0.0442 etc:Thu Mar 10 14:05:13 2022 217 | step: 1980/ 3305 lossCLS:0.0391 lossGSC:0.0137 imps:26.5 lr: 0.0439 etc:Thu Mar 10 14:05:11 2022 218 | Epoch 4/5 219 | step: 1990/ 3305 lossCLS:0.0530 lossGSC:0.0141 imps:15.7 lr: 0.0436 etc:Thu Mar 10 14:05:17 2022 220 | step: 2000/ 3305 lossCLS:0.0502 lossGSC:0.0136 imps:21.0 lr: 0.0433 etc:Thu Mar 10 14:05:16 2022 221 | validating ... {'iou': array([0.79070248, 0.44100464, 0.29220263, 0.55196396, 0.42143935, 222 | 0.50017267, 0.75075787, 0.59033168, 0.65851496, 0.29650318, 223 | 0.60376899, 0.45984486, 0.6578336 , 0.63267883, 0.65056327, 224 | 0.55940637, 0.41198722, 0.6536925 , 0.46891748, 0.51954574, 225 | 0.38766077]), 'miou': 0.5380710964365851} 226 | step: 2010/ 3305 lossCLS:0.0534 lossGSC:0.0139 imps:5.2 lr: 0.0430 etc:Thu Mar 10 14:07:06 2022 227 | step: 2020/ 3305 lossCLS:0.0506 lossGSC:0.0141 imps:6.6 lr: 0.0427 etc:Thu Mar 10 14:07:04 2022 228 | step: 2030/ 3305 lossCLS:0.0541 lossGSC:0.0135 imps:7.9 lr: 0.0424 etc:Thu Mar 10 14:07:01 2022 229 | step: 2040/ 3305 lossCLS:0.0440 lossGSC:0.0144 imps:9.1 lr: 0.0421 etc:Thu Mar 10 14:06:59 2022 230 | step: 2050/ 3305 lossCLS:0.0507 lossGSC:0.0148 imps:10.2 lr: 0.0418 etc:Thu Mar 10 14:06:57 2022 231 | step: 2060/ 3305 lossCLS:0.0419 lossGSC:0.0145 imps:11.1 lr: 0.0415 etc:Thu Mar 10 14:06:55 2022 232 | step: 2070/ 3305 lossCLS:0.0518 lossGSC:0.0142 imps:12.0 lr: 0.0412 etc:Thu Mar 10 14:06:53 2022 233 | step: 2080/ 3305 lossCLS:0.0540 lossGSC:0.0138 imps:12.8 lr: 0.0409 etc:Thu Mar 10 14:06:52 2022 234 | step: 2090/ 3305 lossCLS:0.0466 lossGSC:0.0130 imps:13.5 lr: 0.0406 etc:Thu Mar 10 14:06:50 2022 235 | step: 2100/ 3305 lossCLS:0.0534 lossGSC:0.0138 imps:14.2 lr: 0.0403 etc:Thu Mar 10 14:06:48 2022 236 | step: 2110/ 3305 lossCLS:0.0500 lossGSC:0.0142 imps:14.8 lr: 0.0400 etc:Thu Mar 10 14:06:47 2022 237 | step: 2120/ 3305 lossCLS:0.0468 lossGSC:0.0140 imps:15.3 lr: 0.0397 etc:Thu Mar 10 14:06:45 2022 238 | step: 2130/ 3305 lossCLS:0.0497 lossGSC:0.0141 imps:15.8 lr: 0.0394 etc:Thu Mar 10 14:06:44 2022 239 | step: 2140/ 3305 lossCLS:0.0502 lossGSC:0.0136 imps:16.4 lr: 0.0391 etc:Thu Mar 10 14:06:42 2022 240 | step: 2150/ 3305 lossCLS:0.0472 lossGSC:0.0144 imps:16.9 lr: 0.0388 etc:Thu Mar 10 14:06:40 2022 241 | step: 2160/ 3305 lossCLS:0.0430 lossGSC:0.0143 imps:17.3 lr: 0.0385 etc:Thu Mar 10 14:06:38 2022 242 | step: 2170/ 3305 lossCLS:0.0564 lossGSC:0.0136 imps:17.8 lr: 0.0382 etc:Thu Mar 10 14:06:36 2022 243 | step: 2180/ 3305 lossCLS:0.0493 lossGSC:0.0138 imps:18.2 lr: 0.0379 etc:Thu Mar 10 14:06:34 2022 244 | step: 2190/ 3305 lossCLS:0.0513 lossGSC:0.0142 imps:18.6 lr: 0.0376 etc:Thu Mar 10 14:06:32 2022 245 | step: 2200/ 3305 lossCLS:0.0522 lossGSC:0.0145 imps:18.9 lr: 0.0373 etc:Thu Mar 10 14:06:30 2022 246 | step: 2210/ 3305 lossCLS:0.0537 lossGSC:0.0145 imps:19.2 lr: 0.0370 etc:Thu Mar 10 14:06:28 2022 247 | step: 2220/ 3305 lossCLS:0.0467 lossGSC:0.0137 imps:19.5 lr: 0.0367 etc:Thu Mar 10 14:06:27 2022 248 | step: 2230/ 3305 lossCLS:0.0544 lossGSC:0.0141 imps:19.8 lr: 0.0364 etc:Thu Mar 10 14:06:26 2022 249 | step: 2240/ 3305 lossCLS:0.0547 lossGSC:0.0144 imps:20.1 lr: 0.0361 etc:Thu Mar 10 14:06:24 2022 250 | step: 2250/ 3305 lossCLS:0.0576 lossGSC:0.0137 imps:20.4 lr: 0.0358 etc:Thu Mar 10 14:06:22 2022 251 | step: 2260/ 3305 lossCLS:0.0479 lossGSC:0.0142 imps:20.7 lr: 0.0355 etc:Thu Mar 10 14:06:20 2022 252 | step: 2270/ 3305 lossCLS:0.0447 lossGSC:0.0134 imps:21.0 lr: 0.0352 etc:Thu Mar 10 14:06:18 2022 253 | step: 2280/ 3305 lossCLS:0.0473 lossGSC:0.0150 imps:21.2 lr: 0.0349 etc:Thu Mar 10 14:06:16 2022 254 | step: 2290/ 3305 lossCLS:0.0550 lossGSC:0.0141 imps:21.4 lr: 0.0346 etc:Thu Mar 10 14:06:14 2022 255 | step: 2300/ 3305 lossCLS:0.0500 lossGSC:0.0139 imps:21.6 lr: 0.0343 etc:Thu Mar 10 14:06:13 2022 256 | step: 2310/ 3305 lossCLS:0.0461 lossGSC:0.0137 imps:21.8 lr: 0.0339 etc:Thu Mar 10 14:06:12 2022 257 | step: 2320/ 3305 lossCLS:0.0513 lossGSC:0.0141 imps:22.0 lr: 0.0336 etc:Thu Mar 10 14:06:10 2022 258 | step: 2330/ 3305 lossCLS:0.0438 lossGSC:0.0140 imps:22.2 lr: 0.0333 etc:Thu Mar 10 14:06:09 2022 259 | step: 2340/ 3305 lossCLS:0.0619 lossGSC:0.0138 imps:22.3 lr: 0.0330 etc:Thu Mar 10 14:06:08 2022 260 | step: 2350/ 3305 lossCLS:0.0491 lossGSC:0.0133 imps:22.5 lr: 0.0327 etc:Thu Mar 10 14:06:07 2022 261 | step: 2360/ 3305 lossCLS:0.0495 lossGSC:0.0140 imps:22.6 lr: 0.0324 etc:Thu Mar 10 14:06:05 2022 262 | step: 2370/ 3305 lossCLS:0.0396 lossGSC:0.0134 imps:22.8 lr: 0.0321 etc:Thu Mar 10 14:06:03 2022 263 | step: 2380/ 3305 lossCLS:0.0393 lossGSC:0.0141 imps:23.0 lr: 0.0318 etc:Thu Mar 10 14:06:02 2022 264 | step: 2390/ 3305 lossCLS:0.0612 lossGSC:0.0136 imps:23.2 lr: 0.0315 etc:Thu Mar 10 14:06:00 2022 265 | step: 2400/ 3305 lossCLS:0.0575 lossGSC:0.0144 imps:23.4 lr: 0.0312 etc:Thu Mar 10 14:05:57 2022 266 | step: 2410/ 3305 lossCLS:0.0476 lossGSC:0.0137 imps:23.5 lr: 0.0309 etc:Thu Mar 10 14:05:55 2022 267 | step: 2420/ 3305 lossCLS:0.0491 lossGSC:0.0141 imps:23.7 lr: 0.0305 etc:Thu Mar 10 14:05:54 2022 268 | step: 2430/ 3305 lossCLS:0.0451 lossGSC:0.0144 imps:23.8 lr: 0.0302 etc:Thu Mar 10 14:05:52 2022 269 | step: 2440/ 3305 lossCLS:0.0424 lossGSC:0.0129 imps:23.9 lr: 0.0299 etc:Thu Mar 10 14:05:51 2022 270 | step: 2450/ 3305 lossCLS:0.0415 lossGSC:0.0145 imps:24.0 lr: 0.0296 etc:Thu Mar 10 14:05:50 2022 271 | step: 2460/ 3305 lossCLS:0.0442 lossGSC:0.0133 imps:24.1 lr: 0.0293 etc:Thu Mar 10 14:05:49 2022 272 | step: 2470/ 3305 lossCLS:0.0513 lossGSC:0.0136 imps:24.3 lr: 0.0290 etc:Thu Mar 10 14:05:47 2022 273 | step: 2480/ 3305 lossCLS:0.0511 lossGSC:0.0133 imps:24.4 lr: 0.0287 etc:Thu Mar 10 14:05:45 2022 274 | step: 2490/ 3305 lossCLS:0.0479 lossGSC:0.0136 imps:24.5 lr: 0.0284 etc:Thu Mar 10 14:05:44 2022 275 | step: 2500/ 3305 lossCLS:0.0417 lossGSC:0.0134 imps:24.6 lr: 0.0281 etc:Thu Mar 10 14:05:42 2022 276 | validating ... {'iou': array([0.79404107, 0.46194577, 0.30574783, 0.57871469, 0.4041041 , 277 | 0.4868531 , 0.76179645, 0.58236305, 0.67426671, 0.26738603, 278 | 0.64273618, 0.47403705, 0.66558188, 0.61497181, 0.65747454, 279 | 0.56521033, 0.4200116 , 0.65073858, 0.43636006, 0.55787204, 280 | 0.412032 ]), 'miou': 0.5435354691482936} 281 | step: 2510/ 3305 lossCLS:0.0518 lossGSC:0.0139 imps:21.3 lr: 0.0277 etc:Thu Mar 10 14:06:52 2022 282 | step: 2520/ 3305 lossCLS:0.0421 lossGSC:0.0136 imps:21.5 lr: 0.0274 etc:Thu Mar 10 14:06:51 2022 283 | step: 2530/ 3305 lossCLS:0.0418 lossGSC:0.0128 imps:21.6 lr: 0.0271 etc:Thu Mar 10 14:06:49 2022 284 | step: 2540/ 3305 lossCLS:0.0473 lossGSC:0.0138 imps:21.7 lr: 0.0268 etc:Thu Mar 10 14:06:48 2022 285 | step: 2550/ 3305 lossCLS:0.0396 lossGSC:0.0140 imps:21.8 lr: 0.0265 etc:Thu Mar 10 14:06:47 2022 286 | step: 2560/ 3305 lossCLS:0.0484 lossGSC:0.0137 imps:21.9 lr: 0.0262 etc:Thu Mar 10 14:06:46 2022 287 | step: 2570/ 3305 lossCLS:0.0450 lossGSC:0.0139 imps:22.0 lr: 0.0258 etc:Thu Mar 10 14:06:45 2022 288 | step: 2580/ 3305 lossCLS:0.0405 lossGSC:0.0128 imps:22.1 lr: 0.0255 etc:Thu Mar 10 14:06:43 2022 289 | step: 2590/ 3305 lossCLS:0.0550 lossGSC:0.0138 imps:22.2 lr: 0.0252 etc:Thu Mar 10 14:06:41 2022 290 | step: 2600/ 3305 lossCLS:0.0432 lossGSC:0.0136 imps:22.3 lr: 0.0249 etc:Thu Mar 10 14:06:40 2022 291 | step: 2610/ 3305 lossCLS:0.0379 lossGSC:0.0134 imps:22.4 lr: 0.0246 etc:Thu Mar 10 14:06:38 2022 292 | step: 2620/ 3305 lossCLS:0.0552 lossGSC:0.0139 imps:22.5 lr: 0.0243 etc:Thu Mar 10 14:06:36 2022 293 | step: 2630/ 3305 lossCLS:0.0578 lossGSC:0.0130 imps:22.6 lr: 0.0239 etc:Thu Mar 10 14:06:35 2022 294 | step: 2640/ 3305 lossCLS:0.0464 lossGSC:0.0137 imps:22.7 lr: 0.0236 etc:Thu Mar 10 14:06:33 2022 295 | Epoch 5/5 296 | step: 2650/ 3305 lossCLS:0.0513 lossGSC:0.0140 imps:17.5 lr: 0.0233 etc:Thu Mar 10 14:06:35 2022 297 | step: 2660/ 3305 lossCLS:0.0431 lossGSC:0.0147 imps:23.2 lr: 0.0230 etc:Thu Mar 10 14:06:34 2022 298 | step: 2670/ 3305 lossCLS:0.0467 lossGSC:0.0142 imps:26.3 lr: 0.0227 etc:Thu Mar 10 14:06:32 2022 299 | step: 2680/ 3305 lossCLS:0.0449 lossGSC:0.0145 imps:27.0 lr: 0.0223 etc:Thu Mar 10 14:06:31 2022 300 | step: 2690/ 3305 lossCLS:0.0398 lossGSC:0.0137 imps:27.5 lr: 0.0220 etc:Thu Mar 10 14:06:30 2022 301 | step: 2700/ 3305 lossCLS:0.0420 lossGSC:0.0134 imps:28.0 lr: 0.0217 etc:Thu Mar 10 14:06:29 2022 302 | step: 2710/ 3305 lossCLS:0.0378 lossGSC:0.0143 imps:28.3 lr: 0.0214 etc:Thu Mar 10 14:06:28 2022 303 | step: 2720/ 3305 lossCLS:0.0402 lossGSC:0.0134 imps:28.6 lr: 0.0210 etc:Thu Mar 10 14:06:26 2022 304 | step: 2730/ 3305 lossCLS:0.0498 lossGSC:0.0142 imps:28.7 lr: 0.0207 etc:Thu Mar 10 14:06:25 2022 305 | step: 2740/ 3305 lossCLS:0.0333 lossGSC:0.0133 imps:28.6 lr: 0.0204 etc:Thu Mar 10 14:06:24 2022 306 | step: 2750/ 3305 lossCLS:0.0365 lossGSC:0.0133 imps:28.8 lr: 0.0201 etc:Thu Mar 10 14:06:23 2022 307 | step: 2760/ 3305 lossCLS:0.0362 lossGSC:0.0137 imps:29.0 lr: 0.0197 etc:Thu Mar 10 14:06:22 2022 308 | step: 2770/ 3305 lossCLS:0.0322 lossGSC:0.0133 imps:29.2 lr: 0.0194 etc:Thu Mar 10 14:06:20 2022 309 | step: 2780/ 3305 lossCLS:0.0432 lossGSC:0.0141 imps:29.5 lr: 0.0191 etc:Thu Mar 10 14:06:19 2022 310 | step: 2790/ 3305 lossCLS:0.0396 lossGSC:0.0130 imps:29.7 lr: 0.0188 etc:Thu Mar 10 14:06:17 2022 311 | step: 2800/ 3305 lossCLS:0.0446 lossGSC:0.0136 imps:29.8 lr: 0.0184 etc:Thu Mar 10 14:06:16 2022 312 | step: 2810/ 3305 lossCLS:0.0409 lossGSC:0.0132 imps:29.9 lr: 0.0181 etc:Thu Mar 10 14:06:14 2022 313 | step: 2820/ 3305 lossCLS:0.0488 lossGSC:0.0142 imps:30.0 lr: 0.0178 etc:Thu Mar 10 14:06:13 2022 314 | step: 2830/ 3305 lossCLS:0.0366 lossGSC:0.0142 imps:30.0 lr: 0.0174 etc:Thu Mar 10 14:06:12 2022 315 | step: 2840/ 3305 lossCLS:0.0346 lossGSC:0.0139 imps:29.9 lr: 0.0171 etc:Thu Mar 10 14:06:11 2022 316 | step: 2850/ 3305 lossCLS:0.0403 lossGSC:0.0139 imps:29.8 lr: 0.0168 etc:Thu Mar 10 14:06:10 2022 317 | step: 2860/ 3305 lossCLS:0.0376 lossGSC:0.0137 imps:29.8 lr: 0.0165 etc:Thu Mar 10 14:06:10 2022 318 | step: 2870/ 3305 lossCLS:0.0468 lossGSC:0.0138 imps:29.7 lr: 0.0161 etc:Thu Mar 10 14:06:09 2022 319 | step: 2880/ 3305 lossCLS:0.0470 lossGSC:0.0136 imps:29.8 lr: 0.0158 etc:Thu Mar 10 14:06:08 2022 320 | step: 2890/ 3305 lossCLS:0.0300 lossGSC:0.0126 imps:29.9 lr: 0.0155 etc:Thu Mar 10 14:06:06 2022 321 | step: 2900/ 3305 lossCLS:0.0388 lossGSC:0.0138 imps:30.0 lr: 0.0151 etc:Thu Mar 10 14:06:05 2022 322 | step: 2910/ 3305 lossCLS:0.0487 lossGSC:0.0136 imps:30.1 lr: 0.0148 etc:Thu Mar 10 14:06:03 2022 323 | step: 2920/ 3305 lossCLS:0.0510 lossGSC:0.0136 imps:30.1 lr: 0.0144 etc:Thu Mar 10 14:06:02 2022 324 | step: 2930/ 3305 lossCLS:0.0344 lossGSC:0.0137 imps:30.2 lr: 0.0141 etc:Thu Mar 10 14:06:01 2022 325 | step: 2940/ 3305 lossCLS:0.0303 lossGSC:0.0139 imps:30.3 lr: 0.0138 etc:Thu Mar 10 14:05:59 2022 326 | step: 2950/ 3305 lossCLS:0.0473 lossGSC:0.0145 imps:30.3 lr: 0.0134 etc:Thu Mar 10 14:05:58 2022 327 | step: 2960/ 3305 lossCLS:0.0343 lossGSC:0.0130 imps:30.2 lr: 0.0131 etc:Thu Mar 10 14:05:57 2022 328 | step: 2970/ 3305 lossCLS:0.0383 lossGSC:0.0152 imps:30.2 lr: 0.0127 etc:Thu Mar 10 14:05:57 2022 329 | step: 2980/ 3305 lossCLS:0.0463 lossGSC:0.0131 imps:30.1 lr: 0.0124 etc:Thu Mar 10 14:05:56 2022 330 | step: 2990/ 3305 lossCLS:0.0436 lossGSC:0.0140 imps:30.1 lr: 0.0121 etc:Thu Mar 10 14:05:56 2022 331 | step: 3000/ 3305 lossCLS:0.0381 lossGSC:0.0138 imps:30.1 lr: 0.0117 etc:Thu Mar 10 14:05:55 2022 332 | validating ... {'iou': array([0.78743936, 0.43975395, 0.29363185, 0.55884885, 0.39846377, 333 | 0.48210578, 0.75236995, 0.57089667, 0.69602646, 0.27263182, 334 | 0.63336183, 0.47537731, 0.66632458, 0.61194775, 0.64812002, 335 | 0.550885 , 0.40576804, 0.63791128, 0.44357606, 0.53309847, 336 | 0.4259368 ]), 'miou': 0.537355980206527} 337 | step: 3010/ 3305 lossCLS:0.0463 lossGSC:0.0138 imps:23.6 lr: 0.0114 etc:Thu Mar 10 14:06:52 2022 338 | step: 3020/ 3305 lossCLS:0.0407 lossGSC:0.0137 imps:23.7 lr: 0.0110 etc:Thu Mar 10 14:06:51 2022 339 | step: 3030/ 3305 lossCLS:0.0382 lossGSC:0.0150 imps:23.8 lr: 0.0107 etc:Thu Mar 10 14:06:50 2022 340 | step: 3040/ 3305 lossCLS:0.0386 lossGSC:0.0139 imps:24.0 lr: 0.0103 etc:Thu Mar 10 14:06:49 2022 341 | step: 3050/ 3305 lossCLS:0.0387 lossGSC:0.0140 imps:24.1 lr: 0.0100 etc:Thu Mar 10 14:06:48 2022 342 | step: 3060/ 3305 lossCLS:0.0406 lossGSC:0.0139 imps:24.3 lr: 0.0096 etc:Thu Mar 10 14:06:46 2022 343 | step: 3070/ 3305 lossCLS:0.0330 lossGSC:0.0138 imps:24.4 lr: 0.0093 etc:Thu Mar 10 14:06:45 2022 344 | step: 3080/ 3305 lossCLS:0.0410 lossGSC:0.0141 imps:24.5 lr: 0.0089 etc:Thu Mar 10 14:06:43 2022 345 | step: 3090/ 3305 lossCLS:0.0413 lossGSC:0.0136 imps:24.7 lr: 0.0085 etc:Thu Mar 10 14:06:42 2022 346 | step: 3100/ 3305 lossCLS:0.0495 lossGSC:0.0135 imps:25.0 lr: 0.0082 etc:Thu Mar 10 14:06:38 2022 347 | step: 3110/ 3305 lossCLS:0.0501 lossGSC:0.0135 imps:25.3 lr: 0.0078 etc:Thu Mar 10 14:06:35 2022 348 | step: 3120/ 3305 lossCLS:0.0349 lossGSC:0.0142 imps:25.4 lr: 0.0075 etc:Thu Mar 10 14:06:33 2022 349 | step: 3130/ 3305 lossCLS:0.0374 lossGSC:0.0142 imps:25.5 lr: 0.0071 etc:Thu Mar 10 14:06:32 2022 350 | step: 3140/ 3305 lossCLS:0.0411 lossGSC:0.0139 imps:25.6 lr: 0.0067 etc:Thu Mar 10 14:06:31 2022 351 | step: 3150/ 3305 lossCLS:0.0475 lossGSC:0.0138 imps:25.6 lr: 0.0064 etc:Thu Mar 10 14:06:30 2022 352 | step: 3160/ 3305 lossCLS:0.0379 lossGSC:0.0141 imps:25.7 lr: 0.0060 etc:Thu Mar 10 14:06:29 2022 353 | step: 3170/ 3305 lossCLS:0.0326 lossGSC:0.0142 imps:25.8 lr: 0.0056 etc:Thu Mar 10 14:06:28 2022 354 | step: 3180/ 3305 lossCLS:0.0323 lossGSC:0.0131 imps:25.8 lr: 0.0052 etc:Thu Mar 10 14:06:27 2022 355 | step: 3190/ 3305 lossCLS:0.0356 lossGSC:0.0135 imps:25.9 lr: 0.0049 etc:Thu Mar 10 14:06:26 2022 356 | step: 3200/ 3305 lossCLS:0.0450 lossGSC:0.0133 imps:25.9 lr: 0.0045 etc:Thu Mar 10 14:06:26 2022 357 | step: 3210/ 3305 lossCLS:0.0438 lossGSC:0.0143 imps:26.0 lr: 0.0041 etc:Thu Mar 10 14:06:25 2022 358 | step: 3220/ 3305 lossCLS:0.0343 lossGSC:0.0138 imps:26.1 lr: 0.0037 etc:Thu Mar 10 14:06:24 2022 359 | step: 3230/ 3305 lossCLS:0.0419 lossGSC:0.0137 imps:26.1 lr: 0.0033 etc:Thu Mar 10 14:06:23 2022 360 | step: 3240/ 3305 lossCLS:0.0353 lossGSC:0.0131 imps:26.2 lr: 0.0029 etc:Thu Mar 10 14:06:22 2022 361 | step: 3250/ 3305 lossCLS:0.0356 lossGSC:0.0140 imps:26.2 lr: 0.0025 etc:Thu Mar 10 14:06:22 2022 362 | step: 3260/ 3305 lossCLS:0.0342 lossGSC:0.0138 imps:26.2 lr: 0.0021 etc:Thu Mar 10 14:06:21 2022 363 | step: 3270/ 3305 lossCLS:0.0360 lossGSC:0.0139 imps:26.3 lr: 0.0017 etc:Thu Mar 10 14:06:20 2022 364 | step: 3280/ 3305 lossCLS:0.0335 lossGSC:0.0139 imps:26.3 lr: 0.0012 etc:Thu Mar 10 14:06:19 2022 365 | step: 3290/ 3305 lossCLS:0.0359 lossGSC:0.0139 imps:26.3 lr: 0.0008 etc:Thu Mar 10 14:06:19 2022 366 | step: 3300/ 3305 lossCLS:0.0444 lossGSC:0.0138 imps:26.4 lr: 0.0003 etc:Thu Mar 10 14:06:17 2022 367 | -------------------------------------------------------------------------------- /make_cam.py: -------------------------------------------------------------------------------- 1 | from builtins import bool 2 | import torch 3 | from torch import multiprocessing, cuda 4 | from torch.utils.data import DataLoader 5 | import torch.nn.functional as F 6 | from torch.backends import cudnn 7 | from PIL import Image 8 | import matplotlib.pyplot as plt 9 | import numpy as np 10 | import importlib 11 | import os 12 | import imageio 13 | import argparse 14 | from data import data_voc, data_coco 15 | from tool import torchutils, pyutils 16 | 17 | cudnn.enabled = True 18 | 19 | def overlap(img, hm): 20 | hm = plt.cm.jet(hm)[:, :, :3] 21 | hm = np.array(Image.fromarray((hm*255).astype(np.uint8), 'RGB').resize((img.shape[1], img.shape[0]), Image.BICUBIC)).astype(np.float)*2 22 | if hm.shape == np.array(img).astype(np.float).shape: 23 | out = (hm + np.array(img).astype(np.float)) / 3 24 | out = (out / np.max(out) * 255).astype(np.uint8) 25 | else: 26 | print(hm.shape) 27 | print(np.array(img).shape) 28 | return out 29 | 30 | def draw_heatmap(norm_cam, gt_label, orig_img, save_path, img_name): 31 | gt_cat = np.where(gt_label==1)[0] 32 | for _, gt in enumerate(gt_cat): 33 | heatmap = overlap(orig_img, norm_cam[gt]) 34 | cam_viz_path = os.path.join(save_path, img_name + '_{}.png'.format(gt)) 35 | imageio.imsave(cam_viz_path, heatmap) 36 | 37 | 38 | def _work(process_id, model, dataset, args): 39 | databin = dataset[process_id] 40 | n_gpus = torch.cuda.device_count() 41 | data_loader = DataLoader(databin, shuffle=False, num_workers=args.num_workers // n_gpus, pin_memory=False) 42 | 43 | with torch.no_grad(), cuda.device(process_id): 44 | 45 | model.cuda() 46 | 47 | for iter, pack in enumerate(data_loader): 48 | img_name = pack['name'][0] 49 | label = pack['label'][0] 50 | size = pack['size'] 51 | label = F.pad(label, (1, 0), 'constant', 1.0) 52 | 53 | outputs = [model(img[0].cuda(non_blocking=True), label.cuda(non_blocking=True).unsqueeze(-1).unsqueeze(-1)) for img in pack['img']] 54 | 55 | # multi-scale fusion 56 | IS_CAM_list = [output[1].cpu() for output in outputs] 57 | IS_CAM_list = [F.interpolate(torch.unsqueeze(o, 1), size, mode='bilinear', align_corners=False) for o in IS_CAM_list] 58 | IS_CAM = torch.sum(torch.stack(IS_CAM_list, 0), 0)[:,0] 59 | IS_CAM /= F.adaptive_max_pool2d(IS_CAM, (1, 1)) + 1e-5 60 | IS_CAM = IS_CAM.cpu().numpy() 61 | 62 | # visualize IS-CAM 63 | if args.visualize: 64 | orig_img = np.array(Image.open(pack['img_path'][0]).convert('RGB')) 65 | draw_heatmap(IS_CAM.copy(), label, orig_img, os.path.join(args.session_name, 'visual'), img_name) 66 | 67 | # save IS_CAM 68 | valid_cat = torch.nonzero(label)[:, 0].cpu().numpy() 69 | IS_CAM = IS_CAM[valid_cat] 70 | np.save(os.path.join(args.session_name, 'npy', img_name + '.npy'), {"keys": valid_cat, "IS_CAM": IS_CAM}) 71 | 72 | if process_id == n_gpus - 1 and iter % (len(databin) // 20) == 0: 73 | print("%d " % ((5*iter+1)//(len(databin) // 20)), end='') 74 | 75 | 76 | if __name__ == '__main__': 77 | 78 | parser = argparse.ArgumentParser() 79 | parser.add_argument("--network", default="network.resnet50_SIPE", type=str) 80 | parser.add_argument("--num_workers", default=os.cpu_count()//2, type=int) 81 | parser.add_argument("--session_name", default="exp", type=str) 82 | parser.add_argument("--ckpt", default="final.pth", type=str) 83 | parser.add_argument("--visualize", default=True, type=bool) 84 | parser.add_argument("--dataset", default="voc", type=str) 85 | 86 | args = parser.parse_args() 87 | 88 | os.makedirs(os.path.join(args.session_name, 'npy'), exist_ok=True) 89 | os.makedirs(os.path.join(args.session_name, 'visual'), exist_ok=True) 90 | pyutils.Logger(os.path.join(args.session_name, 'infer.log')) 91 | print(vars(args)) 92 | 93 | assert args.dataset in ['voc', 'coco'], 'Dataset must be voc or coco in this project.' 94 | 95 | if args.dataset == 'voc': 96 | dataset_root = '../PascalVOC2012/VOCdevkit/VOC2012' 97 | model = getattr(importlib.import_module(args.network), 'CAM')(num_cls=21) 98 | dataset = data_voc.VOC12ClsDatasetMSF('data/train_' + args.dataset + '.txt', voc12_root=dataset_root, scales=(1.0, 0.5, 1.5, 2.0)) 99 | 100 | elif args.dataset == 'coco': 101 | dataset_root = "../ms_coco_14&15/images" 102 | model = getattr(importlib.import_module(args.network), 'CAM')(num_cls=81) 103 | dataset = data_coco.COCOClsDatasetMSF('data/train_' + args.dataset + '.txt', coco_root=dataset_root, scales=(1.0, 0.5, 1.5, 2.0)) 104 | 105 | checkpoint = torch.load(args.session_name + '/ckpt/' + args.ckpt) 106 | model.load_state_dict(checkpoint['net'], strict=True) 107 | model.eval() 108 | 109 | n_gpus = torch.cuda.device_count() 110 | 111 | dataset = torchutils.split_dataset(dataset, n_gpus) 112 | 113 | print('[ ', end='') 114 | multiprocessing.spawn(_work, nprocs=n_gpus, args=(model, dataset, args), join=True) 115 | print(']') 116 | 117 | torch.cuda.empty_cache() -------------------------------------------------------------------------------- /make_seg_labels.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from torch import multiprocessing, cuda 3 | from torch.utils.data import DataLoader 4 | import torch.nn.functional as F 5 | from torch.backends import cudnn 6 | 7 | import argparse 8 | import numpy as np 9 | import importlib 10 | import os 11 | import imageio 12 | from PIL import Image 13 | 14 | from data import data_voc 15 | from tool import torchutils, indexing 16 | 17 | cudnn.enabled = True 18 | 19 | palette = [0,0,0, 128,0,0, 0,128,0, 128,128,0, 0,0,128, 128,0,128, 0,128,128, 128,128,128, 20 | 64,0,0, 192,0,0, 64,128,0, 192,128,0, 64,0,128, 192,0,128, 64,128,128, 192,128,128, 21 | 0,64,0, 128,64,0, 0,192,0, 128,192,0, 0,64,128, 128,64,128, 0,192,128, 128,192,128, 22 | 64,64,0, 192,64,0, 64,192,0, 192,192,0] 23 | 24 | def _work(process_id, model, dataset, args): 25 | 26 | n_gpus = torch.cuda.device_count() 27 | databin = dataset[process_id] 28 | data_loader = DataLoader(databin, 29 | shuffle=False, num_workers=args.num_workers // n_gpus, pin_memory=False) 30 | 31 | cam_out_dir = os.path.join(args.session_name, 'npy') 32 | 33 | with torch.no_grad(), cuda.device(process_id): 34 | 35 | model.cuda() 36 | 37 | for iter, pack in enumerate(data_loader): 38 | img_name = pack['name'][0] 39 | orig_img_size = np.array(pack['size']) 40 | 41 | edge, _ = model(pack['img'][0].cuda(non_blocking=True)) 42 | 43 | cam_dict = np.load(cam_out_dir + '/' + img_name + '.npy', allow_pickle=True).item() 44 | cam_dict['IS_CAM'] = cam_dict['IS_CAM'] 45 | keys = cam_dict['keys'] 46 | cams = F.interpolate(torch.tensor(cam_dict['IS_CAM']).unsqueeze(1), edge.shape[1:], mode='bilinear', align_corners=False)[:,0] 47 | cams = np.power(cams, 1.5) 48 | 49 | cam_downsized_values = cams.cuda() 50 | rw = indexing.propagate_to_edge(cam_downsized_values, edge, beta=args.beta, exp_times=args.exp_times, radius=5) 51 | rw_up = F.interpolate(rw, scale_factor=4, mode='bilinear', align_corners=False)[..., 0, :orig_img_size[0], :orig_img_size[1]] 52 | rw_up = rw_up / torch.max(rw_up) 53 | rw_pred = torch.argmax(rw_up, dim=0).cpu().numpy() 54 | rw_pred = keys[rw_pred] 55 | 56 | imageio.imsave(os.path.join(args.sem_seg_out_dir, img_name + '.png'), rw_pred.astype(np.uint8)) 57 | 58 | # show color map 59 | # out = Image.fromarray(rw_pred.astype(np.uint8), mode='P') 60 | # out.putpalette(palette) 61 | # out.save(os.path.join(os.path.join(args.sem_seg_out_dir, img_name + '_palette.png'))) 62 | 63 | if process_id == n_gpus - 1 and iter % (len(databin) // 20) == 0: 64 | print("%d " % ((5*iter+1)//(len(databin) // 20)), end='') 65 | 66 | 67 | if __name__ == '__main__': 68 | 69 | parser = argparse.ArgumentParser() 70 | # Inter-pixel Relation Network (IRNet) 71 | parser.add_argument("--num_workers", default=os.cpu_count()//2, type=int) 72 | parser.add_argument("--infer_list", default="data/trainaug_voc.txt", type=str) 73 | parser.add_argument("--voc12_root", default="../PascalVOC2012/VOCdevkit/VOC2012", type=str) 74 | parser.add_argument("--irn_network", default="network.resnet50_irn", type=str) 75 | parser.add_argument("--session_name", default="exp", type=str) 76 | # Random Walk Params 77 | parser.add_argument("--beta", default=10) 78 | parser.add_argument("--exp_times", default=8, 79 | help="Hyper-parameter that controls the number of random walk iterations," 80 | "The random walk is performed 2^{exp_times}.") 81 | parser.add_argument("--sem_seg_out_dir", default="", type=str) 82 | args = parser.parse_args() 83 | 84 | 85 | model = getattr(importlib.import_module(args.irn_network), 'EdgeDisplacement')() 86 | irn_weights_path = os.path.join(args.session_name, 'ckpt', 'irn.pth') 87 | model.load_state_dict(torch.load(irn_weights_path), strict=False) 88 | model.eval() 89 | 90 | n_gpus = torch.cuda.device_count() 91 | 92 | dataset = data_voc.VOC12ClsDatasetMSF(args.infer_list, voc12_root=args.voc12_root, scales=(1.0,)) 93 | dataset = torchutils.split_dataset(dataset, n_gpus) 94 | 95 | args.sem_seg_out_dir = os.path.join(args.session_name, 'pseudo_label') 96 | os.makedirs(args.sem_seg_out_dir, exist_ok=True) 97 | 98 | print("[", end='') 99 | multiprocessing.spawn(_work, nprocs=n_gpus, args=(model, dataset, args), join=True) 100 | print("]") 101 | 102 | torch.cuda.empty_cache() 103 | -------------------------------------------------------------------------------- /network/resnet50.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import torch.nn.functional as F 3 | import torch.utils.model_zoo as model_zoo 4 | 5 | model_urls = { 6 | 'resnet50': 'https://download.pytorch.org/models/resnet50-19c8e357.pth', 7 | 'resnet101': 'https://download.pytorch.org/models/resnet101-63fe2227.pth', 8 | } 9 | 10 | class FixedBatchNorm(nn.BatchNorm2d): 11 | def forward(self, input): 12 | return F.batch_norm(input, self.running_mean, self.running_var, self.weight, self.bias, 13 | training=False, eps=self.eps) 14 | 15 | 16 | class Bottleneck(nn.Module): 17 | expansion = 4 18 | 19 | def __init__(self, inplanes, planes, stride=1, downsample=None, dilation=1): 20 | super(Bottleneck, self).__init__() 21 | self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=1, bias=False) 22 | self.bn1 = FixedBatchNorm(planes) 23 | self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride, 24 | padding=dilation, bias=False, dilation=dilation) 25 | self.bn2 = FixedBatchNorm(planes) 26 | self.conv3 = nn.Conv2d(planes, planes * 4, kernel_size=1, bias=False) 27 | self.bn3 = FixedBatchNorm(planes * 4) 28 | self.relu = nn.ReLU(inplace=True) 29 | self.downsample = downsample 30 | self.stride = stride 31 | self.dilation = dilation 32 | 33 | def forward(self, x): 34 | residual = x 35 | 36 | out = self.conv1(x) 37 | out = self.bn1(out) 38 | out = self.relu(out) 39 | 40 | out = self.conv2(out) 41 | out = self.bn2(out) 42 | out = self.relu(out) 43 | 44 | out = self.conv3(out) 45 | out = self.bn3(out) 46 | 47 | if self.downsample is not None: 48 | residual = self.downsample(x) 49 | 50 | out += residual 51 | out = self.relu(out) 52 | 53 | return out 54 | 55 | 56 | class ResNet(nn.Module): 57 | 58 | def __init__(self, block, layers, strides=(2, 2, 2, 2), dilations=(1, 1, 1, 1)): 59 | self.inplanes = 64 60 | super(ResNet, self).__init__() 61 | self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=strides[0], padding=3, 62 | bias=False) 63 | self.bn1 = FixedBatchNorm(64) 64 | self.relu = nn.ReLU(inplace=True) 65 | self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1) 66 | self.layer1 = self._make_layer(block, 64, layers[0], stride=1, dilation=dilations[0]) 67 | self.layer2 = self._make_layer(block, 128, layers[1], stride=strides[1], dilation=dilations[1]) 68 | self.layer3 = self._make_layer(block, 256, layers[2], stride=strides[2], dilation=dilations[2]) 69 | self.layer4 = self._make_layer(block, 512, layers[3], stride=strides[3], dilation=dilations[3]) 70 | self.inplanes = 1024 71 | 72 | #self.avgpool = nn.AvgPool2d(7, stride=1) 73 | #self.fc = nn.Linear(512 * block.expansion, 1000) 74 | 75 | 76 | def _make_layer(self, block, planes, blocks, stride=1, dilation=1): 77 | downsample = None 78 | if stride != 1 or self.inplanes != planes * block.expansion: 79 | downsample = nn.Sequential( 80 | nn.Conv2d(self.inplanes, planes * block.expansion, 81 | kernel_size=1, stride=stride, bias=False), 82 | FixedBatchNorm(planes * block.expansion), 83 | ) 84 | 85 | layers = [block(self.inplanes, planes, stride, downsample, dilation=1)] 86 | self.inplanes = planes * block.expansion 87 | for i in range(1, blocks): 88 | layers.append(block(self.inplanes, planes, dilation=dilation)) 89 | 90 | return nn.Sequential(*layers) 91 | 92 | def forward(self, x): 93 | x = self.conv1(x) 94 | x = self.bn1(x) 95 | x = self.relu(x) 96 | x = self.maxpool(x) 97 | 98 | x = self.layer1(x) 99 | x = self.layer2(x) 100 | x = self.layer3(x) 101 | x = self.layer4(x) 102 | 103 | x = self.avgpool(x) 104 | x = x.view(x.size(0), -1) 105 | x = self.fc(x) 106 | 107 | return x 108 | 109 | 110 | def resnet50(pretrained=True, **kwargs): 111 | 112 | model = ResNet(Bottleneck, [3, 4, 6, 3], **kwargs) 113 | if pretrained: 114 | state_dict = model_zoo.load_url(model_urls['resnet50']) 115 | state_dict.pop('fc.weight') 116 | state_dict.pop('fc.bias') 117 | model.load_state_dict(state_dict) 118 | return model 119 | 120 | def resnet101(pretrained=True, **kwargs): 121 | 122 | model = ResNet(Bottleneck, [3, 4, 23, 3], **kwargs) 123 | if pretrained: 124 | state_dict = model_zoo.load_url(model_urls['resnet101']) 125 | state_dict.pop('fc.weight') 126 | state_dict.pop('fc.bias') 127 | model.load_state_dict(state_dict) 128 | return model -------------------------------------------------------------------------------- /network/resnet50_SIPE.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import torch.nn.functional as F 4 | from tool import torchutils 5 | from network import resnet50 6 | 7 | 8 | class Net(nn.Module): 9 | 10 | def __init__(self, num_cls=21): 11 | super(Net, self).__init__() 12 | 13 | self.num_cls = num_cls 14 | 15 | self.resnet50 = resnet50.resnet50(pretrained=True, strides=(2, 2, 2, 1), dilations=(1, 1, 1, 1)) 16 | 17 | self.stage0 = nn.Sequential(self.resnet50.conv1, self.resnet50.bn1, self.resnet50.relu, self.resnet50.maxpool) 18 | self.stage1 = nn.Sequential(self.resnet50.layer1) 19 | self.stage2 = nn.Sequential(self.resnet50.layer2) 20 | self.stage3 = nn.Sequential(self.resnet50.layer3) 21 | self.stage4 = nn.Sequential(self.resnet50.layer4) 22 | 23 | self.side1 = nn.Conv2d(256, 128, 1, bias=False) 24 | self.side2 = nn.Conv2d(512, 128, 1, bias=False) 25 | self.side3 = nn.Conv2d(1024, 256, 1, bias=False) 26 | self.side4 = nn.Conv2d(2048, 256, 1, bias=False) 27 | self.classifier = nn.Conv2d(2048, self.num_cls-1, 1, bias=False) 28 | 29 | self.backbone = nn.ModuleList([self.stage0, self.stage1, self.stage2, self.stage3, self.stage4]) 30 | self.newly_added = nn.ModuleList([self.classifier, self.side1, self.side2, self.side3, self.side4]) 31 | 32 | def get_seed(self, norm_cam, label, feature): 33 | 34 | n,c,h,w = norm_cam.shape 35 | 36 | # iou evalution 37 | seeds = torch.zeros((n,h,w,c)).cuda() 38 | feature_s = feature.view(n,-1,h*w) 39 | feature_s = feature_s/(torch.norm(feature_s,dim=1,keepdim=True)+1e-5) 40 | correlation = F.relu(torch.matmul(feature_s.transpose(2,1), feature_s),inplace=True).unsqueeze(1) #[n,1,h*w,h*w] 41 | # correlation = correlation/torch.max(correlation, dim=-1)[0].unsqueeze(-1) #[n,1,h*w,h*w] 42 | cam_flatten = norm_cam.view(n,-1,h*w).unsqueeze(2) #[n,21,1,h*w] 43 | inter = (correlation * cam_flatten).sum(-1) 44 | union = correlation.sum(-1) + cam_flatten.sum(-1) - inter 45 | miou = (inter/union).view(n,self.num_cls,h,w) #[n,21,h,w] 46 | miou[:,0] = miou[:,0]*0.5 47 | probs = F.softmax(miou, dim=1) 48 | belonging = miou.argmax(1) 49 | seeds = seeds.scatter_(-1, belonging.view(n,h,w,1), 1).permute(0,3,1,2).contiguous() 50 | 51 | seeds = seeds * label 52 | return seeds, probs 53 | 54 | def get_prototype(self, seeds, feature): 55 | n,c,h,w = feature.shape 56 | seeds = F.interpolate(seeds, feature.shape[2:], mode='nearest') 57 | crop_feature = seeds.unsqueeze(2) * feature.unsqueeze(1) # seed:[n,21,1,h,w], feature:[n,1,c,h,w], crop_feature:[n,21,c,h,w] 58 | prototype = F.adaptive_avg_pool2d(crop_feature.view(-1,c,h,w), (1,1)).view(n, self.num_cls, c, 1, 1) # prototypes:[n,21,c,1,1] 59 | return prototype 60 | 61 | def reactivate(self, prototype, feature): 62 | IS_cam = F.relu(torch.cosine_similarity(feature.unsqueeze(1), prototype, dim=2)) # feature:[n,1,c,h,w], prototypes:[n,21,c,1,1], crop_feature:[n,21,h,w] 63 | IS_cam = F.interpolate(IS_cam, feature.shape[2:], mode='bilinear', align_corners=True) 64 | return IS_cam 65 | 66 | def forward(self, x, valid_mask): 67 | 68 | N, C, H, W = x.size() 69 | 70 | # forward 71 | x0 = self.stage0(x) 72 | x1 = self.stage1(x0) 73 | x2 = self.stage2(x1).detach() 74 | x3 = self.stage3(x2) 75 | x4 = self.stage4(x3) 76 | 77 | side1 = self.side1(x1.detach()) 78 | side2 = self.side2(x2.detach()) 79 | side3 = self.side3(x3.detach()) 80 | side4 = self.side4(x4.detach()) 81 | 82 | hie_fea = torch.cat([F.interpolate(side1/(torch.norm(side1,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear'), 83 | F.interpolate(side2/(torch.norm(side2,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear'), 84 | F.interpolate(side3/(torch.norm(side3,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear'), 85 | F.interpolate(side4/(torch.norm(side4,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear')], dim=1) 86 | 87 | sem_feature = x4 88 | cam = self.classifier(x4) 89 | score = F.adaptive_avg_pool2d(cam, 1) 90 | 91 | # initialize background map 92 | norm_cam = F.relu(cam) 93 | norm_cam = norm_cam/(F.adaptive_max_pool2d(norm_cam, (1, 1)) + 1e-5) 94 | cam_bkg = 1-torch.max(norm_cam,dim=1)[0].unsqueeze(1) 95 | norm_cam = torch.cat([cam_bkg, norm_cam], dim=1) 96 | norm_cam = F.interpolate(norm_cam, side3.shape[2:], mode='bilinear', align_corners=True)*valid_mask 97 | 98 | seeds, probs = self.get_seed(norm_cam.clone(), valid_mask.clone(), sem_feature.clone()) 99 | prototypes = self.get_prototype(seeds, hie_fea) 100 | IS_cam = self.reactivate(prototypes, hie_fea) 101 | 102 | 103 | return {"score": score, "cam": norm_cam, "seeds": seeds, "prototypes": prototypes, "IS_cam": IS_cam, "probs":probs} 104 | 105 | def train(self, mode=True): 106 | for p in self.resnet50.conv1.parameters(): 107 | p.requires_grad = False 108 | for p in self.resnet50.bn1.parameters(): 109 | p.requires_grad = False 110 | 111 | def trainable_parameters(self): 112 | return (list(self.backbone.parameters()), list(self.newly_added.parameters())) 113 | 114 | 115 | class CAM(Net): 116 | 117 | def __init__(self, num_cls): 118 | super(CAM, self).__init__(num_cls=num_cls) 119 | self.num_cls = num_cls 120 | 121 | def forward(self, x, label): 122 | 123 | x0 = self.stage0(x) 124 | x1 = self.stage1(x0) 125 | x2 = self.stage2(x1).detach() 126 | x3 = self.stage3(x2) 127 | x4 = self.stage4(x3) 128 | 129 | side1 = self.side1(x1.detach()) 130 | side2 = self.side2(x2.detach()) 131 | side3 = self.side3(x3.detach()) 132 | side4 = self.side4(x4.detach()) 133 | 134 | hie_fea = torch.cat([F.interpolate(side1/(torch.norm(side1,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear'), 135 | F.interpolate(side2/(torch.norm(side2,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear'), 136 | F.interpolate(side3/(torch.norm(side3,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear'), 137 | F.interpolate(side4/(torch.norm(side4,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear')], dim=1) 138 | 139 | cam = self.classifier(x4) 140 | cam = (cam[0] + cam[1].flip(-1)).unsqueeze(0) 141 | hie_fea = (hie_fea[0] + hie_fea[1].flip(-1)).unsqueeze(0) 142 | 143 | norm_cam = F.relu(cam) 144 | norm_cam = norm_cam/(F.adaptive_max_pool2d(norm_cam, (1, 1)) + 1e-5) 145 | cam_bkg = 1-torch.max(norm_cam,dim=1)[0].unsqueeze(1) 146 | norm_cam = torch.cat([cam_bkg, norm_cam], dim=1) 147 | norm_cam = F.interpolate(norm_cam, side3.shape[2:], mode='bilinear', align_corners=True) 148 | 149 | seeds, _ = self.get_seed(norm_cam.clone(), label.unsqueeze(0).clone(), hie_fea.clone()) 150 | prototypes = self.get_prototype(seeds, hie_fea) 151 | IS_cam = self.reactivate(prototypes, hie_fea) 152 | 153 | 154 | return norm_cam[0], IS_cam[0] 155 | -------------------------------------------------------------------------------- /network/resnet50_irn.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import torch.nn.functional as F 4 | from network import resnet50 5 | 6 | 7 | class Net(nn.Module): 8 | 9 | def __init__(self): 10 | super(Net, self).__init__() 11 | 12 | # backbone 13 | self.resnet50 = resnet50.resnet50(pretrained=True, strides=[2, 2, 2, 1]) 14 | 15 | self.stage1 = nn.Sequential(self.resnet50.conv1, self.resnet50.bn1, self.resnet50.relu, self.resnet50.maxpool) 16 | self.stage2 = nn.Sequential(self.resnet50.layer1) 17 | self.stage3 = nn.Sequential(self.resnet50.layer2) 18 | self.stage4 = nn.Sequential(self.resnet50.layer3) 19 | self.stage5 = nn.Sequential(self.resnet50.layer4) 20 | self.mean_shift = Net.MeanShift(2) 21 | 22 | # branch: class boundary detection 23 | self.fc_edge1 = nn.Sequential( 24 | nn.Conv2d(64, 32, 1, bias=False), 25 | nn.GroupNorm(4, 32), 26 | nn.ReLU(inplace=True), 27 | ) 28 | self.fc_edge2 = nn.Sequential( 29 | nn.Conv2d(256, 32, 1, bias=False), 30 | nn.GroupNorm(4, 32), 31 | nn.ReLU(inplace=True), 32 | ) 33 | self.fc_edge3 = nn.Sequential( 34 | nn.Conv2d(512, 32, 1, bias=False), 35 | nn.GroupNorm(4, 32), 36 | nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False), 37 | nn.ReLU(inplace=True), 38 | ) 39 | self.fc_edge4 = nn.Sequential( 40 | nn.Conv2d(1024, 32, 1, bias=False), 41 | nn.GroupNorm(4, 32), 42 | nn.Upsample(scale_factor=4, mode='bilinear', align_corners=False), 43 | nn.ReLU(inplace=True), 44 | ) 45 | self.fc_edge5 = nn.Sequential( 46 | nn.Conv2d(2048, 32, 1, bias=False), 47 | nn.GroupNorm(4, 32), 48 | nn.Upsample(scale_factor=4, mode='bilinear', align_corners=False), 49 | nn.ReLU(inplace=True), 50 | ) 51 | self.fc_edge6 = nn.Conv2d(160, 1, 1, bias=True) 52 | 53 | # branch: displacement field 54 | self.fc_dp1 = nn.Sequential( 55 | nn.Conv2d(64, 64, 1, bias=False), 56 | nn.GroupNorm(8, 64), 57 | nn.ReLU(inplace=True), 58 | ) 59 | self.fc_dp2 = nn.Sequential( 60 | nn.Conv2d(256, 128, 1, bias=False), 61 | nn.GroupNorm(16, 128), 62 | nn.ReLU(inplace=True), 63 | ) 64 | self.fc_dp3 = nn.Sequential( 65 | nn.Conv2d(512, 256, 1, bias=False), 66 | nn.GroupNorm(16, 256), 67 | nn.ReLU(inplace=True), 68 | ) 69 | self.fc_dp4 = nn.Sequential( 70 | nn.Conv2d(1024, 256, 1, bias=False), 71 | nn.GroupNorm(16, 256), 72 | nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False), 73 | nn.ReLU(inplace=True), 74 | ) 75 | self.fc_dp5 = nn.Sequential( 76 | nn.Conv2d(2048, 256, 1, bias=False), 77 | nn.GroupNorm(16, 256), 78 | nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False), 79 | nn.ReLU(inplace=True), 80 | ) 81 | self.fc_dp6 = nn.Sequential( 82 | nn.Conv2d(768, 256, 1, bias=False), 83 | nn.GroupNorm(16, 256), 84 | nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False), 85 | nn.ReLU(inplace=True), 86 | ) 87 | self.fc_dp7 = nn.Sequential( 88 | nn.Conv2d(448, 256, 1, bias=False), 89 | nn.GroupNorm(16, 256), 90 | nn.ReLU(inplace=True), 91 | nn.Conv2d(256, 2, 1, bias=False), 92 | self.mean_shift 93 | ) 94 | 95 | self.backbone = nn.ModuleList([self.stage1, self.stage2, self.stage3, self.stage4, self.stage5]) 96 | self.edge_layers = nn.ModuleList([self.fc_edge1, self.fc_edge2, self.fc_edge3, self.fc_edge4, self.fc_edge5, self.fc_edge6]) 97 | self.dp_layers = nn.ModuleList([self.fc_dp1, self.fc_dp2, self.fc_dp3, self.fc_dp4, self.fc_dp5, self.fc_dp6, self.fc_dp7]) 98 | 99 | class MeanShift(nn.Module): 100 | 101 | def __init__(self, num_features): 102 | super(Net.MeanShift, self).__init__() 103 | self.register_buffer('running_mean', torch.zeros(num_features)) 104 | 105 | def forward(self, input): 106 | if self.training: 107 | return input 108 | return input - self.running_mean.view(1, 2, 1, 1) 109 | 110 | def forward(self, x): 111 | x1 = self.stage1(x).detach() 112 | x2 = self.stage2(x1).detach() 113 | x3 = self.stage3(x2).detach() 114 | x4 = self.stage4(x3).detach() 115 | x5 = self.stage5(x4).detach() 116 | 117 | edge1 = self.fc_edge1(x1) 118 | edge2 = self.fc_edge2(x2) 119 | edge3 = self.fc_edge3(x3)[..., :edge2.size(2), :edge2.size(3)] 120 | edge4 = self.fc_edge4(x4)[..., :edge2.size(2), :edge2.size(3)] 121 | edge5 = self.fc_edge5(x5)[..., :edge2.size(2), :edge2.size(3)] 122 | edge_out = self.fc_edge6(torch.cat([edge1, edge2, edge3, edge4, edge5], dim=1)) 123 | 124 | dp1 = self.fc_dp1(x1) 125 | dp2 = self.fc_dp2(x2) 126 | dp3 = self.fc_dp3(x3) 127 | dp4 = self.fc_dp4(x4)[..., :dp3.size(2), :dp3.size(3)] 128 | dp5 = self.fc_dp5(x5)[..., :dp3.size(2), :dp3.size(3)] 129 | 130 | dp_up3 = self.fc_dp6(torch.cat([dp3, dp4, dp5], dim=1))[..., :dp2.size(2), :dp2.size(3)] 131 | dp_out = self.fc_dp7(torch.cat([dp1, dp2, dp_up3], dim=1)) 132 | 133 | return edge_out, dp_out 134 | 135 | def trainable_parameters(self): 136 | return (tuple(self.edge_layers.parameters()), 137 | tuple(self.dp_layers.parameters())) 138 | 139 | def train(self, mode=True): 140 | super().train(mode) 141 | self.backbone.eval() 142 | 143 | 144 | class AffinityDisplacementLoss(Net): 145 | 146 | path_indices_prefix = "path_indices" 147 | 148 | def __init__(self, path_index): 149 | 150 | super(AffinityDisplacementLoss, self).__init__() 151 | 152 | self.path_index = path_index 153 | 154 | self.n_path_lengths = len(path_index.path_indices) 155 | for i, pi in enumerate(path_index.path_indices): 156 | self.register_buffer(AffinityDisplacementLoss.path_indices_prefix + str(i), torch.from_numpy(pi)) 157 | 158 | self.register_buffer( 159 | 'disp_target', 160 | torch.unsqueeze(torch.unsqueeze(torch.from_numpy(path_index.search_dst).transpose(1, 0), 0), -1).float()) 161 | 162 | def to_affinity(self, edge): 163 | aff_list = [] 164 | edge = edge.view(edge.size(0), -1) 165 | 166 | for i in range(self.n_path_lengths): 167 | ind = self._buffers[AffinityDisplacementLoss.path_indices_prefix + str(i)] 168 | ind_flat = ind.view(-1) 169 | dist = torch.index_select(edge, dim=-1, index=ind_flat) 170 | dist = dist.view(dist.size(0), ind.size(0), ind.size(1), ind.size(2)) 171 | aff = torch.squeeze(1 - F.max_pool2d(dist, (dist.size(2), 1)), dim=2) 172 | aff_list.append(aff) 173 | aff_cat = torch.cat(aff_list, dim=1) 174 | 175 | return aff_cat 176 | 177 | def to_pair_displacement(self, disp): 178 | height, width = disp.size(2), disp.size(3) 179 | radius_floor = self.path_index.radius_floor 180 | 181 | cropped_height = height - radius_floor 182 | cropped_width = width - 2 * radius_floor 183 | 184 | disp_src = disp[:, :, :cropped_height, radius_floor:radius_floor + cropped_width] 185 | 186 | disp_dst = [disp[:, :, dy:dy + cropped_height, radius_floor + dx:radius_floor + dx + cropped_width] 187 | for dy, dx in self.path_index.search_dst] 188 | disp_dst = torch.stack(disp_dst, 2) 189 | 190 | pair_disp = torch.unsqueeze(disp_src, 2) - disp_dst 191 | pair_disp = pair_disp.view(pair_disp.size(0), pair_disp.size(1), pair_disp.size(2), -1) 192 | 193 | return pair_disp 194 | 195 | def to_displacement_loss(self, pair_disp): 196 | return torch.abs(pair_disp - self.disp_target) 197 | 198 | def forward(self, *inputs): 199 | x, return_loss = inputs 200 | edge_out, dp_out = super().forward(x) 201 | 202 | if return_loss is False: 203 | return edge_out, dp_out 204 | 205 | aff = self.to_affinity(torch.sigmoid(edge_out)) 206 | pos_aff_loss = (-1) * torch.log(aff + 1e-5) 207 | neg_aff_loss = (-1) * torch.log(1. + 1e-5 - aff) 208 | 209 | pair_disp = self.to_pair_displacement(dp_out) 210 | dp_fg_loss = self.to_displacement_loss(pair_disp) 211 | dp_bg_loss = torch.abs(pair_disp) 212 | 213 | return pos_aff_loss, neg_aff_loss, dp_fg_loss, dp_bg_loss 214 | 215 | 216 | class EdgeDisplacement(Net): 217 | 218 | def __init__(self, crop_size=512, stride=4): 219 | super(EdgeDisplacement, self).__init__() 220 | self.crop_size = crop_size 221 | self.stride = stride 222 | 223 | def forward(self, x): 224 | feat_size = (x.size(2)-1)//self.stride+1, (x.size(3)-1)//self.stride+1 225 | 226 | x = F.pad(x, [0, self.crop_size-x.size(3), 0, self.crop_size-x.size(2)]) 227 | edge_out, dp_out = super().forward(x) 228 | edge_out = edge_out[..., :feat_size[0], :feat_size[1]] 229 | dp_out = dp_out[..., :feat_size[0], :feat_size[1]] 230 | 231 | edge_out = torch.sigmoid(edge_out[0]/2 + edge_out[1].flip(-1)/2) 232 | dp_out = dp_out[0] 233 | 234 | return edge_out, dp_out 235 | 236 | 237 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | numpy 2 | cython 3 | imageio 4 | scikit-image 5 | opencv-python 6 | chainercv 7 | matplotlib 8 | tqdm 9 | tensorboardx 10 | git+https://github.com/lucasb-eyer/pydensecrf.git -------------------------------------------------------------------------------- /run_coco.sh: -------------------------------------------------------------------------------- 1 | 2 | # Step 1. Train SIPE for localization maps. 3 | 4 | # 1.1 train sipe 5 | python train_resnet50_SIPE.py --dataset coco --session_name exp_coco 6 | # 1.2 obtain localization maps 7 | python make_cam.py --dataset coco --session_name exp_coco 8 | # 1.3 evaluate localization maps 9 | python eval_cam.py --dataset coco --session_name exp_coco 10 | 11 | 12 | # Step 2. Refinement for pseudo labels. 13 | # Note: for MS COCO dataset, we replace IRN with denseCRF due to the large computation cost. 14 | python cam2ir.py --dataset coco --session_name exp_coco -------------------------------------------------------------------------------- /run_voc.sh: -------------------------------------------------------------------------------- 1 | 2 | # Step 1. Train SIPE for localization maps. 3 | 4 | # 1.1 train sipe 5 | python train_resnet50_SIPE.py 6 | # 1.2 obtain localization maps 7 | python make_cam.py 8 | # 1.3 evaluate localization maps 9 | python eval_cam.py 10 | 11 | 12 | # Step 2. Train IRN for pseudo labels. 13 | 14 | # 2.1 generate ir label 15 | python cam2ir.py 16 | # 2.2 train irn 17 | python train_irn.py 18 | # 2.3 make pseudo labels 19 | python make_seg_labels.py -------------------------------------------------------------------------------- /tool/imutils.py: -------------------------------------------------------------------------------- 1 | import random 2 | import numpy as np 3 | 4 | import pydensecrf.densecrf as dcrf 5 | from pydensecrf.utils import unary_from_labels 6 | from PIL import Image 7 | 8 | def pil_resize(img, size, order): 9 | if size[0] == img.shape[0] and size[1] == img.shape[1]: 10 | return img 11 | 12 | if order == 3: 13 | resample = Image.BICUBIC 14 | elif order == 0: 15 | resample = Image.NEAREST 16 | 17 | return np.asarray(Image.fromarray(img).resize(size[::-1], resample)) 18 | 19 | def pil_rescale(img, scale, order): 20 | height, width = img.shape[:2] 21 | target_size = (int(np.round(height*scale)), int(np.round(width*scale))) 22 | return pil_resize(img, target_size, order) 23 | 24 | 25 | def random_resize_long(img, min_long, max_long): 26 | target_long = random.randint(min_long, max_long) 27 | h, w = img.shape[:2] 28 | 29 | if w < h: 30 | scale = target_long / h 31 | else: 32 | scale = target_long / w 33 | 34 | return pil_rescale(img, scale, 3) 35 | 36 | def random_scale(img, scale_range, order): 37 | 38 | target_scale = scale_range[0] + random.random() * (scale_range[1] - scale_range[0]) 39 | 40 | if isinstance(img, tuple): 41 | return (pil_rescale(img[0], target_scale, order[0]), pil_rescale(img[1], target_scale, order[1])) 42 | else: 43 | return pil_rescale(img[0], target_scale, order) 44 | 45 | def random_lr_flip(img): 46 | 47 | if bool(random.getrandbits(1)): 48 | if isinstance(img, tuple): 49 | return [np.fliplr(m) for m in img] 50 | else: 51 | return np.fliplr(img) 52 | else: 53 | return img 54 | 55 | def get_random_crop_box(imgsize, cropsize): 56 | h, w = imgsize 57 | 58 | ch = min(cropsize, h) 59 | cw = min(cropsize, w) 60 | 61 | w_space = w - cropsize 62 | h_space = h - cropsize 63 | 64 | if w_space > 0: 65 | cont_left = 0 66 | img_left = random.randrange(w_space + 1) 67 | else: 68 | cont_left = random.randrange(-w_space + 1) 69 | img_left = 0 70 | 71 | if h_space > 0: 72 | cont_top = 0 73 | img_top = random.randrange(h_space + 1) 74 | else: 75 | cont_top = random.randrange(-h_space + 1) 76 | img_top = 0 77 | 78 | return cont_top, cont_top+ch, cont_left, cont_left+cw, img_top, img_top+ch, img_left, img_left+cw 79 | 80 | def random_crop(images, cropsize, default_values): 81 | 82 | if isinstance(images, np.ndarray): images = (images,) 83 | if isinstance(default_values, int): default_values = (default_values,) 84 | 85 | imgsize = images[0].shape[:2] 86 | box = get_random_crop_box(imgsize, cropsize) 87 | 88 | new_images = [] 89 | for img, f in zip(images, default_values): 90 | 91 | if len(img.shape) == 3: 92 | cont = np.ones((cropsize, cropsize, img.shape[2]), img.dtype)*f 93 | else: 94 | cont = np.ones((cropsize, cropsize), img.dtype)*f 95 | cont[box[0]:box[1], box[2]:box[3]] = img[box[4]:box[5], box[6]:box[7]] 96 | new_images.append(cont) 97 | 98 | if len(new_images) == 1: 99 | new_images = new_images[0] 100 | 101 | return new_images, box 102 | 103 | def top_left_crop(img, cropsize, default_value): 104 | 105 | h, w = img.shape[:2] 106 | 107 | ch = min(cropsize, h) 108 | cw = min(cropsize, w) 109 | 110 | if len(img.shape) == 2: 111 | container = np.ones((cropsize, cropsize), img.dtype)*default_value 112 | else: 113 | container = np.ones((cropsize, cropsize, img.shape[2]), img.dtype)*default_value 114 | 115 | container[:ch, :cw] = img[:ch, :cw] 116 | 117 | return container 118 | 119 | def center_crop(img, cropsize, default_value=0): 120 | 121 | h, w = img.shape[:2] 122 | 123 | ch = min(cropsize, h) 124 | cw = min(cropsize, w) 125 | 126 | sh = h - cropsize 127 | sw = w - cropsize 128 | 129 | if sw > 0: 130 | cont_left = 0 131 | img_left = int(round(sw / 2)) 132 | else: 133 | cont_left = int(round(-sw / 2)) 134 | img_left = 0 135 | 136 | if sh > 0: 137 | cont_top = 0 138 | img_top = int(round(sh / 2)) 139 | else: 140 | cont_top = int(round(-sh / 2)) 141 | img_top = 0 142 | 143 | if len(img.shape) == 2: 144 | container = np.ones((cropsize, cropsize), img.dtype)*default_value 145 | else: 146 | container = np.ones((cropsize, cropsize, img.shape[2]), img.dtype)*default_value 147 | 148 | container[cont_top:cont_top+ch, cont_left:cont_left+cw] = \ 149 | img[img_top:img_top+ch, img_left:img_left+cw] 150 | 151 | return container 152 | 153 | def HWC_to_CHW(img): 154 | return np.transpose(img, (2, 0, 1)) 155 | 156 | def crf_inference_label(img, labels, t=10, n_labels=21, gt_prob=0.7): 157 | 158 | h, w = img.shape[:2] 159 | 160 | d = dcrf.DenseCRF2D(w, h, n_labels) 161 | 162 | unary = unary_from_labels(labels, n_labels, gt_prob=gt_prob, zero_unsure=False) 163 | 164 | d.setUnaryEnergy(unary) 165 | d.addPairwiseGaussian(sxy=3, compat=3) 166 | d.addPairwiseBilateral(sxy=50, srgb=5, rgbim=np.ascontiguousarray(np.copy(img)), compat=10) 167 | 168 | q = d.inference(t) 169 | 170 | return np.argmax(np.array(q).reshape((n_labels, h, w)), axis=0) 171 | 172 | 173 | def get_strided_size(orig_size, stride): 174 | return ((orig_size[0]-1)//stride+1, (orig_size[1]-1)//stride+1) 175 | 176 | 177 | def get_strided_up_size(orig_size, stride): 178 | strided_size = get_strided_size(orig_size, stride) 179 | return strided_size[0]*stride, strided_size[1]*stride 180 | 181 | 182 | def compress_range(arr): 183 | uniques = np.unique(arr) 184 | maximum = np.max(uniques) 185 | 186 | d = np.zeros(maximum+1, np.int32) 187 | d[uniques] = np.arange(uniques.shape[0]) 188 | 189 | out = d[arr] 190 | return out - np.min(out) 191 | 192 | 193 | def colorize_score(score_map, exclude_zero=False, normalize=True, by_hue=False): 194 | import matplotlib.colors 195 | if by_hue: 196 | aranged = np.arange(score_map.shape[0]) / (score_map.shape[0]) 197 | hsv_color = np.stack((aranged, np.ones_like(aranged), np.ones_like(aranged)), axis=-1) 198 | rgb_color = matplotlib.colors.hsv_to_rgb(hsv_color) 199 | 200 | test = rgb_color[np.argmax(score_map, axis=0)] 201 | test = np.expand_dims(np.max(score_map, axis=0), axis=-1) * test 202 | 203 | if normalize: 204 | return test / (np.max(test) + 1e-5) 205 | else: 206 | return test 207 | 208 | else: 209 | VOC_color = np.array([(0, 0, 0), (128, 0, 0), (0, 128, 0), (128, 128, 0), (0, 0, 128), (128, 0, 128), 210 | (0, 128, 128), (128, 128, 128), (64, 0, 0), (192, 0, 0), (64, 128, 0), (192, 128, 0), 211 | (64, 0, 128), (192, 0, 128), (64, 128, 128), (192, 128, 128), (0, 64, 0), (128, 64, 0), 212 | (0, 192, 0), (128, 192, 0), (0, 64, 128), (255, 255, 255)], np.float32) 213 | 214 | if exclude_zero: 215 | VOC_color = VOC_color[1:] 216 | 217 | test = VOC_color[np.argmax(score_map, axis=0)%22] 218 | test = np.expand_dims(np.max(score_map, axis=0), axis=-1) * test 219 | if normalize: 220 | test /= np.max(test) + 1e-5 221 | 222 | return test 223 | 224 | 225 | def colorize_displacement(disp): 226 | 227 | import matplotlib.colors 228 | import math 229 | 230 | a = (np.arctan2(-disp[0], -disp[1]) / math.pi + 1) / 2 231 | 232 | r = np.sqrt(disp[0] ** 2 + disp[1] ** 2) 233 | s = r / np.max(r) 234 | hsv_color = np.stack((a, s, np.ones_like(a)), axis=-1) 235 | rgb_color = matplotlib.colors.hsv_to_rgb(hsv_color) 236 | 237 | return rgb_color 238 | 239 | 240 | def colorize_label(label_map, normalize=True, by_hue=True, exclude_zero=False, outline=False): 241 | 242 | label_map = label_map.astype(np.uint8) 243 | 244 | if by_hue: 245 | import matplotlib.colors 246 | sz = np.max(label_map) 247 | aranged = np.arange(sz) / sz 248 | hsv_color = np.stack((aranged, np.ones_like(aranged), np.ones_like(aranged)), axis=-1) 249 | rgb_color = matplotlib.colors.hsv_to_rgb(hsv_color) 250 | rgb_color = np.concatenate([np.zeros((1, 3)), rgb_color], axis=0) 251 | 252 | test = rgb_color[label_map] 253 | else: 254 | VOC_color = np.array([(0, 0, 0), (128, 0, 0), (0, 128, 0), (128, 128, 0), (0, 0, 128), (128, 0, 128), 255 | (0, 128, 128), (128, 128, 128), (64, 0, 0), (192, 0, 0), (64, 128, 0), (192, 128, 0), 256 | (64, 0, 128), (192, 0, 128), (64, 128, 128), (192, 128, 128), (0, 64, 0), (128, 64, 0), 257 | (0, 192, 0), (128, 192, 0), (0, 64, 128), (255, 255, 255)], np.float32) 258 | 259 | if exclude_zero: 260 | VOC_color = VOC_color[1:] 261 | test = VOC_color[label_map] 262 | if normalize: 263 | test /= np.max(test) 264 | 265 | if outline: 266 | edge = np.greater(np.sum(np.abs(test[:-1, :-1] - test[1:, :-1]), axis=-1) + np.sum(np.abs(test[:-1, :-1] - test[:-1, 1:]), axis=-1), 0) 267 | edge1 = np.pad(edge, ((0, 1), (0, 1)), mode='constant', constant_values=0) 268 | edge2 = np.pad(edge, ((1, 0), (1, 0)), mode='constant', constant_values=0) 269 | edge = np.repeat(np.expand_dims(np.maximum(edge1, edge2), -1), 3, axis=-1) 270 | 271 | test = np.maximum(test, edge) 272 | return test 273 | -------------------------------------------------------------------------------- /tool/indexing.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn.functional as F 3 | import numpy as np 4 | 5 | 6 | class PathIndex: 7 | 8 | def __init__(self, radius, default_size): 9 | self.radius = radius 10 | self.radius_floor = int(np.ceil(radius) - 1) 11 | 12 | self.search_paths, self.search_dst = self.get_search_paths_dst(self.radius) 13 | 14 | self.path_indices, self.src_indices, self.dst_indices = self.get_path_indices(default_size) 15 | 16 | return 17 | 18 | def get_search_paths_dst(self, max_radius=5): 19 | 20 | coord_indices_by_length = [[] for _ in range(max_radius * 4)] 21 | 22 | search_dirs = [] 23 | 24 | for x in range(1, max_radius): 25 | search_dirs.append((0, x)) 26 | 27 | for y in range(1, max_radius): 28 | for x in range(-max_radius + 1, max_radius): 29 | if x * x + y * y < max_radius ** 2: 30 | search_dirs.append((y, x)) 31 | 32 | for dir in search_dirs: 33 | 34 | length_sq = dir[0] ** 2 + dir[1] ** 2 35 | path_coords = [] 36 | 37 | min_y, max_y = sorted((0, dir[0])) 38 | min_x, max_x = sorted((0, dir[1])) 39 | 40 | for y in range(min_y, max_y + 1): 41 | for x in range(min_x, max_x + 1): 42 | 43 | dist_sq = (dir[0] * x - dir[1] * y) ** 2 / length_sq 44 | 45 | if dist_sq < 1: 46 | path_coords.append([y, x]) 47 | 48 | path_coords.sort(key=lambda x: -abs(x[0]) - abs(x[1])) 49 | path_length = len(path_coords) 50 | 51 | coord_indices_by_length[path_length].append(path_coords) 52 | 53 | path_list_by_length = [np.asarray(v) for v in coord_indices_by_length if v] 54 | path_destinations = np.concatenate([p[:, 0] for p in path_list_by_length], axis=0) 55 | 56 | return path_list_by_length, path_destinations 57 | 58 | def get_path_indices(self, size): 59 | 60 | full_indices = np.reshape(np.arange(0, size[0] * size[1], dtype=np.int64), (size[0], size[1])) 61 | 62 | cropped_height = size[0] - self.radius_floor 63 | cropped_width = size[1] - 2 * self.radius_floor 64 | 65 | path_indices = [] 66 | 67 | for paths in self.search_paths: 68 | 69 | path_indices_list = [] 70 | for p in paths: 71 | 72 | coord_indices_list = [] 73 | 74 | for dy, dx in p: 75 | coord_indices = full_indices[dy:dy + cropped_height, 76 | self.radius_floor + dx:self.radius_floor + dx + cropped_width] 77 | coord_indices = np.reshape(coord_indices, [-1]) 78 | 79 | coord_indices_list.append(coord_indices) 80 | 81 | path_indices_list.append(coord_indices_list) 82 | 83 | path_indices.append(np.array(path_indices_list)) 84 | 85 | src_indices = np.reshape(full_indices[:cropped_height, self.radius_floor:self.radius_floor + cropped_width], -1) 86 | dst_indices = np.concatenate([p[:,0] for p in path_indices], axis=0) 87 | 88 | return path_indices, src_indices, dst_indices 89 | 90 | 91 | def edge_to_affinity(edge, paths_indices): 92 | 93 | aff_list = [] 94 | edge = edge.view(edge.size(0), -1) 95 | 96 | for i in range(len(paths_indices)): 97 | if isinstance(paths_indices[i], np.ndarray): 98 | paths_indices[i] = torch.from_numpy(paths_indices[i]) 99 | paths_indices[i] = paths_indices[i].cuda(non_blocking=True) 100 | 101 | for ind in paths_indices: 102 | ind_flat = ind.view(-1) 103 | dist = torch.index_select(edge, dim=-1, index=ind_flat) 104 | dist = dist.view(dist.size(0), ind.size(0), ind.size(1), ind.size(2)) 105 | aff = torch.squeeze(1 - F.max_pool2d(dist, (dist.size(2), 1)), dim=2) 106 | aff_list.append(aff) 107 | aff_cat = torch.cat(aff_list, dim=1) 108 | 109 | return aff_cat 110 | 111 | 112 | def affinity_sparse2dense(affinity_sparse, ind_from, ind_to, n_vertices): 113 | 114 | ind_from = torch.from_numpy(ind_from) 115 | ind_to = torch.from_numpy(ind_to) 116 | 117 | affinity_sparse = affinity_sparse.view(-1).cpu() 118 | ind_from = ind_from.repeat(ind_to.size(0)).view(-1) 119 | ind_to = ind_to.view(-1) 120 | 121 | 122 | indices = torch.stack([ind_from, ind_to]) 123 | indices_tp = torch.stack([ind_to, ind_from]) 124 | 125 | indices_id = torch.stack([torch.arange(0, n_vertices).long(), torch.arange(0, n_vertices).long()]) 126 | 127 | affinity_dense = torch.sparse.FloatTensor(torch.cat([indices, indices_id, indices_tp], dim=1), 128 | torch.cat([affinity_sparse, torch.ones([n_vertices]), affinity_sparse])).to_dense().cuda() 129 | 130 | return affinity_dense 131 | 132 | 133 | def to_transition_matrix(affinity_dense, beta, times): 134 | scaled_affinity = torch.pow(affinity_dense, beta) 135 | 136 | trans_mat = scaled_affinity / torch.sum(scaled_affinity, dim=0, keepdim=True) 137 | for _ in range(times): 138 | trans_mat = torch.matmul(trans_mat, trans_mat) 139 | 140 | return trans_mat 141 | 142 | def propagate_to_edge(x, edge, radius=5, beta=10, exp_times=8): 143 | 144 | height, width = x.shape[-2:] 145 | 146 | hor_padded = width+radius*2 147 | ver_padded = height+radius 148 | 149 | path_index = PathIndex(radius=radius, default_size=(ver_padded, hor_padded)) 150 | 151 | 152 | edge_padded = F.pad(edge, (radius, radius, 0, radius), mode='constant', value=1.0) 153 | 154 | sparse_aff = edge_to_affinity(torch.unsqueeze(edge_padded, 0), 155 | path_index.path_indices) 156 | 157 | dense_aff = affinity_sparse2dense(sparse_aff, path_index.src_indices, 158 | path_index.dst_indices, ver_padded * hor_padded) 159 | 160 | dense_aff = dense_aff.view(ver_padded, hor_padded, ver_padded, hor_padded) 161 | dense_aff = dense_aff[:-radius, radius:-radius, :-radius, radius:-radius] 162 | dense_aff = dense_aff.reshape(height * width, height * width) 163 | 164 | trans_mat = to_transition_matrix(dense_aff, beta=beta, times=exp_times) 165 | 166 | x = x.view(-1, height, width) * (1 - edge) 167 | 168 | rw = torch.matmul(x.view(-1, height * width), trans_mat) 169 | rw = rw.view(rw.size(0), 1, height, width) 170 | 171 | return rw -------------------------------------------------------------------------------- /tool/inferutils.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | import matplotlib.pyplot as plt 4 | from PIL import Image 5 | import shutil 6 | import scipy.misc 7 | import math 8 | 9 | 10 | 11 | def create_class_key_in_dict(dict, cls_nb): 12 | for i in range(cls_nb): 13 | dict[i] = [] 14 | return dict 15 | 16 | 17 | def calculate_class_avg_iou(class_iou_dict): 18 | class_mean_iou_dict = {} 19 | for i in range(20): 20 | class_iou_list = class_iou_dict[i] 21 | if len(class_iou_list) != 0: 22 | class_iou_list_mean = round(sum(class_iou_list)/len(class_iou_list), 4) 23 | else: 24 | class_iou_list_mean = 0. 25 | class_mean_iou_dict[i] = class_iou_list_mean 26 | return class_mean_iou_dict 27 | 28 | 29 | def draw_heatmap(img, hm): 30 | hm = plt.cm.hot(hm)[:, :, :3] 31 | hm = np.array(Image.fromarray((hm*255).astype(np.uint8), 'RGB').resize((img.shape[1], img.shape[0]), Image.BICUBIC)).astype(np.float)*2 32 | if hm.shape == np.array(img).astype(np.float).shape: 33 | out = (hm + np.array(img).astype(np.float)) / 3 34 | out = Image.fromarray((out / np.max(out) * 255).astype(np.uint8), 'RGB') 35 | return hm, out 36 | 37 | 38 | def draw_heatmap_array(img, hm): 39 | hm = plt.cm.hot(hm)[:, :, :3] 40 | hm = np.array(Image.fromarray((hm*255).astype(np.uint8), 'RGB').resize((img.shape[1], img.shape[0]), Image.BICUBIC)).astype(np.float)*2 41 | if hm.shape == np.array(img).astype(np.float).shape: 42 | out = (hm + np.array(img).astype(np.float)) / 3 43 | out = (out / np.max(out) * 255).astype(np.uint8) 44 | return hm, out 45 | 46 | 47 | def cls200_vote(y_200, k_cluster): 48 | topk_subcls = np.argsort(y_200[0].detach().cpu().numpy())[-10:][::-1] 49 | topk_cls = np.array(topk_subcls/k_cluster, dtype=np.uint8) 50 | topk_vote = np.unique(topk_cls, return_counts=True) 51 | 52 | p_cls_sum = [] 53 | for p_cls in topk_vote[0]: 54 | subcls_sum = [] 55 | for prob in np.where(topk_cls == p_cls)[0]: 56 | subcls_sum.append(y_200[0][topk_subcls[prob]].item()) 57 | p_cls_sum.append(sum(subcls_sum)) 58 | 59 | cls200_pred_vote = topk_vote[0][np.where(np.array(p_cls_sum) >= 0.5)[0]] 60 | return cls200_pred_vote 61 | 62 | 63 | def cls200_sum(y_200, k_cluster): 64 | cls20_prob_sum_list = [] 65 | for rou in range(20): 66 | subclass_prob_sum = sum(y_200[0][rou*k_cluster:rou*k_cluster+k_cluster].detach().cpu().numpy()) 67 | cls20_prob_sum_list.append(subclass_prob_sum/10) 68 | 69 | cls200_pred_max = np.where(np.array(cls20_prob_sum_list)>0.05)[0] 70 | return cls200_pred_max 71 | 72 | 73 | def cam_subcls_norm(cam, cls20_gt, k_cluster): 74 | for gt in cls20_gt: 75 | subcls_cam = cam[gt*k_cluster:gt*k_cluster+k_cluster] 76 | 77 | norm_cam = subcls_cam / (np.max(subcls_cam, keepdims=True) + 1e-5) 78 | 79 | subcls_norm_cam = np.asarray(norm_cam) 80 | cam[gt*k_cluster:gt*k_cluster+k_cluster] = subcls_norm_cam 81 | return cam 82 | 83 | 84 | def compute_acc(pred_labels, gt_labels): 85 | pred_correct_count = 0 86 | pred_correct_list = [] 87 | for pred_label in pred_labels: 88 | if pred_label in gt_labels: 89 | pred_correct_count += 1 90 | union = len(gt_labels) + len(pred_labels) - pred_correct_count 91 | acc = round(pred_correct_count/union, 4) 92 | return(acc) 93 | 94 | 95 | def compute_iou(gt_labels, cam_np, gt_np, th, class_iou_dict): 96 | iou_list = [] 97 | for label in gt_labels: 98 | cam = cam_np[label] 99 | gt = gt_np[label] 100 | 101 | gt_target_class = label + 1 102 | 103 | gt_y, gt_x = np.where(gt == gt_target_class) 104 | gt_pixel_nb = gt_y.shape[0] # object 105 | 106 | correct_pixel_nb = 0 107 | 108 | cam_y, cam_x = np.where(cam >= th) 109 | high_response_pixel_nb = cam_y.shape[0] # detected 110 | 111 | for pixel in range(gt_y.shape[0]): 112 | if cam[gt_y[pixel]][gt_x[pixel]] >= th: 113 | correct_pixel_nb += 1 # intersection 114 | else: 115 | continue 116 | 117 | union = gt_pixel_nb + high_response_pixel_nb - correct_pixel_nb 118 | 119 | if high_response_pixel_nb != 0: 120 | iou = round(correct_pixel_nb/union, 4) 121 | else: 122 | iou = 0. 123 | iou_list.append(iou) 124 | if high_response_pixel_nb != 0: 125 | precision = round(correct_pixel_nb/high_response_pixel_nb, 4) 126 | else: 127 | precision = 0. 128 | recall = round(correct_pixel_nb/gt_pixel_nb, 4) 129 | class_iou_dict[label].append(iou) 130 | print(label, iou) 131 | return class_iou_dict, iou_list 132 | 133 | 134 | 135 | 136 | def compute_merge_iou(gt_labels, cam_nor, cam_b4_nor, gt_np, th, k, class_iou_dict): 137 | merged_cam_list = [] 138 | for label in gt_labels: 139 | 140 | cam_b4_nor_ = cam_b4_nor[label*k:label*k+k] # (10, 366, 500) 141 | cam_b4_sum = np.sum(cam_b4_nor_, axis=0) # (366, 500) 142 | merge_cam = cam_b4_sum / np.amax(cam_b4_sum) # (366, 500) np.max(merge_cam)=1.0 143 | 144 | 145 | merged_cam_list.append(merge_cam) 146 | 147 | 148 | gt = gt_np[label] 149 | gt_target_class = label + 1 150 | 151 | gt_y, gt_x = np.where(gt == gt_target_class) 152 | gt_pixel_nb = gt_y.shape[0] # object 153 | 154 | correct_pixel_nb = 0 155 | 156 | cam_y, cam_x = np.where(merge_cam >= th) 157 | high_response_pixel_nb = cam_y.shape[0] # detected 158 | 159 | for pixel in range(gt_y.shape[0]): 160 | if merge_cam[gt_y[pixel]][gt_x[pixel]] >= th: 161 | correct_pixel_nb += 1 # intersection 162 | else: 163 | continue 164 | 165 | union = gt_pixel_nb + high_response_pixel_nb - correct_pixel_nb 166 | 167 | if high_response_pixel_nb != 0: 168 | iou = round(correct_pixel_nb/union, 4) 169 | else: 170 | iou = 0. 171 | if high_response_pixel_nb != 0: 172 | precision = round(correct_pixel_nb/high_response_pixel_nb, 4) 173 | else: 174 | precision = 0. 175 | recall = round(correct_pixel_nb/gt_pixel_nb, 4) 176 | class_iou_dict[label].append(iou) 177 | return class_iou_dict, merged_cam_list 178 | 179 | 180 | 181 | def compute_merge_11_iou(gt_labels, cam_20, cam_200, gt_np, th, k, class_all_iou_dict): 182 | merged_cam_list = [] 183 | for label in gt_labels: 184 | parcls_cam = np.expand_dims(cam_20[label], axis=0) 185 | subcls_cam = cam_200[label*k:label*k+k] 186 | 187 | merge_11_cam = np.concatenate((subcls_cam, parcls_cam), axis=0) 188 | merge_11_cam = np.amax(merge_11_cam, axis=0) 189 | merge_cam = merge_11_cam / np.amax(merge_11_cam) 190 | 191 | merged_cam_list.append(merge_cam) 192 | 193 | gt = gt_np[label] 194 | gt_target_class = label + 1 195 | 196 | gt_y, gt_x = np.where(gt == gt_target_class) 197 | gt_pixel_nb = gt_y.shape[0] 198 | 199 | correct_pixel_nb = 0 200 | 201 | cam_y, cam_x = np.where(merge_cam >= th) 202 | high_response_pixel_nb = cam_y.shape[0] 203 | 204 | for pixel in range(gt_y.shape[0]): 205 | if merge_cam[gt_y[pixel]][gt_x[pixel]] >= th: 206 | correct_pixel_nb += 1 207 | else: 208 | continue 209 | 210 | union = gt_pixel_nb + high_response_pixel_nb - correct_pixel_nb 211 | 212 | if high_response_pixel_nb != 0: 213 | iou = round(correct_pixel_nb/union, 4) 214 | else: 215 | iou = 0. 216 | if high_response_pixel_nb != 0: 217 | precision = round(correct_pixel_nb/high_response_pixel_nb, 4) 218 | else: 219 | precision = 0. 220 | recall = round(correct_pixel_nb/gt_pixel_nb, 4) 221 | class_all_iou_dict[label].append(iou) 222 | 223 | return class_all_iou_dict, merged_cam_list 224 | 225 | 226 | 227 | def compute_ub_iou(gt_labels, cam_np, gt_np, th, k, class_iou_dict): 228 | iou_list = [] 229 | all_subclass_iou_list = [] 230 | for l_num, label in enumerate(gt_labels): 231 | subclass_iou_list = [] 232 | cam = cam_np[label*k:label*k+k] 233 | for num, one in enumerate(cam): 234 | merge_cam = one 235 | gt = gt_np[label] 236 | 237 | gt_target_class = label + 1 238 | 239 | gt_y, gt_x = np.where(gt == gt_target_class) 240 | gt_pixel_nb = gt_y.shape[0] 241 | 242 | correct_pixel_nb = 0 243 | 244 | cam_y, cam_x = np.where(merge_cam >= th) 245 | high_response_pixel_nb = cam_y.shape[0] 246 | 247 | for pixel in range(gt_y.shape[0]): 248 | if merge_cam[gt_y[pixel]][gt_x[pixel]] >= th: 249 | correct_pixel_nb += 1 250 | else: 251 | continue 252 | 253 | union = gt_pixel_nb + high_response_pixel_nb - correct_pixel_nb 254 | 255 | 256 | if high_response_pixel_nb != 0: 257 | iou = round(correct_pixel_nb/union, 4) 258 | else: 259 | iou = 0. 260 | subclass_iou_list.append(iou) 261 | 262 | if high_response_pixel_nb != 0: 263 | precision = round(correct_pixel_nb/high_response_pixel_nb, 4) 264 | else: 265 | precision = 0. 266 | recall = round(correct_pixel_nb/gt_pixel_nb, 4) 267 | 268 | print(label, 'subcls_iou_list: {}'.format(subclass_iou_list)) 269 | max_iou = max(subclass_iou_list) 270 | print(max_iou, subclass_iou_list.index(max(subclass_iou_list))) 271 | class_iou_dict[label].append(max_iou) 272 | iou_list.append(max_iou) 273 | 274 | all_subclass_iou_list.append(subclass_iou_list) 275 | return class_iou_dict, iou_list, all_subclass_iou_list 276 | 277 | 278 | 279 | 280 | def count_maxiou_prob(y_200, cls20_gt, all_subclass_iou_list, class_20_iou_list, k_cluster, subclass_top_iou_list, class_200_ub_iou_list, class_ub_iou_dict, img_name): 281 | for i, gt in enumerate(cls20_gt): 282 | subclass_prob = y_200[0][gt*k_cluster:gt*k_cluster+k_cluster].detach().cpu().numpy() 283 | print('pred_score: {}'.format(subclass_prob)) 284 | 285 | ten_subclass_iou_list = all_subclass_iou_list[i] 286 | ten_subclass_iou_list.append(class_20_iou_list[i]) 287 | 288 | subclass_max_idx = ten_subclass_iou_list.index(max(ten_subclass_iou_list)) 289 | pred_subclass = gt*k_cluster+subclass_max_idx 290 | 291 | if subclass_max_idx != 10: 292 | sort_subclass_prob_idx = np.argsort(subclass_prob)[::-1] 293 | top_k_best_iou = np.where(sort_subclass_prob_idx == subclass_max_idx)[0][0] 294 | subclass_top_iou_list[top_k_best_iou] += 1 295 | else: 296 | top_k_best_iou = 10 297 | subclass_top_iou_list[top_k_best_iou] += 1 298 | 299 | 300 | ub_iou = max(class_20_iou_list[i], class_200_ub_iou_list[i]) 301 | class_ub_iou_dict[cls20_gt[i]].append(ub_iou) 302 | print(subclass_top_iou_list) 303 | 304 | line = '{},{},{},{},{}\n'.format(img_name, pred_subclass, top_k_best_iou, ub_iou, class_20_iou_list) 305 | 306 | return class_ub_iou_dict 307 | 308 | 309 | 310 | def merge_topk_iou(y_200, gt_labels, all_subclass_iou_list, cam_np, gt_np, th, k, class_iou_dict): 311 | merged_cam_list = [] 312 | for num, label in enumerate(gt_labels): 313 | subclass_prob = y_200[0][label*k:label*k+k].detach().cpu().numpy() 314 | subclass_iou_list = all_subclass_iou_list[num][:-1] 315 | 316 | cam = cam_np[label*k:label*k+k] 317 | sort_subcls_prob_idx = np.argsort(subclass_prob)[::-1] 318 | 319 | # print(subclass_prob) 320 | # print(subclass_iou_list) 321 | # print(sort_subcls_prob_idx) 322 | 323 | top_k_list = [0, 1, 2, 4, 9] 324 | top_k_iou_list = [] 325 | for top in top_k_list: 326 | merge_k = np.zeros((top+1, cam.shape[1], cam.shape[2])) 327 | target_subcls_cam_idx = sort_subcls_prob_idx[:top+1] 328 | print(top, merge_k.shape, target_subcls_cam_idx) 329 | 330 | for i, idx in enumerate(target_subcls_cam_idx): 331 | merge_k[i] = cam[idx] 332 | 333 | ## norm -> max per pixel 334 | merge_cam = np.amax(merge_k, axis=0) 335 | 336 | # ## sum -> norm 337 | # merge_cam = np.sum(cam, axis=0) / np.amax(cam) 338 | 339 | merged_cam_list.append(merge_cam) 340 | 341 | 342 | gt = gt_np[label] 343 | gt_target_class = label + 1 344 | 345 | gt_y, gt_x = np.where(gt == gt_target_class) 346 | gt_pixel_nb = gt_y.shape[0] # object 347 | 348 | correct_pixel_nb = 0 349 | 350 | cam_y, cam_x = np.where(merge_cam >= th) 351 | high_response_pixel_nb = cam_y.shape[0] # detected 352 | 353 | for pixel in range(gt_y.shape[0]): 354 | if merge_cam[gt_y[pixel]][gt_x[pixel]] >= th: 355 | correct_pixel_nb += 1 # intersection 356 | else: 357 | continue 358 | 359 | union = gt_pixel_nb + high_response_pixel_nb - correct_pixel_nb 360 | 361 | if high_response_pixel_nb != 0: 362 | iou = round(correct_pixel_nb/union, 4) 363 | else: 364 | iou = 0. 365 | if high_response_pixel_nb != 0: 366 | precision = round(correct_pixel_nb/high_response_pixel_nb, 4) 367 | else: 368 | precision = 0. 369 | recall = round(correct_pixel_nb/gt_pixel_nb, 4) 370 | 371 | top_k_iou_list.append(iou) 372 | 373 | return class_iou_dict, merged_cam_list 374 | 375 | 376 | 377 | def vrf_iou_w_distance(cls20_gt): 378 | cls20_w = np.load('./kmeans_subclass/c20_k10/3rd_round/weight_np/R3_cls20_w.npy') # (20, 4096) 379 | cls200_w = np.load('./kmeans_subclass/c20_k10/3rd_round/weight_np/R3_cls200_w.npy') # (200, 4096) 380 | 381 | bike_w = cls20_w[cls20_gt[0]] 382 | sub_human_w = cls200_w[cls20_gt[1]*10:cls20_gt[1]*10+10] 383 | 384 | sub_w_dis_list = [] 385 | for num, sub in enumerate(sub_human_w): 386 | dist = np.linalg.norm(bike_w-sub) 387 | sub_w_dis_list.append(dist) 388 | print('dist_list: {}'.format(sub_w_dis_list)) 389 | print(sub_w_dis_list.index(min(sub_w_dis_list))) 390 | 391 | 392 | def find_200_pseudo_label(image_name, round_nb): 393 | filename_list_path = './kmeans_subclass/c20_k10/{}_round/train/{}_train_filename_list.txt'.format(round_nb, round_nb) 394 | label_20_npy = np.load( './kmeans_subclass/c20_k10/{}_round/train/{}_train_label_20.npy'.format(round_nb, round_nb)) 395 | label_200_npy = np.load('./kmeans_subclass/c20_k10/{}_round/train/{}_train_label_200.npy'.format(round_nb, round_nb)) 396 | 397 | with open(filename_list_path, 'r') as f: 398 | filename_list = f.read().split('\n') 399 | f.close() 400 | 401 | image_idx = filename_list.index(image_name) 402 | label_20 = label_20_npy[image_idx] 403 | label_200 = label_200_npy[image_idx] 404 | 405 | 406 | return label_20, label_200 407 | 408 | 409 | def cam_npy_to_cam_dict(cam_np, label): 410 | cam_dict = {} 411 | idxs = np.where(label==1)[0] 412 | 413 | for idx in idxs: 414 | cam_dict[idx] = cam_np[idx] 415 | 416 | return cam_dict 417 | 418 | 419 | def response_to_label(cam_npy): 420 | seg_map = cam_npy.transpose(1,2,0) 421 | seg_map = np.asarray(np.argmax(seg_map, axis=2), dtype=np.int) 422 | 423 | return seg_map 424 | 425 | 426 | def get_accum_from_dict(par_cls, clust_dict): 427 | accum = 0 428 | for m in range(par_cls): 429 | accum += clust_dict[m] 430 | return accum 431 | 432 | 433 | 434 | def dict2npy(cam_dict, gt_label, th): 435 | gt_cat = np.where(gt_label==1)[0] 436 | 437 | orig_img_size = cam_dict[gt_cat[0]].shape 438 | 439 | bg_score = [np.ones_like(cam_dict[gt_cat[0]])*th] 440 | cam_npy = np.zeros((20, orig_img_size[0], orig_img_size[1])) 441 | 442 | for gt in gt_cat: 443 | cam_npy[gt] = cam_dict[gt] 444 | 445 | cam_npy = np.concatenate((bg_score, cam_npy), axis=0) 446 | return cam_npy 447 | 448 | 449 | def cam_npy_to_label_map(cam_npy): 450 | seg_map = cam_npy.transpose(1,2,0) 451 | seg_map = np.asarray(np.argmax(seg_map, axis=2), dtype=np.int) 452 | return seg_map 453 | 454 | 455 | 456 | def cam_npy_to_cam_dict(cam_npy, label): 457 | cam_dict = {} 458 | for i in range(len(label)): 459 | if label[i] > 1e-5: 460 | cam_dict[i] = cam_npy[i] 461 | return cam_dict 462 | 463 | 464 | def cls200_cam_to_cls20_entropy(no_norm_cam_200, k_cluster, norm_cam, save_path, img_name, orig_img, gt_label, save_entropy_heatmap): 465 | gt_cat = np.where(gt_label==1)[0] 466 | 467 | cam_200_entropy_path = '{}/entropy/cls_200/{}'.format(save_path, img_name) 468 | if save_entropy_heatmap == 1: 469 | if os.path.isdir(cam_200_entropy_path): 470 | shutil.rmtree(cam_200_entropy_path) 471 | os.mkdir(cam_200_entropy_path) 472 | 473 | entropy_npy = np.zeros((norm_cam.shape[0], norm_cam.shape[1], norm_cam.shape[2])) 474 | 475 | 476 | for i in range(20): 477 | sub_cams = no_norm_cam_200[i*k_cluster:i*k_cluster+k_cluster] 478 | 479 | sub_cams_sum = np.sum(sub_cams, axis=0) 480 | 481 | sub_cams_sum_10 = sub_cams_sum[np.newaxis, :] 482 | sub_cams_sum_10 = np.repeat(sub_cams_sum_10, k_cluster, axis=0) 483 | prob = sub_cams/(sub_cams_sum_10 + 1e-5) 484 | 485 | prob_log = np.log(prob + 1e-5) / np.log(k_cluster) 486 | 487 | entropy_norm = -(np.sum(prob*prob_log, axis=0)) # entropy normalization 488 | 489 | entropy_norm[entropy_norm<0]=0 490 | entropy_npy[i] = entropy_norm 491 | 492 | if save_entropy_heatmap == 1: 493 | if i in gt_cat: 494 | hm, heatmap = draw_heatmap(orig_img, entropy_norm) 495 | scipy.misc.imsave('{}/entropy/cls_200/{}/{}_{}.png'.format(save_path, img_name, img_name, i), heatmap) 496 | 497 | return entropy_npy 498 | 499 | 500 | def create_folder(inference_dir_path): 501 | 502 | if os.path.exists(inference_dir_path) == True: 503 | shutil.rmtree(inference_dir_path) 504 | 505 | os.mkdir(inference_dir_path) 506 | os.mkdir(os.path.join(inference_dir_path + '/heatmap')) 507 | os.mkdir(os.path.join(inference_dir_path + '/heatmap/cls_20')) 508 | os.mkdir(os.path.join(inference_dir_path + '/heatmap/cls_200')) 509 | os.mkdir(os.path.join(inference_dir_path + '/output_CAM_npy')) 510 | os.mkdir(os.path.join(inference_dir_path + '/output_CAM_npy/cls_20')) 511 | os.mkdir(os.path.join(inference_dir_path + '/output_CAM_npy/cls_200')) 512 | os.mkdir(os.path.join(inference_dir_path + '/IOU')) 513 | os.mkdir(os.path.join(inference_dir_path + '/crf')) 514 | os.mkdir(os.path.join(inference_dir_path + '/crf/out_la_crf')) 515 | os.mkdir(os.path.join(inference_dir_path + '/crf/out_ha_crf')) 516 | 517 | 518 | 519 | 520 | def draw_single_heatmap(norm_cam, gt_label, orig_img, save_path, img_name): 521 | gt_cat = np.where(gt_label==1)[0] 522 | heatmap_list = [] 523 | mask_list = [] 524 | for i, gt in enumerate(gt_cat): 525 | hm, heatmap = draw_heatmap_array(orig_img, norm_cam[gt]) 526 | cam_viz_path = os.path.join(save_path,'heatmap/cls_20', img_name + '_{}.png'.format(gt)) 527 | scipy.misc.imsave(cam_viz_path, heatmap) 528 | 529 | norm_cam_gt = norm_cam[gt] 530 | norm_cam_gt[norm_cam_gt<=0.15]=0 531 | norm_cam_gt[norm_cam_gt>0.15]=255 532 | 533 | heatmap = np.transpose(heatmap, (2, 1, 0)) 534 | heatmap_list.append(heatmap) 535 | mask_list.append(norm_cam_gt) 536 | 537 | return heatmap_list, mask_list 538 | 539 | 540 | 541 | 542 | def draw_heatmap_cls200(norm_cam, gt_label, orig_img): 543 | gt_cat = np.where(gt_label==1)[0] 544 | heatmap_list = [] 545 | for i, gt in enumerate(gt_cat): 546 | heatmap_cat_list = [] 547 | for x in range(10): 548 | hm, heatmap = draw_heatmap_array(orig_img, norm_cam[gt*10+x]) 549 | heatmap = np.transpose(heatmap, (2, 1, 0)) 550 | heatmap_cat_list.append(heatmap) 551 | heatmap_list.append(heatmap_cat_list) 552 | 553 | return heatmap_list 554 | 555 | 556 | def draw_heatmap_cls200_merge(norm_cam, gt_label, orig_img, img_name): 557 | gt_cat = np.where(gt_label==1)[0] 558 | heatmap_list = [] 559 | for i, gt in enumerate(gt_cat): 560 | hm, heatmap = draw_heatmap_array(orig_img, norm_cam[gt]) 561 | scipy.misc.imsave('/home/julia/julia_data/wsss/best/heatmap/cls_200/merge_{}.png'.format(img_name), heatmap) 562 | heatmap = np.transpose(heatmap, (2, 1, 0)) 563 | heatmap_list.append(heatmap) 564 | 565 | return heatmap_list 566 | 567 | 568 | def draw_heatmap_cls200_entropy(norm_cam, gt_label, orig_img): 569 | gt_cat = np.where(gt_label==1)[0] 570 | heatmap_list = [] 571 | mask_list = [] 572 | for i, gt in enumerate(gt_cat): 573 | hm, heatmap = draw_heatmap_array(orig_img, norm_cam[gt]) 574 | 575 | norm_cam_gt = norm_cam[gt] 576 | norm_cam_gt[norm_cam_gt<=0.6]=0 577 | norm_cam_gt[norm_cam_gt>0.6]=255 578 | 579 | heatmap = np.transpose(heatmap, (2, 1, 0)) 580 | heatmap_list.append(heatmap) 581 | mask_list.append(norm_cam_gt) 582 | 583 | return heatmap_list, mask_list 584 | 585 | 586 | def combine_four_images(files, img_name, gt, save_path): 587 | result = Image.new("RGB", (1200, 800)) 588 | 589 | for index, file in enumerate(files): 590 | img = file 591 | img.thumbnail((400, 400), Image.ANTIALIAS) 592 | x = index // 2 * 400 593 | y = index % 2 * 400 594 | w , h = img.size 595 | result.paste(img, (x, y, x + w, y + h)) 596 | result.save(os.path.expanduser('./{}/combine_maps/{}_{}.jpg'.format(save_path, img_name, gt))) 597 | 598 | 599 | def save_combine_response_maps(cam_20_heatmap, cam_200_merge_heatmap, cam_200_entropy_heatmap, cam_20_map, cam_200_entropy_map, orig_img, gt_label, img_name, save_path): 600 | gt_cat = np.where(gt_label==1)[0] 601 | orig_img_out = Image.fromarray(orig_img.astype(np.uint8), 'RGB') 602 | print(len(cam_20_heatmap), len(cam_200_merge_heatmap), len(cam_200_entropy_heatmap)) 603 | 604 | for num, gt in enumerate(gt_cat): 605 | cls20_out = Image.fromarray(np.transpose(cam_20_heatmap[num], (2, 1, 0)).astype(np.uint8), 'RGB') 606 | cls200_merge_out = Image.fromarray(np.transpose(cam_200_merge_heatmap[num], (2, 1, 0)).astype(np.uint8), 'RGB') 607 | cls200_entropy_out = Image.fromarray(np.transpose(cam_200_entropy_heatmap[num], (2, 1, 0)).astype(np.uint8), 'RGB') 608 | cam_20_map_out = Image.fromarray(cam_20_map[num].astype(np.uint8)) 609 | cam_200_entropy_map_out = Image.fromarray(cam_200_entropy_map[num].astype(np.uint8)) 610 | 611 | image_list = [orig_img_out, cls200_merge_out, cls20_out, cls200_entropy_out, cam_20_map_out, cam_200_entropy_map_out] 612 | combine_four_images(image_list, img_name, gt, save_path) -------------------------------------------------------------------------------- /tool/iouutils.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import os 3 | 4 | 5 | CAT_LIST = ['background', 'aeroplane', 'bicycle', 'bird', 'boat', 6 | 'bottle', 'bus', 'car', 'cat', 'chair', 7 | 'cow', 'diningtable', 'dog', 'horse', 8 | 'motorbike', 'person', 'pottedplant', 9 | 'sheep', 'sofa', 'train', 10 | 'tvmonitor', 'meanIOU'] 11 | 12 | def _fast_hist(label_true, label_pred, n_class): 13 | mask = (label_true >= 0) & (label_true < n_class) 14 | hist = np.bincount( 15 | n_class * label_true[mask].astype(int) + label_pred[mask], 16 | minlength=n_class ** 2, 17 | ).reshape(n_class, n_class) 18 | return hist 19 | 20 | 21 | 22 | def scores(label_trues, label_preds, n_class): 23 | hist = np.zeros((n_class, n_class)) 24 | for lt, lp in zip(label_trues, label_preds): 25 | hist += _fast_hist(lt.flatten(), lp.flatten(), n_class) 26 | acc = np.diag(hist).sum() / hist.sum() 27 | acc_cls = np.diag(hist) / hist.sum(axis=1) 28 | acc_cls = np.nanmean(acc_cls) 29 | iu = np.diag(hist) / (hist.sum(axis=1) + hist.sum(axis=0) - np.diag(hist)) 30 | valid = hist.sum(axis=1) > 0 # added 31 | mean_iu = np.nanmean(iu[valid]) 32 | freq = hist.sum(axis=1) / hist.sum() 33 | fwavacc = (freq[freq > 0] * iu[freq > 0]).sum() 34 | cls_iu = dict(zip(range(n_class), iu)) 35 | 36 | return { 37 | "Pixel Accuracy": acc, 38 | "Mean Accuracy": acc_cls, 39 | "Frequency Weighted IoU": fwavacc, 40 | "Mean IoU": mean_iu, 41 | "Class IoU": cls_iu, 42 | } 43 | 44 | 45 | def record_score(score, save_path, iou_type): 46 | score_list = [] 47 | 48 | for i in range(21): 49 | score_list.append(score['Class IoU'][i]) 50 | aveJ = score['Mean IoU'] 51 | 52 | with open('{}/iou_{}.txt'.format(save_path, iou_type), 'w') as f: 53 | for num, cls_iou in enumerate(score_list): 54 | print('class {:2d} {:12} IU {:.2f}'.format(num, CAT_LIST[num], round(cls_iou, 3))) 55 | f.write('class {:2d} {:12} IU {:.2f}'.format(num, CAT_LIST[num], round(cls_iou, 3)) + '\n') 56 | print('meanIOU: ' + str(aveJ) + '\n') 57 | f.write('meanIOU: ' + str(aveJ) + '\n') -------------------------------------------------------------------------------- /tool/pyutils.py: -------------------------------------------------------------------------------- 1 | 2 | import numpy as np 3 | import time 4 | import sys 5 | 6 | class Logger(object): 7 | def __init__(self, outfile): 8 | self.terminal = sys.stdout 9 | self.log = open(outfile, "w") 10 | sys.stdout = self 11 | 12 | def write(self, message): 13 | self.terminal.write(message) 14 | self.log.write(message) 15 | 16 | def flush(self): 17 | self.terminal.flush() 18 | 19 | 20 | class AverageMeter: 21 | def __init__(self, *keys): 22 | self.__data = dict() 23 | for k in keys: 24 | self.__data[k] = [0.0, 0] 25 | 26 | def add(self, dict): 27 | for k, v in dict.items(): 28 | if k not in self.__data: 29 | self.__data[k] = [0.0, 0] 30 | self.__data[k][0] += v 31 | self.__data[k][1] += 1 32 | 33 | def get(self, *keys): 34 | if len(keys) == 1: 35 | return self.__data[keys[0]][0] / self.__data[keys[0]][1] 36 | else: 37 | v_list = [self.__data[k][0] / self.__data[k][1] for k in keys] 38 | return tuple(v_list) 39 | 40 | def pop(self, key=None): 41 | if key is None: 42 | for k in self.__data.keys(): 43 | self.__data[k] = [0.0, 0] 44 | else: 45 | v = self.get(key) 46 | self.__data[key] = [0.0, 0] 47 | return v 48 | 49 | 50 | class Timer: 51 | def __init__(self, starting_msg = None): 52 | self.start = time.time() 53 | self.stage_start = self.start 54 | 55 | if starting_msg is not None: 56 | print(starting_msg, time.ctime(time.time())) 57 | 58 | def __enter__(self): 59 | return self 60 | 61 | def __exit__(self, exc_type, exc_val, exc_tb): 62 | return 63 | 64 | def update_progress(self, progress): 65 | self.elapsed = time.time() - self.start 66 | self.est_total = self.elapsed / progress 67 | self.est_remaining = self.est_total - self.elapsed 68 | self.est_finish = int(self.start + self.est_total) 69 | 70 | 71 | def str_est_finish(self): 72 | return str(time.ctime(self.est_finish)) 73 | 74 | def get_stage_elapsed(self): 75 | return time.time() - self.stage_start 76 | 77 | def reset_stage(self): 78 | self.stage_start = time.time() 79 | 80 | def lapse(self): 81 | out = time.time() - self.stage_start 82 | self.stage_start = time.time() 83 | return out 84 | 85 | 86 | def to_one_hot(sparse_integers, maximum_val=None, dtype=np.bool): 87 | 88 | if maximum_val is None: 89 | maximum_val = np.max(sparse_integers) + 1 90 | 91 | src_shape = sparse_integers.shape 92 | 93 | flat_src = np.reshape(sparse_integers, [-1]) 94 | src_size = flat_src.shape[0] 95 | 96 | one_hot = np.zeros((maximum_val, src_size), dtype) 97 | one_hot[flat_src, np.arange(src_size)] = 1 98 | 99 | one_hot = np.reshape(one_hot, [maximum_val] + list(src_shape)) 100 | 101 | return one_hot 102 | 103 | 104 | from multiprocessing.pool import ThreadPool 105 | 106 | class BatchThreader: 107 | 108 | def __init__(self, func, args_list, batch_size, prefetch_size=4, processes=12): 109 | self.batch_size = batch_size 110 | self.prefetch_size = prefetch_size 111 | 112 | self.pool = ThreadPool(processes=processes) 113 | self.async_result = [] 114 | 115 | self.func = func 116 | self.left_args_list = args_list 117 | self.n_tasks = len(args_list) 118 | 119 | # initial work 120 | self.__start_works(self.__get_n_pending_works()) 121 | 122 | 123 | def __start_works(self, times): 124 | for _ in range(times): 125 | args = self.left_args_list.pop(0) 126 | self.async_result.append( 127 | self.pool.apply_async(self.func, args)) 128 | 129 | 130 | def __get_n_pending_works(self): 131 | return min((self.prefetch_size + 1) * self.batch_size - len(self.async_result) 132 | , len(self.left_args_list)) 133 | 134 | 135 | 136 | def pop_results(self): 137 | 138 | n_inwork = len(self.async_result) 139 | 140 | n_fetch = min(n_inwork, self.batch_size) 141 | rtn = [self.async_result.pop(0).get() 142 | for _ in range(n_fetch)] 143 | 144 | to_fill = self.__get_n_pending_works() 145 | if to_fill == 0: 146 | self.pool.close() 147 | else: 148 | self.__start_works(to_fill) 149 | 150 | return rtn 151 | 152 | 153 | 154 | 155 | def get_indices_of_pairs(radius, size): 156 | 157 | search_dist = [] 158 | 159 | for x in range(1, radius): 160 | search_dist.append((0, x)) 161 | 162 | for y in range(1, radius): 163 | for x in range(-radius + 1, radius): 164 | if x * x + y * y < radius * radius: 165 | search_dist.append((y, x)) 166 | 167 | radius_floor = radius - 1 168 | 169 | full_indices = np.reshape(np.arange(0, size[0]*size[1], dtype=np.int64), 170 | (size[0], size[1])) 171 | 172 | cropped_height = size[0] - radius_floor 173 | cropped_width = size[1] - 2 * radius_floor 174 | 175 | indices_from = np.reshape(full_indices[:-radius_floor, radius_floor:-radius_floor], 176 | [-1]) 177 | 178 | indices_to_list = [] 179 | 180 | for dy, dx in search_dist: 181 | indices_to = full_indices[dy:dy + cropped_height, 182 | radius_floor + dx:radius_floor + dx + cropped_width] 183 | indices_to = np.reshape(indices_to, [-1]) 184 | 185 | indices_to_list.append(indices_to) 186 | 187 | concat_indices_to = np.concatenate(indices_to_list, axis=0) 188 | 189 | return indices_from, concat_indices_to 190 | -------------------------------------------------------------------------------- /tool/torchutils.py: -------------------------------------------------------------------------------- 1 | 2 | import torch 3 | from torch.utils.data import Dataset, Subset 4 | from PIL import Image 5 | import os.path 6 | import random 7 | 8 | import numpy as np 9 | from tool import imutils 10 | 11 | class PolyOptimizerSGD(torch.optim.SGD): 12 | 13 | def __init__(self, params, lr, weight_decay, max_step, momentum=0.9): 14 | super().__init__(params, lr, weight_decay) 15 | 16 | self.global_step = 0 17 | self.max_step = max_step 18 | self.momentum = momentum 19 | 20 | self.__initial_lr = [group['lr'] for group in self.param_groups] 21 | 22 | 23 | def step(self, closure=None): 24 | 25 | if self.global_step < self.max_step: 26 | lr_mult = (1 - self.global_step / self.max_step) ** self.momentum 27 | 28 | for i in range(len(self.param_groups)): 29 | self.param_groups[i]['lr'] = self.__initial_lr[i] * lr_mult 30 | 31 | super().step(closure) 32 | 33 | self.global_step += 1 34 | 35 | 36 | class PolyOptimizerAdam(torch.optim.Adam): 37 | 38 | def __init__(self, params, lr, weight_decay, max_step, betas=[0.5, 0.99]): 39 | super().__init__(params, lr, betas, weight_decay) 40 | 41 | self.global_step = 0 42 | self.max_step = max_step 43 | self.betas = betas 44 | 45 | self.__initial_lr = [group['lr'] for group in self.param_groups] 46 | 47 | 48 | def step(self, closure=None): 49 | 50 | if self.global_step < self.max_step: 51 | lr_mult = (1 - self.global_step / self.max_step) ** self.betas[0] 52 | 53 | for i in range(len(self.param_groups)): 54 | self.param_groups[i]['lr'] = self.__initial_lr[i] * lr_mult 55 | 56 | super().step(closure) 57 | 58 | self.global_step += 1 59 | 60 | 61 | class SGDROptimizer(torch.optim.SGD): 62 | def __init__(self, params, steps_per_epoch, lr=0, weight_decay=0, epoch_start=1, restart_mult=2): 63 | super().__init__(params, lr, weight_decay) 64 | 65 | self.global_step = 0 66 | self.local_step = 0 67 | self.total_restart = 0 68 | 69 | self.max_step = steps_per_epoch * epoch_start 70 | self.restart_mult = restart_mult 71 | 72 | self.__initial_lr = [group['lr'] for group in self.param_groups] 73 | 74 | 75 | def step(self, closure=None): 76 | 77 | if self.local_step >= self.max_step: 78 | self.local_step = 0 79 | self.max_step *= self.restart_mult 80 | self.total_restart += 1 81 | 82 | lr_mult = (1 + math.cos(math.pi * self.local_step / self.max_step))/2 / (self.total_restart + 1) 83 | 84 | for i in range(len(self.param_groups)): 85 | self.param_groups[i]['lr'] = self.__initial_lr[i] * lr_mult 86 | 87 | super().step(closure) 88 | 89 | self.local_step += 1 90 | self.global_step += 1 91 | 92 | 93 | def split_dataset(dataset, n_splits): 94 | 95 | return [Subset(dataset, np.arange(i, len(dataset), n_splits)) for i in range(n_splits)] 96 | 97 | 98 | def gap2d(x, keepdims=False): 99 | out = torch.mean(x.view(x.size(0), x.size(1), -1), -1) 100 | if keepdims: 101 | out = out.view(out.size(0), out.size(1), 1, 1) 102 | 103 | return out 104 | 105 | def gap2d_pos(x, keepdims=False): 106 | out = torch.sum(x.view(x.size(0), x.size(1), -1), -1) / (torch.sum(x>0)+1e-12) 107 | if keepdims: 108 | out = out.view(out.size(0), out.size(1), 1, 1) 109 | 110 | return out 111 | 112 | def gsp2d(x, keepdims=False): 113 | out = torch.sum(x.view(x.size(0), x.size(1), -1), -1) 114 | if keepdims: 115 | out = out.view(out.size(0), out.size(1), 1, 1) 116 | 117 | return out 118 | 119 | class BatchNorm2dFixed(torch.nn.Module): 120 | 121 | def __init__(self, num_features, eps=1e-5): 122 | super(BatchNorm2dFixed, self).__init__() 123 | self.num_features = num_features 124 | self.eps = eps 125 | self.weight = torch.nn.Parameter(torch.Tensor(num_features)) 126 | self.bias = torch.nn.Parameter(torch.Tensor(num_features)) 127 | self.register_buffer('running_mean', torch.zeros(num_features)) 128 | self.register_buffer('running_var', torch.ones(num_features)) 129 | 130 | 131 | def forward(self, input): 132 | 133 | return F.batch_norm( 134 | input, self.running_mean, self.running_var, self.weight, self.bias, 135 | False, eps=self.eps) 136 | 137 | def __call__(self, x): 138 | return self.forward(x) 139 | 140 | -------------------------------------------------------------------------------- /tool/visualization.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import torch 3 | import torch.nn.functional as F 4 | import cv2 5 | import pydensecrf.densecrf as dcrf 6 | from pydensecrf.utils import unary_from_softmax 7 | 8 | def color_pro(pro, img=None, mode='hwc'): 9 | H, W = pro.shape 10 | pro_255 = (pro*255).astype(np.uint8) 11 | pro_255 = np.expand_dims(pro_255,axis=2) 12 | color = cv2.applyColorMap(pro_255,cv2.COLORMAP_JET) 13 | color = cv2.cvtColor(color, cv2.COLOR_BGR2RGB) 14 | if img is not None: 15 | rate = 0.5 16 | if mode == 'hwc': 17 | assert img.shape[0] == H and img.shape[1] == W 18 | color = cv2.addWeighted(img,rate,color,1-rate,0) 19 | elif mode == 'chw': 20 | assert img.shape[1] == H and img.shape[2] == W 21 | img = np.transpose(img,(1,2,0)) 22 | color = cv2.addWeighted(img,rate,color,1-rate,0) 23 | color = np.transpose(color,(2,0,1)) 24 | else: 25 | if mode == 'chw': 26 | color = np.transpose(color,(2,0,1)) 27 | return color 28 | 29 | def generate_vis(p, gt, img, func_label2color, threshold=0.1, norm=True): 30 | # All the input should be numpy.array 31 | # img should be 0-255 uint8 32 | C, H, W = p.shape 33 | 34 | if norm: 35 | prob = max_norm(p, 'numpy') 36 | else: 37 | prob = p 38 | if gt is not None: 39 | prob = prob * gt 40 | prob[prob<=0] = 1e-7 41 | if threshold is not None: 42 | prob[0,:,:] = np.power(1-np.max(prob[1:,:,:],axis=0,keepdims=True), 4) 43 | 44 | CAM = ColorCAM(prob, img) 45 | 46 | return CAM 47 | 48 | def max_norm(p, version='torch', e=1e-5): 49 | if version is 'torch': 50 | if p.dim() == 3: 51 | C, H, W = p.size() 52 | p = F.relu(p) 53 | max_v = torch.max(p.view(C,-1),dim=-1)[0].view(C,1,1) 54 | min_v = torch.min(p.view(C,-1),dim=-1)[0].view(C,1,1) 55 | p = F.relu(p-min_v-e)/(max_v-min_v+e) 56 | elif p.dim() == 4: 57 | N, C, H, W = p.size() 58 | p = F.relu(p) 59 | max_v = torch.max(p.view(N,C,-1),dim=-1)[0].view(N,C,1,1) 60 | min_v = torch.min(p.view(N,C,-1),dim=-1)[0].view(N,C,1,1) 61 | p = F.relu(p-min_v-e)/(max_v-min_v+e) 62 | elif version is 'numpy' or version is 'np': 63 | if p.ndim == 3: 64 | C, H, W = p.shape 65 | p[p<0] = 0 66 | max_v = np.max(p,(1,2),keepdims=True) 67 | min_v = np.min(p,(1,2),keepdims=True) 68 | p[p 255] = 255 132 | img_8[img_8 < 0] = 0 133 | img_8 = img_8.astype(np.uint8) 134 | img_8 = cv2.resize(img_8, (img_8.shape[1]//2, img_8.shape[0]//2)).transpose((2,0,1)) 135 | 136 | return img_8 137 | -------------------------------------------------------------------------------- /train_irn.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import torch 3 | import os 4 | from torch.backends import cudnn 5 | cudnn.enabled = True 6 | from torch.utils.data import DataLoader 7 | from data import data_voc 8 | from tool import pyutils, torchutils, indexing 9 | import importlib 10 | from PIL import ImageFile 11 | ImageFile.LOAD_TRUNCATED_IMAGES = True 12 | def run(args): 13 | ir_label_out_dir = os.path.join(args.session_name, 'ir_label') 14 | 15 | path_index = indexing.PathIndex(radius=10, default_size=(args.irn_crop_size // 4, args.irn_crop_size // 4)) 16 | 17 | model = getattr(importlib.import_module(args.irn_network), 'AffinityDisplacementLoss')(path_index) 18 | 19 | train_dataset = data_voc.VOC12AffinityDataset(args.train_list, 20 | label_dir=ir_label_out_dir, 21 | voc12_root=args.voc12_root, 22 | indices_from=path_index.src_indices, 23 | indices_to=path_index.dst_indices, 24 | hor_flip=True, 25 | crop_size=args.irn_crop_size, 26 | crop_method="random", 27 | rescale=(0.5, 1.5) 28 | ) 29 | train_data_loader = DataLoader(train_dataset, batch_size=args.irn_batch_size, 30 | shuffle=True, num_workers=args.num_workers, pin_memory=True, drop_last=True) 31 | 32 | max_step = (len(train_dataset) // args.irn_batch_size) * args.irn_num_epoches 33 | 34 | param_groups = model.trainable_parameters() 35 | optimizer = torchutils.PolyOptimizerSGD([ 36 | {'params': param_groups[0], 'lr': 1*args.irn_learning_rate, 'weight_decay': args.irn_weight_decay}, 37 | {'params': param_groups[1], 'lr': 10*args.irn_learning_rate, 'weight_decay': args.irn_weight_decay} 38 | ], lr=args.irn_learning_rate, weight_decay=args.irn_weight_decay, max_step=max_step) 39 | 40 | model = torch.nn.DataParallel(model).cuda() 41 | model.train() 42 | 43 | avg_meter = pyutils.AverageMeter() 44 | 45 | timer = pyutils.Timer() 46 | 47 | for ep in range(args.irn_num_epoches): 48 | 49 | print('Epoch %d/%d' % (ep+1, args.irn_num_epoches)) 50 | 51 | for iter, pack in enumerate(train_data_loader): 52 | 53 | img = pack['img'].cuda(non_blocking=True) 54 | bg_pos_label = pack['aff_bg_pos_label'].cuda(non_blocking=True) 55 | fg_pos_label = pack['aff_fg_pos_label'].cuda(non_blocking=True) 56 | neg_label = pack['aff_neg_label'].cuda(non_blocking=True) 57 | 58 | pos_aff_loss, neg_aff_loss, dp_fg_loss, dp_bg_loss = model(img, True) 59 | 60 | bg_pos_aff_loss = torch.sum(bg_pos_label * pos_aff_loss) / (torch.sum(bg_pos_label) + 1e-5) 61 | fg_pos_aff_loss = torch.sum(fg_pos_label * pos_aff_loss) / (torch.sum(fg_pos_label) + 1e-5) 62 | pos_aff_loss = bg_pos_aff_loss / 2 + fg_pos_aff_loss / 2 63 | neg_aff_loss = torch.sum(neg_label * neg_aff_loss) / (torch.sum(neg_label) + 1e-5) 64 | 65 | dp_fg_loss = torch.sum(dp_fg_loss * torch.unsqueeze(fg_pos_label, 1)) / (2 * torch.sum(fg_pos_label) + 1e-5) 66 | dp_bg_loss = torch.sum(dp_bg_loss * torch.unsqueeze(bg_pos_label, 1)) / (2 * torch.sum(bg_pos_label) + 1e-5) 67 | 68 | avg_meter.add({'loss1': pos_aff_loss.item(), 'loss2': neg_aff_loss.item(), 69 | 'loss3': dp_fg_loss.item(), 'loss4': dp_bg_loss.item()}) 70 | 71 | total_loss = (pos_aff_loss + neg_aff_loss) / 2 + (dp_fg_loss + dp_bg_loss) / 2 72 | 73 | optimizer.zero_grad() 74 | total_loss.backward() 75 | optimizer.step() 76 | 77 | if (optimizer.global_step - 1) % 10 == 0: 78 | timer.update_progress(optimizer.global_step / max_step) 79 | 80 | print('step:%5d/%5d' % (optimizer.global_step - 1, max_step), 81 | 'loss:%.4f %.4f %.4f %.4f' % ( 82 | avg_meter.pop('loss1'), avg_meter.pop('loss2'), avg_meter.pop('loss3'), avg_meter.pop('loss4')), 83 | 'imps:%.1f' % ((iter + 1) * args.irn_batch_size / timer.get_stage_elapsed()), 84 | 'lr: %.4f' % (optimizer.param_groups[0]['lr']), 85 | 'etc:%s' % (timer.str_est_finish()), flush=True) 86 | else: 87 | timer.reset_stage() 88 | 89 | infer_dataset = data_voc.VOC12ImageDataset(args.infer_list, 90 | voc12_root=args.voc12_root, 91 | crop_size=args.irn_crop_size, 92 | crop_method="top_left") 93 | infer_data_loader = DataLoader(infer_dataset, batch_size=args.irn_batch_size, 94 | shuffle=False, num_workers=args.num_workers, pin_memory=True, drop_last=True) 95 | 96 | model.eval() 97 | print('Analyzing displacements mean ... ', end='') 98 | 99 | dp_mean_list = [] 100 | 101 | with torch.no_grad(): 102 | for iter, pack in enumerate(infer_data_loader): 103 | img = pack['img'].cuda(non_blocking=True) 104 | 105 | aff, dp = model(img, False) 106 | 107 | dp_mean_list.append(torch.mean(dp, dim=(0, 2, 3)).cpu()) 108 | 109 | model.module.mean_shift.running_mean = torch.mean(torch.stack(dp_mean_list), dim=0) 110 | print('done.') 111 | 112 | torch.save(model.module.state_dict(), os.path.join(args.session_name, 'ckpt', 'irn.pth')) 113 | torch.cuda.empty_cache() 114 | 115 | if __name__ == '__main__': 116 | 117 | parser = argparse.ArgumentParser() 118 | # Inter-pixel Relation Network (IRNet) 119 | parser.add_argument("--num_workers", default=1, type=int) 120 | parser.add_argument("--train_list", default="data/trainaug_voc.txt", type=str) 121 | parser.add_argument("--infer_list", default="data/train_voc.txt", type=str) 122 | parser.add_argument("--session_name", default="exp", type=str) 123 | parser.add_argument("--voc12_root", default="../PascalVOC2012/VOCdevkit/VOC2012", type=str) 124 | 125 | parser.add_argument("--irn_network", default="network.resnet50_irn", type=str) 126 | parser.add_argument("--irn_crop_size", default=512, type=int) 127 | parser.add_argument("--irn_batch_size", default=32, type=int) 128 | parser.add_argument("--irn_num_epoches", default=3, type=int) 129 | parser.add_argument("--irn_learning_rate", default=0.1, type=float) 130 | parser.add_argument("--irn_weight_decay", default=1e-4, type=float) 131 | args = parser.parse_args() 132 | 133 | run(args) 134 | -------------------------------------------------------------------------------- /train_resnet50_SIPE.py: -------------------------------------------------------------------------------- 1 | import torch, os 2 | from torch.backends import cudnn 3 | cudnn.enabled = True 4 | from torch.utils.data import DataLoader 5 | import torch.nn.functional as F 6 | import argparse 7 | import importlib 8 | import numpy as np 9 | from tensorboardX import SummaryWriter 10 | from data import data_coco, data_voc 11 | from chainercv.datasets import VOCSemanticSegmentationDataset 12 | from chainercv.evaluations import calc_semantic_segmentation_confusion 13 | from tool import pyutils, torchutils, visualization, imutils 14 | import random 15 | 16 | def validate(model, data_loader): 17 | 18 | gt_dataset = VOCSemanticSegmentationDataset(split='train', data_dir="../PascalVOC2012/VOCdevkit/VOC2012") 19 | labels = [gt_dataset.get_example_by_keys(i, (1,))[0] for i in range(len(gt_dataset))] 20 | 21 | print('validating ... ', flush=True, end='') 22 | 23 | model.eval() 24 | 25 | with torch.no_grad(): 26 | preds = [] 27 | for iter, pack in enumerate(data_loader): 28 | img = pack['img'].cuda() 29 | label = pack['label'].cuda(non_blocking=True).unsqueeze(-1).unsqueeze(-1) 30 | label = F.pad(label, (0, 0, 0, 0, 1, 0), 'constant', 1.0) 31 | 32 | outputs = model.forward(img, label) 33 | IS_cam = outputs['IS_cam'] 34 | IS_cam = F.interpolate(IS_cam, img.shape[2:], mode='bilinear') 35 | IS_cam = IS_cam/(F.adaptive_max_pool2d(IS_cam, (1, 1)) + 1e-5) 36 | cls_labels_bkg = torch.argmax(IS_cam, 1) 37 | 38 | preds.append(cls_labels_bkg[0].cpu().numpy().copy()) 39 | 40 | confusion = calc_semantic_segmentation_confusion(preds, labels) 41 | gtj = confusion.sum(axis=1) 42 | resj = confusion.sum(axis=0) 43 | gtjresj = np.diag(confusion) 44 | denominator = gtj + resj - gtjresj 45 | iou = gtjresj / denominator 46 | print({'iou': iou, 'miou': np.nanmean(iou)}) 47 | 48 | model.train() 49 | 50 | return np.nanmean(iou) 51 | 52 | 53 | def setup_seed(seed): 54 | print("random seed is set to", seed) 55 | torch.manual_seed(seed) 56 | torch.cuda.manual_seed_all(seed) 57 | np.random.seed(seed) 58 | random.seed(seed) 59 | torch.backends.cudnn.deterministic = True 60 | 61 | 62 | def train(): 63 | 64 | parser = argparse.ArgumentParser() 65 | parser.add_argument("--batch_size", default=16, type=int) 66 | parser.add_argument("--max_epoches", default=5, type=int) 67 | parser.add_argument("--network", default="network.resnet50_SIPE", type=str) 68 | parser.add_argument("--lr", default=0.1, type=float) 69 | parser.add_argument("--num_workers", default=8, type=int) 70 | parser.add_argument("--wt_dec", default=1e-4, type=float) 71 | parser.add_argument("--session_name", default="exp", type=str) 72 | parser.add_argument("--crop_size", default=512, type=int) 73 | parser.add_argument("--print_freq", default=10, type=int) 74 | parser.add_argument("--tf_freq", default=500, type=int) 75 | parser.add_argument("--val_freq", default=500, type=int) 76 | parser.add_argument("--dataset", default="voc", type=str) 77 | parser.add_argument("--dataset_root", default="../PascalVOC2012/VOCdevkit/VOC2012", type=str) 78 | parser.add_argument("--seed", default=15, type=int) 79 | args = parser.parse_args() 80 | 81 | setup_seed(args.seed) 82 | os.makedirs(args.session_name, exist_ok=True) 83 | os.makedirs(os.path.join(args.session_name, 'runs'), exist_ok=True) 84 | os.makedirs(os.path.join(args.session_name, 'ckpt'), exist_ok=True) 85 | pyutils.Logger(os.path.join(args.session_name, args.session_name + '.log')) 86 | tblogger = SummaryWriter(os.path.join(args.session_name, 'runs')) 87 | 88 | assert args.dataset in ['voc', 'coco'], 'Dataset must be voc or coco in this project.' 89 | 90 | if args.dataset == 'voc': 91 | dataset_root = '../PascalVOC2012/VOCdevkit/VOC2012' 92 | model = getattr(importlib.import_module(args.network), 'Net')(num_cls=21) 93 | train_dataset = data_voc.VOC12ClsDataset('data/trainaug_' + args.dataset + '.txt', voc12_root=dataset_root, 94 | resize_long=(320, 640), hor_flip=True, 95 | crop_size=512, crop_method="random") 96 | train_data_loader = DataLoader(train_dataset, batch_size=args.batch_size, 97 | shuffle=True, num_workers=args.num_workers, pin_memory=True, drop_last=True) 98 | max_step = (len(train_dataset) // args.batch_size) * args.max_epoches 99 | 100 | val_dataset = data_voc.VOC12ClsDataset('data/train_' + args.dataset + '.txt', voc12_root=dataset_root) 101 | val_data_loader = DataLoader(val_dataset, batch_size=1, shuffle=False, num_workers=args.num_workers, pin_memory=True, drop_last=True) 102 | 103 | elif args.dataset == 'coco': 104 | args.tf_freq = 99999 105 | args.val_freq = 99999 106 | dataset_root = '../ms_coco_14&15/images' 107 | model = getattr(importlib.import_module(args.network), 'Net')(num_cls=81) 108 | train_dataset = data_coco.COCOClsDataset('data/train_' + args.dataset + '.txt', coco_root=dataset_root, 109 | resize_long=(320, 640), hor_flip=True, 110 | crop_size=512, crop_method="random") 111 | train_data_loader = DataLoader(train_dataset, batch_size=args.batch_size, 112 | shuffle=True, num_workers=args.num_workers, pin_memory=True, drop_last=True) 113 | max_step = (len(train_dataset) // args.batch_size) * args.max_epoches 114 | 115 | val_dataset = data_coco.COCOClsDataset('data/train_' + args.dataset + '.txt', coco_root=dataset_root) 116 | val_data_loader = DataLoader(val_dataset, batch_size=1, shuffle=False, num_workers=args.num_workers, pin_memory=True, drop_last=True) 117 | 118 | param_groups = model.trainable_parameters() 119 | optimizer = torchutils.PolyOptimizerSGD([ 120 | {'params': param_groups[0], 'lr': args.lr, 'weight_decay': args.wt_dec}, 121 | {'params': param_groups[1], 'lr': 10 * args.lr, 'weight_decay': args.wt_dec} 122 | ], lr=args.lr, weight_decay=args.wt_dec, max_step=max_step) 123 | 124 | model = torch.nn.DataParallel(model).cuda() 125 | model.train() 126 | 127 | avg_meter = pyutils.AverageMeter() 128 | timer = pyutils.Timer() 129 | 130 | bestiou = 0 131 | for ep in range(args.max_epoches): 132 | 133 | print('Epoch %d/%d' % (ep + 1, args.max_epoches)) 134 | 135 | for step, pack in enumerate(train_data_loader): 136 | img = pack['img'].cuda() 137 | n, c, h, w = img.shape 138 | label = pack['label'].cuda(non_blocking=True).unsqueeze(-1).unsqueeze(-1) 139 | 140 | valid_mask = pack['valid_mask'].cuda() 141 | valid_mask[:,1:] = valid_mask[:,1:] * label 142 | valid_mask_lowres = F.interpolate(valid_mask, size=(h//16, w//16), mode='nearest') 143 | 144 | outputs = model.forward(img, valid_mask_lowres) 145 | score = outputs['score'] 146 | norm_cam = outputs['cam'] 147 | IS_cam = outputs['IS_cam'] 148 | 149 | lossCLS = F.multilabel_soft_margin_loss(score, label) 150 | 151 | IS_cam = IS_cam / (F.adaptive_max_pool2d(IS_cam, (1, 1)) + 1e-5) 152 | lossGSC = torch.mean(torch.abs(norm_cam - IS_cam)) 153 | 154 | losses = lossCLS + lossGSC 155 | avg_meter.add({'lossCLS': lossCLS.item(), 'lossGSC': lossGSC.item()}) 156 | 157 | optimizer.zero_grad() 158 | losses.backward() 159 | torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1) 160 | optimizer.step() 161 | 162 | if (optimizer.global_step - 1) % args.print_freq == 0: 163 | timer.update_progress(optimizer.global_step / max_step) 164 | 165 | print('step:%5d/%5d' % (optimizer.global_step - 1, max_step), 166 | 'lossCLS:%.4f' % (avg_meter.pop('lossCLS')), 167 | 'lossGSC:%.4f' % (avg_meter.pop('lossGSC')), 168 | 'imps:%.1f' % ((step + 1) * args.batch_size / timer.get_stage_elapsed()), 169 | 'lr: %.4f' % (optimizer.param_groups[0]['lr']), 170 | 'etc:%s' % (timer.str_est_finish()), flush=True) 171 | 172 | # tf record 173 | tblogger.add_scalar('lossCLS', lossCLS, optimizer.global_step) 174 | tblogger.add_scalar('lossGSC', lossGSC, optimizer.global_step) 175 | tblogger.add_scalar('lr', optimizer.param_groups[0]['lr'], optimizer.global_step) 176 | 177 | if (optimizer.global_step - 1) % args.tf_freq == 0: 178 | # visualization 179 | img_8 = visualization.convert_to_tf(img[0]) 180 | norm_cam = F.interpolate(norm_cam,img_8.shape[1:],mode='bilinear')[0].detach().cpu().numpy() 181 | IS_cam = F.interpolate(IS_cam,img_8.shape[1:],mode='bilinear')[0].detach().cpu().numpy() 182 | CAM = visualization.generate_vis(norm_cam, None, img_8, func_label2color=visualization.VOClabel2colormap, threshold=None, norm=False) 183 | IS_CAM = visualization.generate_vis(IS_cam, None, img_8, func_label2color=visualization.VOClabel2colormap, threshold=None, norm=False) 184 | 185 | # tf record 186 | tblogger.add_images('CAM', CAM, optimizer.global_step) 187 | tblogger.add_images('IS_CAM', IS_CAM, optimizer.global_step) 188 | 189 | if (optimizer.global_step-1) % args.val_freq == 0 and optimizer.global_step > 10: 190 | miou = validate(model, val_data_loader) 191 | torch.save({'net':model.module.state_dict()}, os.path.join(args.session_name, 'ckpt', 'iter_' + str(optimizer.global_step) + '.pth')) 192 | if miou > bestiou: 193 | bestiou = miou 194 | torch.save({'net':model.module.state_dict()}, os.path.join(args.session_name, 'ckpt', 'best.pth')) 195 | 196 | else: 197 | timer.reset_stage() 198 | 199 | torch.save({'net':model.module.state_dict()}, os.path.join(args.session_name, 'ckpt', 'final.pth')) 200 | torch.cuda.empty_cache() 201 | 202 | if __name__ == '__main__': 203 | 204 | train() 205 | --------------------------------------------------------------------------------