├── LICENSE ├── README.md ├── cam2ir.py ├── data ├── cls_labels_voc.npy ├── data_coco.py └── trainaug_voc.txt ├── eval_cam.py ├── eval_sem_seg.py ├── exp ├── eval_exp.log ├── exp.log └── infer.log ├── fig └── framework1.png ├── make_cam.py ├── make_seg_labels.py ├── network ├── resnet50.py ├── resnet50_SFC.py ├── resnet50_SIPE.py └── resnet50_irn.py ├── requirements.txt ├── run_coco.sh ├── run_voc.sh ├── run_voc2.sh ├── train_irn.py └── train_resnet50_SFC.py /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 chenqi1126 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # SFC: Shared Feature Calibration in Weakly Supervised Semantic Segmentation 2 | 3 | framework 4 | 5 | The implementation of [**SFC: Shared Feature Calibration in Weakly Supervised Semantic Segmentation**](https://arxiv.org/pdf/2401.11719.pdf), Xinqiao Zhao*, Feilong Tang*, Xiaoyang Wang, Jimin Xiao, AAAI 2024. 6 | 7 | ## Abstract 8 | Image-level weakly supervised semantic segmentation has received increasing attention due to its low annotation cost. Existing methods mainly rely on Class Activation Mapping (CAM) to obtain pseudo labels for training semantic segmentation models. In this work, we demonstrate that the long-tailed distribution in the training set can cause the CAM calculated through the classifier weights over-activated for the head classes and under-activated for the tail classes due to the shared features among different classes. This degrades the CAM quality and further influences the final semantic segmentation performances. To address this issue, we propose a Shared Feature Calibration (SFC) method for CAM generation. Specifically, we leverage the class prototypes which carry positive shared features and propose a Multi-Scaled ROCon-Weighted (MSRW) consistency loss for narrowing the gap between the CAMs generated through the classifier weights and class prototypes during training, and further combine these CAMs during inference, thereby solving the head-class over-activation and tail-class under-activation issues. Experimental results show that our SFC can significantly improve the CAM boundaries and achieve new state-of-the-art performances. 9 | 10 | ## Environment 11 | 12 | - Python >= 3.6.6 13 | - Pytorch >= 1.6.0 14 | - Torchvision 15 | 16 | ## Usage 17 | 18 | #### Step 1. Prepare Dataset 19 | 20 | - PASCAL VOC 2012: [Download](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/). 21 | - MS COCO 2014: [Image](https://cocodataset.org/#home) and [Label](https://drive.google.com/file/d/1Pm_OH8an5MzZh56QKTcdlXNI3RNmZB9d/view?usp=sharing). 22 | 23 | #### Step 2. Train SFC 24 | 25 | ```bash 26 | # PASCAL VOC 2012 27 | bash run_voc.sh 28 | 29 | # MS COCO 2014 30 | bash run_coco.sh 31 | ``` 32 | 33 | #### Step 3. Train Fully Supervised Segmentation Models 34 | 35 | To train fully supervised segmentation models, we refer to [deeplab-pytorch](https://github.com/kazuto1011/deeplab-pytorch) and [seamv1](https://github.com/YudeWang/semantic-segmentation-codebase/tree/main/experiment/seamv1-pseudovoc). 36 | 37 | ## Citation 38 | ``` 39 | @inproceedings{zhao2024sfc, 40 | title={Sfc: Shared feature calibration in weakly supervised semantic segmentation}, 41 | author={Zhao, Xinqiao and Tang, Feilong and Wang, Xiaoyang and Xiao, Jimin}, 42 | booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, 43 | volume={38}, 44 | number={7}, 45 | pages={7525--7533}, 46 | year={2024} 47 | } 48 | ``` 49 | -------------------------------------------------------------------------------- /cam2ir.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | import imageio 4 | from torch import multiprocessing 5 | from torch.utils.data import DataLoader 6 | from data import data_voc, data_coco 7 | from tool import torchutils, imutils 8 | import argparse 9 | import torch 10 | 11 | def _work(process_id, infer_dataset, args): 12 | 13 | databin = infer_dataset[process_id] 14 | infer_data_loader = DataLoader(databin, shuffle=False, num_workers=0, pin_memory=False) 15 | cam_out_dir = os.path.join(args.session_name, 'npy') 16 | ir_label_out_dir = os.path.join(args.session_name, 'ir_label') 17 | 18 | for iter, pack in enumerate(infer_data_loader): 19 | img_name = pack['name'][0] 20 | img = pack['img'][0].numpy() 21 | cam_dict = np.load(os.path.join(cam_out_dir, img_name + '.npy'), allow_pickle=True).item() 22 | # cams = cam_dict['IS_CAM'][1:,...] 23 | # cams = cam_dict['IS_CAM'] 24 | keys = cam_dict['keys'] 25 | 26 | cams = cam_dict['IS_CAM1'] + cam_dict['IS_CAM']*0.7 27 | 28 | # 1. find confident fg & bg 29 | fg_conf_cam = cams 30 | fg_conf_cam = np.argmax(fg_conf_cam, axis=0) 31 | pred = imutils.crf_inference_label(img, fg_conf_cam, n_labels=keys.shape[0]) 32 | fg_conf = keys[pred] 33 | 34 | bg_conf_cam = cams 35 | bg_conf_cam[0] = bg_conf_cam[0]*0.5 36 | bg_conf_cam = np.argmax(bg_conf_cam, axis=0) 37 | pred = imutils.crf_inference_label(img, bg_conf_cam, n_labels=keys.shape[0]) 38 | bg_conf = keys[pred] 39 | 40 | # 2. combine confident fg & bg 41 | conf = fg_conf.copy() 42 | conf[fg_conf == 0] = 255 43 | conf[bg_conf + fg_conf == 0] = 0 44 | 45 | imageio.imwrite(os.path.join(ir_label_out_dir, img_name + '.png'), conf.astype(np.uint8)) 46 | 47 | if process_id == args.num_workers - 1 and iter % (len(databin) // 20) == 0: 48 | print("%d " % ((5 * iter + 1) // (len(databin) // 20)), end='') 49 | 50 | 51 | if __name__ == '__main__': 52 | 53 | parser = argparse.ArgumentParser() 54 | parser.add_argument("--dataset", default="voc", type=str) 55 | parser.add_argument("--num_workers", default=os.cpu_count()//2, type=int) 56 | parser.add_argument("--session_name", default='exp', type=str) 57 | args = parser.parse_args() 58 | 59 | assert args.dataset in ['voc', 'coco'], 'Dataset must be voc or coco in this project.' 60 | 61 | if args.dataset == 'voc': 62 | dataset_root = '/data/tfl/VOCdevkit/VOC2012' 63 | dataset = data_voc.VOC12ImageDataset('data/trainaug_' + args.dataset + '.txt', voc12_root=dataset_root, img_normal=None, to_torch=False) 64 | 65 | elif args.dataset == 'coco': 66 | dataset_root = "../../dataset/ms_coco_14&15/images" 67 | dataset = data_coco.COCOImageDataset('data/train_' + args.dataset + '.txt', coco_root=dataset_root, img_normal=None, to_torch=False) 68 | 69 | dataset = torchutils.split_dataset(dataset, args.num_workers) 70 | os.makedirs(os.path.join(args.session_name, 'ir_label'), exist_ok=True) 71 | 72 | print('[ ', end='') 73 | multiprocessing.spawn(_work, nprocs=args.num_workers, args=(dataset, args), join=True) 74 | print(']') 75 | -------------------------------------------------------------------------------- /data/cls_labels_voc.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Barrett-python/SFC/4f624a92aea10d8be2049f6cff53639d31507136/data/cls_labels_voc.npy -------------------------------------------------------------------------------- /data/data_coco.py: -------------------------------------------------------------------------------- 1 | 2 | import numpy as np 3 | import torch 4 | from torch.utils.data import Dataset 5 | import PIL.Image 6 | import os.path 7 | import scipy.misc 8 | import imageio 9 | from torchvision import transforms 10 | from tool import pyutils, imutils, torchutils 11 | import pdb 12 | import cv2 13 | import random 14 | 15 | IMG_FOLDER_NAME = "JPEGImages" 16 | MASK_FOLDER_NAME = "SegmentationClass" 17 | 18 | 19 | def load_image_label_list_from_npy(img_name_list): 20 | 21 | cls_labels_dict = np.load('data/cls_labels_coco.npy', allow_pickle=True).item() 22 | 23 | return [cls_labels_dict[img_name] for img_name in img_name_list] 24 | 25 | 26 | def get_img_path(img_name, coco_root): 27 | return os.path.join(coco_root, 'train2014', img_name + '.jpg') 28 | 29 | def get_mask_path(mask_name, coco_root): 30 | return os.path.join(coco_root, MASK_FOLDER_NAME, mask_name + '.png') 31 | 32 | def load_img_name_list(dataset_path): 33 | 34 | img_gt_name_list = open(dataset_path).read().splitlines() 35 | img_name_list = [img_gt_name for img_gt_name in img_gt_name_list] 36 | 37 | return img_name_list 38 | 39 | 40 | class TorchvisionNormalize(): 41 | def __init__(self, mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)): 42 | self.mean = mean 43 | self.std = std 44 | 45 | def __call__(self, img): 46 | imgarr = np.asarray(img) 47 | proc_img = np.empty_like(imgarr, np.float32) 48 | 49 | proc_img[..., 0] = (imgarr[..., 0] / 255. - self.mean[0]) / self.std[0] 50 | proc_img[..., 1] = (imgarr[..., 1] / 255. - self.mean[1]) / self.std[1] 51 | proc_img[..., 2] = (imgarr[..., 2] / 255. - self.mean[2]) / self.std[2] 52 | 53 | return proc_img 54 | 55 | class COCOImageDataset(Dataset): 56 | def __init__(self, img_name_list_path, coco_root, resize=None, 57 | resize_long=None, rescale=None, img_normal=TorchvisionNormalize(), hor_flip=False, 58 | crop_size=None, crop_method=None, to_torch=True): 59 | self.img_name_list = load_img_name_list(img_name_list_path) 60 | self.coco_root = coco_root 61 | 62 | self.resize = resize 63 | self.resize_long = resize_long 64 | self.rescale = rescale 65 | self.crop_size = crop_size 66 | self.img_normal = img_normal 67 | self.hor_flip = hor_flip 68 | self.crop_method = crop_method 69 | self.to_torch = to_torch 70 | 71 | def __len__(self): 72 | return len(self.img_name_list) 73 | 74 | 75 | def __getitem__(self, idx): 76 | name = self.img_name_list[idx] 77 | 78 | img = np.array(PIL.Image.open(get_img_path(name, self.coco_root)).convert('RGB')) 79 | 80 | if self.resize: 81 | img = imutils.pil_resize(img, size=self.resize, order=3) 82 | 83 | if self.resize_long: 84 | img = imutils.random_resize_long(img, self.resize_long[0], self.resize_long[1]) 85 | 86 | if self.rescale: 87 | img = imutils.random_scale(img, scale_range=self.rescale, order=3) 88 | 89 | if self.img_normal: 90 | img = self.img_normal(img) 91 | 92 | if self.hor_flip: 93 | img = imutils.random_lr_flip(img) 94 | 95 | if self.crop_size: 96 | valid_mask = torch.zeros((81, self.crop_size, self.crop_size)) 97 | if self.crop_method == "random": 98 | img, box = imutils.random_crop(img, self.crop_size, 0) 99 | valid_mask[:,box[0]:box[1],box[2]:box[3]] = 1 100 | else: 101 | img = imutils.top_left_crop(img, self.crop_size, 0) 102 | valid_mask = torch.ones((81, self.crop_size, self.crop_size)) 103 | 104 | else: 105 | valid_mask = torch.ones((81, img.shape[0], img.shape[1])) 106 | 107 | if self.to_torch: 108 | img = np.ascontiguousarray(imutils.HWC_to_CHW(img)) 109 | 110 | return {'name': name, 'img': img, 'valid_mask': valid_mask} 111 | 112 | class COCOClsDataset(COCOImageDataset): 113 | 114 | def __init__(self, img_name_list_path, coco_root, resize=None, 115 | resize_long=None, rescale=None, img_normal=TorchvisionNormalize(), hor_flip=False, 116 | crop_size=None, crop_method=None): 117 | super().__init__(img_name_list_path, coco_root, resize, 118 | resize_long, rescale, img_normal, hor_flip, 119 | crop_size, crop_method) 120 | self.label_list = load_image_label_list_from_npy(self.img_name_list) 121 | 122 | def __getitem__(self, idx): 123 | out = super().__getitem__(idx) 124 | 125 | out['label'] = torch.from_numpy(self.label_list[idx]) 126 | 127 | return out 128 | 129 | class COCOClsDatasetMSF(COCOClsDataset): 130 | 131 | def __init__(self, img_name_list_path, coco_root, img_normal=TorchvisionNormalize(), scales=(1.0,)): 132 | super().__init__(img_name_list_path, coco_root) 133 | self.scales = scales 134 | 135 | def __getitem__(self, idx): 136 | name = self.img_name_list[idx] 137 | img = np.asarray(PIL.Image.open(get_img_path(name, self.coco_root)).convert('RGB')) 138 | # img = imageio.imread(get_img_path(name, self.coco_root)).convert('RGB') 139 | 140 | ms_img_list = [] 141 | for s in self.scales: 142 | if s == 1: 143 | s_img = img 144 | else: 145 | s_img = imutils.pil_rescale(img, s, order=3) 146 | s_img = self.img_normal(s_img) 147 | s_img = imutils.HWC_to_CHW(s_img) 148 | ms_img_list.append(np.stack([s_img, np.flip(s_img, -1)], axis=0)) 149 | 150 | if len(self.scales) == 1: 151 | ms_img_list = ms_img_list[0] 152 | 153 | # mask = PIL.Image.open(get_mask_path(name, self.coco_root)) 154 | 155 | out = {"name": name, "img": ms_img_list, "size": (img.shape[0], img.shape[1]), 156 | "label": torch.from_numpy(self.label_list[idx]), "img_path": get_img_path(name, self.coco_root)} 157 | 158 | 159 | return out 160 | 161 | class COCOSegmentationDataset(Dataset): 162 | 163 | def __init__(self, img_name_list_path, label_dir, crop_size, coco_root, 164 | rescale=None, img_normal=TorchvisionNormalize(), hor_flip=False, 165 | crop_method = 'random'): 166 | 167 | self.img_name_list = load_img_name_list(img_name_list_path) 168 | self.coco_root = coco_root 169 | 170 | self.label_dir = label_dir 171 | 172 | self.rescale = rescale 173 | self.crop_size = crop_size 174 | self.img_normal = img_normal 175 | self.hor_flip = hor_flip 176 | self.crop_method = crop_method 177 | 178 | def __len__(self): 179 | return len(self.img_name_list) 180 | 181 | def __getitem__(self, idx): 182 | name = self.img_name_list[idx] 183 | 184 | img = PIL.Image.open(get_img_path(name, self.coco_root)).convert('RGB') 185 | label = imageio.imread(os.path.join(self.label_dir, name + '.png')) 186 | 187 | img = np.asarray(img) 188 | 189 | if self.rescale: 190 | img, label = imutils.random_scale((img, label), scale_range=self.rescale, order=(3, 0)) 191 | 192 | if self.img_normal: 193 | img = self.img_normal(img) 194 | 195 | if self.hor_flip: 196 | img, label = imutils.random_lr_flip((img, label)) 197 | 198 | 199 | if self.crop_method == "random": 200 | img, label = imutils.random_crop((img, label), self.crop_size, (0, 255)) 201 | else: 202 | img = imutils.top_left_crop(img, self.crop_size, 0) 203 | label = imutils.top_left_crop(label, self.crop_size, 255) 204 | 205 | label = label.astype(np.long) 206 | img = imutils.HWC_to_CHW(img) 207 | return {'name': name, 'img': img, 'label': label} 208 | 209 | class GetAffinityLabelFromIndices(): 210 | 211 | def __init__(self, indices_from, indices_to): 212 | 213 | self.indices_from = indices_from 214 | self.indices_to = indices_to 215 | 216 | def __call__(self, segm_map): 217 | 218 | segm_map_flat = np.reshape(segm_map, -1) 219 | 220 | segm_label_from = np.expand_dims(segm_map_flat[self.indices_from], axis=0) 221 | segm_label_to = segm_map_flat[self.indices_to] 222 | 223 | valid_label = np.logical_and(np.less(segm_label_from, 81), np.less(segm_label_to, 81)) 224 | 225 | equal_label = np.equal(segm_label_from, segm_label_to) 226 | 227 | pos_affinity_label = np.logical_and(equal_label, valid_label) 228 | 229 | bg_pos_affinity_label = np.logical_and(pos_affinity_label, np.equal(segm_label_from, 0)).astype(np.float32) 230 | fg_pos_affinity_label = np.logical_and(pos_affinity_label, np.greater(segm_label_from, 0)).astype(np.float32) 231 | 232 | neg_affinity_label = np.logical_and(np.logical_not(equal_label), valid_label).astype(np.float32) 233 | 234 | return torch.from_numpy(bg_pos_affinity_label), torch.from_numpy(fg_pos_affinity_label), \ 235 | torch.from_numpy(neg_affinity_label) 236 | 237 | class COCOAffinityDataset(COCOSegmentationDataset): 238 | def __init__(self, img_name_list_path, label_dir, crop_size, coco_root, 239 | indices_from, indices_to, 240 | rescale=None, img_normal=TorchvisionNormalize(), hor_flip=False, crop_method=None): 241 | super().__init__(img_name_list_path, label_dir, crop_size, coco_root, rescale, img_normal, hor_flip, crop_method=crop_method) 242 | 243 | self.extract_aff_lab_func = GetAffinityLabelFromIndices(indices_from, indices_to) 244 | 245 | def __len__(self): 246 | return len(self.img_name_list) 247 | 248 | def __getitem__(self, idx): 249 | out = super().__getitem__(idx) 250 | 251 | reduced_label = imutils.pil_rescale(out['label'].astype(np.uint8), 0.25, 0) 252 | 253 | out['aff_bg_pos_label'], out['aff_fg_pos_label'], out['aff_neg_label'] = self.extract_aff_lab_func(reduced_label) 254 | 255 | return out 256 | -------------------------------------------------------------------------------- /eval_cam.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import os 3 | import argparse 4 | from PIL import Image 5 | from tool import pyutils 6 | from data import data_voc, data_coco 7 | from tqdm import tqdm 8 | from torch.utils.data import DataLoader 9 | from chainercv.evaluations import calc_semantic_segmentation_confusion 10 | import torch 11 | def run(args, predict_dir, num_cls): 12 | preds = [] 13 | masks = [] 14 | n_images = 0 15 | for iter, pack in tqdm(enumerate(dataloader)): 16 | n_images += 1 17 | cam_dict = np.load(os.path.join(predict_dir, pack['name'][0] + '.npy'), allow_pickle=True).item() 18 | keys = cam_dict['keys'] 19 | 20 | cams = cam_dict['IS_CAM'] 21 | 22 | cls_labels = np.argmax(cams, axis=0) 23 | cls_labels = keys[cls_labels] 24 | preds.append(cls_labels.copy()) 25 | mask = np.array(Image.open(os.path.join(args.gt_path, pack['name'][0] + '.png'))) 26 | masks.append(mask.copy()) 27 | 28 | confusion = calc_semantic_segmentation_confusion(preds, masks)[:num_cls, :num_cls] 29 | 30 | gtj = confusion.sum(axis=1) 31 | resj = confusion.sum(axis=0) 32 | gtjresj = np.diag(confusion) 33 | denominator = gtj + resj - gtjresj 34 | iou = gtjresj / denominator 35 | print({'iou': iou, 'miou': np.nanmean(iou)}) 36 | 37 | if __name__ == '__main__': 38 | 39 | parser = argparse.ArgumentParser() 40 | parser.add_argument("--dataset", default="voc", type=str) 41 | parser.add_argument("--gt_path", default='/data/tfl/VOCdevkit/VOC2012/SegmentationClass', type=str) 42 | parser.add_argument('--session_name', default="exp", type=str) 43 | args = parser.parse_args() 44 | 45 | assert args.dataset in ['voc', 'coco'], 'Dataset must be voc or coco in this project.' 46 | 47 | if args.dataset == 'voc': 48 | dataset_root = '/data/tfl/VOCdevkit/VOC2012' 49 | num_cls = 21 50 | dataset = data_voc.VOC12ImageDataset('data/train_' + args.dataset + '.txt', voc12_root=dataset_root, img_normal=None, to_torch=False) 51 | 52 | elif args.dataset == 'coco': 53 | args.gt_path = "../ms_coco_14&15/SegmentationClass/train2014/" 54 | dataset_root = "../ms_coco_14&15/images" 55 | num_cls = 81 56 | dataset = data_coco.COCOImageDataset('data/train_' + args.dataset + '.txt', coco_root=dataset_root, img_normal=None, to_torch=False) 57 | 58 | dataloader = DataLoader(dataset, batch_size=1, shuffle=False, num_workers=0, pin_memory=True, drop_last=True) 59 | pyutils.Logger(os.path.join(args.session_name, 'eval_' + args.session_name + '.log')) 60 | run(args, args.session_name + "/npy", num_cls) -------------------------------------------------------------------------------- /eval_sem_seg.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import os 3 | from chainercv.datasets import VOCSemanticSegmentationDataset 4 | from chainercv.evaluations import calc_semantic_segmentation_confusion 5 | import imageio 6 | import argparse 7 | 8 | def run(args): 9 | dataset = VOCSemanticSegmentationDataset(split=args.chainer_eval_set, data_dir=args.voc12_root) 10 | 11 | preds = [] 12 | labels = [] 13 | n_img = 0 14 | for i, id in enumerate(dataset.ids): 15 | cls_labels = imageio.imread(os.path.join(args.sem_seg_out_dir, id + '.png')).astype(np.uint8) 16 | cls_labels[cls_labels == 255] = 0 17 | preds.append(cls_labels.copy()) 18 | labels.append(dataset.get_example_by_keys(i, (1,))[0]) 19 | n_img += 1 20 | 21 | confusion = calc_semantic_segmentation_confusion(preds, labels)[:21, :21] 22 | 23 | gtj = confusion.sum(axis=1) 24 | resj = confusion.sum(axis=0) 25 | gtjresj = np.diag(confusion) 26 | denominator = gtj + resj - gtjresj 27 | fp = 1. - gtj / denominator 28 | fn = 1. - resj / denominator 29 | iou = gtjresj / denominator 30 | print("total images", n_img) 31 | print(fp[0], fn[0]) 32 | print(np.mean(fp[1:]), np.mean(fn[1:])) 33 | 34 | print({'iou': iou, 'miou': np.nanmean(iou)}) 35 | 36 | if __name__ == '__main__': 37 | 38 | parser = argparse.ArgumentParser() 39 | parser.add_argument("--voc12_root", default='/data/tfl/VOCdevkit/VOC2012', type=str) 40 | parser.add_argument("--gt_dir", default='/data/tfl/VOCdevkit/VOC2012/SegmentationClass', type=str) 41 | parser.add_argument("--chainer_eval_set", default="train", type=str) 42 | parser.add_argument("--sem_seg_out_dir", default="exp/pseudo_label", type=str) # pseudo_label ir_label 43 | args = parser.parse_args() 44 | run(args) 45 | -------------------------------------------------------------------------------- /exp/eval_exp.log: -------------------------------------------------------------------------------- 1 | {'iou': array([0.86857637, 0.59245783, 0.41768468, 0.70426828, 0.52627075, 2 | 0.55415618, 0.78760482, 0.74837461, 0.78535905, 0.27629848, 3 | 0.71548724, 0.39263215, 0.76212652, 0.738706 , 0.74758258, 4 | 0.68541648, 0.53580758, 0.78630819, 0.53048859, 0.6172028 , 5 | 0.48462043]), 'miou': 0.6313061713098745} 6 | -------------------------------------------------------------------------------- /exp/exp.log: -------------------------------------------------------------------------------- 1 | Epoch 1/6 2 | step: 0/ 3966 lossCLS:0.7158 lossGSC:0.0503 loss_consistency:0.0010 imps:1.4 lr: 0.1000 etc:Thu May 18 03:45:32 2023 3 | step: 10/ 3966 lossCLS:0.2786 lossGSC:0.0182 loss_consistency:0.0005 imps:8.7 lr: 0.0998 etc:Wed May 17 16:52:19 2023 4 | step: 20/ 3966 lossCLS:0.2405 lossGSC:0.0114 loss_consistency:0.0006 imps:12.1 lr: 0.0995 etc:Wed May 17 16:18:05 2023 5 | step: 30/ 3966 lossCLS:0.2214 lossGSC:0.0149 loss_consistency:0.0013 imps:14.1 lr: 0.0993 etc:Wed May 17 16:05:47 2023 6 | step: 40/ 3966 lossCLS:0.2060 lossGSC:0.0195 loss_consistency:0.0023 imps:15.4 lr: 0.0991 etc:Wed May 17 15:59:31 2023 7 | step: 50/ 3966 lossCLS:0.1673 lossGSC:0.0201 loss_consistency:0.0025 imps:16.4 lr: 0.0989 etc:Wed May 17 15:55:22 2023 8 | step: 60/ 3966 lossCLS:0.1709 lossGSC:0.0250 loss_consistency:0.0027 imps:17.1 lr: 0.0986 etc:Wed May 17 15:52:49 2023 9 | step: 70/ 3966 lossCLS:0.1233 lossGSC:0.0238 loss_consistency:0.0034 imps:17.6 lr: 0.0984 etc:Wed May 17 15:50:57 2023 10 | step: 80/ 3966 lossCLS:0.1416 lossGSC:0.0212 loss_consistency:0.0026 imps:18.0 lr: 0.0982 etc:Wed May 17 15:49:41 2023 11 | step: 90/ 3966 lossCLS:0.1308 lossGSC:0.0232 loss_consistency:0.0029 imps:18.3 lr: 0.0980 etc:Wed May 17 15:48:40 2023 12 | step: 100/ 3966 lossCLS:0.1145 lossGSC:0.0232 loss_consistency:0.0030 imps:18.6 lr: 0.0977 etc:Wed May 17 15:47:43 2023 13 | step: 110/ 3966 lossCLS:0.1239 lossGSC:0.0233 loss_consistency:0.0034 imps:18.6 lr: 0.0975 etc:Wed May 17 15:47:34 2023 14 | step: 120/ 3966 lossCLS:0.1101 lossGSC:0.0258 loss_consistency:0.0033 imps:18.9 lr: 0.0973 etc:Wed May 17 15:46:55 2023 15 | step: 130/ 3966 lossCLS:0.1339 lossGSC:0.0224 loss_consistency:0.0030 imps:19.0 lr: 0.0970 etc:Wed May 17 15:46:25 2023 16 | step: 140/ 3966 lossCLS:0.1140 lossGSC:0.0256 loss_consistency:0.0033 imps:19.2 lr: 0.0968 etc:Wed May 17 15:45:56 2023 17 | step: 150/ 3966 lossCLS:0.1105 lossGSC:0.0230 loss_consistency:0.0031 imps:19.4 lr: 0.0966 etc:Wed May 17 15:45:29 2023 18 | step: 160/ 3966 lossCLS:0.1009 lossGSC:0.0225 loss_consistency:0.0029 imps:19.5 lr: 0.0964 etc:Wed May 17 15:45:04 2023 19 | step: 170/ 3966 lossCLS:0.1167 lossGSC:0.0238 loss_consistency:0.0029 imps:19.6 lr: 0.0961 etc:Wed May 17 15:44:41 2023 20 | step: 180/ 3966 lossCLS:0.1108 lossGSC:0.0236 loss_consistency:0.0035 imps:19.8 lr: 0.0959 etc:Wed May 17 15:44:17 2023 21 | step: 190/ 3966 lossCLS:0.1056 lossGSC:0.0232 loss_consistency:0.0028 imps:19.9 lr: 0.0957 etc:Wed May 17 15:43:58 2023 22 | step: 200/ 3966 lossCLS:0.1185 lossGSC:0.0236 loss_consistency:0.0029 imps:20.0 lr: 0.0954 etc:Wed May 17 15:43:46 2023 23 | step: 210/ 3966 lossCLS:0.1062 lossGSC:0.0245 loss_consistency:0.0031 imps:20.0 lr: 0.0952 etc:Wed May 17 15:43:45 2023 24 | step: 220/ 3966 lossCLS:0.1031 lossGSC:0.0248 loss_consistency:0.0033 imps:20.1 lr: 0.0950 etc:Wed May 17 15:43:33 2023 25 | step: 230/ 3966 lossCLS:0.1023 lossGSC:0.0247 loss_consistency:0.0035 imps:20.2 lr: 0.0948 etc:Wed May 17 15:43:18 2023 26 | step: 240/ 3966 lossCLS:0.0998 lossGSC:0.0238 loss_consistency:0.0034 imps:20.2 lr: 0.0945 etc:Wed May 17 15:43:08 2023 27 | step: 250/ 3966 lossCLS:0.0844 lossGSC:0.0231 loss_consistency:0.0029 imps:20.3 lr: 0.0943 etc:Wed May 17 15:42:55 2023 28 | step: 260/ 3966 lossCLS:0.0825 lossGSC:0.0243 loss_consistency:0.0031 imps:20.4 lr: 0.0941 etc:Wed May 17 15:42:46 2023 29 | step: 270/ 3966 lossCLS:0.1137 lossGSC:0.0226 loss_consistency:0.0032 imps:20.4 lr: 0.0939 etc:Wed May 17 15:42:34 2023 30 | step: 280/ 3966 lossCLS:0.0998 lossGSC:0.0239 loss_consistency:0.0032 imps:20.5 lr: 0.0936 etc:Wed May 17 15:42:26 2023 31 | step: 290/ 3966 lossCLS:0.0907 lossGSC:0.0238 loss_consistency:0.0031 imps:20.6 lr: 0.0934 etc:Wed May 17 15:42:17 2023 32 | step: 300/ 3966 lossCLS:0.0924 lossGSC:0.0229 loss_consistency:0.0029 imps:20.6 lr: 0.0932 etc:Wed May 17 15:42:09 2023 33 | validating ... 34 | 35 | {'iou': array([0.8159207 , 0.50855972, 0.30974405, 0.61692984, 0.29964064, 36 | 0.45532438, 0.58372135, 0.57254118, 0.64133958, 0.17211568, 37 | 0.6655454 , 0.32601863, 0.64505792, 0.63945221, 0.68257063, 38 | 0.57989761, 0.44439519, 0.7683756 , 0.39670825, 0.47826575, 39 | 0.40826887]), 'miou': 0.5243044383905388} 40 | 41 | 42 | {'iou1': array([0.79762992, 0.42641051, 0.28993205, 0.48705599, 0.23642002, 43 | 0.40356063, 0.59991362, 0.5892653 , 0.69397157, 0.16198049, 44 | 0.47071778, 0.34266324, 0.66768016, 0.59344402, 0.68540185, 45 | 0.57575127, 0.3619911 , 0.69119538, 0.33996339, 0.41314576, 46 | 0.22423357]), 'miou': 0.47868226658103097} 47 | 48 | 49 | {'iou2': array([0.80677531, 0.46748511, 0.29983805, 0.55199291, 0.26803033, 50 | 0.4294425 , 0.59181748, 0.58090324, 0.66765557, 0.16704809, 51 | 0.56813159, 0.33434094, 0.65636904, 0.61644812, 0.68398624, 52 | 0.57782444, 0.40319315, 0.72978549, 0.36833582, 0.44570575, 53 | 0.31625122]), 'miou': 0.5014933524857849} 54 | step: 310/ 3966 lossCLS:0.0989 lossGSC:0.0223 loss_consistency:0.0029 imps:15.5 lr: 0.0929 etc:Wed May 17 15:58:51 2023 55 | step: 320/ 3966 lossCLS:0.0996 lossGSC:0.0225 loss_consistency:0.0028 imps:15.7 lr: 0.0927 etc:Wed May 17 15:58:10 2023 56 | step: 330/ 3966 lossCLS:0.0960 lossGSC:0.0231 loss_consistency:0.0029 imps:15.9 lr: 0.0925 etc:Wed May 17 15:57:33 2023 57 | step: 340/ 3966 lossCLS:0.0978 lossGSC:0.0231 loss_consistency:0.0031 imps:16.0 lr: 0.0923 etc:Wed May 17 15:56:57 2023 58 | step: 350/ 3966 lossCLS:0.0878 lossGSC:0.0231 loss_consistency:0.0032 imps:16.1 lr: 0.0920 etc:Wed May 17 15:56:22 2023 59 | step: 360/ 3966 lossCLS:0.0995 lossGSC:0.0229 loss_consistency:0.0031 imps:16.3 lr: 0.0918 etc:Wed May 17 15:55:50 2023 60 | step: 370/ 3966 lossCLS:0.0934 lossGSC:0.0237 loss_consistency:0.0030 imps:16.4 lr: 0.0916 etc:Wed May 17 15:55:18 2023 61 | step: 380/ 3966 lossCLS:0.0848 lossGSC:0.0228 loss_consistency:0.0028 imps:16.5 lr: 0.0913 etc:Wed May 17 15:54:48 2023 62 | step: 390/ 3966 lossCLS:0.1066 lossGSC:0.0224 loss_consistency:0.0030 imps:16.7 lr: 0.0911 etc:Wed May 17 15:54:21 2023 63 | step: 400/ 3966 lossCLS:0.0910 lossGSC:0.0239 loss_consistency:0.0034 imps:16.8 lr: 0.0909 etc:Wed May 17 15:53:54 2023 64 | step: 410/ 3966 lossCLS:0.0980 lossGSC:0.0230 loss_consistency:0.0029 imps:16.8 lr: 0.0906 etc:Wed May 17 15:53:37 2023 65 | step: 420/ 3966 lossCLS:0.0842 lossGSC:0.0227 loss_consistency:0.0030 imps:17.0 lr: 0.0904 etc:Wed May 17 15:53:13 2023 66 | step: 430/ 3966 lossCLS:0.1107 lossGSC:0.0226 loss_consistency:0.0030 imps:17.1 lr: 0.0902 etc:Wed May 17 15:52:49 2023 67 | step: 440/ 3966 lossCLS:0.0872 lossGSC:0.0225 loss_consistency:0.0031 imps:17.2 lr: 0.0900 etc:Wed May 17 15:52:27 2023 68 | step: 450/ 3966 lossCLS:0.0918 lossGSC:0.0237 loss_consistency:0.0030 imps:17.3 lr: 0.0897 etc:Wed May 17 15:52:05 2023 69 | step: 460/ 3966 lossCLS:0.0893 lossGSC:0.0229 loss_consistency:0.0030 imps:17.4 lr: 0.0895 etc:Wed May 17 15:51:46 2023 70 | step: 470/ 3966 lossCLS:0.1030 lossGSC:0.0225 loss_consistency:0.0029 imps:17.4 lr: 0.0893 etc:Wed May 17 15:51:30 2023 71 | step: 480/ 3966 lossCLS:0.1055 lossGSC:0.0214 loss_consistency:0.0028 imps:17.5 lr: 0.0890 etc:Wed May 17 15:51:10 2023 72 | step: 490/ 3966 lossCLS:0.0764 lossGSC:0.0220 loss_consistency:0.0026 imps:17.6 lr: 0.0888 etc:Wed May 17 15:50:55 2023 73 | step: 500/ 3966 lossCLS:0.0894 lossGSC:0.0221 loss_consistency:0.0029 imps:17.7 lr: 0.0886 etc:Wed May 17 15:50:38 2023 74 | step: 510/ 3966 lossCLS:0.0929 lossGSC:0.0226 loss_consistency:0.0031 imps:17.7 lr: 0.0883 etc:Wed May 17 15:50:31 2023 75 | step: 520/ 3966 lossCLS:0.0984 lossGSC:0.0207 loss_consistency:0.0026 imps:17.8 lr: 0.0881 etc:Wed May 17 15:50:16 2023 76 | step: 530/ 3966 lossCLS:0.0867 lossGSC:0.0220 loss_consistency:0.0029 imps:17.9 lr: 0.0879 etc:Wed May 17 15:50:01 2023 77 | step: 540/ 3966 lossCLS:0.0911 lossGSC:0.0209 loss_consistency:0.0030 imps:17.9 lr: 0.0877 etc:Wed May 17 15:49:48 2023 78 | step: 550/ 3966 lossCLS:0.1040 lossGSC:0.0220 loss_consistency:0.0032 imps:18.0 lr: 0.0874 etc:Wed May 17 15:49:35 2023 79 | step: 560/ 3966 lossCLS:0.0789 lossGSC:0.0213 loss_consistency:0.0029 imps:18.1 lr: 0.0872 etc:Wed May 17 15:49:22 2023 80 | step: 570/ 3966 lossCLS:0.0993 lossGSC:0.0212 loss_consistency:0.0029 imps:18.1 lr: 0.0870 etc:Wed May 17 15:49:10 2023 81 | step: 580/ 3966 lossCLS:0.1008 lossGSC:0.0214 loss_consistency:0.0029 imps:18.2 lr: 0.0867 etc:Wed May 17 15:48:58 2023 82 | step: 590/ 3966 lossCLS:0.0868 lossGSC:0.0208 loss_consistency:0.0030 imps:18.2 lr: 0.0865 etc:Wed May 17 15:48:48 2023 83 | step: 600/ 3966 lossCLS:0.0899 lossGSC:0.0213 loss_consistency:0.0031 imps:18.3 lr: 0.0863 etc:Wed May 17 15:48:41 2023 84 | validating ... 85 | 86 | {'iou': array([0.83013197, 0.50048558, 0.31048701, 0.64171842, 0.47090569, 87 | 0.50247833, 0.77163616, 0.67161048, 0.74861707, 0.27240696, 88 | 0.67788894, 0.46857354, 0.720428 , 0.63226134, 0.6643529 , 89 | 0.61154332, 0.4816432 , 0.72455376, 0.43130083, 0.56644079, 90 | 0.4158772 ]), 'miou': 0.576921023673644} 91 | 92 | 93 | {'iou1': array([0.82077746, 0.47593899, 0.28768813, 0.57396223, 0.40409852, 94 | 0.46826298, 0.78773698, 0.68041934, 0.74072954, 0.22120991, 95 | 0.6012253 , 0.38148667, 0.67851561, 0.64432738, 0.67728556, 96 | 0.62152711, 0.37767585, 0.68226164, 0.40034003, 0.53159925, 97 | 0.17330194]), 'miou': 0.5347795443807041} 98 | 99 | 100 | {'iou2': array([0.82545472, 0.48821229, 0.29908757, 0.60784033, 0.4375021 , 101 | 0.48537065, 0.77968657, 0.67601491, 0.74467331, 0.24680844, 102 | 0.63955712, 0.4250301 , 0.6994718 , 0.63829436, 0.67081923, 103 | 0.61653522, 0.42965953, 0.7034077 , 0.41582043, 0.54902002, 104 | 0.29458957]), 'miou': 0.555850284027174} 105 | step: 610/ 3966 lossCLS:0.0846 lossGSC:0.0201 loss_consistency:0.0028 imps:15.7 lr: 0.0860 etc:Wed May 17 15:58:14 2023 106 | step: 620/ 3966 lossCLS:0.0818 lossGSC:0.0209 loss_consistency:0.0028 imps:15.7 lr: 0.0858 etc:Wed May 17 15:58:01 2023 107 | step: 630/ 3966 lossCLS:0.0831 lossGSC:0.0216 loss_consistency:0.0030 imps:15.8 lr: 0.0856 etc:Wed May 17 15:57:46 2023 108 | step: 640/ 3966 lossCLS:0.0829 lossGSC:0.0207 loss_consistency:0.0031 imps:15.9 lr: 0.0854 etc:Wed May 17 15:57:31 2023 109 | step: 650/ 3966 lossCLS:0.0709 lossGSC:0.0215 loss_consistency:0.0032 imps:15.9 lr: 0.0851 etc:Wed May 17 15:57:20 2023 110 | step: 660/ 3966 lossCLS:0.0876 lossGSC:0.0208 loss_consistency:0.0030 imps:15.9 lr: 0.0849 etc:Wed May 17 15:57:09 2023 111 | Epoch 2/6 112 | step: 670/ 3966 lossCLS:0.0675 lossGSC:0.0216 loss_consistency:0.0028 imps:2.9 lr: 0.0847 etc:Wed May 17 16:01:32 2023 113 | step: 680/ 3966 lossCLS:0.0774 lossGSC:0.0217 loss_consistency:0.0031 imps:5.2 lr: 0.0844 etc:Wed May 17 16:01:15 2023 114 | step: 690/ 3966 lossCLS:0.0689 lossGSC:0.0213 loss_consistency:0.0031 imps:6.9 lr: 0.0842 etc:Wed May 17 16:00:59 2023 115 | step: 700/ 3966 lossCLS:0.0815 lossGSC:0.0208 loss_consistency:0.0030 imps:8.2 lr: 0.0840 etc:Wed May 17 16:00:46 2023 116 | step: 710/ 3966 lossCLS:0.0664 lossGSC:0.0211 loss_consistency:0.0029 imps:9.0 lr: 0.0837 etc:Wed May 17 16:00:49 2023 117 | step: 720/ 3966 lossCLS:0.0900 lossGSC:0.0199 loss_consistency:0.0028 imps:9.9 lr: 0.0835 etc:Wed May 17 16:00:35 2023 118 | step: 730/ 3966 lossCLS:0.0947 lossGSC:0.0211 loss_consistency:0.0030 imps:10.6 lr: 0.0833 etc:Wed May 17 16:00:21 2023 119 | step: 740/ 3966 lossCLS:0.0792 lossGSC:0.0194 loss_consistency:0.0029 imps:11.3 lr: 0.0830 etc:Wed May 17 16:00:04 2023 120 | step: 750/ 3966 lossCLS:0.0860 lossGSC:0.0216 loss_consistency:0.0031 imps:12.0 lr: 0.0828 etc:Wed May 17 15:59:49 2023 121 | step: 760/ 3966 lossCLS:0.0814 lossGSC:0.0209 loss_consistency:0.0030 imps:12.5 lr: 0.0826 etc:Wed May 17 15:59:34 2023 122 | step: 770/ 3966 lossCLS:0.0909 lossGSC:0.0207 loss_consistency:0.0028 imps:13.0 lr: 0.0823 etc:Wed May 17 15:59:21 2023 123 | step: 780/ 3966 lossCLS:0.0827 lossGSC:0.0199 loss_consistency:0.0029 imps:13.4 lr: 0.0821 etc:Wed May 17 15:59:08 2023 124 | step: 790/ 3966 lossCLS:0.0835 lossGSC:0.0216 loss_consistency:0.0029 imps:13.8 lr: 0.0819 etc:Wed May 17 15:58:53 2023 125 | step: 800/ 3966 lossCLS:0.0686 lossGSC:0.0212 loss_consistency:0.0030 imps:14.1 lr: 0.0816 etc:Wed May 17 15:58:41 2023 126 | step: 810/ 3966 lossCLS:0.0783 lossGSC:0.0208 loss_consistency:0.0031 imps:14.2 lr: 0.0814 etc:Wed May 17 15:58:39 2023 127 | step: 820/ 3966 lossCLS:0.0747 lossGSC:0.0194 loss_consistency:0.0026 imps:14.5 lr: 0.0812 etc:Wed May 17 15:58:26 2023 128 | step: 830/ 3966 lossCLS:0.0646 lossGSC:0.0206 loss_consistency:0.0027 imps:14.8 lr: 0.0810 etc:Wed May 17 15:58:13 2023 129 | step: 840/ 3966 lossCLS:0.0664 lossGSC:0.0212 loss_consistency:0.0030 imps:15.0 lr: 0.0807 etc:Wed May 17 15:58:04 2023 130 | step: 850/ 3966 lossCLS:0.0597 lossGSC:0.0184 loss_consistency:0.0025 imps:15.2 lr: 0.0805 etc:Wed May 17 15:57:52 2023 131 | step: 860/ 3966 lossCLS:0.0896 lossGSC:0.0200 loss_consistency:0.0028 imps:15.5 lr: 0.0803 etc:Wed May 17 15:57:38 2023 132 | step: 870/ 3966 lossCLS:0.0799 lossGSC:0.0202 loss_consistency:0.0028 imps:15.7 lr: 0.0800 etc:Wed May 17 15:57:25 2023 133 | step: 880/ 3966 lossCLS:0.0645 lossGSC:0.0202 loss_consistency:0.0029 imps:15.9 lr: 0.0798 etc:Wed May 17 15:57:12 2023 134 | step: 890/ 3966 lossCLS:0.0776 lossGSC:0.0196 loss_consistency:0.0028 imps:16.1 lr: 0.0796 etc:Wed May 17 15:57:00 2023 135 | step: 900/ 3966 lossCLS:0.0721 lossGSC:0.0192 loss_consistency:0.0029 imps:16.3 lr: 0.0793 etc:Wed May 17 15:56:47 2023 136 | validating ... 137 | 138 | {'iou': array([0.82478687, 0.45878913, 0.29693545, 0.62772646, 0.45280615, 139 | 0.54628295, 0.72297328, 0.67795989, 0.69621917, 0.26793294, 140 | 0.68345381, 0.42440665, 0.728706 , 0.64531895, 0.67693483, 141 | 0.59000945, 0.45703823, 0.71426985, 0.41377167, 0.52631137, 142 | 0.43427821]), 'miou': 0.5650910150059694} 143 | 144 | 145 | {'iou1': array([0.81856184, 0.47768882, 0.27704355, 0.57432848, 0.42111088, 146 | 0.49576961, 0.73012997, 0.67853865, 0.72125729, 0.23608615, 147 | 0.58887772, 0.39238306, 0.69648063, 0.62923475, 0.67719854, 148 | 0.59630178, 0.30780887, 0.69717818, 0.39478443, 0.50932025, 149 | 0.26316121]), 'miou': 0.5325354617886855} 150 | 151 | 152 | {'iou2': array([0.82167436, 0.46823898, 0.2869895 , 0.60102747, 0.43695852, 153 | 0.52102628, 0.72655162, 0.67824927, 0.70873823, 0.25200954, 154 | 0.63616577, 0.40839486, 0.71259332, 0.63727685, 0.67706669, 155 | 0.59315562, 0.38242355, 0.70572402, 0.40427805, 0.51781581, 156 | 0.34871971]), 'miou': 0.5488132383973274} 157 | step: 910/ 3966 lossCLS:0.0630 lossGSC:0.0202 loss_consistency:0.0030 imps:11.2 lr: 0.0791 etc:Wed May 17 16:04:57 2023 158 | step: 920/ 3966 lossCLS:0.0860 lossGSC:0.0192 loss_consistency:0.0027 imps:11.4 lr: 0.0789 etc:Wed May 17 16:04:42 2023 159 | step: 930/ 3966 lossCLS:0.0770 lossGSC:0.0196 loss_consistency:0.0030 imps:11.5 lr: 0.0786 etc:Wed May 17 16:04:30 2023 160 | step: 940/ 3966 lossCLS:0.0741 lossGSC:0.0195 loss_consistency:0.0029 imps:11.7 lr: 0.0784 etc:Wed May 17 16:04:16 2023 161 | step: 950/ 3966 lossCLS:0.0921 lossGSC:0.0190 loss_consistency:0.0028 imps:11.9 lr: 0.0782 etc:Wed May 17 16:04:04 2023 162 | step: 960/ 3966 lossCLS:0.0744 lossGSC:0.0193 loss_consistency:0.0026 imps:12.0 lr: 0.0779 etc:Wed May 17 16:03:52 2023 163 | step: 970/ 3966 lossCLS:0.0785 lossGSC:0.0191 loss_consistency:0.0025 imps:12.2 lr: 0.0777 etc:Wed May 17 16:03:41 2023 164 | step: 980/ 3966 lossCLS:0.0877 lossGSC:0.0198 loss_consistency:0.0029 imps:12.3 lr: 0.0775 etc:Wed May 17 16:03:30 2023 165 | step: 990/ 3966 lossCLS:0.0696 lossGSC:0.0200 loss_consistency:0.0031 imps:12.5 lr: 0.0772 etc:Wed May 17 16:03:20 2023 166 | step: 1000/ 3966 lossCLS:0.0805 lossGSC:0.0201 loss_consistency:0.0028 imps:12.6 lr: 0.0770 etc:Wed May 17 16:03:11 2023 167 | step: 1010/ 3966 lossCLS:0.0614 lossGSC:0.0185 loss_consistency:0.0028 imps:12.7 lr: 0.0768 etc:Wed May 17 16:03:02 2023 168 | step: 1020/ 3966 lossCLS:0.0851 lossGSC:0.0191 loss_consistency:0.0027 imps:12.8 lr: 0.0765 etc:Wed May 17 16:02:50 2023 169 | step: 1030/ 3966 lossCLS:0.0642 lossGSC:0.0176 loss_consistency:0.0026 imps:13.0 lr: 0.0763 etc:Wed May 17 16:02:38 2023 170 | step: 1040/ 3966 lossCLS:0.0675 lossGSC:0.0197 loss_consistency:0.0030 imps:13.1 lr: 0.0761 etc:Wed May 17 16:02:30 2023 171 | step: 1050/ 3966 lossCLS:0.0717 lossGSC:0.0190 loss_consistency:0.0028 imps:13.2 lr: 0.0758 etc:Wed May 17 16:02:21 2023 172 | step: 1060/ 3966 lossCLS:0.0721 lossGSC:0.0186 loss_consistency:0.0026 imps:13.3 lr: 0.0756 etc:Wed May 17 16:02:11 2023 173 | step: 1070/ 3966 lossCLS:0.0736 lossGSC:0.0183 loss_consistency:0.0024 imps:13.4 lr: 0.0754 etc:Wed May 17 16:02:02 2023 174 | step: 1080/ 3966 lossCLS:0.0687 lossGSC:0.0187 loss_consistency:0.0027 imps:13.5 lr: 0.0751 etc:Wed May 17 16:01:53 2023 175 | step: 1090/ 3966 lossCLS:0.0775 lossGSC:0.0188 loss_consistency:0.0030 imps:13.6 lr: 0.0749 etc:Wed May 17 16:01:41 2023 176 | step: 1100/ 3966 lossCLS:0.0688 lossGSC:0.0194 loss_consistency:0.0027 imps:13.6 lr: 0.0747 etc:Wed May 17 16:01:39 2023 177 | step: 1110/ 3966 lossCLS:0.0770 lossGSC:0.0177 loss_consistency:0.0028 imps:13.7 lr: 0.0744 etc:Wed May 17 16:01:31 2023 178 | step: 1120/ 3966 lossCLS:0.0861 lossGSC:0.0197 loss_consistency:0.0029 imps:13.8 lr: 0.0742 etc:Wed May 17 16:01:20 2023 179 | step: 1130/ 3966 lossCLS:0.0754 lossGSC:0.0174 loss_consistency:0.0025 imps:13.9 lr: 0.0739 etc:Wed May 17 16:01:08 2023 180 | step: 1140/ 3966 lossCLS:0.0826 lossGSC:0.0187 loss_consistency:0.0029 imps:14.0 lr: 0.0737 etc:Wed May 17 16:00:56 2023 181 | step: 1150/ 3966 lossCLS:0.0709 lossGSC:0.0177 loss_consistency:0.0027 imps:14.1 lr: 0.0735 etc:Wed May 17 16:00:47 2023 182 | step: 1160/ 3966 lossCLS:0.0802 lossGSC:0.0183 loss_consistency:0.0028 imps:14.2 lr: 0.0732 etc:Wed May 17 16:00:35 2023 183 | step: 1170/ 3966 lossCLS:0.0635 lossGSC:0.0187 loss_consistency:0.0027 imps:14.3 lr: 0.0730 etc:Wed May 17 16:00:24 2023 184 | step: 1180/ 3966 lossCLS:0.0739 lossGSC:0.0184 loss_consistency:0.0028 imps:14.4 lr: 0.0728 etc:Wed May 17 16:00:13 2023 185 | step: 1190/ 3966 lossCLS:0.0755 lossGSC:0.0174 loss_consistency:0.0027 imps:14.5 lr: 0.0725 etc:Wed May 17 16:00:05 2023 186 | step: 1200/ 3966 lossCLS:0.0904 lossGSC:0.0188 loss_consistency:0.0028 imps:14.6 lr: 0.0723 etc:Wed May 17 15:59:54 2023 187 | validating ... 188 | 189 | {'iou': array([0.83842727, 0.50885131, 0.29633594, 0.63423171, 0.47134788, 190 | 0.55406806, 0.7801141 , 0.69537085, 0.75021065, 0.25508846, 191 | 0.69485064, 0.3834699 , 0.72785637, 0.67030418, 0.68951144, 192 | 0.60288601, 0.50984909, 0.73262551, 0.42775873, 0.54956587, 193 | 0.41304081]), 'miou': 0.5802745134765667} 194 | 195 | 196 | {'iou1': array([0.8338415 , 0.52297809, 0.30343407, 0.59860773, 0.46464763, 197 | 0.50638296, 0.75756836, 0.67760683, 0.74406336, 0.23191948, 198 | 0.69176876, 0.35510168, 0.70755793, 0.67532617, 0.65181717, 199 | 0.60769373, 0.45598862, 0.73045021, 0.43167034, 0.53310495, 200 | 0.37793635]), 'miou': 0.5647364721279633} 201 | 202 | 203 | {'iou2': array([0.83613439, 0.5159147 , 0.299885 , 0.61641972, 0.46799776, 204 | 0.53022551, 0.76884123, 0.68648884, 0.74713701, 0.24350397, 205 | 0.6933097 , 0.36928579, 0.71770715, 0.67281517, 0.67066431, 206 | 0.60528987, 0.48291885, 0.73153786, 0.42971453, 0.54133541, 207 | 0.39548858]), 'miou': 0.572505492802265} 208 | step: 1210/ 3966 lossCLS:0.0821 lossGSC:0.0188 loss_consistency:0.0026 imps:12.8 lr: 0.0721 etc:Wed May 17 16:04:36 2023 209 | step: 1220/ 3966 lossCLS:0.0675 lossGSC:0.0176 loss_consistency:0.0026 imps:12.9 lr: 0.0718 etc:Wed May 17 16:04:29 2023 210 | step: 1230/ 3966 lossCLS:0.0746 lossGSC:0.0171 loss_consistency:0.0026 imps:12.9 lr: 0.0716 etc:Wed May 17 16:04:19 2023 211 | step: 1240/ 3966 lossCLS:0.0759 lossGSC:0.0174 loss_consistency:0.0025 imps:13.0 lr: 0.0714 etc:Wed May 17 16:04:08 2023 212 | step: 1250/ 3966 lossCLS:0.0678 lossGSC:0.0184 loss_consistency:0.0028 imps:13.1 lr: 0.0711 etc:Wed May 17 16:04:00 2023 213 | step: 1260/ 3966 lossCLS:0.0692 lossGSC:0.0174 loss_consistency:0.0026 imps:13.2 lr: 0.0709 etc:Wed May 17 16:03:50 2023 214 | step: 1270/ 3966 lossCLS:0.0753 lossGSC:0.0190 loss_consistency:0.0031 imps:13.2 lr: 0.0707 etc:Wed May 17 16:03:39 2023 215 | step: 1280/ 3966 lossCLS:0.0690 lossGSC:0.0177 loss_consistency:0.0027 imps:13.3 lr: 0.0704 etc:Wed May 17 16:03:31 2023 216 | step: 1290/ 3966 lossCLS:0.0829 lossGSC:0.0179 loss_consistency:0.0027 imps:13.4 lr: 0.0702 etc:Wed May 17 16:03:21 2023 217 | step: 1300/ 3966 lossCLS:0.0751 lossGSC:0.0188 loss_consistency:0.0028 imps:13.5 lr: 0.0699 etc:Wed May 17 16:03:10 2023 218 | step: 1310/ 3966 lossCLS:0.0772 lossGSC:0.0178 loss_consistency:0.0027 imps:13.5 lr: 0.0697 etc:Wed May 17 16:03:04 2023 219 | step: 1320/ 3966 lossCLS:0.0728 lossGSC:0.0170 loss_consistency:0.0027 imps:13.6 lr: 0.0695 etc:Wed May 17 16:02:54 2023 220 | Epoch 3/6 221 | step: 1330/ 3966 lossCLS:0.0667 lossGSC:0.0188 loss_consistency:0.0030 imps:13.0 lr: 0.0692 etc:Wed May 17 16:02:58 2023 222 | step: 1340/ 3966 lossCLS:0.0702 lossGSC:0.0176 loss_consistency:0.0026 imps:15.8 lr: 0.0690 etc:Wed May 17 16:02:50 2023 223 | step: 1350/ 3966 lossCLS:0.0583 lossGSC:0.0193 loss_consistency:0.0032 imps:17.1 lr: 0.0688 etc:Wed May 17 16:02:41 2023 224 | step: 1360/ 3966 lossCLS:0.0671 lossGSC:0.0174 loss_consistency:0.0026 imps:17.6 lr: 0.0685 etc:Wed May 17 16:02:34 2023 225 | step: 1370/ 3966 lossCLS:0.0666 lossGSC:0.0186 loss_consistency:0.0028 imps:18.0 lr: 0.0683 etc:Wed May 17 16:02:26 2023 226 | step: 1380/ 3966 lossCLS:0.0704 lossGSC:0.0170 loss_consistency:0.0026 imps:18.2 lr: 0.0681 etc:Wed May 17 16:02:18 2023 227 | step: 1390/ 3966 lossCLS:0.0743 lossGSC:0.0180 loss_consistency:0.0030 imps:18.5 lr: 0.0678 etc:Wed May 17 16:02:10 2023 228 | step: 1400/ 3966 lossCLS:0.0625 lossGSC:0.0172 loss_consistency:0.0026 imps:18.7 lr: 0.0676 etc:Wed May 17 16:02:02 2023 229 | step: 1410/ 3966 lossCLS:0.0590 lossGSC:0.0183 loss_consistency:0.0028 imps:18.6 lr: 0.0673 etc:Wed May 17 16:01:57 2023 230 | step: 1420/ 3966 lossCLS:0.0686 lossGSC:0.0174 loss_consistency:0.0027 imps:18.8 lr: 0.0671 etc:Wed May 17 16:01:49 2023 231 | step: 1430/ 3966 lossCLS:0.0544 lossGSC:0.0173 loss_consistency:0.0026 imps:18.9 lr: 0.0669 etc:Wed May 17 16:01:40 2023 232 | step: 1440/ 3966 lossCLS:0.0641 lossGSC:0.0175 loss_consistency:0.0027 imps:19.0 lr: 0.0666 etc:Wed May 17 16:01:32 2023 233 | step: 1450/ 3966 lossCLS:0.0619 lossGSC:0.0182 loss_consistency:0.0027 imps:19.2 lr: 0.0664 etc:Wed May 17 16:01:24 2023 234 | step: 1460/ 3966 lossCLS:0.0655 lossGSC:0.0183 loss_consistency:0.0030 imps:19.4 lr: 0.0662 etc:Wed May 17 16:01:14 2023 235 | step: 1470/ 3966 lossCLS:0.0646 lossGSC:0.0187 loss_consistency:0.0030 imps:19.5 lr: 0.0659 etc:Wed May 17 16:01:05 2023 236 | step: 1480/ 3966 lossCLS:0.0722 lossGSC:0.0177 loss_consistency:0.0028 imps:19.7 lr: 0.0657 etc:Wed May 17 16:00:56 2023 237 | step: 1490/ 3966 lossCLS:0.0640 lossGSC:0.0166 loss_consistency:0.0023 imps:19.7 lr: 0.0654 etc:Wed May 17 16:00:49 2023 238 | step: 1500/ 3966 lossCLS:0.0582 lossGSC:0.0175 loss_consistency:0.0029 imps:19.8 lr: 0.0652 etc:Wed May 17 16:00:40 2023 239 | validating ... 240 | 241 | {'iou': array([0.84009575, 0.47830181, 0.31657987, 0.62176673, 0.44831255, 242 | 0.52922699, 0.76897855, 0.70924456, 0.77060863, 0.25244181, 243 | 0.68094106, 0.39204921, 0.70056653, 0.66097005, 0.70000076, 244 | 0.60531607, 0.52210309, 0.73557253, 0.42226496, 0.57703919, 245 | 0.46033368]), 'miou': 0.5806054472375874} 246 | 247 | 248 | {'iou1': array([0.83416609, 0.50053261, 0.31309655, 0.5844378 , 0.44610398, 249 | 0.48481147, 0.77315808, 0.67975172, 0.73508844, 0.24649716, 250 | 0.67599587, 0.36832479, 0.67404498, 0.65968132, 0.67328268, 251 | 0.61272827, 0.42723809, 0.70792252, 0.42190171, 0.56023698, 252 | 0.39205056]), 'miou': 0.5605262697454202} 253 | 254 | 255 | {'iou2': array([0.83713092, 0.48941721, 0.31483821, 0.60310226, 0.44720826, 256 | 0.50701923, 0.77106832, 0.69449814, 0.75284854, 0.24946949, 257 | 0.67846846, 0.380187 , 0.68730575, 0.66032569, 0.68664172, 258 | 0.60902217, 0.47467059, 0.72174752, 0.42208333, 0.56863809, 259 | 0.42619212]), 'miou': 0.5705658584915039} 260 | step: 1510/ 3966 lossCLS:0.0573 lossGSC:0.0168 loss_consistency:0.0025 imps:11.7 lr: 0.0650 etc:Wed May 17 16:05:11 2023 261 | step: 1520/ 3966 lossCLS:0.0689 lossGSC:0.0169 loss_consistency:0.0027 imps:12.0 lr: 0.0647 etc:Wed May 17 16:05:03 2023 262 | step: 1530/ 3966 lossCLS:0.0720 lossGSC:0.0166 loss_consistency:0.0025 imps:12.2 lr: 0.0645 etc:Wed May 17 16:04:56 2023 263 | step: 1540/ 3966 lossCLS:0.0628 lossGSC:0.0177 loss_consistency:0.0028 imps:12.4 lr: 0.0643 etc:Wed May 17 16:04:47 2023 264 | step: 1550/ 3966 lossCLS:0.0500 lossGSC:0.0180 loss_consistency:0.0028 imps:12.6 lr: 0.0640 etc:Wed May 17 16:04:39 2023 265 | step: 1560/ 3966 lossCLS:0.0687 lossGSC:0.0168 loss_consistency:0.0026 imps:12.8 lr: 0.0638 etc:Wed May 17 16:04:33 2023 266 | step: 1570/ 3966 lossCLS:0.0552 lossGSC:0.0170 loss_consistency:0.0026 imps:12.9 lr: 0.0635 etc:Wed May 17 16:04:26 2023 267 | step: 1580/ 3966 lossCLS:0.0534 lossGSC:0.0176 loss_consistency:0.0027 imps:13.1 lr: 0.0633 etc:Wed May 17 16:04:19 2023 268 | step: 1590/ 3966 lossCLS:0.0580 lossGSC:0.0171 loss_consistency:0.0024 imps:13.2 lr: 0.0631 etc:Wed May 17 16:04:14 2023 269 | step: 1600/ 3966 lossCLS:0.0614 lossGSC:0.0167 loss_consistency:0.0026 imps:13.4 lr: 0.0628 etc:Wed May 17 16:04:05 2023 270 | step: 1610/ 3966 lossCLS:0.0688 lossGSC:0.0174 loss_consistency:0.0026 imps:13.5 lr: 0.0626 etc:Wed May 17 16:04:00 2023 271 | step: 1620/ 3966 lossCLS:0.0760 lossGSC:0.0169 loss_consistency:0.0026 imps:13.7 lr: 0.0623 etc:Wed May 17 16:03:53 2023 272 | step: 1630/ 3966 lossCLS:0.0639 lossGSC:0.0160 loss_consistency:0.0025 imps:13.8 lr: 0.0621 etc:Wed May 17 16:03:45 2023 273 | step: 1640/ 3966 lossCLS:0.0710 lossGSC:0.0166 loss_consistency:0.0027 imps:13.9 lr: 0.0619 etc:Wed May 17 16:03:40 2023 274 | step: 1650/ 3966 lossCLS:0.0511 lossGSC:0.0177 loss_consistency:0.0026 imps:14.0 lr: 0.0616 etc:Wed May 17 16:03:34 2023 275 | step: 1660/ 3966 lossCLS:0.0616 lossGSC:0.0171 loss_consistency:0.0025 imps:14.1 lr: 0.0614 etc:Wed May 17 16:03:28 2023 276 | step: 1670/ 3966 lossCLS:0.0608 lossGSC:0.0181 loss_consistency:0.0028 imps:14.2 lr: 0.0611 etc:Wed May 17 16:03:22 2023 277 | step: 1680/ 3966 lossCLS:0.0644 lossGSC:0.0178 loss_consistency:0.0029 imps:14.4 lr: 0.0609 etc:Wed May 17 16:03:15 2023 278 | step: 1690/ 3966 lossCLS:0.0649 lossGSC:0.0164 loss_consistency:0.0026 imps:14.4 lr: 0.0607 etc:Wed May 17 16:03:10 2023 279 | step: 1700/ 3966 lossCLS:0.0633 lossGSC:0.0169 loss_consistency:0.0028 imps:14.6 lr: 0.0604 etc:Wed May 17 16:03:02 2023 280 | step: 1710/ 3966 lossCLS:0.0616 lossGSC:0.0172 loss_consistency:0.0026 imps:14.6 lr: 0.0602 etc:Wed May 17 16:02:58 2023 281 | step: 1720/ 3966 lossCLS:0.0577 lossGSC:0.0160 loss_consistency:0.0025 imps:14.7 lr: 0.0599 etc:Wed May 17 16:02:50 2023 282 | step: 1730/ 3966 lossCLS:0.0598 lossGSC:0.0165 loss_consistency:0.0025 imps:14.8 lr: 0.0597 etc:Wed May 17 16:02:42 2023 283 | step: 1740/ 3966 lossCLS:0.0774 lossGSC:0.0156 loss_consistency:0.0023 imps:15.0 lr: 0.0595 etc:Wed May 17 16:02:34 2023 284 | step: 1750/ 3966 lossCLS:0.0675 lossGSC:0.0167 loss_consistency:0.0025 imps:15.1 lr: 0.0592 etc:Wed May 17 16:02:26 2023 285 | step: 1760/ 3966 lossCLS:0.0763 lossGSC:0.0176 loss_consistency:0.0028 imps:15.2 lr: 0.0590 etc:Wed May 17 16:02:19 2023 286 | step: 1770/ 3966 lossCLS:0.0797 lossGSC:0.0168 loss_consistency:0.0028 imps:15.3 lr: 0.0587 etc:Wed May 17 16:02:12 2023 287 | step: 1780/ 3966 lossCLS:0.0654 lossGSC:0.0171 loss_consistency:0.0028 imps:15.4 lr: 0.0585 etc:Wed May 17 16:02:04 2023 288 | step: 1790/ 3966 lossCLS:0.0799 lossGSC:0.0161 loss_consistency:0.0026 imps:15.5 lr: 0.0583 etc:Wed May 17 16:01:56 2023 289 | step: 1800/ 3966 lossCLS:0.0481 lossGSC:0.0162 loss_consistency:0.0027 imps:15.6 lr: 0.0580 etc:Wed May 17 16:01:48 2023 290 | validating ... 291 | 292 | {'iou': array([0.84217588, 0.49276733, 0.31798652, 0.61806494, 0.41099132, 293 | 0.52587653, 0.79359053, 0.71032828, 0.71378333, 0.2694185 , 294 | 0.69512487, 0.39415782, 0.72330796, 0.66462331, 0.69613967, 295 | 0.6282019 , 0.49995245, 0.74296524, 0.44604778, 0.60357233, 296 | 0.44473275]), 'miou': 0.5825623449857934} 297 | 298 | 299 | {'iou1': array([0.83240655, 0.50507963, 0.31982235, 0.59192297, 0.40495583, 300 | 0.49293961, 0.77498362, 0.66910404, 0.70141713, 0.25013147, 301 | 0.6824585 , 0.35907307, 0.6835626 , 0.66746205, 0.67777183, 302 | 0.62679019, 0.35172117, 0.72486171, 0.4271763 , 0.59148845, 303 | 0.36839059]), 'miou': 0.5573104600667508} 304 | 305 | 306 | {'iou2': array([0.83729122, 0.49892348, 0.31890444, 0.60499395, 0.40797357, 307 | 0.50940807, 0.78428708, 0.68971616, 0.70760023, 0.25977499, 308 | 0.68879169, 0.37661545, 0.70343528, 0.66604268, 0.68695575, 309 | 0.62749605, 0.42583681, 0.73391347, 0.43661204, 0.59753039, 310 | 0.40656167]), 'miou': 0.5699364025262721} 311 | step: 1810/ 3966 lossCLS:0.0597 lossGSC:0.0165 loss_consistency:0.0028 imps:12.4 lr: 0.0578 etc:Wed May 17 16:06:28 2023 312 | step: 1820/ 3966 lossCLS:0.0671 lossGSC:0.0181 loss_consistency:0.0029 imps:12.5 lr: 0.0575 etc:Wed May 17 16:06:21 2023 313 | step: 1830/ 3966 lossCLS:0.0649 lossGSC:0.0168 loss_consistency:0.0027 imps:12.6 lr: 0.0573 etc:Wed May 17 16:06:14 2023 314 | step: 1840/ 3966 lossCLS:0.0763 lossGSC:0.0168 loss_consistency:0.0025 imps:12.7 lr: 0.0571 etc:Wed May 17 16:06:08 2023 315 | step: 1850/ 3966 lossCLS:0.0630 lossGSC:0.0165 loss_consistency:0.0026 imps:12.7 lr: 0.0568 etc:Wed May 17 16:06:01 2023 316 | step: 1860/ 3966 lossCLS:0.0740 lossGSC:0.0164 loss_consistency:0.0027 imps:12.8 lr: 0.0566 etc:Wed May 17 16:05:54 2023 317 | step: 1870/ 3966 lossCLS:0.0663 lossGSC:0.0169 loss_consistency:0.0029 imps:12.9 lr: 0.0563 etc:Wed May 17 16:05:48 2023 318 | step: 1880/ 3966 lossCLS:0.0804 lossGSC:0.0170 loss_consistency:0.0027 imps:13.0 lr: 0.0561 etc:Wed May 17 16:05:41 2023 319 | step: 1890/ 3966 lossCLS:0.0581 lossGSC:0.0163 loss_consistency:0.0025 imps:13.1 lr: 0.0558 etc:Wed May 17 16:05:34 2023 320 | step: 1900/ 3966 lossCLS:0.0560 lossGSC:0.0153 loss_consistency:0.0024 imps:13.1 lr: 0.0556 etc:Wed May 17 16:05:28 2023 321 | step: 1910/ 3966 lossCLS:0.0644 lossGSC:0.0162 loss_consistency:0.0025 imps:13.2 lr: 0.0554 etc:Wed May 17 16:05:23 2023 322 | step: 1920/ 3966 lossCLS:0.0707 lossGSC:0.0166 loss_consistency:0.0028 imps:13.3 lr: 0.0551 etc:Wed May 17 16:05:16 2023 323 | step: 1930/ 3966 lossCLS:0.0544 lossGSC:0.0162 loss_consistency:0.0024 imps:13.3 lr: 0.0549 etc:Wed May 17 16:05:10 2023 324 | step: 1940/ 3966 lossCLS:0.0666 lossGSC:0.0158 loss_consistency:0.0025 imps:13.4 lr: 0.0546 etc:Wed May 17 16:05:03 2023 325 | step: 1950/ 3966 lossCLS:0.0626 lossGSC:0.0166 loss_consistency:0.0028 imps:13.5 lr: 0.0544 etc:Wed May 17 16:04:55 2023 326 | step: 1960/ 3966 lossCLS:0.0559 lossGSC:0.0159 loss_consistency:0.0027 imps:13.6 lr: 0.0541 etc:Wed May 17 16:04:47 2023 327 | step: 1970/ 3966 lossCLS:0.0651 lossGSC:0.0151 loss_consistency:0.0025 imps:13.7 lr: 0.0539 etc:Wed May 17 16:04:38 2023 328 | step: 1980/ 3966 lossCLS:0.0647 lossGSC:0.0162 loss_consistency:0.0028 imps:13.8 lr: 0.0537 etc:Wed May 17 16:04:30 2023 329 | Epoch 4/6 330 | step: 1990/ 3966 lossCLS:0.0567 lossGSC:0.0158 loss_consistency:0.0027 imps:13.6 lr: 0.0534 etc:Wed May 17 16:04:30 2023 331 | step: 2000/ 3966 lossCLS:0.0588 lossGSC:0.0164 loss_consistency:0.0023 imps:17.2 lr: 0.0532 etc:Wed May 17 16:04:23 2023 332 | step: 2010/ 3966 lossCLS:0.0487 lossGSC:0.0168 loss_consistency:0.0029 imps:17.9 lr: 0.0529 etc:Wed May 17 16:04:17 2023 333 | step: 2020/ 3966 lossCLS:0.0552 lossGSC:0.0164 loss_consistency:0.0027 imps:18.7 lr: 0.0527 etc:Wed May 17 16:04:10 2023 334 | step: 2030/ 3966 lossCLS:0.0497 lossGSC:0.0170 loss_consistency:0.0028 imps:19.1 lr: 0.0524 etc:Wed May 17 16:04:03 2023 335 | step: 2040/ 3966 lossCLS:0.0621 lossGSC:0.0158 loss_consistency:0.0027 imps:19.6 lr: 0.0522 etc:Wed May 17 16:03:56 2023 336 | step: 2050/ 3966 lossCLS:0.0601 lossGSC:0.0164 loss_consistency:0.0027 imps:19.9 lr: 0.0520 etc:Wed May 17 16:03:49 2023 337 | step: 2060/ 3966 lossCLS:0.0434 lossGSC:0.0155 loss_consistency:0.0025 imps:20.1 lr: 0.0517 etc:Wed May 17 16:03:42 2023 338 | step: 2070/ 3966 lossCLS:0.0622 lossGSC:0.0164 loss_consistency:0.0026 imps:19.9 lr: 0.0515 etc:Wed May 17 16:03:37 2023 339 | step: 2080/ 3966 lossCLS:0.0529 lossGSC:0.0167 loss_consistency:0.0027 imps:19.9 lr: 0.0512 etc:Wed May 17 16:03:32 2023 340 | step: 2090/ 3966 lossCLS:0.0564 lossGSC:0.0160 loss_consistency:0.0026 imps:19.8 lr: 0.0510 etc:Wed May 17 16:03:27 2023 341 | step: 2100/ 3966 lossCLS:0.0550 lossGSC:0.0152 loss_consistency:0.0025 imps:19.8 lr: 0.0507 etc:Wed May 17 16:03:21 2023 342 | validating ... 343 | 344 | {'iou': array([0.85032833, 0.51332302, 0.31919096, 0.63664504, 0.46806668, 345 | 0.56346521, 0.79281851, 0.71648677, 0.68867796, 0.28862696, 346 | 0.69684183, 0.42866438, 0.72728497, 0.6713585 , 0.7101475 , 347 | 0.62118545, 0.50187226, 0.73644441, 0.42581665, 0.61871121, 348 | 0.45832656]), 'miou': 0.5921087223808661} 349 | 350 | 351 | {'iou1': array([0.84335544, 0.52699017, 0.32521025, 0.60784772, 0.47299548, 352 | 0.51603083, 0.77833694, 0.6903454 , 0.69490856, 0.28254687, 353 | 0.68728003, 0.38915703, 0.68169657, 0.66888571, 0.69571898, 354 | 0.62889692, 0.45465097, 0.70974379, 0.43577763, 0.60499606, 355 | 0.41092954]), 'miou': 0.5764905183407623} 356 | 357 | 358 | {'iou2': array([0.84684188, 0.52015659, 0.3222006 , 0.62224638, 0.47053108, 359 | 0.53974802, 0.78557773, 0.70341608, 0.69179326, 0.28558692, 360 | 0.69206093, 0.40891071, 0.70449077, 0.6701221 , 0.70293324, 361 | 0.62504118, 0.47826161, 0.7230941 , 0.43079714, 0.61185364, 362 | 0.43462805]), 'miou': 0.5842996203608142} 363 | step: 2110/ 3966 lossCLS:0.0545 lossGSC:0.0156 loss_consistency:0.0027 imps:11.1 lr: 0.0505 etc:Wed May 17 16:05:48 2023 364 | step: 2120/ 3966 lossCLS:0.0448 lossGSC:0.0168 loss_consistency:0.0027 imps:11.5 lr: 0.0502 etc:Wed May 17 16:05:42 2023 365 | step: 2130/ 3966 lossCLS:0.0585 lossGSC:0.0167 loss_consistency:0.0028 imps:11.8 lr: 0.0500 etc:Wed May 17 16:05:37 2023 366 | step: 2140/ 3966 lossCLS:0.0662 lossGSC:0.0161 loss_consistency:0.0026 imps:12.1 lr: 0.0498 etc:Wed May 17 16:05:31 2023 367 | step: 2150/ 3966 lossCLS:0.0458 lossGSC:0.0167 loss_consistency:0.0026 imps:12.4 lr: 0.0495 etc:Wed May 17 16:05:25 2023 368 | step: 2160/ 3966 lossCLS:0.0649 lossGSC:0.0155 loss_consistency:0.0026 imps:12.6 lr: 0.0493 etc:Wed May 17 16:05:19 2023 369 | step: 2170/ 3966 lossCLS:0.0514 lossGSC:0.0163 loss_consistency:0.0026 imps:12.9 lr: 0.0490 etc:Wed May 17 16:05:12 2023 370 | step: 2180/ 3966 lossCLS:0.0449 lossGSC:0.0151 loss_consistency:0.0027 imps:13.2 lr: 0.0488 etc:Wed May 17 16:05:06 2023 371 | step: 2190/ 3966 lossCLS:0.0516 lossGSC:0.0155 loss_consistency:0.0025 imps:13.4 lr: 0.0485 etc:Wed May 17 16:05:00 2023 372 | step: 2200/ 3966 lossCLS:0.0591 lossGSC:0.0160 loss_consistency:0.0029 imps:13.6 lr: 0.0483 etc:Wed May 17 16:04:54 2023 373 | step: 2210/ 3966 lossCLS:0.0504 lossGSC:0.0172 loss_consistency:0.0027 imps:13.8 lr: 0.0480 etc:Wed May 17 16:04:49 2023 374 | step: 2220/ 3966 lossCLS:0.0595 lossGSC:0.0164 loss_consistency:0.0025 imps:13.9 lr: 0.0478 etc:Wed May 17 16:04:44 2023 375 | step: 2230/ 3966 lossCLS:0.0528 lossGSC:0.0150 loss_consistency:0.0025 imps:14.1 lr: 0.0475 etc:Wed May 17 16:04:38 2023 376 | step: 2240/ 3966 lossCLS:0.0614 lossGSC:0.0166 loss_consistency:0.0029 imps:14.3 lr: 0.0473 etc:Wed May 17 16:04:31 2023 377 | step: 2250/ 3966 lossCLS:0.0595 lossGSC:0.0153 loss_consistency:0.0024 imps:14.5 lr: 0.0470 etc:Wed May 17 16:04:24 2023 378 | step: 2260/ 3966 lossCLS:0.0638 lossGSC:0.0152 loss_consistency:0.0024 imps:14.7 lr: 0.0468 etc:Wed May 17 16:04:18 2023 379 | step: 2270/ 3966 lossCLS:0.0576 lossGSC:0.0162 loss_consistency:0.0028 imps:14.8 lr: 0.0466 etc:Wed May 17 16:04:11 2023 380 | step: 2280/ 3966 lossCLS:0.0595 lossGSC:0.0159 loss_consistency:0.0028 imps:15.0 lr: 0.0463 etc:Wed May 17 16:04:05 2023 381 | step: 2290/ 3966 lossCLS:0.0540 lossGSC:0.0152 loss_consistency:0.0025 imps:15.1 lr: 0.0461 etc:Wed May 17 16:03:59 2023 382 | step: 2300/ 3966 lossCLS:0.0482 lossGSC:0.0160 loss_consistency:0.0027 imps:15.3 lr: 0.0458 etc:Wed May 17 16:03:54 2023 383 | step: 2310/ 3966 lossCLS:0.0809 lossGSC:0.0155 loss_consistency:0.0028 imps:15.3 lr: 0.0456 etc:Wed May 17 16:03:49 2023 384 | step: 2320/ 3966 lossCLS:0.0547 lossGSC:0.0159 loss_consistency:0.0026 imps:15.5 lr: 0.0453 etc:Wed May 17 16:03:43 2023 385 | step: 2330/ 3966 lossCLS:0.0631 lossGSC:0.0154 loss_consistency:0.0025 imps:15.6 lr: 0.0451 etc:Wed May 17 16:03:38 2023 386 | step: 2340/ 3966 lossCLS:0.0433 lossGSC:0.0162 loss_consistency:0.0028 imps:15.7 lr: 0.0448 etc:Wed May 17 16:03:31 2023 387 | step: 2350/ 3966 lossCLS:0.0458 lossGSC:0.0163 loss_consistency:0.0027 imps:15.8 lr: 0.0446 etc:Wed May 17 16:03:25 2023 388 | step: 2360/ 3966 lossCLS:0.0454 lossGSC:0.0154 loss_consistency:0.0024 imps:15.9 lr: 0.0443 etc:Wed May 17 16:03:21 2023 389 | step: 2370/ 3966 lossCLS:0.0550 lossGSC:0.0153 loss_consistency:0.0027 imps:16.0 lr: 0.0441 etc:Wed May 17 16:03:16 2023 390 | step: 2380/ 3966 lossCLS:0.0486 lossGSC:0.0161 loss_consistency:0.0025 imps:16.1 lr: 0.0438 etc:Wed May 17 16:03:11 2023 391 | step: 2390/ 3966 lossCLS:0.0571 lossGSC:0.0157 loss_consistency:0.0025 imps:16.1 lr: 0.0436 etc:Wed May 17 16:03:07 2023 392 | step: 2400/ 3966 lossCLS:0.0545 lossGSC:0.0148 loss_consistency:0.0024 imps:16.2 lr: 0.0433 etc:Wed May 17 16:03:02 2023 393 | validating ... 394 | 395 | {'iou': array([0.85406864, 0.53799957, 0.32323121, 0.63257051, 0.47628076, 396 | 0.53379515, 0.76818543, 0.71854665, 0.74587783, 0.2879212 , 397 | 0.71166302, 0.35892956, 0.73243346, 0.67912541, 0.72340078, 398 | 0.63877723, 0.50232107, 0.75997318, 0.46524252, 0.61504522, 399 | 0.46351976]), 'miou': 0.5966146740997195} 400 | 401 | 402 | {'iou1': array([0.84299057, 0.54115956, 0.3248568 , 0.5976753 , 0.47429864, 403 | 0.49654033, 0.74590817, 0.68020395, 0.7218887 , 0.2654708 , 404 | 0.694977 , 0.3432559 , 0.68614834, 0.67673807, 0.68666959, 405 | 0.633793 , 0.45977124, 0.72230108, 0.46768063, 0.60753417, 406 | 0.38796246]), 'miou': 0.5741821097039053} 407 | 408 | 409 | {'iou2': array([0.84852961, 0.53957957, 0.32404401, 0.61512291, 0.4752897 , 410 | 0.51516774, 0.7570468 , 0.6993753 , 0.73388327, 0.276696 , 411 | 0.70332001, 0.35109273, 0.7092909 , 0.67793174, 0.70503519, 412 | 0.63628511, 0.48104616, 0.74113713, 0.46646158, 0.61128969, 413 | 0.42574111]), 'miou': 0.5853983919018124} 414 | step: 2410/ 3966 lossCLS:0.0475 lossGSC:0.0156 loss_consistency:0.0025 imps:13.1 lr: 0.0431 etc:Wed May 17 16:05:47 2023 415 | step: 2420/ 3966 lossCLS:0.0500 lossGSC:0.0155 loss_consistency:0.0028 imps:13.2 lr: 0.0428 etc:Wed May 17 16:05:40 2023 416 | step: 2430/ 3966 lossCLS:0.0480 lossGSC:0.0158 loss_consistency:0.0025 imps:13.3 lr: 0.0426 etc:Wed May 17 16:05:33 2023 417 | step: 2440/ 3966 lossCLS:0.0452 lossGSC:0.0152 loss_consistency:0.0025 imps:13.4 lr: 0.0423 etc:Wed May 17 16:05:28 2023 418 | step: 2450/ 3966 lossCLS:0.0603 lossGSC:0.0158 loss_consistency:0.0027 imps:13.5 lr: 0.0421 etc:Wed May 17 16:05:22 2023 419 | step: 2460/ 3966 lossCLS:0.0498 lossGSC:0.0152 loss_consistency:0.0025 imps:13.6 lr: 0.0418 etc:Wed May 17 16:05:17 2023 420 | step: 2470/ 3966 lossCLS:0.0517 lossGSC:0.0162 loss_consistency:0.0025 imps:13.7 lr: 0.0416 etc:Wed May 17 16:05:12 2023 421 | step: 2480/ 3966 lossCLS:0.0647 lossGSC:0.0159 loss_consistency:0.0027 imps:13.8 lr: 0.0413 etc:Wed May 17 16:05:06 2023 422 | step: 2490/ 3966 lossCLS:0.0517 lossGSC:0.0159 loss_consistency:0.0026 imps:13.9 lr: 0.0411 etc:Wed May 17 16:05:01 2023 423 | step: 2500/ 3966 lossCLS:0.0410 lossGSC:0.0155 loss_consistency:0.0026 imps:14.0 lr: 0.0408 etc:Wed May 17 16:04:56 2023 424 | step: 2510/ 3966 lossCLS:0.0603 lossGSC:0.0156 loss_consistency:0.0026 imps:14.0 lr: 0.0406 etc:Wed May 17 16:04:51 2023 425 | step: 2520/ 3966 lossCLS:0.0579 lossGSC:0.0149 loss_consistency:0.0025 imps:14.1 lr: 0.0403 etc:Wed May 17 16:04:45 2023 426 | step: 2530/ 3966 lossCLS:0.0620 lossGSC:0.0148 loss_consistency:0.0024 imps:14.2 lr: 0.0401 etc:Wed May 17 16:04:38 2023 427 | step: 2540/ 3966 lossCLS:0.0420 lossGSC:0.0155 loss_consistency:0.0025 imps:14.3 lr: 0.0398 etc:Wed May 17 16:04:32 2023 428 | step: 2550/ 3966 lossCLS:0.0598 lossGSC:0.0151 loss_consistency:0.0025 imps:14.4 lr: 0.0396 etc:Wed May 17 16:04:26 2023 429 | step: 2560/ 3966 lossCLS:0.0554 lossGSC:0.0153 loss_consistency:0.0025 imps:14.5 lr: 0.0393 etc:Wed May 17 16:04:20 2023 430 | step: 2570/ 3966 lossCLS:0.0629 lossGSC:0.0157 loss_consistency:0.0025 imps:14.6 lr: 0.0391 etc:Wed May 17 16:04:15 2023 431 | step: 2580/ 3966 lossCLS:0.0566 lossGSC:0.0151 loss_consistency:0.0023 imps:14.7 lr: 0.0388 etc:Wed May 17 16:04:09 2023 432 | step: 2590/ 3966 lossCLS:0.0476 lossGSC:0.0153 loss_consistency:0.0025 imps:14.7 lr: 0.0386 etc:Wed May 17 16:04:03 2023 433 | step: 2600/ 3966 lossCLS:0.0570 lossGSC:0.0152 loss_consistency:0.0026 imps:14.8 lr: 0.0383 etc:Wed May 17 16:03:57 2023 434 | step: 2610/ 3966 lossCLS:0.0581 lossGSC:0.0147 loss_consistency:0.0025 imps:14.9 lr: 0.0381 etc:Wed May 17 16:03:53 2023 435 | step: 2620/ 3966 lossCLS:0.0497 lossGSC:0.0165 loss_consistency:0.0027 imps:14.9 lr: 0.0378 etc:Wed May 17 16:03:48 2023 436 | step: 2630/ 3966 lossCLS:0.0501 lossGSC:0.0153 loss_consistency:0.0028 imps:15.0 lr: 0.0376 etc:Wed May 17 16:03:43 2023 437 | step: 2640/ 3966 lossCLS:0.0757 lossGSC:0.0150 loss_consistency:0.0025 imps:15.1 lr: 0.0373 etc:Wed May 17 16:03:37 2023 438 | Epoch 5/6 439 | step: 2650/ 3966 lossCLS:0.0436 lossGSC:0.0152 loss_consistency:0.0025 imps:12.1 lr: 0.0371 etc:Wed May 17 16:03:39 2023 440 | step: 2660/ 3966 lossCLS:0.0525 lossGSC:0.0156 loss_consistency:0.0026 imps:15.6 lr: 0.0368 etc:Wed May 17 16:03:35 2023 441 | step: 2670/ 3966 lossCLS:0.0429 lossGSC:0.0147 loss_consistency:0.0026 imps:17.2 lr: 0.0365 etc:Wed May 17 16:03:30 2023 442 | step: 2680/ 3966 lossCLS:0.0550 lossGSC:0.0150 loss_consistency:0.0026 imps:17.9 lr: 0.0363 etc:Wed May 17 16:03:25 2023 443 | step: 2690/ 3966 lossCLS:0.0382 lossGSC:0.0155 loss_consistency:0.0025 imps:18.6 lr: 0.0360 etc:Wed May 17 16:03:20 2023 444 | step: 2700/ 3966 lossCLS:0.0394 lossGSC:0.0151 loss_consistency:0.0026 imps:18.8 lr: 0.0358 etc:Wed May 17 16:03:16 2023 445 | validating ... 446 | 447 | {'iou': array([0.84206132, 0.49080618, 0.31268232, 0.61288875, 0.43793074, 448 | 0.526948 , 0.77221276, 0.70105285, 0.71312037, 0.27761498, 449 | 0.69019009, 0.39337568, 0.71236653, 0.65793087, 0.70765127, 450 | 0.61361035, 0.49401736, 0.7304857 , 0.43576067, 0.57218764, 451 | 0.45612083]), 'miou': 0.5786197744887622} 452 | 453 | 454 | {'iou1': array([0.83299355, 0.49933694, 0.30937804, 0.59117928, 0.45189718, 455 | 0.47716686, 0.76306089, 0.65433884, 0.69129891, 0.25936781, 456 | 0.67468045, 0.35678274, 0.66516043, 0.65060159, 0.67575059, 457 | 0.61680785, 0.42849515, 0.69690124, 0.42784895, 0.56556982, 458 | 0.40113851]), 'miou': 0.5566550283050145} 459 | 460 | 461 | {'iou2': array([0.83752743, 0.49507156, 0.31103018, 0.60203401, 0.44491396, 462 | 0.50205743, 0.76763682, 0.67769584, 0.70220964, 0.2684914 , 463 | 0.68243527, 0.37507921, 0.68876348, 0.65426623, 0.69170093, 464 | 0.6152091 , 0.46125626, 0.71369347, 0.43180481, 0.56887873, 465 | 0.42862967]), 'miou': 0.5676374013968883} 466 | step: 2710/ 3966 lossCLS:0.0322 lossGSC:0.0158 loss_consistency:0.0026 imps:7.7 lr: 0.0355 etc:Wed May 17 16:05:13 2023 467 | step: 2720/ 3966 lossCLS:0.0525 lossGSC:0.0162 loss_consistency:0.0026 imps:8.4 lr: 0.0353 etc:Wed May 17 16:05:07 2023 468 | step: 2730/ 3966 lossCLS:0.0384 lossGSC:0.0159 loss_consistency:0.0029 imps:9.1 lr: 0.0350 etc:Wed May 17 16:05:00 2023 469 | step: 2740/ 3966 lossCLS:0.0365 lossGSC:0.0163 loss_consistency:0.0028 imps:9.7 lr: 0.0348 etc:Wed May 17 16:04:54 2023 470 | step: 2750/ 3966 lossCLS:0.0533 lossGSC:0.0156 loss_consistency:0.0030 imps:10.2 lr: 0.0345 etc:Wed May 17 16:04:49 2023 471 | step: 2760/ 3966 lossCLS:0.0450 lossGSC:0.0156 loss_consistency:0.0025 imps:10.7 lr: 0.0343 etc:Wed May 17 16:04:43 2023 472 | step: 2770/ 3966 lossCLS:0.0395 lossGSC:0.0153 loss_consistency:0.0028 imps:11.1 lr: 0.0340 etc:Wed May 17 16:04:37 2023 473 | step: 2780/ 3966 lossCLS:0.0427 lossGSC:0.0151 loss_consistency:0.0025 imps:11.5 lr: 0.0337 etc:Wed May 17 16:04:34 2023 474 | step: 2790/ 3966 lossCLS:0.0484 lossGSC:0.0143 loss_consistency:0.0024 imps:11.8 lr: 0.0335 etc:Wed May 17 16:04:29 2023 475 | step: 2800/ 3966 lossCLS:0.0447 lossGSC:0.0154 loss_consistency:0.0026 imps:12.2 lr: 0.0332 etc:Wed May 17 16:04:24 2023 476 | step: 2810/ 3966 lossCLS:0.0547 lossGSC:0.0147 loss_consistency:0.0026 imps:12.5 lr: 0.0330 etc:Wed May 17 16:04:20 2023 477 | step: 2820/ 3966 lossCLS:0.0378 lossGSC:0.0151 loss_consistency:0.0024 imps:12.8 lr: 0.0327 etc:Wed May 17 16:04:14 2023 478 | step: 2830/ 3966 lossCLS:0.0384 lossGSC:0.0146 loss_consistency:0.0024 imps:13.0 lr: 0.0325 etc:Wed May 17 16:04:10 2023 479 | step: 2840/ 3966 lossCLS:0.0546 lossGSC:0.0145 loss_consistency:0.0026 imps:13.3 lr: 0.0322 etc:Wed May 17 16:04:04 2023 480 | step: 2850/ 3966 lossCLS:0.0490 lossGSC:0.0148 loss_consistency:0.0027 imps:13.5 lr: 0.0319 etc:Wed May 17 16:04:00 2023 481 | step: 2860/ 3966 lossCLS:0.0479 lossGSC:0.0145 loss_consistency:0.0029 imps:13.8 lr: 0.0317 etc:Wed May 17 16:03:54 2023 482 | step: 2870/ 3966 lossCLS:0.0457 lossGSC:0.0145 loss_consistency:0.0024 imps:14.0 lr: 0.0314 etc:Wed May 17 16:03:49 2023 483 | step: 2880/ 3966 lossCLS:0.0423 lossGSC:0.0152 loss_consistency:0.0026 imps:14.2 lr: 0.0312 etc:Wed May 17 16:03:44 2023 484 | step: 2890/ 3966 lossCLS:0.0390 lossGSC:0.0155 loss_consistency:0.0027 imps:14.4 lr: 0.0309 etc:Wed May 17 16:03:39 2023 485 | step: 2900/ 3966 lossCLS:0.0492 lossGSC:0.0159 loss_consistency:0.0025 imps:14.7 lr: 0.0307 etc:Wed May 17 16:03:33 2023 486 | step: 2910/ 3966 lossCLS:0.0468 lossGSC:0.0152 loss_consistency:0.0028 imps:14.8 lr: 0.0304 etc:Wed May 17 16:03:29 2023 487 | step: 2920/ 3966 lossCLS:0.0450 lossGSC:0.0153 loss_consistency:0.0026 imps:15.0 lr: 0.0301 etc:Wed May 17 16:03:24 2023 488 | step: 2930/ 3966 lossCLS:0.0435 lossGSC:0.0156 loss_consistency:0.0028 imps:15.1 lr: 0.0299 etc:Wed May 17 16:03:20 2023 489 | step: 2940/ 3966 lossCLS:0.0371 lossGSC:0.0151 loss_consistency:0.0026 imps:15.3 lr: 0.0296 etc:Wed May 17 16:03:16 2023 490 | step: 2950/ 3966 lossCLS:0.0515 lossGSC:0.0148 loss_consistency:0.0024 imps:15.4 lr: 0.0294 etc:Wed May 17 16:03:12 2023 491 | step: 2960/ 3966 lossCLS:0.0404 lossGSC:0.0146 loss_consistency:0.0024 imps:15.5 lr: 0.0291 etc:Wed May 17 16:03:08 2023 492 | step: 2970/ 3966 lossCLS:0.0449 lossGSC:0.0145 loss_consistency:0.0024 imps:15.5 lr: 0.0288 etc:Wed May 17 16:03:06 2023 493 | step: 2980/ 3966 lossCLS:0.0453 lossGSC:0.0149 loss_consistency:0.0025 imps:15.6 lr: 0.0286 etc:Wed May 17 16:03:02 2023 494 | step: 2990/ 3966 lossCLS:0.0530 lossGSC:0.0155 loss_consistency:0.0028 imps:15.7 lr: 0.0283 etc:Wed May 17 16:02:58 2023 495 | step: 3000/ 3966 lossCLS:0.0420 lossGSC:0.0141 loss_consistency:0.0026 imps:15.8 lr: 0.0281 etc:Wed May 17 16:02:55 2023 496 | validating ... 497 | 498 | {'iou': array([0.84865585, 0.52684416, 0.3101953 , 0.64045752, 0.4790437 , 499 | 0.5204992 , 0.74876459, 0.69106126, 0.68278203, 0.27981038, 500 | 0.68799484, 0.37988764, 0.71247031, 0.67059577, 0.71186212, 501 | 0.6276136 , 0.48543848, 0.72658721, 0.4055996 , 0.61896726, 502 | 0.45421318]), 'miou': 0.5813973336543536} 503 | 504 | 505 | {'iou1': array([0.83805814, 0.5229057 , 0.30979858, 0.60511717, 0.48219734, 506 | 0.47469909, 0.73885349, 0.65736356, 0.66519449, 0.26960292, 507 | 0.66741835, 0.35955524, 0.66669224, 0.65543309, 0.67773677, 508 | 0.62535645, 0.42570341, 0.68878913, 0.40290509, 0.59218152, 509 | 0.38143046]), 'miou': 0.5574758202149029} 510 | 511 | 512 | {'iou2': array([0.84335699, 0.52487493, 0.30999694, 0.62278734, 0.48062052, 513 | 0.49759915, 0.74380904, 0.67421241, 0.67398826, 0.27470665, 514 | 0.6777066 , 0.36972144, 0.68958127, 0.66301443, 0.69479945, 515 | 0.62648502, 0.45557095, 0.70768817, 0.40425235, 0.60557439, 516 | 0.41782182]), 'miou': 0.5694365769346283} 517 | step: 3010/ 3966 lossCLS:0.0387 lossGSC:0.0145 loss_consistency:0.0027 imps:12.9 lr: 0.0278 etc:Wed May 17 16:04:43 2023 518 | step: 3020/ 3966 lossCLS:0.0536 lossGSC:0.0156 loss_consistency:0.0026 imps:13.1 lr: 0.0275 etc:Wed May 17 16:04:38 2023 519 | step: 3030/ 3966 lossCLS:0.0514 lossGSC:0.0155 loss_consistency:0.0025 imps:13.2 lr: 0.0273 etc:Wed May 17 16:04:32 2023 520 | step: 3040/ 3966 lossCLS:0.0390 lossGSC:0.0139 loss_consistency:0.0022 imps:13.4 lr: 0.0270 etc:Wed May 17 16:04:26 2023 521 | step: 3050/ 3966 lossCLS:0.0478 lossGSC:0.0146 loss_consistency:0.0023 imps:13.5 lr: 0.0267 etc:Wed May 17 16:04:21 2023 522 | step: 3060/ 3966 lossCLS:0.0583 lossGSC:0.0141 loss_consistency:0.0024 imps:13.6 lr: 0.0265 etc:Wed May 17 16:04:16 2023 523 | step: 3070/ 3966 lossCLS:0.0508 lossGSC:0.0140 loss_consistency:0.0023 imps:13.8 lr: 0.0262 etc:Wed May 17 16:04:10 2023 524 | step: 3080/ 3966 lossCLS:0.0410 lossGSC:0.0149 loss_consistency:0.0026 imps:13.9 lr: 0.0260 etc:Wed May 17 16:04:05 2023 525 | step: 3090/ 3966 lossCLS:0.0439 lossGSC:0.0146 loss_consistency:0.0022 imps:14.0 lr: 0.0257 etc:Wed May 17 16:04:00 2023 526 | step: 3100/ 3966 lossCLS:0.0526 lossGSC:0.0152 loss_consistency:0.0027 imps:14.1 lr: 0.0254 etc:Wed May 17 16:03:56 2023 527 | step: 3110/ 3966 lossCLS:0.0416 lossGSC:0.0149 loss_consistency:0.0024 imps:14.2 lr: 0.0252 etc:Wed May 17 16:03:52 2023 528 | step: 3120/ 3966 lossCLS:0.0553 lossGSC:0.0144 loss_consistency:0.0024 imps:14.3 lr: 0.0249 etc:Wed May 17 16:03:47 2023 529 | step: 3130/ 3966 lossCLS:0.0409 lossGSC:0.0151 loss_consistency:0.0026 imps:14.4 lr: 0.0246 etc:Wed May 17 16:03:42 2023 530 | step: 3140/ 3966 lossCLS:0.0389 lossGSC:0.0140 loss_consistency:0.0023 imps:14.5 lr: 0.0244 etc:Wed May 17 16:03:37 2023 531 | step: 3150/ 3966 lossCLS:0.0458 lossGSC:0.0146 loss_consistency:0.0026 imps:14.6 lr: 0.0241 etc:Wed May 17 16:03:32 2023 532 | step: 3160/ 3966 lossCLS:0.0558 lossGSC:0.0145 loss_consistency:0.0025 imps:14.7 lr: 0.0238 etc:Wed May 17 16:03:27 2023 533 | step: 3170/ 3966 lossCLS:0.0447 lossGSC:0.0149 loss_consistency:0.0026 imps:14.8 lr: 0.0236 etc:Wed May 17 16:03:23 2023 534 | step: 3180/ 3966 lossCLS:0.0360 lossGSC:0.0151 loss_consistency:0.0027 imps:14.9 lr: 0.0233 etc:Wed May 17 16:03:18 2023 535 | step: 3190/ 3966 lossCLS:0.0331 lossGSC:0.0144 loss_consistency:0.0024 imps:15.0 lr: 0.0230 etc:Wed May 17 16:03:13 2023 536 | step: 3200/ 3966 lossCLS:0.0412 lossGSC:0.0142 loss_consistency:0.0025 imps:15.1 lr: 0.0228 etc:Wed May 17 16:03:09 2023 537 | step: 3210/ 3966 lossCLS:0.0561 lossGSC:0.0147 loss_consistency:0.0025 imps:15.1 lr: 0.0225 etc:Wed May 17 16:03:06 2023 538 | step: 3220/ 3966 lossCLS:0.0524 lossGSC:0.0132 loss_consistency:0.0024 imps:15.2 lr: 0.0222 etc:Wed May 17 16:03:03 2023 539 | step: 3230/ 3966 lossCLS:0.0650 lossGSC:0.0150 loss_consistency:0.0027 imps:15.2 lr: 0.0220 etc:Wed May 17 16:03:00 2023 540 | step: 3240/ 3966 lossCLS:0.0440 lossGSC:0.0136 loss_consistency:0.0025 imps:15.3 lr: 0.0217 etc:Wed May 17 16:02:56 2023 541 | step: 3250/ 3966 lossCLS:0.0545 lossGSC:0.0148 loss_consistency:0.0026 imps:15.4 lr: 0.0214 etc:Wed May 17 16:02:53 2023 542 | step: 3260/ 3966 lossCLS:0.0498 lossGSC:0.0146 loss_consistency:0.0026 imps:15.4 lr: 0.0212 etc:Wed May 17 16:02:50 2023 543 | step: 3270/ 3966 lossCLS:0.0437 lossGSC:0.0141 loss_consistency:0.0026 imps:15.5 lr: 0.0209 etc:Wed May 17 16:02:46 2023 544 | step: 3280/ 3966 lossCLS:0.0378 lossGSC:0.0152 loss_consistency:0.0026 imps:15.5 lr: 0.0206 etc:Wed May 17 16:02:43 2023 545 | step: 3290/ 3966 lossCLS:0.0487 lossGSC:0.0146 loss_consistency:0.0027 imps:15.6 lr: 0.0203 etc:Wed May 17 16:02:39 2023 546 | step: 3300/ 3966 lossCLS:0.0573 lossGSC:0.0155 loss_consistency:0.0025 imps:15.6 lr: 0.0201 etc:Wed May 17 16:02:35 2023 547 | validating ... 548 | 549 | {'iou': array([0.8509339 , 0.54780349, 0.31059451, 0.65035346, 0.48142227, 550 | 0.54417585, 0.75375424, 0.70244381, 0.69654393, 0.2883562 , 551 | 0.69615245, 0.39353266, 0.70667371, 0.67835877, 0.70792417, 552 | 0.61584147, 0.47621585, 0.7403559 , 0.4350587 , 0.60544243, 553 | 0.46104892]), 'miou': 0.587761269923074} 554 | 555 | 556 | {'iou1': array([0.8416697 , 0.54718231, 0.31730883, 0.61691047, 0.48911521, 557 | 0.49711072, 0.73980523, 0.65086238, 0.67132374, 0.27921045, 558 | 0.67715639, 0.36857675, 0.66251221, 0.66129375, 0.67969557, 559 | 0.62060096, 0.42730735, 0.69871682, 0.4378442 , 0.59720124, 560 | 0.41883166]), 'miou': 0.5666779017546635} 561 | 562 | 563 | {'iou2': array([0.8463018 , 0.5474929 , 0.31395167, 0.63363196, 0.48526874, 564 | 0.52064329, 0.74677974, 0.67665309, 0.68393383, 0.28378332, 565 | 0.68665442, 0.3810547 , 0.68459296, 0.66982626, 0.69380987, 566 | 0.61822121, 0.4517616 , 0.71953636, 0.43645145, 0.60132183, 567 | 0.43994029]), 'miou': 0.5772195858388687} 568 | Epoch 6/6 569 | step: 3310/ 3966 lossCLS:0.0512 lossGSC:0.0146 loss_consistency:0.0024 imps:12.0 lr: 0.0198 etc:Wed May 17 16:04:28 2023 570 | step: 3320/ 3966 lossCLS:0.0507 lossGSC:0.0143 loss_consistency:0.0024 imps:16.5 lr: 0.0195 etc:Wed May 17 16:04:23 2023 571 | step: 3330/ 3966 lossCLS:0.0406 lossGSC:0.0150 loss_consistency:0.0026 imps:18.2 lr: 0.0193 etc:Wed May 17 16:04:19 2023 572 | step: 3340/ 3966 lossCLS:0.0486 lossGSC:0.0154 loss_consistency:0.0025 imps:19.3 lr: 0.0190 etc:Wed May 17 16:04:14 2023 573 | step: 3350/ 3966 lossCLS:0.0355 lossGSC:0.0146 loss_consistency:0.0026 imps:19.8 lr: 0.0187 etc:Wed May 17 16:04:10 2023 574 | step: 3360/ 3966 lossCLS:0.0344 lossGSC:0.0141 loss_consistency:0.0026 imps:20.2 lr: 0.0184 etc:Wed May 17 16:04:05 2023 575 | step: 3370/ 3966 lossCLS:0.0369 lossGSC:0.0145 loss_consistency:0.0023 imps:20.5 lr: 0.0182 etc:Wed May 17 16:04:00 2023 576 | step: 3380/ 3966 lossCLS:0.0474 lossGSC:0.0138 loss_consistency:0.0024 imps:20.8 lr: 0.0179 etc:Wed May 17 16:03:55 2023 577 | step: 3390/ 3966 lossCLS:0.0454 lossGSC:0.0145 loss_consistency:0.0026 imps:21.0 lr: 0.0176 etc:Wed May 17 16:03:51 2023 578 | step: 3400/ 3966 lossCLS:0.0388 lossGSC:0.0147 loss_consistency:0.0024 imps:21.3 lr: 0.0173 etc:Wed May 17 16:03:46 2023 579 | step: 3410/ 3966 lossCLS:0.0455 lossGSC:0.0137 loss_consistency:0.0025 imps:21.2 lr: 0.0171 etc:Wed May 17 16:03:42 2023 580 | step: 3420/ 3966 lossCLS:0.0473 lossGSC:0.0138 loss_consistency:0.0025 imps:21.3 lr: 0.0168 etc:Wed May 17 16:03:37 2023 581 | step: 3430/ 3966 lossCLS:0.0401 lossGSC:0.0145 loss_consistency:0.0026 imps:21.4 lr: 0.0165 etc:Wed May 17 16:03:33 2023 582 | step: 3440/ 3966 lossCLS:0.0351 lossGSC:0.0148 loss_consistency:0.0024 imps:21.4 lr: 0.0162 etc:Wed May 17 16:03:29 2023 583 | step: 3450/ 3966 lossCLS:0.0341 lossGSC:0.0141 loss_consistency:0.0025 imps:21.5 lr: 0.0160 etc:Wed May 17 16:03:24 2023 584 | step: 3460/ 3966 lossCLS:0.0565 lossGSC:0.0154 loss_consistency:0.0030 imps:21.5 lr: 0.0157 etc:Wed May 17 16:03:20 2023 585 | step: 3470/ 3966 lossCLS:0.0377 lossGSC:0.0148 loss_consistency:0.0026 imps:21.6 lr: 0.0154 etc:Wed May 17 16:03:16 2023 586 | step: 3480/ 3966 lossCLS:0.0343 lossGSC:0.0144 loss_consistency:0.0024 imps:21.6 lr: 0.0151 etc:Wed May 17 16:03:12 2023 587 | step: 3490/ 3966 lossCLS:0.0381 lossGSC:0.0149 loss_consistency:0.0026 imps:21.6 lr: 0.0148 etc:Wed May 17 16:03:07 2023 588 | step: 3500/ 3966 lossCLS:0.0304 lossGSC:0.0147 loss_consistency:0.0027 imps:21.6 lr: 0.0146 etc:Wed May 17 16:03:03 2023 589 | step: 3510/ 3966 lossCLS:0.0352 lossGSC:0.0147 loss_consistency:0.0026 imps:21.5 lr: 0.0143 etc:Wed May 17 16:03:00 2023 590 | step: 3520/ 3966 lossCLS:0.0345 lossGSC:0.0139 loss_consistency:0.0022 imps:21.4 lr: 0.0140 etc:Wed May 17 16:02:57 2023 591 | step: 3530/ 3966 lossCLS:0.0319 lossGSC:0.0154 loss_consistency:0.0026 imps:21.4 lr: 0.0137 etc:Wed May 17 16:02:53 2023 592 | step: 3540/ 3966 lossCLS:0.0472 lossGSC:0.0142 loss_consistency:0.0026 imps:21.4 lr: 0.0134 etc:Wed May 17 16:02:50 2023 593 | step: 3550/ 3966 lossCLS:0.0347 lossGSC:0.0144 loss_consistency:0.0026 imps:21.3 lr: 0.0131 etc:Wed May 17 16:02:47 2023 594 | step: 3560/ 3966 lossCLS:0.0444 lossGSC:0.0152 loss_consistency:0.0026 imps:21.1 lr: 0.0129 etc:Wed May 17 16:02:45 2023 595 | step: 3570/ 3966 lossCLS:0.0459 lossGSC:0.0142 loss_consistency:0.0024 imps:21.0 lr: 0.0126 etc:Wed May 17 16:02:42 2023 596 | step: 3580/ 3966 lossCLS:0.0390 lossGSC:0.0141 loss_consistency:0.0024 imps:21.0 lr: 0.0123 etc:Wed May 17 16:02:38 2023 597 | step: 3590/ 3966 lossCLS:0.0352 lossGSC:0.0144 loss_consistency:0.0024 imps:20.9 lr: 0.0120 etc:Wed May 17 16:02:36 2023 598 | step: 3600/ 3966 lossCLS:0.0461 lossGSC:0.0144 loss_consistency:0.0023 imps:20.9 lr: 0.0117 etc:Wed May 17 16:02:32 2023 599 | validating ... 600 | 601 | {'iou': array([0.85134051, 0.53954292, 0.31547449, 0.65857537, 0.47923428, 602 | 0.54327516, 0.74510557, 0.71078296, 0.70242589, 0.30309304, 603 | 0.70155359, 0.35995472, 0.71316768, 0.67869783, 0.70953779, 604 | 0.6282945 , 0.49357692, 0.74903515, 0.43493794, 0.59235522, 605 | 0.46425632]), 'miou': 0.5892484677770329} 606 | 607 | 608 | {'iou1': array([0.84031708, 0.53868594, 0.3138149 , 0.61866711, 0.48293811, 609 | 0.49213456, 0.73283103, 0.65827622, 0.67119435, 0.28530479, 610 | 0.68105412, 0.33893297, 0.66705396, 0.66403253, 0.67590682, 611 | 0.62830987, 0.44310052, 0.70436631, 0.43580384, 0.58677813, 612 | 0.4125866 ]), 'miou': 0.5653376069460302} 613 | 614 | 615 | {'iou2': array([0.84582879, 0.53911443, 0.31464469, 0.63862124, 0.4810862 , 616 | 0.51770486, 0.7389683 , 0.68452959, 0.68681012, 0.29419892, 617 | 0.69130386, 0.34944384, 0.69011082, 0.67136518, 0.6927223 , 618 | 0.62830218, 0.46833872, 0.72670073, 0.43537089, 0.58956668, 619 | 0.43842146]), 'miou': 0.5772930373615315} 620 | step: 3610/ 3966 lossCLS:0.0400 lossGSC:0.0142 loss_consistency:0.0026 imps:15.1 lr: 0.0114 etc:Wed May 17 16:04:07 2023 621 | step: 3620/ 3966 lossCLS:0.0394 lossGSC:0.0139 loss_consistency:0.0023 imps:15.3 lr: 0.0111 etc:Wed May 17 16:04:03 2023 622 | step: 3630/ 3966 lossCLS:0.0334 lossGSC:0.0140 loss_consistency:0.0024 imps:15.4 lr: 0.0108 etc:Wed May 17 16:03:59 2023 623 | step: 3640/ 3966 lossCLS:0.0352 lossGSC:0.0152 loss_consistency:0.0026 imps:15.6 lr: 0.0106 etc:Wed May 17 16:03:54 2023 624 | step: 3650/ 3966 lossCLS:0.0485 lossGSC:0.0143 loss_consistency:0.0024 imps:15.7 lr: 0.0103 etc:Wed May 17 16:03:50 2023 625 | step: 3660/ 3966 lossCLS:0.0421 lossGSC:0.0148 loss_consistency:0.0026 imps:15.9 lr: 0.0100 etc:Wed May 17 16:03:46 2023 626 | step: 3670/ 3966 lossCLS:0.0287 lossGSC:0.0132 loss_consistency:0.0024 imps:16.0 lr: 0.0097 etc:Wed May 17 16:03:41 2023 627 | step: 3680/ 3966 lossCLS:0.0494 lossGSC:0.0148 loss_consistency:0.0026 imps:16.1 lr: 0.0094 etc:Wed May 17 16:03:37 2023 628 | step: 3690/ 3966 lossCLS:0.0430 lossGSC:0.0155 loss_consistency:0.0026 imps:16.3 lr: 0.0091 etc:Wed May 17 16:03:33 2023 629 | step: 3700/ 3966 lossCLS:0.0345 lossGSC:0.0136 loss_consistency:0.0024 imps:16.4 lr: 0.0088 etc:Wed May 17 16:03:28 2023 630 | step: 3710/ 3966 lossCLS:0.0374 lossGSC:0.0144 loss_consistency:0.0025 imps:16.5 lr: 0.0085 etc:Wed May 17 16:03:25 2023 631 | step: 3720/ 3966 lossCLS:0.0388 lossGSC:0.0142 loss_consistency:0.0024 imps:16.6 lr: 0.0082 etc:Wed May 17 16:03:20 2023 632 | step: 3730/ 3966 lossCLS:0.0425 lossGSC:0.0149 loss_consistency:0.0023 imps:16.7 lr: 0.0079 etc:Wed May 17 16:03:16 2023 633 | step: 3740/ 3966 lossCLS:0.0334 lossGSC:0.0156 loss_consistency:0.0026 imps:16.8 lr: 0.0076 etc:Wed May 17 16:03:12 2023 634 | step: 3750/ 3966 lossCLS:0.0355 lossGSC:0.0152 loss_consistency:0.0025 imps:16.9 lr: 0.0073 etc:Wed May 17 16:03:08 2023 635 | step: 3760/ 3966 lossCLS:0.0439 lossGSC:0.0142 loss_consistency:0.0025 imps:17.0 lr: 0.0070 etc:Wed May 17 16:03:04 2023 636 | step: 3770/ 3966 lossCLS:0.0407 lossGSC:0.0150 loss_consistency:0.0026 imps:17.1 lr: 0.0067 etc:Wed May 17 16:03:00 2023 637 | step: 3780/ 3966 lossCLS:0.0432 lossGSC:0.0156 loss_consistency:0.0026 imps:17.2 lr: 0.0064 etc:Wed May 17 16:02:56 2023 638 | step: 3790/ 3966 lossCLS:0.0321 lossGSC:0.0143 loss_consistency:0.0024 imps:17.2 lr: 0.0061 etc:Wed May 17 16:02:52 2023 639 | step: 3800/ 3966 lossCLS:0.0377 lossGSC:0.0146 loss_consistency:0.0026 imps:17.3 lr: 0.0057 etc:Wed May 17 16:02:49 2023 640 | step: 3810/ 3966 lossCLS:0.0322 lossGSC:0.0145 loss_consistency:0.0027 imps:17.3 lr: 0.0054 etc:Wed May 17 16:02:47 2023 641 | step: 3820/ 3966 lossCLS:0.0423 lossGSC:0.0151 loss_consistency:0.0026 imps:17.3 lr: 0.0051 etc:Wed May 17 16:02:44 2023 642 | step: 3830/ 3966 lossCLS:0.0324 lossGSC:0.0149 loss_consistency:0.0027 imps:17.4 lr: 0.0048 etc:Wed May 17 16:02:41 2023 643 | step: 3840/ 3966 lossCLS:0.0373 lossGSC:0.0144 loss_consistency:0.0026 imps:17.4 lr: 0.0045 etc:Wed May 17 16:02:39 2023 644 | step: 3850/ 3966 lossCLS:0.0429 lossGSC:0.0146 loss_consistency:0.0023 imps:17.4 lr: 0.0042 etc:Wed May 17 16:02:36 2023 645 | step: 3860/ 3966 lossCLS:0.0325 lossGSC:0.0133 loss_consistency:0.0022 imps:17.5 lr: 0.0038 etc:Wed May 17 16:02:33 2023 646 | step: 3870/ 3966 lossCLS:0.0307 lossGSC:0.0139 loss_consistency:0.0024 imps:17.5 lr: 0.0035 etc:Wed May 17 16:02:31 2023 647 | step: 3880/ 3966 lossCLS:0.0477 lossGSC:0.0138 loss_consistency:0.0023 imps:17.5 lr: 0.0032 etc:Wed May 17 16:02:28 2023 648 | step: 3890/ 3966 lossCLS:0.0468 lossGSC:0.0140 loss_consistency:0.0026 imps:17.6 lr: 0.0028 etc:Wed May 17 16:02:25 2023 649 | step: 3900/ 3966 lossCLS:0.0447 lossGSC:0.0147 loss_consistency:0.0025 imps:17.6 lr: 0.0025 etc:Wed May 17 16:02:22 2023 650 | validating ... 651 | 652 | {'iou': array([0.85174836, 0.5422535 , 0.31013453, 0.64288914, 0.49461818, 653 | 0.53288195, 0.74483105, 0.69623926, 0.69371591, 0.29480375, 654 | 0.69856666, 0.35365025, 0.71079008, 0.67598771, 0.70619216, 655 | 0.61506762, 0.48395538, 0.74144284, 0.43065725, 0.60151721, 656 | 0.45906223]), 'miou': 0.5848097628798994} 657 | 658 | 659 | {'iou1': array([0.84202992, 0.53953365, 0.31154715, 0.61293742, 0.49806334, 660 | 0.48614425, 0.73315915, 0.65438936, 0.66903546, 0.28105996, 661 | 0.67852403, 0.34081464, 0.66811753, 0.66348064, 0.67311052, 662 | 0.62107273, 0.43885245, 0.69846425, 0.42471386, 0.59285971, 663 | 0.40739304]), 'miou': 0.5635858608745427} 664 | 665 | 666 | {'iou2': array([0.84688914, 0.54089358, 0.31084084, 0.62791328, 0.49634076, 667 | 0.5095131 , 0.7389951 , 0.67531431, 0.68137569, 0.28793185, 668 | 0.68854534, 0.34723245, 0.68945381, 0.66973418, 0.68965134, 669 | 0.61807017, 0.46140391, 0.71995355, 0.42768556, 0.59718846, 670 | 0.43322763]), 'miou': 0.5741978118772211} 671 | step: 3910/ 3966 lossCLS:0.0384 lossGSC:0.0139 loss_consistency:0.0027 imps:15.3 lr: 0.0022 etc:Wed May 17 16:03:47 2023 672 | step: 3920/ 3966 lossCLS:0.0501 lossGSC:0.0140 loss_consistency:0.0026 imps:15.3 lr: 0.0018 etc:Wed May 17 16:03:44 2023 673 | step: 3930/ 3966 lossCLS:0.0421 lossGSC:0.0138 loss_consistency:0.0023 imps:15.4 lr: 0.0015 etc:Wed May 17 16:03:41 2023 674 | step: 3940/ 3966 lossCLS:0.0345 lossGSC:0.0139 loss_consistency:0.0026 imps:15.4 lr: 0.0011 etc:Wed May 17 16:03:38 2023 675 | step: 3950/ 3966 lossCLS:0.0302 lossGSC:0.0139 loss_consistency:0.0024 imps:15.5 lr: 0.0007 etc:Wed May 17 16:03:35 2023 676 | step: 3960/ 3966 lossCLS:0.0363 lossGSC:0.0149 loss_consistency:0.0027 imps:15.5 lr: 0.0003 etc:Wed May 17 16:03:31 2023 677 | -------------------------------------------------------------------------------- /exp/infer.log: -------------------------------------------------------------------------------- 1 | {'network': 'network.resnet50_SIPE_Best', 'num_workers': 20, 'session_name': 'exp', 'ckpt': 'best.pth', 'visualize': True, 'dataset': 'voc'} 2 | [ ] 3 | -------------------------------------------------------------------------------- /fig/framework1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Barrett-python/SFC/4f624a92aea10d8be2049f6cff53639d31507136/fig/framework1.png -------------------------------------------------------------------------------- /make_cam.py: -------------------------------------------------------------------------------- 1 | from builtins import bool 2 | import torch 3 | from torch import multiprocessing, cuda 4 | from torch.utils.data import DataLoader 5 | import torch.nn.functional as F 6 | from torch.backends import cudnn 7 | from PIL import Image 8 | import matplotlib.pyplot as plt 9 | import numpy as np 10 | import importlib 11 | import os 12 | import imageio 13 | import argparse 14 | from data import data_voc, data_coco 15 | from tool import torchutils, pyutils 16 | from tool import torchutils, imutils 17 | import torch 18 | cudnn.enabled = True 19 | 20 | def overlap(img, hm): 21 | hm = plt.cm.jet(hm)[:, :, :3] 22 | hm = np.array(Image.fromarray((hm*255).astype(np.uint8), 'RGB').resize((img.shape[1], img.shape[0]), Image.BICUBIC)).astype(np.float)*2 23 | if hm.shape == np.array(img).astype(np.float).shape: 24 | out = (hm + np.array(img).astype(np.float)) / 3 25 | out = (out / np.max(out) * 255).astype(np.uint8) 26 | else: 27 | print(hm.shape) 28 | print(np.array(img).shape) 29 | return out 30 | 31 | def draw_heatmap(norm_cam, gt_label, orig_img, save_path, img_name): 32 | gt_cat = np.where(gt_label==1)[0] 33 | for _, gt in enumerate(gt_cat): 34 | heatmap = overlap(orig_img, norm_cam[gt]) 35 | cam_viz_path = os.path.join(save_path, img_name + '_{}.png'.format(gt)) 36 | imageio.imsave(cam_viz_path, heatmap) 37 | 38 | 39 | def _work(process_id, model, dataset, args): 40 | databin = dataset[process_id] 41 | n_gpus = torch.cuda.device_count() 42 | data_loader = DataLoader(databin, shuffle=False, num_workers=args.num_workers // n_gpus, pin_memory=False) 43 | 44 | with torch.no_grad(), cuda.device(process_id): 45 | 46 | model.cuda() 47 | 48 | for iter, pack in enumerate(data_loader): 49 | img_name = pack['name'][0] # 2007_000032 50 | label = pack['label'][0] # torch.Size([21]) 51 | size = pack['size'] # tensor([281]) 1:tensor([500]) 52 | label = F.pad(label, (1, 0), 'constant', 1.0) # torch.Size([21]) 53 | 54 | outputs = [model(img[0].cuda(non_blocking=True), label.cuda(non_blocking=True).unsqueeze(-1).unsqueeze(-1)) for img in pack['img']] 55 | 56 | # multi-scale fusion 57 | IS_CAM_list = [output[1].cpu() for output in outputs] # 4 * torch.Size([21, 18, 32]) 58 | IS_CAM_list = [F.interpolate(torch.unsqueeze(o, 1), size, mode='bilinear', align_corners=False) for o in IS_CAM_list] # torch.Size([21, 1, 281, 500]) 59 | IS_CAM = torch.sum(torch.stack(IS_CAM_list, 0), 0)[:,0] # torch.Size([21, 281, 500]) 60 | 61 | IS_CAM /= F.adaptive_max_pool2d(IS_CAM, (1, 1)) + 1e-5 # torch.Size([21, 281, 500]) 62 | 63 | IS_CAM = IS_CAM + np.pad(IS_CAM[1:, ...], ((1, 0), (0, 0), (0, 0)), mode='constant', constant_values=0.4) 64 | IS_CAM = IS_CAM.cpu().numpy() 65 | prototye = IS_CAM 66 | # save IS_CAM 67 | valid_cat = torch.nonzero(label)[:, 0].cpu().numpy() 68 | IS_CAM = IS_CAM[valid_cat] 69 | 70 | IS_CAM_list1 = [output[0].cpu() for output in outputs] # 4 * torch.Size([21, 18, 32]) 71 | IS_CAM_list1 = [F.interpolate(torch.unsqueeze(o, 1), size, mode='bilinear', align_corners=False) for o in IS_CAM_list1] # torch.Size([21, 1, 281, 500]) 72 | IS_CAM1 = torch.sum(torch.stack(IS_CAM_list1, 0), 0)[:,0] # torch.Size([21, 281, 500]) 73 | IS_CAM1 /= F.adaptive_max_pool2d(IS_CAM1, (1, 1)) + 1e-5 # torch.Size([21, 281, 500]) 74 | IS_CAM1 = IS_CAM1 + np.pad(IS_CAM1[1:, ...], ((1, 0), (0, 0), (0, 0)), mode='constant', constant_values=0.4) 75 | IS_CAM1 = IS_CAM1.cpu().numpy() 76 | cam_classifier = IS_CAM1 77 | # save IS_CAM 78 | valid_cat = torch.nonzero(label)[:, 0].cpu().numpy() 79 | IS_CAM1 = IS_CAM1[valid_cat] 80 | 81 | # final_cam = prototye*0.3 + cam_classifier 82 | # if args.visualize: 83 | # orig_img = np.array(Image.open(pack['img_path'][0]).convert('RGB')) 84 | # draw_heatmap(cam_classifier.copy(), label, orig_img, os.path.join(args.session_name, 'visual_classifier'), img_name) 85 | # draw_heatmap(prototye.copy(), label, orig_img, os.path.join(args.session_name, 'visual_prototye'), img_name) 86 | # draw_heatmap(final_cam.copy(), label, orig_img, os.path.join(args.session_name, 'final_cam'), img_name) 87 | 88 | np.save(os.path.join(args.session_name, 'npy', img_name + '.npy'), {"keys": valid_cat, "IS_CAM": IS_CAM, "IS_CAM1": IS_CAM1}) 89 | 90 | if process_id == n_gpus - 1 and iter % (len(databin) // 20) == 0: 91 | print("%d " % ((5*iter+1)//(len(databin) // 20)), end='') 92 | 93 | 94 | if __name__ == '__main__': 95 | 96 | parser = argparse.ArgumentParser() 97 | parser.add_argument("--network", default="network.resnet50_SIPE_Best", type=str) 98 | parser.add_argument("--num_workers", default=os.cpu_count()//2, type=int) 99 | parser.add_argument("--session_name", default="exp", type=str) 100 | parser.add_argument("--ckpt", default="best.pth", type=str) 101 | parser.add_argument("--visualize", default=True, type=bool) 102 | parser.add_argument("--dataset", default="voc", type=str) 103 | 104 | args = parser.parse_args() 105 | 106 | os.makedirs(os.path.join(args.session_name, 'npy'), exist_ok=True) 107 | os.makedirs(os.path.join(args.session_name, 'visual_classifier'), exist_ok=True) 108 | os.makedirs(os.path.join(args.session_name, 'visual_prototye'), exist_ok=True) 109 | 110 | pyutils.Logger(os.path.join(args.session_name, 'infer.log')) 111 | print(vars(args)) 112 | 113 | assert args.dataset in ['voc', 'coco'], 'Dataset must be voc or coco in this project.' 114 | 115 | if args.dataset == 'voc': 116 | dataset_root = '/data/tfl/VOCdevkit/VOC2012' 117 | model = getattr(importlib.import_module(args.network), 'CAM')(num_cls=21) 118 | dataset = data_voc.VOC12ClsDatasetMSF('data/trainaug_' + args.dataset + '.txt', voc12_root=dataset_root, scales=(1.0, 0.5, 1.5, 2.0)) 119 | 120 | elif args.dataset == 'coco': 121 | dataset_root = "/data/tfl/COCO/" 122 | model = getattr(importlib.import_module(args.network), 'CAM')(num_cls=81) 123 | dataset = data_coco.COCOClsDatasetMSF('data/train_' + args.dataset + '.txt', coco_root=dataset_root, scales=(1.0, 0.5, 1.5, 2.0)) 124 | 125 | checkpoint = torch.load(args.session_name + '/ckpt/' + args.ckpt) 126 | model.load_state_dict(checkpoint['net'], strict=True) 127 | model.eval() 128 | 129 | n_gpus = torch.cuda.device_count() 130 | 131 | dataset = torchutils.split_dataset(dataset, n_gpus) 132 | 133 | print('[ ', end='') 134 | multiprocessing.spawn(_work, nprocs=n_gpus, args=(model, dataset, args), join=True) 135 | print(']') 136 | 137 | torch.cuda.empty_cache() -------------------------------------------------------------------------------- /make_seg_labels.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from torch import multiprocessing, cuda 3 | from torch.utils.data import DataLoader 4 | import torch.nn.functional as F 5 | from torch.backends import cudnn 6 | 7 | import argparse 8 | import numpy as np 9 | import importlib 10 | import os 11 | import imageio 12 | from PIL import Image 13 | 14 | from data import data_voc 15 | from tool import torchutils, indexing 16 | 17 | cudnn.enabled = True 18 | 19 | palette = [0,0,0, 128,0,0, 0,128,0, 128,128,0, 0,0,128, 128,0,128, 0,128,128, 128,128,128, 20 | 64,0,0, 192,0,0, 64,128,0, 192,128,0, 64,0,128, 192,0,128, 64,128,128, 192,128,128, 21 | 0,64,0, 128,64,0, 0,192,0, 128,192,0, 0,64,128, 128,64,128, 0,192,128, 128,192,128, 22 | 64,64,0, 192,64,0, 64,192,0, 192,192,0] 23 | 24 | def _work(process_id, model, dataset, args): 25 | 26 | n_gpus = torch.cuda.device_count() 27 | databin = dataset[process_id] 28 | data_loader = DataLoader(databin, 29 | shuffle=False, num_workers=args.num_workers // n_gpus, pin_memory=False) 30 | 31 | cam_out_dir = os.path.join(args.session_name, 'npy') 32 | 33 | with torch.no_grad(), cuda.device(process_id): 34 | 35 | model.cuda() 36 | 37 | for iter, pack in enumerate(data_loader): 38 | img_name = pack['name'][0] 39 | orig_img_size = np.array(pack['size']) 40 | 41 | edge, _ = model(pack['img'][0].cuda(non_blocking=True)) 42 | 43 | cam_dict = np.load(cam_out_dir + '/' + img_name + '.npy', allow_pickle=True).item() 44 | 45 | cams = cam_dict['IS_CAM1'] + cam_dict['IS_CAM']*0.3 46 | 47 | keys = cam_dict['keys'] 48 | cams = F.interpolate(torch.tensor(cams).unsqueeze(1), edge.shape[1:], mode='bilinear', align_corners=False)[:,0] 49 | cams = np.power(cams, 1.3) 50 | cam_downsized_values = cams[1:, ...].cuda() 51 | rw = indexing.propagate_to_edge(cam_downsized_values, edge, beta=args.beta, exp_times=args.exp_times, radius=5) 52 | rw_up = F.interpolate(rw, scale_factor=4, mode='bilinear', align_corners=False)[..., 0, :orig_img_size[0], :orig_img_size[1]] 53 | rw_up = rw_up / torch.max(rw_up) 54 | rw_up_bg = F.pad(rw_up, (0, 0, 0, 0, 1, 0), value=0.25) 55 | 56 | cams = F.interpolate(torch.tensor(cams).unsqueeze(1), edge.shape[1:], mode='bilinear', align_corners=False)[:,0] 57 | cams = np.power(cams, 1.3) 58 | cam_downsized_values = cams.cuda() 59 | rw = indexing.propagate_to_edge(cam_downsized_values, edge, beta=args.beta, exp_times=args.exp_times, radius=5) 60 | rw_up = F.interpolate(rw, scale_factor=4, mode='bilinear', align_corners=False)[..., 0, :orig_img_size[0], :orig_img_size[1]] 61 | rw_up = rw_up / torch.max(rw_up) 62 | 63 | rw_pred = torch.argmax(rw_up_bg + rw_up, dim=0).cpu().numpy() 64 | rw_pred = keys[rw_pred] 65 | imageio.imsave(os.path.join(args.sem_seg_out_dir, img_name + '.png'), rw_pred.astype(np.uint8)) 66 | 67 | 68 | # show color map 69 | out = Image.fromarray(rw_pred.astype(np.uint8), mode='P') 70 | out.putpalette(palette) 71 | out.save(os.path.join(os.path.join(args.sem_seg_out_dir, img_name + '_palette.png'))) 72 | 73 | if process_id == n_gpus - 1 and iter % (len(databin) // 20) == 0: 74 | print("%d " % ((5*iter+1)//(len(databin) // 20)), end='') 75 | 76 | 77 | if __name__ == '__main__': 78 | 79 | parser = argparse.ArgumentParser() 80 | # Inter-pixel Relation Network (IRNet) 81 | parser.add_argument("--num_workers", default=os.cpu_count()//2, type=int) 82 | parser.add_argument("--infer_list", default="data/train_voc.txt", type=str) 83 | parser.add_argument("--voc12_root", default="/data/tfl/VOCdevkit/VOC2012", type=str) 84 | parser.add_argument("--irn_network", default="network.resnet50_irn", type=str) 85 | parser.add_argument("--session_name", default="exp", type=str) 86 | # Random Walk Params 87 | parser.add_argument("--beta", default=10) 88 | parser.add_argument("--exp_times", default=8, 89 | help="Hyper-parameter that controls the number of random walk iterations," 90 | "The random walk is performed 2^{exp_times}.") 91 | parser.add_argument("--sem_seg_out_dir", default="", type=str) 92 | args = parser.parse_args() 93 | 94 | model = getattr(importlib.import_module(args.irn_network), 'EdgeDisplacement')() 95 | irn_weights_path = os.path.join(args.session_name, 'ckpt', 'irn.pth') 96 | model.load_state_dict(torch.load(irn_weights_path), strict=False) 97 | model.eval() 98 | n_gpus = torch.cuda.device_count() 99 | dataset = data_voc.VOC12ClsDatasetMSF(args.infer_list, voc12_root=args.voc12_root, scales=(1.0,)) 100 | dataset = torchutils.split_dataset(dataset, n_gpus) 101 | 102 | args.sem_seg_out_dir = os.path.join(args.session_name, 'pseudo_label') 103 | os.makedirs(args.sem_seg_out_dir, exist_ok=True) 104 | 105 | print("[", end='') 106 | multiprocessing.spawn(_work, nprocs=n_gpus, args=(model, dataset, args), join=True) 107 | print("]") 108 | 109 | torch.cuda.empty_cache() 110 | -------------------------------------------------------------------------------- /network/resnet50.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import torch.nn.functional as F 3 | import torch.utils.model_zoo as model_zoo 4 | 5 | model_urls = { 6 | 'resnet50': 'https://download.pytorch.org/models/resnet50-19c8e357.pth', 7 | 'resnet101': 'https://download.pytorch.org/models/resnet101-63fe2227.pth', 8 | } 9 | 10 | class FixedBatchNorm(nn.BatchNorm2d): 11 | def forward(self, input): 12 | return F.batch_norm(input, self.running_mean, self.running_var, self.weight, self.bias, 13 | training=False, eps=self.eps) 14 | 15 | 16 | class Bottleneck(nn.Module): 17 | expansion = 4 18 | 19 | def __init__(self, inplanes, planes, stride=1, downsample=None, dilation=1): 20 | super(Bottleneck, self).__init__() 21 | self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=1, bias=False) 22 | self.bn1 = FixedBatchNorm(planes) 23 | self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride, 24 | padding=dilation, bias=False, dilation=dilation) 25 | self.bn2 = FixedBatchNorm(planes) 26 | self.conv3 = nn.Conv2d(planes, planes * 4, kernel_size=1, bias=False) 27 | self.bn3 = FixedBatchNorm(planes * 4) 28 | self.relu = nn.ReLU(inplace=True) 29 | self.downsample = downsample 30 | self.stride = stride 31 | self.dilation = dilation 32 | 33 | def forward(self, x): 34 | residual = x 35 | 36 | out = self.conv1(x) 37 | out = self.bn1(out) 38 | out = self.relu(out) 39 | 40 | out = self.conv2(out) 41 | out = self.bn2(out) 42 | out = self.relu(out) 43 | 44 | out = self.conv3(out) 45 | out = self.bn3(out) 46 | 47 | if self.downsample is not None: 48 | residual = self.downsample(x) 49 | 50 | out += residual 51 | out = self.relu(out) 52 | 53 | return out 54 | 55 | 56 | class ResNet(nn.Module): 57 | 58 | def __init__(self, block, layers, strides=(2, 2, 2, 2), dilations=(1, 1, 1, 1)): 59 | self.inplanes = 64 60 | super(ResNet, self).__init__() 61 | self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=strides[0], padding=3, 62 | bias=False) 63 | self.bn1 = FixedBatchNorm(64) 64 | self.relu = nn.ReLU(inplace=True) 65 | self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1) 66 | self.layer1 = self._make_layer(block, 64, layers[0], stride=1, dilation=dilations[0]) 67 | self.layer2 = self._make_layer(block, 128, layers[1], stride=strides[1], dilation=dilations[1]) 68 | self.layer3 = self._make_layer(block, 256, layers[2], stride=strides[2], dilation=dilations[2]) 69 | self.layer4 = self._make_layer(block, 512, layers[3], stride=strides[3], dilation=dilations[3]) 70 | self.inplanes = 1024 71 | 72 | #self.avgpool = nn.AvgPool2d(7, stride=1) 73 | #self.fc = nn.Linear(512 * block.expansion, 1000) 74 | 75 | 76 | def _make_layer(self, block, planes, blocks, stride=1, dilation=1): 77 | downsample = None 78 | if stride != 1 or self.inplanes != planes * block.expansion: 79 | downsample = nn.Sequential( 80 | nn.Conv2d(self.inplanes, planes * block.expansion, 81 | kernel_size=1, stride=stride, bias=False), 82 | FixedBatchNorm(planes * block.expansion), 83 | ) 84 | 85 | layers = [block(self.inplanes, planes, stride, downsample, dilation=1)] 86 | self.inplanes = planes * block.expansion 87 | for i in range(1, blocks): 88 | layers.append(block(self.inplanes, planes, dilation=dilation)) 89 | 90 | return nn.Sequential(*layers) 91 | 92 | def forward(self, x): 93 | x = self.conv1(x) 94 | x = self.bn1(x) 95 | x = self.relu(x) 96 | x = self.maxpool(x) 97 | 98 | x = self.layer1(x) 99 | x = self.layer2(x) 100 | x = self.layer3(x) 101 | x = self.layer4(x) 102 | 103 | x = self.avgpool(x) 104 | x = x.view(x.size(0), -1) 105 | x = self.fc(x) 106 | 107 | return x 108 | 109 | 110 | def resnet50(pretrained=True, **kwargs): 111 | 112 | model = ResNet(Bottleneck, [3, 4, 6, 3], **kwargs) 113 | if pretrained: 114 | state_dict = model_zoo.load_url(model_urls['resnet50']) 115 | state_dict.pop('fc.weight') 116 | state_dict.pop('fc.bias') 117 | model.load_state_dict(state_dict) 118 | return model 119 | 120 | def resnet101(pretrained=True, **kwargs): 121 | 122 | model = ResNet(Bottleneck, [3, 4, 23, 3], **kwargs) 123 | if pretrained: 124 | state_dict = model_zoo.load_url(model_urls['resnet101']) 125 | state_dict.pop('fc.weight') 126 | state_dict.pop('fc.bias') 127 | model.load_state_dict(state_dict) 128 | return model -------------------------------------------------------------------------------- /network/resnet50_SFC.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import torch.nn.functional as F 4 | from tool import torchutils 5 | from network import resnet50 6 | 7 | 8 | 9 | class Net(nn.Module): 10 | 11 | def __init__(self, num_cls=21): 12 | super(Net, self).__init__() 13 | 14 | self.num_cls = num_cls 15 | 16 | self.resnet50 = resnet50.resnet50(pretrained=True, strides=(2, 2, 2, 1), dilations=(1, 1, 1, 1)) 17 | 18 | self.stage0 = nn.Sequential(self.resnet50.conv1, self.resnet50.bn1, self.resnet50.relu, self.resnet50.maxpool) 19 | self.stage1 = nn.Sequential(self.resnet50.layer1) 20 | self.stage2 = nn.Sequential(self.resnet50.layer2) 21 | self.stage3 = nn.Sequential(self.resnet50.layer3) 22 | self.stage4 = nn.Sequential(self.resnet50.layer4) 23 | 24 | self.side1 = nn.Conv2d(256, 128, 1, bias=False) 25 | self.side2 = nn.Conv2d(512, 128, 1, bias=False) 26 | self.side3 = nn.Conv2d(1024, 256, 1, bias=False) 27 | self.side4 = nn.Conv2d(2048, 256, 1, bias=False) 28 | self.classifier = nn.Conv2d(2048, self.num_cls-1, 1, bias=False) 29 | 30 | self.f9 = torch.nn.Conv2d(3+2048, 2048, 1, bias=False) 31 | 32 | self.backbone = nn.ModuleList([self.stage0, self.stage1, self.stage2, self.stage3, self.stage4]) 33 | self.newly_added = nn.ModuleList([self.classifier, self.f9, self.side1, self.side2, self.side3, self.side4]) 34 | 35 | def PCM(self, cam, f): 36 | n,c,h,w = f.size() 37 | cam = F.interpolate(cam, (h,w), mode='bilinear', align_corners=True).view(n,-1,h*w) 38 | f = self.f9(f) 39 | f = f.view(n,-1,h*w) 40 | f = f/(torch.norm(f,dim=1,keepdim=True)+1e-5) 41 | aff = F.relu(torch.matmul(f.transpose(1,2), f),inplace=True) 42 | aff = aff/(torch.sum(aff,dim=1,keepdim=True)+1e-5) 43 | cam_rv = torch.matmul(cam, aff).view(n,-1,h,w) 44 | cam_rv = torch.matmul(cam_rv.view(n,-1,h*w), aff).view(n,-1,h,w) 45 | return cam_rv 46 | 47 | def prototype(self, norm_cam, feature, valid_mask): 48 | n,c,h,w = norm_cam.shape 49 | norm_cam[:,0] = norm_cam[:,0]*0.3 50 | seeds = torch.zeros((n,h,w,c)).cuda() 51 | belonging = norm_cam.argmax(1) 52 | seeds = seeds.scatter_(-1, belonging.view(n,h,w,1), 1).permute(0,3,1,2).contiguous() 53 | seeds = seeds * valid_mask # 4, 21, 32, 32 54 | 55 | n,c,h,w = feature.shape # hie 56 | seeds = F.interpolate(seeds, feature.shape[2:], mode='nearest') 57 | crop_feature = seeds.unsqueeze(2) * feature.unsqueeze(1) #.clone().detach() # seed:[n,21,1,h,w], feature:[n,1,4c,h,w], crop_feature:[n,21,4c,h,w] 58 | prototype = F.adaptive_avg_pool2d(crop_feature.view(-1,c,h,w), (1,1)).view(n, self.num_cls, c, 1, 1) # prototypes:[n,21,c,1,1] 59 | 60 | IS_cam = F.relu(torch.cosine_similarity(feature.unsqueeze(1), prototype, dim=2)) # feature:[n,1,4c,h,w], prototypes:[n,21,4c,1,1], crop_feature:[n,21,h,w] 61 | IS_cam = F.interpolate(IS_cam, feature.shape[2:], mode='bilinear', align_corners=True) 62 | return IS_cam 63 | 64 | def forward(self, x, valid_mask, my_label= None, epoch=None, index=None, train= None): 65 | 66 | x0 = self.stage0(x) 67 | x1 = self.stage1(x0) 68 | x2 = self.stage2(x1).detach() 69 | x3 = self.stage3(x2) 70 | x4 = self.stage4(x3) 71 | side1 = self.side1(x1.detach()) 72 | side2 = self.side2(x2.detach()) 73 | side3 = self.side3(x3.detach()) 74 | side4 = self.side4(x4.detach()) 75 | 76 | # print(self.side1.weight) 77 | sem_feature = x4 78 | cam = self.classifier(x4) 79 | score = F.adaptive_avg_pool2d(cam, 1) 80 | norm_cam = F.relu(cam) 81 | norm_cam = norm_cam/(F.adaptive_max_pool2d(norm_cam, (1, 1)) + 1e-5) 82 | cam_bkg = 1-torch.max(norm_cam,dim=1)[0].unsqueeze(1) 83 | norm_cam = torch.cat([cam_bkg, norm_cam], dim=1) 84 | norm_cam = F.interpolate(norm_cam, side3.shape[2:], mode='bilinear', align_corners=True)*valid_mask 85 | orignal_cam = norm_cam 86 | 87 | hie_fea = torch.cat([F.interpolate(side1/(torch.norm(side1,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear'), 88 | F.interpolate(side2/(torch.norm(side2,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear'), 89 | F.interpolate(side3/(torch.norm(side3,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear'), 90 | F.interpolate(side4/(torch.norm(side4,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear')], dim=1) 91 | 92 | norm_cam = self.PCM(norm_cam, torch.cat([F.interpolate(x, side3.shape[2:],mode='bilinear',align_corners=True), sem_feature], dim=1)) 93 | IS_cam = self.prototype(norm_cam.clone(), hie_fea.clone(), valid_mask.clone()) 94 | 95 | return {"score": score, "cam1": norm_cam, "cam2": IS_cam, "orignal_cam": orignal_cam} 96 | 97 | def train(self, mode=True): 98 | for p in self.resnet50.conv1.parameters(): 99 | p.requires_grad = False 100 | for p in self.resnet50.bn1.parameters(): 101 | p.requires_grad = False 102 | 103 | def trainable_parameters(self): 104 | return (list(self.backbone.parameters()), list(self.newly_added.parameters())) 105 | 106 | class CAM(Net): 107 | 108 | def __init__(self, num_cls): 109 | super(CAM, self).__init__(num_cls=num_cls) 110 | self.num_cls = num_cls 111 | 112 | def forward(self, x, label): 113 | 114 | x0 = self.stage0(x) 115 | x1 = self.stage1(x0) 116 | x2 = self.stage2(x1).detach() 117 | x3 = self.stage3(x2) 118 | x4 = self.stage4(x3) 119 | 120 | side1 = self.side1(x1.detach()) 121 | side2 = self.side2(x2.detach()) 122 | side3 = self.side3(x3.detach()) 123 | side4 = self.side4(x4.detach()) 124 | 125 | hie_fea = torch.cat([F.interpolate(side1/(torch.norm(side1,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear'), 126 | F.interpolate(side2/(torch.norm(side2,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear'), 127 | F.interpolate(side3/(torch.norm(side3,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear'), 128 | F.interpolate(side4/(torch.norm(side4,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear')], dim=1) 129 | 130 | cam = self.classifier(x4) 131 | cam = (cam[0] + cam[1].flip(-1)).unsqueeze(0) 132 | hie_fea = (hie_fea[0] + hie_fea[1].flip(-1)).unsqueeze(0) 133 | norm_cam = F.relu(cam) 134 | norm_cam = norm_cam/(F.adaptive_max_pool2d(norm_cam, (1, 1)) + 1e-5) 135 | cam_bkg = 1-torch.max(norm_cam,dim=1)[0].unsqueeze(1) 136 | norm_cam = torch.cat([cam_bkg, norm_cam], dim=1) 137 | norm_cam = F.interpolate(norm_cam, side3.shape[2:], mode='bilinear', align_corners=True) 138 | orignal_cam = norm_cam 139 | x = (x[0] + x[1].flip(-1)).unsqueeze(0) 140 | x4 = (x4[0] + x4[1].flip(-1)).unsqueeze(0) 141 | x3 = (x3[0] + x3[1].flip(-1)).unsqueeze(0) 142 | 143 | norm_cam = self.PCM(norm_cam, torch.cat([F.interpolate(x,side3.shape[2:],mode='bilinear',align_corners=True), x4], dim=1)) 144 | IS_cam = self.prototype(norm_cam.clone(), hie_fea.clone(), label.unsqueeze(0).clone()) 145 | return norm_cam[0], IS_cam[0], orignal_cam[0] -------------------------------------------------------------------------------- /network/resnet50_SIPE.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import torch.nn.functional as F 4 | from tool import torchutils 5 | from network import resnet50 6 | from sklearn.cluster import KMeans 7 | 8 | 9 | class Net(nn.Module): 10 | 11 | def __init__(self, num_cls=21): 12 | super(Net, self).__init__() 13 | 14 | self.num_cls = num_cls 15 | 16 | self.resnet50 = resnet50.resnet50(pretrained=True, strides=(2, 2, 2, 1), dilations=(1, 1, 1, 1)) 17 | 18 | self.stage0 = nn.Sequential(self.resnet50.conv1, self.resnet50.bn1, self.resnet50.relu, self.resnet50.maxpool) 19 | self.stage1 = nn.Sequential(self.resnet50.layer1) 20 | self.stage2 = nn.Sequential(self.resnet50.layer2) 21 | self.stage3 = nn.Sequential(self.resnet50.layer3) 22 | self.stage4 = nn.Sequential(self.resnet50.layer4) 23 | 24 | self.side1 = nn.Conv2d(256, 128, 1, bias=False) 25 | self.side2 = nn.Conv2d(512, 128, 1, bias=False) 26 | self.side3 = nn.Conv2d(1024, 256, 1, bias=False) 27 | self.side4 = nn.Conv2d(2048, 256, 1, bias=False) 28 | self.classifier = nn.Conv2d(2048, self.num_cls-1, 1, bias=False) 29 | 30 | self.f9 = torch.nn.Conv2d(3+2048+128, 2048, 1, bias=False) 31 | 32 | self.in_channels = 2048 33 | self.final_in_channels = 128 34 | self.lenk = 128 35 | self.inter_channels = 128 36 | self.Q = nn.Conv2d(in_channels=self.in_channels, out_channels=self.inter_channels, kernel_size=1, stride=1, padding=0) 37 | self.K = nn.Linear(in_features = self.lenk, out_features = self.inter_channels) 38 | self.V = nn.Linear(in_features = self.lenk, out_features = self.inter_channels) 39 | self.aggre = nn.Conv2d(in_channels=self.inter_channels, out_channels=self.inter_channels, kernel_size=1, stride=1, padding=0) 40 | self.concat_project = nn.Conv2d(in_channels=self.in_channels+self.inter_channels, out_channels=self.final_in_channels, kernel_size=1, stride=1, padding=0) 41 | self.conv = nn.Conv2d(in_channels=768, out_channels=self.inter_channels, kernel_size=1, stride=1, padding=0) 42 | 43 | self.feat_dim = 128 44 | self.queue_len = 10 45 | self.momentum = 0.99 46 | for i in range(0, 20): 47 | self.register_buffer("queue" + str(i), torch.randn(self.queue_len, self.feat_dim)) 48 | self.register_buffer("queue_ptr" + str(i), torch.zeros(1, dtype=torch.long)) 49 | 50 | self.backbone = nn.ModuleList([self.stage0, self.stage1, self.stage2, self.stage3, self.stage4]) 51 | self.newly_added = nn.ModuleList([self.classifier, self.f9, self.side1, self.side2, self.side3, self.side4, 52 | self.Q, self.K, self.V, self.aggre, self.concat_project 53 | ]) 54 | 55 | def PCM(self, cam, f, valid_mask): 56 | n,c,h,w = f.size() 57 | cam = F.interpolate(cam, (h,w), mode='bilinear', align_corners=True).view(n,-1,h*w) 58 | f = self.f9(f) 59 | f = f.view(n,-1,h*w) 60 | f = f/(torch.norm(f,dim=1,keepdim=True)+1e-5) 61 | aff = F.relu(torch.matmul(f.transpose(1,2), f),inplace=True) 62 | aff = aff/(torch.sum(aff,dim=1,keepdim=True)+1e-5) 63 | cam_rv = torch.matmul(cam, aff).view(n,-1,h,w) 64 | cam_rv = torch.matmul(cam_rv.view(n,-1,h*w), aff).view(n,-1,h,w) 65 | return cam_rv 66 | 67 | def Bank(self, sem_feature): 68 | feat_memory = getattr(self, "queue0") 69 | for k in range(1, 20): 70 | feat_memory = torch.cat((feat_memory, getattr(self, "queue" + str(k))), 0) 71 | query_projector = self.Q(sem_feature.clone().detach()) 72 | # b, c, h, w = query_projector.shape 73 | # query = query_projector.permute(0, 2, 3, 1).view(b, w*h, c) # 16, 1024, 128 74 | # key = self.K(feat_memory).permute(1, 0) 75 | # Affinity = F.relu(torch.matmul(query, key)) 76 | # Affinity = Affinity/(torch.sum(Affinity,dim=1,keepdim=True)+1e-5) 77 | # value = self.V(feat_memory) 78 | # out_feature = torch.matmul(Affinity, value).permute(0, 2, 1).view(b, c, h, w) 79 | # query_ = self.aggre(out_feature) 80 | # refine_feature = self.concat_project(torch.cat((sem_feature, query_), 1)) 81 | 82 | # sem_feature = torch.cat([sem_feature, query_projector], dim=1) 83 | 84 | return sem_feature, query_projector 85 | 86 | 87 | def prototype(self, norm_cam, feature, valid_mask): 88 | n,c,h,w = norm_cam.shape 89 | norm_cam[:,0] = norm_cam[:,0]*0.2 90 | seeds = torch.zeros((n,h,w,c)).cuda() 91 | belonging = norm_cam.argmax(1) 92 | seeds = seeds.scatter_(-1, belonging.view(n,h,w,1), 1).permute(0,3,1,2).contiguous() 93 | seeds = seeds * valid_mask # 4, 21, 32, 32 94 | 95 | n,c,h,w = feature.shape 96 | seeds = F.interpolate(seeds, feature.shape[2:], mode='nearest') 97 | # crop_feature = seeds.unsqueeze(2) * feature.unsqueeze(1) #.clone().detach() # seed:[n,21,1,h,w], feature:[n,1,c,h,w], crop_feature:[n,21,c,h,w] 98 | # prototype = F.adaptive_avg_pool2d(crop_feature.view(-1,c,h,w), (1,1)).view(n, self.num_cls, c, 1, 1) # prototypes:[n,21,c,1,1] 99 | 100 | feat_memory = getattr(self, "queue0").unsqueeze(0) 101 | for k in range(1, 20): 102 | feat_memory = torch.cat((feat_memory, getattr(self, "queue" + str(k)).unsqueeze(0)), 0) 103 | prototype = torch.mean(feat_memory,1).unsqueeze(0) 104 | 105 | feature = self.conv(feature) 106 | prototype = prototype.unsqueeze(-1).unsqueeze(-1) 107 | if n == 1: 108 | prototype = F.normalize(prototype, dim=-1) 109 | 110 | IS_cam = F.relu(torch.cosine_similarity(feature.unsqueeze(1), prototype, dim=2)) # feature:[n,1,c,h,w], prototypes:[n,21,c,1,1], crop_feature:[n,21,h,w] 111 | IS_cam = F.interpolate(IS_cam, feature.shape[2:], mode='bilinear', align_corners=True) 112 | 113 | norm_cam = F.relu(IS_cam) 114 | norm_cam = norm_cam/(F.adaptive_max_pool2d(norm_cam, (1, 1)) + 1e-5) 115 | cam_bkg = 1-torch.max(norm_cam,dim=1)[0].unsqueeze(1) 116 | norm_cam = torch.cat([cam_bkg, norm_cam], dim=1)*valid_mask 117 | 118 | return norm_cam 119 | 120 | 121 | 122 | 123 | def forward(self, x, valid_mask,my_label= None, epoch=None, index=None, train= None): 124 | 125 | x0 = self.stage0(x) 126 | x1 = self.stage1(x0) 127 | x2 = self.stage2(x1).detach() 128 | x3 = self.stage3(x2) 129 | x4 = self.stage4(x3) 130 | side1 = self.side1(x1.detach()) 131 | side2 = self.side2(x2.detach()) 132 | side3 = self.side3(x3.detach()) 133 | side4 = self.side4(x4.detach()) 134 | 135 | 136 | sem_feature = x4 137 | cam = self.classifier(x4) 138 | score = F.adaptive_avg_pool2d(cam, 1) 139 | norm_cam = F.relu(cam) 140 | norm_cam = norm_cam/(F.adaptive_max_pool2d(norm_cam, (1, 1)) + 1e-5) 141 | cam_bkg = 1-torch.max(norm_cam,dim=1)[0].unsqueeze(1) 142 | norm_cam = torch.cat([cam_bkg, norm_cam], dim=1) 143 | norm_cam = F.interpolate(norm_cam, side3.shape[2:], mode='bilinear', align_corners=True)*valid_mask 144 | orignal_cam = norm_cam 145 | 146 | hie_fea = torch.cat([F.interpolate(side1/(torch.norm(side1,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear'), 147 | F.interpolate(side2/(torch.norm(side2,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear'), 148 | F.interpolate(side3/(torch.norm(side3,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear'), 149 | F.interpolate(side4/(torch.norm(side4,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear')], dim=1) 150 | 151 | sem_feature, query_projector = self.Bank(sem_feature) 152 | norm_cam = self.PCM(norm_cam, torch.cat([F.interpolate(x, side3.shape[2:],mode='bilinear',align_corners=True), sem_feature], dim=1), valid_mask.clone()) 153 | 154 | IS_cam = self.prototype(norm_cam.clone(), hie_fea.clone(), valid_mask.clone()) 155 | 156 | if x.size()[0]!=1: 157 | for j in range(0, x.size()[0]): 158 | self._dequeue_and_enqueue(query_projector[j].clone().detach(), IS_cam[j], my_label[j], score[j]) 159 | 160 | return {"score": score, "cam1": norm_cam, "cam2": IS_cam, "orignal_cam": orignal_cam} 161 | 162 | def _dequeue_and_enqueue(self, x, map, label, probs): 163 | map = F.softmax(map, dim = 0) 164 | for ind, cla in enumerate(label): 165 | if cla == 1: 166 | if(probs[ind] > 0.99): 167 | mask = map[ind] > (torch.mean(map[ind])) 168 | x = x * mask.float() 169 | embedding = x.reshape(x.shape[0], -1).sum(1)/mask.float().sum() 170 | queue_i = getattr(self, "queue" + str(ind)) 171 | queue_ptr_i = getattr(self, "queue_ptr" + str(ind)) 172 | ptr = int(queue_ptr_i) 173 | queue_i[ptr:ptr + 1] = queue_i[ptr:ptr + 1] * self.momentum + embedding * (1 - self.momentum) 174 | ptr = (ptr + 1) % self.queue_len # move pointer 175 | queue_ptr_i[0] = ptr 176 | 177 | def train(self, mode=True): 178 | for p in self.resnet50.conv1.parameters(): 179 | p.requires_grad = False 180 | for p in self.resnet50.bn1.parameters(): 181 | p.requires_grad = False 182 | 183 | def trainable_parameters(self): 184 | return (list(self.backbone.parameters()), list(self.newly_added.parameters())) 185 | 186 | class CAM(Net): 187 | 188 | def __init__(self, num_cls): 189 | super(CAM, self).__init__(num_cls=num_cls) 190 | self.num_cls = num_cls 191 | 192 | def forward(self, x, label): 193 | 194 | x0 = self.stage0(x) 195 | x1 = self.stage1(x0) 196 | x2 = self.stage2(x1).detach() 197 | x3 = self.stage3(x2) 198 | x4 = self.stage4(x3) 199 | 200 | side1 = self.side1(x1.detach()) 201 | side2 = self.side2(x2.detach()) 202 | side3 = self.side3(x3.detach()) 203 | side4 = self.side4(x4.detach()) 204 | 205 | hie_fea = torch.cat([F.interpolate(side1/(torch.norm(side1,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear'), 206 | F.interpolate(side2/(torch.norm(side2,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear'), 207 | F.interpolate(side3/(torch.norm(side3,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear'), 208 | F.interpolate(side4/(torch.norm(side4,dim=1,keepdim=True)+1e-5), side3.shape[2:], mode='bilinear')], dim=1) 209 | 210 | cam = self.classifier(x4) 211 | cam = (cam[0] + cam[1].flip(-1)).unsqueeze(0) 212 | hie_fea = (hie_fea[0] + hie_fea[1].flip(-1)).unsqueeze(0) 213 | norm_cam = F.relu(cam) 214 | norm_cam = norm_cam/(F.adaptive_max_pool2d(norm_cam, (1, 1)) + 1e-5) 215 | cam_bkg = 1-torch.max(norm_cam,dim=1)[0].unsqueeze(1) 216 | norm_cam = torch.cat([cam_bkg, norm_cam], dim=1) 217 | norm_cam = F.interpolate(norm_cam, side3.shape[2:], mode='bilinear', align_corners=True) 218 | 219 | x4, _ = self.Bank(x4) 220 | 221 | x = (x[0] + x[1].flip(-1)).unsqueeze(0) 222 | x4 = (x4[0] + x4[1].flip(-1)).unsqueeze(0) 223 | x3 = (x3[0] + x3[1].flip(-1)).unsqueeze(0) 224 | 225 | norm_cam = self.PCM(norm_cam, torch.cat([F.interpolate(x,side3.shape[2:],mode='bilinear',align_corners=True), x4], dim=1), label.unsqueeze(0).clone()) 226 | IS_cam = self.prototype(norm_cam.clone(), hie_fea.clone(), label.unsqueeze(0).clone()) 227 | 228 | return norm_cam[0], IS_cam[0] -------------------------------------------------------------------------------- /network/resnet50_irn.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import torch.nn.functional as F 4 | from network import resnet50 5 | 6 | 7 | class Net(nn.Module): 8 | 9 | def __init__(self): 10 | super(Net, self).__init__() 11 | 12 | # backbone 13 | self.resnet50 = resnet50.resnet50(pretrained=True, strides=[2, 2, 2, 1]) 14 | 15 | self.stage1 = nn.Sequential(self.resnet50.conv1, self.resnet50.bn1, self.resnet50.relu, self.resnet50.maxpool) 16 | self.stage2 = nn.Sequential(self.resnet50.layer1) 17 | self.stage3 = nn.Sequential(self.resnet50.layer2) 18 | self.stage4 = nn.Sequential(self.resnet50.layer3) 19 | self.stage5 = nn.Sequential(self.resnet50.layer4) 20 | self.mean_shift = Net.MeanShift(2) 21 | 22 | # branch: class boundary detection 23 | self.fc_edge1 = nn.Sequential( 24 | nn.Conv2d(64, 32, 1, bias=False), 25 | nn.GroupNorm(4, 32), 26 | nn.ReLU(inplace=True), 27 | ) 28 | self.fc_edge2 = nn.Sequential( 29 | nn.Conv2d(256, 32, 1, bias=False), 30 | nn.GroupNorm(4, 32), 31 | nn.ReLU(inplace=True), 32 | ) 33 | self.fc_edge3 = nn.Sequential( 34 | nn.Conv2d(512, 32, 1, bias=False), 35 | nn.GroupNorm(4, 32), 36 | nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False), 37 | nn.ReLU(inplace=True), 38 | ) 39 | self.fc_edge4 = nn.Sequential( 40 | nn.Conv2d(1024, 32, 1, bias=False), 41 | nn.GroupNorm(4, 32), 42 | nn.Upsample(scale_factor=4, mode='bilinear', align_corners=False), 43 | nn.ReLU(inplace=True), 44 | ) 45 | self.fc_edge5 = nn.Sequential( 46 | nn.Conv2d(2048, 32, 1, bias=False), 47 | nn.GroupNorm(4, 32), 48 | nn.Upsample(scale_factor=4, mode='bilinear', align_corners=False), 49 | nn.ReLU(inplace=True), 50 | ) 51 | self.fc_edge6 = nn.Conv2d(160, 1, 1, bias=True) 52 | 53 | # branch: displacement field 54 | self.fc_dp1 = nn.Sequential( 55 | nn.Conv2d(64, 64, 1, bias=False), 56 | nn.GroupNorm(8, 64), 57 | nn.ReLU(inplace=True), 58 | ) 59 | self.fc_dp2 = nn.Sequential( 60 | nn.Conv2d(256, 128, 1, bias=False), 61 | nn.GroupNorm(16, 128), 62 | nn.ReLU(inplace=True), 63 | ) 64 | self.fc_dp3 = nn.Sequential( 65 | nn.Conv2d(512, 256, 1, bias=False), 66 | nn.GroupNorm(16, 256), 67 | nn.ReLU(inplace=True), 68 | ) 69 | self.fc_dp4 = nn.Sequential( 70 | nn.Conv2d(1024, 256, 1, bias=False), 71 | nn.GroupNorm(16, 256), 72 | nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False), 73 | nn.ReLU(inplace=True), 74 | ) 75 | self.fc_dp5 = nn.Sequential( 76 | nn.Conv2d(2048, 256, 1, bias=False), 77 | nn.GroupNorm(16, 256), 78 | nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False), 79 | nn.ReLU(inplace=True), 80 | ) 81 | self.fc_dp6 = nn.Sequential( 82 | nn.Conv2d(768, 256, 1, bias=False), 83 | nn.GroupNorm(16, 256), 84 | nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False), 85 | nn.ReLU(inplace=True), 86 | ) 87 | self.fc_dp7 = nn.Sequential( 88 | nn.Conv2d(448, 256, 1, bias=False), 89 | nn.GroupNorm(16, 256), 90 | nn.ReLU(inplace=True), 91 | nn.Conv2d(256, 2, 1, bias=False), 92 | self.mean_shift 93 | ) 94 | 95 | self.backbone = nn.ModuleList([self.stage1, self.stage2, self.stage3, self.stage4, self.stage5]) 96 | self.edge_layers = nn.ModuleList([self.fc_edge1, self.fc_edge2, self.fc_edge3, self.fc_edge4, self.fc_edge5, self.fc_edge6]) 97 | self.dp_layers = nn.ModuleList([self.fc_dp1, self.fc_dp2, self.fc_dp3, self.fc_dp4, self.fc_dp5, self.fc_dp6, self.fc_dp7]) 98 | 99 | class MeanShift(nn.Module): 100 | 101 | def __init__(self, num_features): 102 | super(Net.MeanShift, self).__init__() 103 | self.register_buffer('running_mean', torch.zeros(num_features)) 104 | 105 | def forward(self, input): 106 | if self.training: 107 | return input 108 | return input - self.running_mean.view(1, 2, 1, 1) 109 | 110 | def forward(self, x): 111 | x1 = self.stage1(x).detach() 112 | x2 = self.stage2(x1).detach() 113 | x3 = self.stage3(x2).detach() 114 | x4 = self.stage4(x3).detach() 115 | x5 = self.stage5(x4).detach() 116 | 117 | edge1 = self.fc_edge1(x1) 118 | edge2 = self.fc_edge2(x2) 119 | edge3 = self.fc_edge3(x3)[..., :edge2.size(2), :edge2.size(3)] 120 | edge4 = self.fc_edge4(x4)[..., :edge2.size(2), :edge2.size(3)] 121 | edge5 = self.fc_edge5(x5)[..., :edge2.size(2), :edge2.size(3)] 122 | edge_out = self.fc_edge6(torch.cat([edge1, edge2, edge3, edge4, edge5], dim=1)) 123 | 124 | dp1 = self.fc_dp1(x1) 125 | dp2 = self.fc_dp2(x2) 126 | dp3 = self.fc_dp3(x3) 127 | dp4 = self.fc_dp4(x4)[..., :dp3.size(2), :dp3.size(3)] 128 | dp5 = self.fc_dp5(x5)[..., :dp3.size(2), :dp3.size(3)] 129 | 130 | dp_up3 = self.fc_dp6(torch.cat([dp3, dp4, dp5], dim=1))[..., :dp2.size(2), :dp2.size(3)] 131 | dp_out = self.fc_dp7(torch.cat([dp1, dp2, dp_up3], dim=1)) 132 | 133 | return edge_out, dp_out 134 | 135 | def trainable_parameters(self): 136 | return (tuple(self.edge_layers.parameters()), 137 | tuple(self.dp_layers.parameters())) 138 | 139 | def train(self, mode=True): 140 | super().train(mode) 141 | self.backbone.eval() 142 | 143 | 144 | class AffinityDisplacementLoss(Net): 145 | 146 | path_indices_prefix = "path_indices" 147 | 148 | def __init__(self, path_index): 149 | 150 | super(AffinityDisplacementLoss, self).__init__() 151 | 152 | self.path_index = path_index 153 | 154 | self.n_path_lengths = len(path_index.path_indices) 155 | for i, pi in enumerate(path_index.path_indices): 156 | self.register_buffer(AffinityDisplacementLoss.path_indices_prefix + str(i), torch.from_numpy(pi)) 157 | 158 | self.register_buffer( 159 | 'disp_target', 160 | torch.unsqueeze(torch.unsqueeze(torch.from_numpy(path_index.search_dst).transpose(1, 0), 0), -1).float()) 161 | 162 | def to_affinity(self, edge): 163 | aff_list = [] 164 | edge = edge.view(edge.size(0), -1) 165 | 166 | for i in range(self.n_path_lengths): 167 | ind = self._buffers[AffinityDisplacementLoss.path_indices_prefix + str(i)] 168 | ind_flat = ind.view(-1) 169 | dist = torch.index_select(edge, dim=-1, index=ind_flat) 170 | dist = dist.view(dist.size(0), ind.size(0), ind.size(1), ind.size(2)) 171 | aff = torch.squeeze(1 - F.max_pool2d(dist, (dist.size(2), 1)), dim=2) 172 | aff_list.append(aff) 173 | aff_cat = torch.cat(aff_list, dim=1) 174 | 175 | return aff_cat 176 | 177 | def to_pair_displacement(self, disp): 178 | height, width = disp.size(2), disp.size(3) 179 | radius_floor = self.path_index.radius_floor 180 | 181 | cropped_height = height - radius_floor 182 | cropped_width = width - 2 * radius_floor 183 | 184 | disp_src = disp[:, :, :cropped_height, radius_floor:radius_floor + cropped_width] 185 | 186 | disp_dst = [disp[:, :, dy:dy + cropped_height, radius_floor + dx:radius_floor + dx + cropped_width] 187 | for dy, dx in self.path_index.search_dst] 188 | disp_dst = torch.stack(disp_dst, 2) 189 | 190 | pair_disp = torch.unsqueeze(disp_src, 2) - disp_dst 191 | pair_disp = pair_disp.view(pair_disp.size(0), pair_disp.size(1), pair_disp.size(2), -1) 192 | 193 | return pair_disp 194 | 195 | def to_displacement_loss(self, pair_disp): 196 | return torch.abs(pair_disp - self.disp_target) 197 | 198 | def forward(self, *inputs): 199 | x, return_loss = inputs 200 | edge_out, dp_out = super().forward(x) 201 | 202 | if return_loss is False: 203 | return edge_out, dp_out 204 | 205 | aff = self.to_affinity(torch.sigmoid(edge_out)) 206 | pos_aff_loss = (-1) * torch.log(aff + 1e-5) 207 | neg_aff_loss = (-1) * torch.log(1. + 1e-5 - aff) 208 | 209 | pair_disp = self.to_pair_displacement(dp_out) 210 | dp_fg_loss = self.to_displacement_loss(pair_disp) 211 | dp_bg_loss = torch.abs(pair_disp) 212 | 213 | return pos_aff_loss, neg_aff_loss, dp_fg_loss, dp_bg_loss 214 | 215 | 216 | class EdgeDisplacement(Net): 217 | 218 | def __init__(self, crop_size=512, stride=4): 219 | super(EdgeDisplacement, self).__init__() 220 | self.crop_size = crop_size 221 | self.stride = stride 222 | 223 | def forward(self, x): 224 | feat_size = (x.size(2)-1)//self.stride+1, (x.size(3)-1)//self.stride+1 225 | 226 | x = F.pad(x, [0, self.crop_size-x.size(3), 0, self.crop_size-x.size(2)]) 227 | edge_out, dp_out = super().forward(x) 228 | edge_out = edge_out[..., :feat_size[0], :feat_size[1]] 229 | dp_out = dp_out[..., :feat_size[0], :feat_size[1]] 230 | 231 | edge_out = torch.sigmoid(edge_out[0]/2 + edge_out[1].flip(-1)/2) 232 | dp_out = dp_out[0] 233 | 234 | return edge_out, dp_out 235 | 236 | 237 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | numpy 2 | cython 3 | imageio 4 | scikit-image 5 | opencv-python 6 | chainercv 7 | matplotlib 8 | tqdm 9 | tensorboardx 10 | git+https://github.com/lucasb-eyer/pydensecrf.git -------------------------------------------------------------------------------- /run_coco.sh: -------------------------------------------------------------------------------- 1 | 2 | # Step 1. Train SIPE for localization maps. 3 | 4 | # 1.1 train sipe 5 | CUDA_VISIBLE_DEVICES=2,3 python train_resnet50_SIPE.py --dataset coco --session_name exp_coco 6 | # 1.2 obtain localization maps 7 | # CUDA_VISIBLE_DEVICES=2,3 python make_cam.py --dataset coco --session_name exp_coco 8 | # 1.3 evaluate localization maps 9 | # CUDA_VISIBLE_DEVICES=2,3 python eval_cam.py --dataset coco --session_name exp_coco 10 | 11 | 12 | # Step 2. Refinement for pseudo labels. 13 | # Note: for MS COCO dataset, we replace IRN with denseCRF due to the large computation cost. 14 | # CUDA_VISIBLE_DEVICES=2,3 python cam2ir.py --dataset coco --session_name exp_coco 15 | 16 | -------------------------------------------------------------------------------- /run_voc.sh: -------------------------------------------------------------------------------- 1 | 2 | # Step 1. Train SFC for localization maps. 3 | 4 | # 1.1 train SFC 5 | CUDA_VISIBLE_DEVICES=0 python train_resnet50_SFC.py 6 | ## # 1.2 obtain localization maps 7 | CUDA_VISIBLE_DEVICES=0,1,2,3 python make_cam.py 8 | #### # 1.3 evaluate localization maps 9 | CUDA_VISIBLE_DEVICES=0,1,2,3 python eval_cam.py 10 | # 11 | # Step 2. Train IRN for pseudo labels. 12 | # 2.1 generate ir label 13 | CUDA_VISIBLE_DEVICES=0,1,2,3 python cam2ir.py 14 | # # 2.2 train irn 15 | CUDA_VISIBLE_DEVICES=2 python train_irn.py 16 | # 2.3 make pseudo labels 17 | CUDA_VISIBLE_DEVICES=0,1,2,3 python make_seg_labels.py 18 | CUDA_VISIBLE_DEVICES=0 python eval_sem_seg.py 19 | 20 | -------------------------------------------------------------------------------- /run_voc2.sh: -------------------------------------------------------------------------------- 1 | 2 | # Step 1. Train SIPE for localization maps. 3 | 4 | # 1.1 train sipe 5 | #CUDA_VISIBLE_DEVICES=2 python train_resnet50_SIPE.py 6 | # # 1.2 obtain localization maps 7 | CUDA_VISIBLE_DEVICES=2 python make_cam.py 8 | # # 1.3 evaluate localization maps 9 | CUDA_VISIBLE_DEVICES=2 python eval_cam.py 10 | 11 | # # Step 2. Train IRN for pseudo labels. 12 | # # 2.1 generate ir label 13 | CUDA_VISIBLE_DEVICES=2 python cam2ir.py 14 | # # # 2.2 train irn 15 | CUDA_VISIBLE_DEVICES=2 python train_irn.py 16 | # # # 2.3 make pseudo labels 17 | CUDA_VISIBLE_DEVICES=2 python make_seg_labels.py 18 | 19 | CUDA_VISIBLE_DEVICES=2 python eval_sem_seg.py 20 | 21 | -------------------------------------------------------------------------------- /train_irn.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import torch 3 | import os 4 | from torch.backends import cudnn 5 | cudnn.enabled = True 6 | from torch.utils.data import DataLoader 7 | from data import data_voc 8 | from tool import pyutils, torchutils, indexing 9 | import importlib 10 | from PIL import ImageFile 11 | ImageFile.LOAD_TRUNCATED_IMAGES = True 12 | def run(args): 13 | ir_label_out_dir = os.path.join(args.session_name, 'ir_label') 14 | 15 | path_index = indexing.PathIndex(radius=10, default_size=(args.irn_crop_size // 4, args.irn_crop_size // 4)) 16 | 17 | model = getattr(importlib.import_module(args.irn_network), 'AffinityDisplacementLoss')(path_index) 18 | 19 | train_dataset = data_voc.VOC12AffinityDataset(args.train_list, 20 | label_dir=ir_label_out_dir, 21 | voc12_root=args.voc12_root, 22 | indices_from=path_index.src_indices, 23 | indices_to=path_index.dst_indices, 24 | hor_flip=True, 25 | crop_size=args.irn_crop_size, 26 | crop_method="random", 27 | rescale=(0.5, 1.0, 1.5) 28 | ) 29 | train_data_loader = DataLoader(train_dataset, batch_size=args.irn_batch_size, 30 | shuffle=True, num_workers=args.num_workers, pin_memory=True, drop_last=True) 31 | 32 | max_step = (len(train_dataset) // args.irn_batch_size) * args.irn_num_epoches 33 | 34 | param_groups = model.trainable_parameters() 35 | optimizer = torchutils.PolyOptimizerSGD([ 36 | {'params': param_groups[0], 'lr': 1*args.irn_learning_rate, 'weight_decay': args.irn_weight_decay}, 37 | {'params': param_groups[1], 'lr': 10*args.irn_learning_rate, 'weight_decay': args.irn_weight_decay} 38 | ], lr=args.irn_learning_rate, weight_decay=args.irn_weight_decay, max_step=max_step) 39 | 40 | model = torch.nn.DataParallel(model).cuda() 41 | model.train() 42 | 43 | avg_meter = pyutils.AverageMeter() 44 | 45 | timer = pyutils.Timer() 46 | 47 | for ep in range(args.irn_num_epoches): 48 | 49 | print('Epoch %d/%d' % (ep+1, args.irn_num_epoches)) 50 | 51 | for iter, pack in enumerate(train_data_loader): 52 | 53 | img = pack['img'].cuda(non_blocking=True) 54 | bg_pos_label = pack['aff_bg_pos_label'].cuda(non_blocking=True) 55 | fg_pos_label = pack['aff_fg_pos_label'].cuda(non_blocking=True) 56 | neg_label = pack['aff_neg_label'].cuda(non_blocking=True) 57 | 58 | pos_aff_loss, neg_aff_loss, dp_fg_loss, dp_bg_loss = model(img, True) 59 | 60 | bg_pos_aff_loss = torch.sum(bg_pos_label * pos_aff_loss) / (torch.sum(bg_pos_label) + 1e-5) 61 | fg_pos_aff_loss = torch.sum(fg_pos_label * pos_aff_loss) / (torch.sum(fg_pos_label) + 1e-5) 62 | pos_aff_loss = bg_pos_aff_loss / 2 + fg_pos_aff_loss / 2 63 | neg_aff_loss = torch.sum(neg_label * neg_aff_loss) / (torch.sum(neg_label) + 1e-5) 64 | 65 | dp_fg_loss = torch.sum(dp_fg_loss * torch.unsqueeze(fg_pos_label, 1)) / (2 * torch.sum(fg_pos_label) + 1e-5) 66 | dp_bg_loss = torch.sum(dp_bg_loss * torch.unsqueeze(bg_pos_label, 1)) / (2 * torch.sum(bg_pos_label) + 1e-5) 67 | 68 | avg_meter.add({'loss1': pos_aff_loss.item(), 'loss2': neg_aff_loss.item(), 69 | 'loss3': dp_fg_loss.item(), 'loss4': dp_bg_loss.item()}) 70 | 71 | total_loss = (pos_aff_loss + neg_aff_loss) / 2 + (dp_fg_loss + dp_bg_loss) / 2 72 | 73 | optimizer.zero_grad() 74 | total_loss.backward() 75 | optimizer.step() 76 | 77 | if (optimizer.global_step - 1) % 10 == 0: 78 | timer.update_progress(optimizer.global_step / max_step) 79 | 80 | print('step:%5d/%5d' % (optimizer.global_step - 1, max_step), 81 | 'loss:%.4f %.4f %.4f %.4f' % ( 82 | avg_meter.pop('loss1'), avg_meter.pop('loss2'), avg_meter.pop('loss3'), avg_meter.pop('loss4')), 83 | 'imps:%.1f' % ((iter + 1) * args.irn_batch_size / timer.get_stage_elapsed()), 84 | 'lr: %.4f' % (optimizer.param_groups[0]['lr']), 85 | 'etc:%s' % (timer.str_est_finish()), flush=True) 86 | else: 87 | timer.reset_stage() 88 | 89 | infer_dataset = data_voc.VOC12ImageDataset(args.infer_list, 90 | voc12_root=args.voc12_root, 91 | crop_size=args.irn_crop_size, 92 | crop_method="top_left") 93 | infer_data_loader = DataLoader(infer_dataset, batch_size=args.irn_batch_size, 94 | shuffle=False, num_workers=args.num_workers, pin_memory=True, drop_last=True) 95 | 96 | model.eval() 97 | print('Analyzing displacements mean ... ', end='') 98 | 99 | dp_mean_list = [] 100 | 101 | with torch.no_grad(): 102 | for iter, pack in enumerate(infer_data_loader): 103 | img = pack['img'].cuda(non_blocking=True) 104 | 105 | aff, dp = model(img, False) 106 | 107 | dp_mean_list.append(torch.mean(dp, dim=(0, 2, 3)).cpu()) 108 | 109 | model.module.mean_shift.running_mean = torch.mean(torch.stack(dp_mean_list), dim=0) 110 | print('done.') 111 | 112 | torch.save(model.module.state_dict(), os.path.join(args.session_name, 'ckpt', 'irn.pth')) 113 | torch.cuda.empty_cache() 114 | 115 | if __name__ == '__main__': 116 | 117 | parser = argparse.ArgumentParser() 118 | # Inter-pixel Relation Network (IRNet) 119 | parser.add_argument("--num_workers", default=1, type=int) 120 | parser.add_argument("--train_list", default="data/trainaug_voc.txt", type=str) 121 | parser.add_argument("--infer_list", default="data/train_voc.txt", type=str) 122 | parser.add_argument("--session_name", default="exp", type=str) 123 | parser.add_argument("--voc12_root", default="/data/tfl/VOCdevkit/VOC2012", type=str) 124 | 125 | parser.add_argument("--irn_network", default="network.resnet50_irn", type=str) 126 | parser.add_argument("--irn_crop_size", default=512, type=int) 127 | parser.add_argument("--irn_batch_size", default=32, type=int) 128 | parser.add_argument("--irn_num_epoches", default=3, type=int) 129 | parser.add_argument("--irn_learning_rate", default=0.1, type=float) 130 | parser.add_argument("--irn_weight_decay", default=1e-4, type=float) 131 | args = parser.parse_args() 132 | 133 | run(args) 134 | -------------------------------------------------------------------------------- /train_resnet50_SFC.py: -------------------------------------------------------------------------------- 1 | import torch, os 2 | from torch.backends import cudnn 3 | cudnn.enabled = True 4 | from torch.utils.data import DataLoader 5 | import torch.nn.functional as F 6 | import argparse 7 | import importlib 8 | import numpy as np 9 | from tensorboardX import SummaryWriter 10 | from data import data_coco, data_voc 11 | from chainercv.datasets import VOCSemanticSegmentationDataset 12 | from chainercv.evaluations import calc_semantic_segmentation_confusion 13 | from tool import pyutils, torchutils, visualization, imutils 14 | import random 15 | 16 | def validate(model, data_loader, global_step,tblogger): 17 | gt_dataset = VOCSemanticSegmentationDataset(split='train', data_dir="/data/tfl/VOCdevkit/VOC2012") 18 | labels = [gt_dataset.get_example_by_keys(i, (1,))[0] for i in range(len(gt_dataset))] 19 | print('validating ... ', flush=True, end='') 20 | model.eval() 21 | with torch.no_grad(): 22 | preds = [] 23 | preds1 = [] 24 | preds2 = [] 25 | for iter, pack in enumerate(data_loader): 26 | img = pack['img'].cuda() 27 | label = pack['label'].cuda(non_blocking=True).unsqueeze(-1).unsqueeze(-1) 28 | label = F.pad(label, (0, 0, 0, 0, 1, 0), 'constant', 1.0) 29 | outputs = model.forward(img, label, pack['label'].cuda(non_blocking=True)) 30 | IS_cam1 = outputs['cam1'] 31 | IS_cam1 = F.interpolate(IS_cam1, img.shape[2:], mode='bilinear') 32 | IS_cam1 = IS_cam1/(F.adaptive_max_pool2d(IS_cam1, (1, 1)) + 1e-5) 33 | cls_labels_bkg1 = torch.argmax(IS_cam1, 1) 34 | preds1.append(cls_labels_bkg1[0].cpu().numpy().copy()) 35 | 36 | IS_cam = outputs['cam2'] 37 | IS_cam = F.interpolate(IS_cam, img.shape[2:], mode='bilinear') 38 | IS_cam = IS_cam/(F.adaptive_max_pool2d(IS_cam, (1, 1)) + 1e-5) 39 | cls_labels_bkg = torch.argmax(IS_cam, 1) 40 | preds.append(cls_labels_bkg[0].cpu().numpy().copy()) 41 | 42 | 43 | confusion = calc_semantic_segmentation_confusion(preds1, labels) 44 | gtj = confusion.sum(axis=1) 45 | resj = confusion.sum(axis=0) 46 | gtjresj = np.diag(confusion) 47 | denominator = gtj + resj - gtjresj 48 | iou = gtjresj / denominator 49 | 50 | confusion = calc_semantic_segmentation_confusion(preds, labels) 51 | gtj = confusion.sum(axis=1) 52 | resj = confusion.sum(axis=0) 53 | gtjresj = np.diag(confusion) 54 | denominator = gtj + resj - gtjresj 55 | iou1 = gtjresj / denominator 56 | 57 | print('\n') 58 | print({'iou': iou, 'miou': np.nanmean(iou)}) 59 | print('\n') 60 | print({'iou1': iou1, 'miou': np.nanmean(iou1)}) 61 | model.train() 62 | 63 | return np.nanmean(iou) + np.nanmean(iou1) 64 | 65 | def setup_seed(seed): 66 | print("random seed is set to", seed) 67 | torch.manual_seed(seed) 68 | torch.cuda.manual_seed_all(seed) 69 | np.random.seed(seed) 70 | random.seed(seed) 71 | torch.backends.cudnn.deterministic = True 72 | torch.backends.cudnn.benchmark = False 73 | os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':4096:8' 74 | os.environ['PYTHONHASHSEED'] = str(seed) 75 | 76 | def shuffle_batch(x, y,z): 77 | index = torch.randperm(x.size(0)) 78 | x = x[index] 79 | y = y[index] 80 | z= z[index] 81 | return x, y, z 82 | 83 | 84 | def train(): 85 | parser = argparse.ArgumentParser() 86 | parser.add_argument("--batch_size", default=16, type=int) 87 | parser.add_argument("--max_epoches", default=6, type=int) 88 | parser.add_argument("--network", default="network.resnet50_SFC", type=str) 89 | parser.add_argument("--lr", default=0.1, type=float) 90 | parser.add_argument("--num_workers", default=8, type=int) 91 | parser.add_argument("--wt_dec", default=1e-4, type=float) 92 | parser.add_argument("--session_name", default="exp", type=str) 93 | parser.add_argument("--crop_size", default=512, type=int) 94 | parser.add_argument("--print_freq", default=10, type=int) 95 | parser.add_argument("--tf_freq", default=100, type=int) 96 | parser.add_argument("--val_freq", default=300, type=int) 97 | parser.add_argument("--dataset", default="voc", type=str) 98 | parser.add_argument("--dataset_root", default="/data/tfl/VOCdevkit/VOC2012", type=str) 99 | parser.add_argument("--seed", default=10, type=int) 100 | args = parser.parse_args() 101 | setup_seed(args.seed) 102 | os.makedirs(args.session_name, exist_ok=True) 103 | os.makedirs(os.path.join(args.session_name, 'runs'), exist_ok=True) 104 | os.makedirs(os.path.join(args.session_name, 'ckpt'), exist_ok=True) 105 | pyutils.Logger(os.path.join(args.session_name, args.session_name + '.log')) 106 | tblogger = SummaryWriter(os.path.join(args.session_name, 'runs')) 107 | 108 | assert args.dataset in ['voc', 'coco'], 'Dataset must be voc or coco in this project.' 109 | 110 | if args.dataset == 'voc': 111 | dataset_root = '/data/tfl/VOCdevkit/VOC2012' 112 | model = getattr(importlib.import_module(args.network), 'Net')(num_cls=21) 113 | train_dataset = data_voc.VOC12ClsDataset('data/trainaug_' + args.dataset + '.txt', voc12_root=dataset_root, 114 | resize_long=(320, 640), hor_flip=True, 115 | crop_size=512, crop_method="random") 116 | train_data_loader = DataLoader(train_dataset, batch_size=args.batch_size, shuffle=True, num_workers=args.num_workers, pin_memory=True, drop_last=True) 117 | max_step = (len(train_dataset) // args.batch_size) * args.max_epoches 118 | 119 | val_dataset = data_voc.VOC12ClsDataset('data/train_' + args.dataset + '.txt', voc12_root=dataset_root) 120 | val_data_loader = DataLoader(val_dataset, batch_size=1, shuffle=False, num_workers=args.num_workers, pin_memory=True, drop_last=True) 121 | 122 | param_groups = model.trainable_parameters() 123 | optimizer = torchutils.PolyOptimizerSGD([ 124 | {'params': param_groups[0], 'lr': args.lr, 'weight_decay': args.wt_dec}, 125 | {'params': param_groups[1], 'lr': 10 * args.lr, 'weight_decay': args.wt_dec} 126 | ], lr=args.lr, weight_decay=args.wt_dec, max_step=max_step) 127 | 128 | model = torch.nn.DataParallel(model).cuda() 129 | model.train() 130 | bestiou = 0 131 | avg_meter = pyutils.AverageMeter() 132 | timer = pyutils.Timer() 133 | slot = torch.zeros([20,3,512,512]) 134 | vm = torch.zeros([20,21,512,512]) 135 | sltarget = torch.zeros([20,20]) 136 | target2 = torch.linspace(0, 19, 20, dtype=torch.long).cuda() 137 | for ep in range(args.max_epoches): 138 | 139 | print('Epoch %d/%d' % (ep + 1, args.max_epoches)) 140 | index = 0 141 | for step, pack in enumerate(train_data_loader): 142 | 143 | scale_factor = 0.5 144 | img1 = pack['img'].cuda() 145 | label = pack['label'].cuda(non_blocking=True) 146 | valid_mask = pack['valid_mask'].cuda() 147 | img1, label, valid_mask = shuffle_batch(img1, label, valid_mask) 148 | ba = len(label) 149 | for i in range(ba): 150 | argl = (label[i,:].squeeze()==1).nonzero(as_tuple =False).squeeze() 151 | slot[argl] = img1[i].cpu() 152 | vm[argl] = valid_mask[i].cpu() 153 | sltarget[argl] = label[i].cpu() 154 | label = label.cuda() 155 | img1 = img1.cuda() 156 | valid_mask = valid_mask.cuda() 157 | slot_c, target2_c, vm_c = shuffle_batch(slot, target2, vm) 158 | first = random.randint(0,19) 159 | second = random.randint(0,19) 160 | third = random.randint(0,19) 161 | four = random.randint(0,19) 162 | 163 | slot_c = slot.cuda() 164 | target2_c = sltarget.cuda() 165 | vm_c = vm.cuda() 166 | img1 = torch.cat([img1, slot_c[first].unsqueeze(0), slot_c[second].unsqueeze(0), slot_c[third].unsqueeze(0), slot_c[four].unsqueeze(0)],dim=0) 167 | label = torch.cat([label, target2_c[first].unsqueeze(0), target2_c[second].unsqueeze(0), target2_c[third].unsqueeze(0), target2_c[four].unsqueeze(0)],dim=0) 168 | valid_mask = torch.cat([valid_mask, vm_c[first].unsqueeze(0), vm_c[second].unsqueeze(0), vm_c[third].unsqueeze(0), vm_c[four].unsqueeze(0)],dim=0) 169 | 170 | img2 = F.interpolate(img1,scale_factor=scale_factor,mode='bilinear',align_corners=True) 171 | N, c, h, w = img1.shape 172 | my_label = label 173 | label = label.cuda(non_blocking=True).unsqueeze(2).unsqueeze(3) 174 | valid_mask[:,1:] = valid_mask[:,1:] * label 175 | valid_mask_lowres0 = F.interpolate(valid_mask, size=(h//16, w//16), mode='nearest') 176 | valid_mask_lowres1 = F.interpolate(valid_mask, size=(h//32, w//32), mode='nearest') 177 | bg_score = torch.ones((N,1)).cuda() 178 | label = torch.cat((bg_score.unsqueeze(-1).unsqueeze(-1), label), dim=1) 179 | outputs1 = model.forward(img1, valid_mask_lowres0, my_label) 180 | outputs2 = model.forward(img2, valid_mask_lowres1, my_label) 181 | index += args.batch_size 182 | 183 | label1, cam1, cam_rv1, orignal_cam1 = outputs1['score'], outputs1['cam1'], outputs1['cam2'], outputs1['orignal_cam'] 184 | loss_cls1 = F.multilabel_soft_margin_loss(label1, label[:,1:,:,:]) 185 | cam_rv1 = cam_rv1 / (F.adaptive_max_pool2d(cam_rv1, (1, 1)) + 1e-5) 186 | 187 | lo = torch.abs(cam1-cam_rv1) 188 | cons = lo[:,1:,:,:] 189 | 190 | ben = [66642, 13956, 13316, 12784, 11044, 8134, 8062, 7726, 7582, 191 | 7536, 7536, 7626, 7698, 7704, 7792, 8032, 8308, 9036, 192 | 10524, 11118] 193 | 194 | Images_num = [4155, 1228, 1188, 1150, 1005, 714, 705, 649, 613, 590, 195 | 567, 522, 504, 503, 492, 468, 445, 393, 196 | 300, 267] 197 | Images_num = torch.Tensor(Images_num) 198 | Images_num = Images_num+660*4*3/20 199 | a = torch.Tensor(ben) 200 | a = a /Images_num 201 | a/=a.mean() 202 | 203 | cons[:,14,:,:] = cons[:,14,:,:]*a[0] # 4.0486026 person 204 | cons[:,8,:,:] = cons[:,8,:,:]*a[1] # 0.4921020 chair 205 | cons[:,11,:,:] = cons[:,11,:,:]*a[2] # 0.443499 dog 206 | cons[:,6,:,:] = cons[:,6,:,:]*a[3] # 0.28434 car 207 | cons[:,7,:,:] = cons[:,7,:,:]*a[4] # 0.39732 cat 208 | cons[:,4,:,:] = cons[:,4,:,:]*a[5] # 0.13244 bottle 209 | cons[:,2,:,:] = cons[:,2,:,:]*a[6] # 0.143377 bird 210 | cons[:,17,:,:] = cons[:,17,:,:]*a[7] # 0.211421 sofa 211 | cons[:,10,:,:] = cons[:,10,:,:]*a[8] # 0.25516 diningtable 212 | cons[:,0,:,:] = cons[:,0,:,:]*a[9] # 0.28311 aeroplane 213 | cons[:,19,:,:] = cons[:,19,:,:]*a[10] # 0.311057 tvmonitor 214 | cons[:,15,:,:] = cons[:,15,:,:]*a[11] # 0.365735 pottedplant 215 | cons[:,1,:,:] = cons[:,1,:,:]*a[12] # 0.387606 bicycle 216 | cons[:,18,:,:] = cons[:,18,:,:]*a[13] # 0.636182 train 217 | cons[:,13,:,:] = cons[:,13,:,:]*a[14] # 0.672764 motorbike 218 | cons[:,3,:,:] = cons[:,3,:,:]*a[15] # 0.388821 boat 219 | cons[:,12,:,:] = cons[:,12,:,:]*a[16] # 0.402187 horse 220 | cons[:,5,:,:] = cons[:,5,:,:]*a[17] # 0.431348 bus 221 | cons[:,16,:,:] = cons[:,16,:,:]*a[18] # 1.743333 sheet 222 | cons[:,9,:,:] = cons[:,9,:,:]*a[19] # 2.0823970 cow 223 | 224 | lo[:,1:,:,:] = cons 225 | lossGSC = torch.mean(torch.abs(lo))*1.6 226 | 227 | cam1 = F.interpolate(cam1,scale_factor=scale_factor,mode='bilinear',align_corners=True)*label 228 | label2, cam2, cam_rv2, orignal_cam2 = outputs2['score'], outputs2['cam1'], outputs2['cam2'], outputs2['orignal_cam'] 229 | loss_cls2 = F.multilabel_soft_margin_loss(label2, label[:,1:,:,:]) 230 | cam2 = cam2*label 231 | lossCLS = (loss_cls1 + loss_cls2)/2 232 | cons = torch.abs(cam1[:,1:,:,:]-cam2[:,1:,:,:]) 233 | cons[:,14,:,:] = cons[:,14,:,:]*a[0] # 4.0486026 person 234 | cons[:,8,:,:] = cons[:,8,:,:]*a[1] # 0.4921020 chair 235 | cons[:,11,:,:] = cons[:,11,:,:]*a[2] # 0.443499 dog 236 | cons[:,6,:,:] = cons[:,6,:,:]*a[3] # 0.28434 car 237 | cons[:,7,:,:] = cons[:,7,:,:]*a[4] # 0.39732 cat 238 | cons[:,4,:,:] = cons[:,4,:,:]*a[5] # 0.13244 bottle 239 | cons[:,2,:,:] = cons[:,2,:,:]*a[6] # 0.143377 bird 240 | cons[:,17,:,:] = cons[:,17,:,:]*a[7] # 0.211421 sofa 241 | cons[:,10,:,:] = cons[:,10,:,:]*a[8] # 0.25516 diningtable 242 | cons[:,0,:,:] = cons[:,0,:,:]*a[9] # 0.28311 aeroplane 243 | cons[:,19,:,:] = cons[:,19,:,:]*a[10] # 0.311057 tvmonitor 244 | cons[:,15,:,:] = cons[:,15,:,:]*a[11] # 0.365735 pottedplant 245 | cons[:,1,:,:] = cons[:,1,:,:]*a[12] # 0.387606 bicycle 246 | cons[:,18,:,:] = cons[:,18,:,:]*a[13] # 0.636182 train 247 | cons[:,13,:,:] = cons[:,13,:,:]*a[14] # 0.672764 motorbike 248 | cons[:,3,:,:] = cons[:,3,:,:]*a[15] # 0.388821 boat 249 | cons[:,12,:,:] = cons[:,12,:,:]*a[16] # 0.402187 horse 250 | cons[:,5,:,:] = cons[:,5,:,:]*a[17] # 0.431348 bus 251 | cons[:,16,:,:] = cons[:,16,:,:]*a[18] # 1.743333 sheet 252 | cons[:,9,:,:] = cons[:,9,:,:]*a[19] # 2.0823970 cow 253 | 254 | loss_consistency = torch.mean(cons)*0.8 255 | losses = lossCLS + lossGSC + loss_consistency 256 | avg_meter.add({'lossCLS': lossCLS.item(), 'lossGSC': lossGSC.item(), 'loss_consistency': loss_consistency.item(), 257 | }) 258 | optimizer.zero_grad() 259 | losses.backward() 260 | torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1) 261 | optimizer.step() 262 | 263 | if (optimizer.global_step - 1) % args.print_freq == 0: 264 | timer.update_progress(optimizer.global_step / max_step) 265 | 266 | print('step:%5d/%5d' % (optimizer.global_step - 1, max_step), 267 | 'lossCLS:%.4f' % (avg_meter.pop('lossCLS')), 268 | 'lossGSC:%.4f' % (avg_meter.pop('lossGSC')), 269 | 'loss_consistency:%.4f' % (avg_meter.pop('loss_consistency')), 270 | 'imps:%.1f' % ((step + 1) * args.batch_size / timer.get_stage_elapsed()), 271 | 'lr: %.4f' % (optimizer.param_groups[0]['lr']), 272 | 'etc:%s' % (timer.str_est_finish()), flush=True) 273 | 274 | # tf record 275 | tblogger.add_scalar('lossCLS', lossCLS, optimizer.global_step) 276 | tblogger.add_scalar('lossGSC', lossGSC, optimizer.global_step) 277 | tblogger.add_scalar('loss_er', loss_consistency, optimizer.global_step) 278 | tblogger.add_scalar('lr', optimizer.param_groups[0]['lr'], optimizer.global_step) 279 | 280 | if (optimizer.global_step - 1) % args.tf_freq == 0: 281 | # visualization 282 | img_1 = visualization.convert_to_tf(img1[0]) 283 | norm_cam = F.interpolate(orignal_cam1,img_1.shape[1:],mode='bilinear')[0].detach().cpu().numpy() 284 | cam_rv1 = F.interpolate(cam_rv1,img_1.shape[1:],mode='bilinear')[0].detach().cpu().numpy() 285 | CAM1 = visualization.generate_vis(norm_cam, None, img_1, func_label2color=visualization.VOClabel2colormap, threshold=None, norm=False) 286 | prototype_CAM1 = visualization.generate_vis(cam_rv1, None, img_1, func_label2color=visualization.VOClabel2colormap, threshold=None, norm=False) 287 | 288 | img_2 = visualization.convert_to_tf(img2[0]) 289 | norm_cam2 = F.interpolate(orignal_cam2, img_2.shape[1:],mode='bilinear')[0].detach().cpu().numpy() 290 | cam_rv2 = F.interpolate(cam_rv2, img_2.shape[1:],mode='bilinear')[0].detach().cpu().numpy() 291 | CAM2 = visualization.generate_vis(norm_cam2, None, img_2, func_label2color=visualization.VOClabel2colormap, threshold=None, norm=False) 292 | prototype_CAM2 = visualization.generate_vis(cam_rv2, None, img_2, func_label2color=visualization.VOClabel2colormap, threshold=None, norm=False) 293 | 294 | tblogger.add_images('CAM', CAM1, optimizer.global_step) 295 | tblogger.add_images('prototype_CAM1', prototype_CAM1, optimizer.global_step) 296 | tblogger.add_images('CAM2', CAM2, optimizer.global_step) 297 | tblogger.add_images('prototype_CAM2', prototype_CAM2, optimizer.global_step) 298 | 299 | if (optimizer.global_step-1) % args.val_freq == 0 and optimizer.global_step > 10: 300 | miou = validate(model, val_data_loader, optimizer.global_step, tblogger) 301 | torch.save({'net':model.module.state_dict()}, os.path.join(args.session_name, 'ckpt', 'iter_' + str(optimizer.global_step) + '.pth')) 302 | if miou > bestiou: 303 | bestiou = miou 304 | torch.save({'net':model.module.state_dict()}, os.path.join(args.session_name, 'ckpt', 'best.pth')) 305 | else: 306 | timer.reset_stage() 307 | 308 | torch.save({'net':model.module.state_dict()}, os.path.join(args.session_name, 'ckpt', 'final.pth')) 309 | torch.cuda.empty_cache() 310 | 311 | if __name__ == '__main__': 312 | train() 313 | --------------------------------------------------------------------------------