├── .gitignore ├── README.md ├── Toy_Example.md ├── compute_iou.py ├── compute_iou_train.py ├── convert_robot_color.py ├── dataset ├── __init__.py ├── autoaugment.py ├── cityscapes_dataset.py ├── cityscapes_list │ ├── .DS_Store │ ├── info.json │ ├── label.txt │ ├── train.txt │ ├── train_label.txt │ └── val.txt ├── cityscapes_pseudo_dataset.py ├── cityscapes_train_dataset.py ├── drone_dataset.py ├── drone_list │ └── train.txt ├── gta5_dataset.py ├── gta5_list │ ├── gta5_short.py │ ├── info.json │ ├── train.txt │ └── train_short.txt ├── robot_dataset.py ├── robot_list │ ├── info.json │ ├── label.txt │ ├── train.txt │ ├── train_label.txt │ └── val.txt ├── robot_pseudo_dataset.py ├── synthia_dataset.py └── synthia_list │ └── train.txt ├── discuss_plabel.py ├── discuss_plabel_MC_dropout.py ├── evaluate_cityscapes.py ├── evaluate_cityscapes_folder.py ├── evaluate_cityscapes_train.py ├── evaluate_gta5.py ├── evaluate_gta5_folder.py ├── evaluate_robot.py ├── evaluate_robot_train.py ├── focalloss.py ├── generate_plabel_cityscapes.py ├── generate_plabel_cityscapes_99.py ├── generate_plabel_cityscapes_SYNTHIA.py ├── generate_plabel_robot.py ├── model ├── __init__.py ├── blurpool.py ├── deeplab.py ├── deeplab_multi.py ├── deeplab_vgg.py ├── discriminator.py └── ms_discriminator.py ├── pipeline.png ├── sam.py ├── swa_utils.py ├── test.py ├── train_ft.py ├── train_ft_robot.py ├── train_ft_synthia.py ├── train_ms.py ├── train_ms_GTA_SYNTHIA.py ├── train_ms_drone.py ├── train_ms_robot.py ├── train_ms_synthia.py ├── trainer_ms.py ├── trainer_ms_variance.py ├── try_run.py ├── utils ├── __init__.py ├── autoaugment.py ├── clear_model.py ├── loss.py └── tool.py └── visualize_noisy_label.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | *.pth 3 | *.png 4 | *.jpg 5 | *.yaml 6 | 7 | data 8 | 9 | log/ 10 | data/ 11 | result/ 12 | snapshots/ 13 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## AdaBoost_Seg 2 | 3 | ![Python 3.6](https://img.shields.io/badge/python-3.6-green.svg) 4 | [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT) 5 | 6 | ![](pipeline.png) 7 | In this repo, we provide the code for the paper [Adaptive Boosting for Domain Adaptation: Towards Robust Predictions in Scene Segmentation](https://arxiv.org/abs/2103.15685). 8 | 9 | [[Paper]](https://zdzheng.xyz/files/TIP_Adaboost.pdf) [[中文解读]](https://zhuanlan.zhihu.com/p/593571554) 10 | 11 | ## Initial Model 12 | The original DeepLab link of ucmerced is failed. Please use the following link. 13 | 14 | [Google Drive] https://drive.google.com/file/d/1BMTTMCNkV98pjZh_rU0Pp47zeVqF3MEc/view?usp=share_link 15 | 16 | [One Drive] https://1drv.ms/u/s!Avx-MJllNj5b3SqR7yurCxTgIUOK?e=A1dq3m 17 | 18 | or use 19 | ``` 20 | pip install gdown 21 | pip install --upgrade gdown 22 | gdown 1BMTTMCNkV98pjZh_rU0Pp47zeVqF3MEc 23 | ``` 24 | 25 | ## News 26 | - Will more “hard” samples to have a negative effect? How to simulate? (https://github.com/layumi/AdaBoost_Seg/blob/master/Toy_Example.md) 27 | - Evaluation on semi-supervised Cifar10 (https://github.com/layumi/Cifar10-Adaboost) 28 | - A brief illustration of AdaBoost in English at [[Here]](https://cmp.felk.cvut.cz/~sochmj1/adaboost_talk.pdf) and in Chinese [[中文介绍]](https://zhuanlan.zhihu.com/p/368077560) 29 | 30 | ## Tips 31 | 1. When adopting this method to other fields, we suggest to tune the sampling weight with temperature to suit your task and dataset. In this paper, we do not change it, and keep it as 1. 32 | 33 | 2. In our recent experiment, we can achieve a better performance 49.72% (MRNet+Ours) than the number reported in the paper. We think that when Aggrregated Model converges, the adboost sampler updates slowly, which also compromises the performance. If we give more weights to recent snapshots for updating sampler, it works better. 34 | 35 | ```bash 36 | python train_ms.py --snapshot-dir ./snapshots/ReRUN_Adaboost_SWA_SE_GN_batchsize2_1024x512_pp_ms_me0_classbalance7_kl0.1_lr2_drop0.1_seg0.5_swa0_recent --drop 0.1 --warm-up 5000 --batch-size 2 --learning-rate 2e-4 --crop-size 1024,512 --lambda-seg 0.5 --lambda-adv-target1 0.0002 --lambda-adv-target2 0.001 --lambda-me-target 0 --lambda-kl-target 0.1 --norm-style gn --class-balance --only-hard-label 80 --max-value 7 --gpu-ids 0 --often-balance --use-se --swa --swa_start 0 --adaboost --recent 37 | ``` 38 | 39 | ## Table of contents 40 | * [Prerequisites](#prerequisites) 41 | * [Prepare Data](#prepare-data) 42 | * [Training](#training) 43 | * [Testing](#testing) 44 | * [Trained Model](#trained-model) 45 | * [The Key Code](#the-key-code) 46 | * [Related Works](#related-works) 47 | * [Citation](#citation) 48 | 49 | ### Prerequisites 50 | - Python 3.6 51 | - GPU Memory >= 14G (e.g.,RTX6000 or V100) 52 | - Pytorch 53 | 54 | 55 | ### Prepare Data 56 | Download [GTA5] and [Cityscapes] to run the basic code. 57 | Alternatively, you could download extra two datasets from [SYNTHIA] and [OxfordRobotCar]. 58 | 59 | - Download [The GTA5 Dataset]( https://download.visinf.tu-darmstadt.de/data/from_games/ ) 60 | 61 | - Download [The SYNTHIA Dataset]( http://synthia-dataset.net/download/808/) SYNTHIA-RAND-CITYSCAPES (CVPR16) 62 | 63 | - Download [The Cityscapes Dataset]( https://www.cityscapes-dataset.com/ ) 64 | 65 | - Download [The Oxford RobotCar Dataset]( http://www.nec-labs.com/~mas/adapt-seg/adapt-seg.html ) 66 | 67 | The data folder is structured as follows: 68 | ``` 69 | ├── data/ 70 | │ ├── Cityscapes/ 71 | | | ├── data/ 72 | | | ├── gtFine/ 73 | | | ├── leftImg8bit/ 74 | │ ├── GTA5/ 75 | | | ├── images/ 76 | | | ├── labels/ 77 | | | ├── ... 78 | │ ├── synthia/ 79 | | | ├── RGB/ 80 | | | ├── GT/ 81 | | | ├── Depth/ 82 | | | ├── ... 83 | │ └── Oxford_Robot_ICCV19 84 | | | ├── train/ 85 | | | ├── ... 86 | ``` 87 | 88 | ### Training 89 | 90 | - **GTA5 to Cityscapes** (ResNet-101) 91 | 92 | Stage-I: (around 49.0%) 93 | ```bash 94 | python train_ms.py --snapshot-dir ./snapshots/ReRUN_Adaboost_SWA_SE_GN_batchsize2_1024x512_pp_ms_me0_classbalance7_kl0.1_lr2_drop0.1_seg0.5_swa0 --drop 0.1 --warm-up 5000 --batch-size 2 --learning-rate 2e-4 --crop-size 1024,512 --lambda-seg 0.5 --lambda-adv-target1 0.0002 --lambda-adv-target2 0.001 --lambda-me-target 0 --lambda-kl-target 0.1 --norm-style gn --class-balance --only-hard-label 80 --max-value 7 --gpu-ids 0 --often-balance --use-se --swa --swa_start 0 --adaboost 95 | ``` 96 | 97 | Generate Pseudo Label: 98 | ```bash 99 | python generate_plabel_cityscapes.py --restore ./snapshots/ReRUN_Adaboost_SWA_SE_GN_batchsize2_1024x512_pp_ms_me0_classbalance7_kl0.1_lr2_drop0.1_seg0.5_swa0/GTA5_40000_average.pth 100 | ``` 101 | 102 | Stage-II (with recitfying pseudo label): (around 50.9%) 103 | ```bash 104 | python train_ft.py --snapshot-dir ./snapshots/Adaboost_1280x640_restore_ft48_GN_batchsize2_960x480_pp_ms_me0_classbalance7_kl0_lr4_drop0.2_seg0.5_BN_80_255_0.8_Noaug_swa2.5W_t97 --restore-from ./snapshots/ReRUN_Adaboost_SWA_SE_GN_batchsize2_1024x512_pp_ms_me0_classbalance7_kl0.1_lr2_drop0.1_seg0.5_swa0/GTA5_40000_average.pth --drop 0.2 --warm-up 5000 --batch-size 2 --learning-rate 4e-4 --crop-size 960,480 --lambda-seg 0.5 --lambda-adv-target1 0 --lambda-adv-target2 0 --lambda-me-target 0 --lambda-kl-target 0 --norm-style gn --class-balance --only-hard-label 80 --max-value 7 --gpu-ids 0 --often-balance --use-se --input-size 1280,640 --train_bn --autoaug False --swa --adaboost --swa_start 25000 --threshold 97 105 | ``` 106 | 107 | - **SYNTHIA to Cityscapes** 108 | 109 | Stage-I: 110 | ```bash 111 | python train_ms_synthia.py --snapshot-dir ./snapshots/AdaBoost_SWA_SY_SE_GN_batchsize2_1024x512_pp_ms_me0_classbalance7_kl0.1_lr2_drop0.1_seg0.5_power0.5 --drop 0.1 --warm-up 5000 --batch-size 2 --learning-rate 2e-4 --crop-size 1024,512 --lambda-seg 0.5 --lambda-adv-target1 0.0002 --lambda-adv-target2 0.001 --lambda-me-target 0 --lambda-kl-target 0.1 --norm-style gn --class-balance --only-hard-label 80 --max-value 7 --gpu-ids 0 --often-balance --use-se --swa --swa_start 0 --adaboost 112 | ``` 113 | 114 | Generate Pseudo Label: 115 | ```bash 116 | python generate_plabel_cityscapes_SYNTHIA.py --restore ./snapshots/AdaBoost_SWA_SY_SE_GN_batchsize2_1024x512_pp_ms_me0_classbalance7_kl0.1_lr2_drop0.1_seg0.5_power0.5/GTA5_50000_average.pth 117 | ``` 118 | 119 | Stage-II: 120 | ```bash 121 | python train_ft_synthia.py --snapshot-dir ./snapshots/Cosine_Adaboost_SY_1280x640_restore_ft_GN_batchsize8_512x256_pp_ms_me0_classbalance7_kl0.1_lr8_drop0.1_seg0.5_BN_255_Noaug_t777_swa2.5W --restore ./snapshots/AdaBoost_SWA_SY_SE_GN_batchsize2_1024x512_pp_ms_me0_classbalance7_kl0.1_lr2_drop0.1_seg0.5_power0.5/GTA5_50000_average.pth --drop 0.1 --warm-up 5000 --batch-size 8 --learning-rate 8e-4 --crop-size 512,256 --lambda-seg 0.5 --lambda-adv-target1 0 --lambda-adv-target2 0 --lambda-me-target 0 --lambda-kl-target 0 --norm-style gn --class-balance --only-hard-label 50 --max-value 7 --gpu-ids 0 --often-balance --use-se --input-size 1280,640 --autoaug False --swa --swa_start 25000 --threshold 777 --adaboost --train_bn --cosine 122 | ``` 123 | 124 | - **Cityscapes to Oxford RobotCar** 125 | 126 | Stage-I: (around 73.80%) higher than paper. 127 | ```bash 128 | python train_ms_robot.py --snapshot-dir ./snapshots/Adaboost_SWA3W_Robot_SE_GN_batchsize6_adapative_kl0.1_sam_lr6 --drop 0.1 --warm-up 5000 --batch-size 6 --learning-rate 6e-4 --crop-size 800,400 --lambda-seg 0.5 --lambda-adv-target1 0.0002 --lambda-adv-target2 0.001 --lambda-me-target 0 --lambda-kl-target 0.1 --norm-style gn --class-balance --only-hard-label 80 --max-value 7 --gpu-ids 0,1,2 --often-balance --use-se --swa --swa_start 30000 --adaboost --sam 129 | ``` 130 | 131 | Generate Pseudo Label: 132 | ```bash 133 | python generate_plabel_robot.py --restore ./snapshots/Adaboost_SWA3W_Robot_SE_GN_batchsize6_adapative_kl0.1_sam_lr6/GTA5_70000_average.pth 134 | ``` 135 | 136 | Stage-II: (around 75.62%) 137 | ```bash 138 | python train_ft_robot.py --snapshot-dir ./snapshots/Adaboost_0.9RB_b3_lr3_800x432_97_swa0W_T80 --restore-from ./snapshots/Adaboost_SWA3W_Robot_SE_GN_batchsize6_adapative_kl0.1_sam_lr6/GTA5_70000_average.pth --drop 0.1 --warm-up 5000 --batch-size 3 --learning-rate 3e-4 --crop-size 800,432 --lambda-seg 0.5 --lambda-adv-target1 0 --lambda-adv-target2 0 --lambda-me-target 0 --lambda-kl-target 0 --norm-style gn --class-balance --only-hard-label 50 --max-value 7 --gpu-ids 0,1,2 --often-balance --use-se --input-size 1280,960 --train_bn --adaboost --swa --swa_start 0 --threshold 0.8 --autoaug False 139 | ``` 140 | 141 | ### Ablation Studies 142 | 143 | - GTA5 to Cityscapes (VGG-16) 144 | 145 | Stage-I: (around 39.5%) 146 | ```bash 147 | python train_ms.py --snapshot-dir ./snapshots/255VGGBN_Adaboost_SWA_SE_GN_batchsize3_1024x512_pp_ms_me0_classbalance7_kl0.1_lr3_drop0.1_seg0.5_swa0_auto --drop 0.1 --warm-up 5000 --batch-size 3 --learning-rate 3e-4 --crop-size 1024,512 --lambda-seg 0.5 --lambda-adv-target1 0.0002 --lambda-adv-target2 0.001 --lambda-me-target 0 --lambda-kl-target 0.1 --norm-style gn --class-balance --only-hard-label 80 --max-value 7 --gpu-ids 0,1,2 --often-balance --use-se --swa --swa_start 0 --adaboost --model DeepVGG --autoaug 148 | ``` 149 | 150 | ### Testing 151 | ```bash 152 | python evaluate_cityscapes.py --restore-from ./snapshots/ReRUN_Adaboost_SWA_SE_GN_batchsize2_1024x512_pp_ms_me0_classbalance7_kl0.1_lr2_drop0.1_seg0.5_swa0/GTA5_40000_average.pth 153 | ``` 154 | 155 | ### Trained Model 156 | The trained model is available at [Wait] 157 | 158 | - The folder with `SY` in name is for SYNTHIA-to-Cityscapes 159 | - The folder with `RB` in name is for Cityscapes-to-Robot Car 160 | 161 | ### The Key Code 162 | Core code is relatively simple, and could be directly applied to other works. 163 | 164 | Adaptive Data Sampler: https://github.com/layumi/AdaBoost_Seg/blob/master/train_ms.py#L429-L436 165 | 166 | Student Aggregation: https://github.com/layumi/AdaBoost_Seg/blob/master/train_ms.py#L415-L427 167 | 168 | 169 | ### Related Works 170 | We also would like to thank great works as follows: 171 | - https://github.com/layumi/Seg-Uncertainty 172 | - https://github.com/wasidennis/AdaptSegNet 173 | - https://github.com/RoyalVane/CLAN 174 | - https://github.com/yzou2/CRST 175 | 176 | ### Citation 177 | ```bibtex 178 | @article{zheng2021adaboost, 179 | title={Adaptive Boosting for Domain Adaptation: Towards Robust Predictions in Scene Segmentation}, 180 | author={Zheng, Zhedong and Yang, Yi}, 181 | journal={IEEE Transactions on Image Processing}, 182 | doi={10.1109/TIP.2022.3195642}, 183 | note={\mbox{doi}:\url{10.1109/TIP.2022.3195642}}, 184 | year={2021} 185 | } 186 | ``` 187 | -------------------------------------------------------------------------------- /Toy_Example.md: -------------------------------------------------------------------------------- 1 | ## Will more “hard” samples to have a negative effect? 2 | 3 | Yes. We intend to obtain complementary snapshots as the final model and some single snapshots may overfit the negative sample, yielding a worse performance. 4 | This phenomenon also occurs in the conventional Adaboost algorithm. 5 | 6 | Given x=[0,1,2,3,4,5,6], y = [1, -1,-1,-1, 1, 1,1], we train one binary classifier mapping x to y: 7 | 8 | - In the first round, we will get the binary classifier. G1(x) = -1 (if x<3.5), 1 (if x≥3.5), only one sample is wrong. 9 | The error rate is 1/7 = 14.3%. Then we update the data weights as [0.4, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1] 10 | (Here just show the extreme case. Weight updating strategy can be different. ) 11 | 12 | 13 | 14 | - In the second round, we will get a new binary classifier. G2(x) = -1 (if x<0), 1 (if x≥0) 15 | The error is 3/7 = 42.9%, which is worse than the error rate in the first round. 16 | 17 | 18 | Here the toy example just to show one worst case in Adaboost that the weak classifier of certain epoch can be worse than the model trained in the early stage. On the other hand, it is also worth noting that the second-round classifier is complementary to the first-round model for the first data (x=0). Similarly, in our practice, the model focuses on hard negative data and may perform worse in the late stage. Despite the probability of overfitting the negatives, such snapshots are complementary to the snapshot trained in the early stage and help the ensembled model keep the performance improvement. 19 | -------------------------------------------------------------------------------- /compute_iou.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import argparse 3 | import json 4 | from PIL import Image 5 | from os.path import join 6 | 7 | 8 | def fast_hist(a, b, n): 9 | k = (a >= 0) & (a < n) 10 | return np.bincount(n * a[k].astype(int) + b[k], minlength=n ** 2).reshape(n, n) 11 | 12 | 13 | def per_class_iu(hist): 14 | return np.diag(hist) / (hist.sum(1) + hist.sum(0) - np.diag(hist)) 15 | 16 | 17 | def label_mapping(input, mapping): 18 | output = np.copy(input) 19 | for ind in range(len(mapping)): 20 | output[input == mapping[ind][0]] = mapping[ind][1] 21 | return np.array(output, dtype=np.int64) 22 | 23 | 24 | def compute_mIoU(gt_dir, pred_dir, devkit_dir=''): 25 | """ 26 | Compute IoU given the predicted colorized images and 27 | """ 28 | with open(join(devkit_dir, 'info.json'), 'r') as fp: 29 | info = json.load(fp) 30 | num_classes = np.int(info['classes']) 31 | name_classes = np.array(info['label'], dtype=np.str) 32 | mapping = np.array(info['label2train'], dtype=np.int) 33 | hist = np.zeros((num_classes, num_classes)) 34 | 35 | image_path_list = join(devkit_dir, 'val.txt') 36 | label_path_list = join(devkit_dir, 'label.txt') 37 | gt_imgs = open(label_path_list, 'r').read().splitlines() 38 | gt_imgs = [join(gt_dir, x) for x in gt_imgs] 39 | pred_imgs = open(image_path_list, 'r').read().splitlines() 40 | pred_imgs = [join(pred_dir, x.split('/')[-1]) for x in pred_imgs] 41 | 42 | for ind in range(len(gt_imgs)): 43 | pred = np.array(Image.open(pred_imgs[ind])) 44 | label = np.array(Image.open(gt_imgs[ind])) 45 | label = label_mapping(label, mapping) 46 | if len(label.shape) == 3 and label.shape[2]==4: 47 | label = label[:,:,0] 48 | if len(label.flatten()) != len(pred.flatten()): 49 | print(('Skipping: len(gt) = {:d}, len(pred) = {:d}, {:s}, {:s}'.format(len(label.flatten()), len(pred.flatten()), gt_imgs[ind], pred_imgs[ind]))) 50 | continue 51 | hist += fast_hist(label.flatten(), pred.flatten(), num_classes) 52 | #if ind > 0 and ind % 10 == 0: 53 | # print(('{:d} / {:d}: {:0.2f}'.format(ind, len(gt_imgs), 100*np.mean(per_class_iu(hist))))) 54 | 55 | mIoUs = per_class_iu(hist) 56 | for ind_class in range(num_classes): 57 | if pred_dir.endswith('_p') or pred_dir.endswith('_a'): 58 | break 59 | print(('===>' + name_classes[ind_class] + ':\t' + str(round(mIoUs[ind_class] * 100, 2)))) 60 | 61 | if pred_dir.endswith('_p'): 62 | print(('Main Classifier: ===> mIoU: ' + str(round(np.nanmean(mIoUs) * 100, 2)))) 63 | elif pred_dir.endswith('_a'): 64 | print(('Auxiliary Classifier: ===> mIoU: ' + str(round(np.nanmean(mIoUs) * 100, 2)))) 65 | else: 66 | print(('Num classes', num_classes)) 67 | print(('Fusion: ===> mIoU: ' + str(round(np.nanmean(mIoUs) * 100, 2)))) 68 | return mIoUs 69 | 70 | 71 | def main(args): 72 | compute_mIoU(args.gt_dir, args.pred_dir, args.devkit_dir) 73 | 74 | 75 | if __name__ == "__main__": 76 | parser = argparse.ArgumentParser() 77 | parser.add_argument('gt_dir', type=str, help='directory which stores CityScapes val gt images') 78 | parser.add_argument('pred_dir', type=str, help='directory which stores CityScapes val pred images') 79 | parser.add_argument('--devkit_dir', default='dataset/cityscapes_list', help='base directory of cityscapes') 80 | args = parser.parse_args() 81 | main(args) 82 | -------------------------------------------------------------------------------- /compute_iou_train.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import argparse 3 | import json 4 | from PIL import Image 5 | from os.path import join 6 | 7 | 8 | def fast_hist(a, b, n): 9 | k = (a >= 0) & (a < n) 10 | return np.bincount(n * a[k].astype(int) + b[k], minlength=n ** 2).reshape(n, n) 11 | 12 | 13 | def per_class_iu(hist): 14 | return np.diag(hist) / (hist.sum(1) + hist.sum(0) - np.diag(hist)) 15 | 16 | 17 | def label_mapping(input, mapping): 18 | output = np.copy(input) 19 | for ind in range(len(mapping)): 20 | output[input == mapping[ind][0]] = mapping[ind][1] 21 | return np.array(output, dtype=np.int64) 22 | 23 | 24 | def compute_mIoU(gt_dir, pred_dir, devkit_dir=''): 25 | """ 26 | Compute IoU given the predicted colorized images and 27 | """ 28 | with open(join(devkit_dir, 'info.json'), 'r') as fp: 29 | info = json.load(fp) 30 | num_classes = np.int(info['classes']) 31 | print(('Num classes', num_classes)) 32 | name_classes = np.array(info['label'], dtype=np.str) 33 | mapping = np.array(info['label2train'], dtype=np.int) 34 | hist = np.zeros((num_classes, num_classes)) 35 | 36 | image_path_list = join(devkit_dir, 'train_short.txt') 37 | label_path_list = join(devkit_dir, 'train_short.txt') 38 | pred_imgs = open(image_path_list, 'r').read().splitlines() 39 | pred_imgs = [join(pred_dir, x.split('/')[-1]) for x in pred_imgs] 40 | gt_imgs = open(label_path_list, 'r').read().splitlines() 41 | gt_imgs = [join(gt_dir, x) for x in gt_imgs] 42 | 43 | for ind in range(len(gt_imgs)): 44 | pred = np.array(Image.open(pred_imgs[ind])) 45 | label = np.array(Image.open(gt_imgs[ind]).resize((1280,640), Image.NEAREST)) 46 | label = label_mapping(label, mapping) 47 | if len(label.shape) == 3 and label.shape[2]==4: 48 | label = label[:,:,0] 49 | if len(label.flatten()) != len(pred.flatten()): 50 | print(('Skipping: len(gt) = {:d}, len(pred) = {:d}, {:s}, {:s}'.format(len(label.flatten()), len(pred.flatten()), gt_imgs[ind], pred_imgs[ind]))) 51 | continue 52 | hist += fast_hist(label.flatten(), pred.flatten(), num_classes) 53 | if ind > 0 and ind % 10 == 0: 54 | print(('{:d} / {:d}: {:0.2f}'.format(ind, len(gt_imgs), 100*np.mean(per_class_iu(hist))))) 55 | 56 | mIoUs = per_class_iu(hist) 57 | for ind_class in range(num_classes): 58 | print(('===>' + name_classes[ind_class] + ':\t' + str(round(mIoUs[ind_class] * 100, 2)))) 59 | print(('===> mIoU: ' + str(round(np.nanmean(mIoUs) * 100, 2)))) 60 | return mIoUs 61 | 62 | 63 | def main(args): 64 | compute_mIoU(args.gt_dir, args.pred_dir, args.devkit_dir) 65 | 66 | 67 | if __name__ == "__main__": 68 | parser = argparse.ArgumentParser() 69 | parser.add_argument('gt_dir', type=str, help='directory which stores CityScapes val gt images') 70 | parser.add_argument('pred_dir', type=str, help='directory which stores CityScapes val pred images') 71 | parser.add_argument('--devkit_dir', default='dataset/cityscapes_list', help='base directory of cityscapes') 72 | args = parser.parse_args() 73 | main(args) 74 | -------------------------------------------------------------------------------- /convert_robot_color.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import argparse 3 | import json 4 | from PIL import Image 5 | from os.path import join 6 | from evaluate_robot import colorize_mask,save 7 | from compute_iou import label_mapping 8 | 9 | def main(gt_dir='./data/Oxford_Robot_ICCV19/anno', devkit_dir = './dataset/robot_list/'): 10 | """ 11 | Compute IoU given the predicted colorized images and 12 | """ 13 | with open(join(devkit_dir, 'info.json'), 'r') as fp: 14 | info = json.load(fp) 15 | image_path_list = join(devkit_dir, 'val.txt') 16 | label_path_list = join(devkit_dir, 'label.txt') 17 | mapping = np.array(info['label2train'], dtype=np.int) 18 | gt_imgs = open(label_path_list, 'r').read().splitlines() 19 | gt_imgs = [join(gt_dir, x) for x in gt_imgs] 20 | 21 | for ind in range(len(gt_imgs)): 22 | label = np.array(Image.open(gt_imgs[ind])) 23 | label = label_mapping(label, mapping) 24 | label = label[:,:,0].astype(np.uint8) 25 | name_tmp = gt_imgs[ind].replace('anno','anno_color') 26 | save([label, name_tmp]) 27 | 28 | return 29 | 30 | 31 | 32 | if __name__ == "__main__": 33 | main() 34 | -------------------------------------------------------------------------------- /dataset/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/layumi/AdaBoost_Seg/2624fe9c7e0c877097248b7fa9d3cf3c367f73c3/dataset/__init__.py -------------------------------------------------------------------------------- /dataset/cityscapes_dataset.py: -------------------------------------------------------------------------------- 1 | import os 2 | import os.path as osp 3 | import numpy as np 4 | import random 5 | import matplotlib 6 | matplotlib.use('agg') 7 | import matplotlib.pyplot as plt 8 | import collections 9 | import torch 10 | import torchvision 11 | from torch.utils import data 12 | from PIL import Image, ImageFile 13 | from dataset.autoaugment import ImageNetPolicy 14 | import time 15 | 16 | ImageFile.LOAD_TRUNCATED_IMAGES = True 17 | 18 | class cityscapesDataSet(data.Dataset): 19 | def __init__(self, root, list_path, max_iters=None, resize_size=(1024, 512), crop_size=(512, 1024), mean=(128, 128, 128), scale=False, mirror=True, ignore_label=255, set='val', autoaug=False): 20 | self.root = root 21 | self.list_path = list_path 22 | self.crop_size = crop_size 23 | self.scale = scale 24 | self.ignore_label = ignore_label 25 | self.mean = mean 26 | self.is_mirror = mirror 27 | self.resize_size = resize_size 28 | self.autoaug = autoaug 29 | self.h = crop_size[0] 30 | self.w = crop_size[1] 31 | # self.mean_bgr = np.array([104.00698793, 116.66876762, 122.67891434]) 32 | self.img_ids = [i_id.strip() for i_id in open(list_path)] 33 | if not max_iters==None: 34 | self.img_ids = self.img_ids * int(np.ceil(float(max_iters) / len(self.img_ids))) 35 | self.files = [] 36 | self.set = set 37 | # for split in ["train", "trainval", "val"]: 38 | 39 | #https://github.com/mcordts/cityscapesScripts/blob/master/cityscapesscripts/helpers/labels.py 40 | self.id_to_trainid = {7: 0, 8: 1, 11: 2, 12: 3, 13: 4, 17: 5, 41 | 19: 6, 20: 7, 21: 8, 22: 9, 23: 10, 24: 11, 25: 12, 42 | 26: 13, 27: 14, 28: 15, 31: 16, 32: 17, 33: 18} 43 | 44 | for name in self.img_ids: 45 | img_file = osp.join(self.root, "leftImg8bit/%s/%s" % (self.set, name)) 46 | label_file = osp.join(self.root, "gtFine/%s/%s" % (self.set, name.replace('leftImg8bit', 'gtFine_labelIds') )) 47 | self.files.append({ 48 | "img": img_file, 49 | "label": label_file, 50 | "name": name 51 | }) 52 | 53 | def __len__(self): 54 | return len(self.files) 55 | 56 | def __getitem__(self, index): 57 | #tt = time.time() 58 | datafiles = self.files[index] 59 | name = datafiles["name"] 60 | 61 | image, label = Image.open(datafiles["img"]).convert('RGB'), Image.open(datafiles["label"]) 62 | # resize 63 | if self.scale: 64 | random_scale = 0.8 + random.random()*0.4 # 0.8 - 1.2 65 | image = image.resize( ( round(self.resize_size[0] * random_scale), round(self.resize_size[1] * random_scale)) , Image.BICUBIC) 66 | label = label.resize( ( round(self.resize_size[0] * random_scale), round(self.resize_size[1] * random_scale)) , Image.NEAREST) 67 | else: 68 | image, label = image.resize(self.resize_size, Image.BICUBIC), label.resize(self.resize_size, Image.NEAREST) 69 | 70 | if self.autoaug: 71 | policy = ImageNetPolicy() 72 | image = policy(image) 73 | 74 | image, label = np.asarray(image, np.float32), np.asarray(label, np.uint8) 75 | 76 | # re-assign labels to match the format of Cityscapes 77 | label_copy = 255 * np.ones(label.shape, dtype=np.uint8) 78 | for k, v in list(self.id_to_trainid.items()): 79 | label_copy[label == k] = v 80 | 81 | size = image.shape 82 | image = image[:, :, ::-1] # change to BGR 83 | image -= self.mean 84 | image = image.transpose((2, 0, 1)) 85 | x1 = random.randint(0, image.shape[1] - self.h) 86 | y1 = random.randint(0, image.shape[2] - self.w) 87 | image = image[:, x1:x1+self.h, y1:y1+self.w] 88 | label_copy = label_copy[x1:x1+self.h, y1:y1+self.w] 89 | 90 | if self.is_mirror and random.random() < 0.5: 91 | image = np.flip(image, axis = 2) 92 | label_copy = np.flip(label_copy, axis = 1) 93 | #print('Time used: {} sec'.format(time.time()-tt)) 94 | return image.copy(), label_copy.copy(), np.array(size), name 95 | 96 | 97 | if __name__ == '__main__': 98 | dst = cityscapesDataSet('./data/Cityscapes/data', './dataset/cityscapes_list/train.txt', mean=(0,0,0), set = 'train') 99 | trainloader = data.DataLoader(dst, batch_size=4) 100 | for i, data in enumerate(trainloader): 101 | imgs, _, _, _ = data 102 | if i == 0: 103 | img = torchvision.utils.make_grid(imgs).numpy() 104 | img = np.transpose(img, (1, 2, 0)) 105 | img = img[:, :, ::-1] 106 | img = Image.fromarray(np.uint8(img) ) 107 | img.save('Cityscape_Demo.jpg') 108 | break 109 | -------------------------------------------------------------------------------- /dataset/cityscapes_list/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/layumi/AdaBoost_Seg/2624fe9c7e0c877097248b7fa9d3cf3c367f73c3/dataset/cityscapes_list/.DS_Store -------------------------------------------------------------------------------- /dataset/cityscapes_list/info.json: -------------------------------------------------------------------------------- 1 | { 2 | "classes":19, 3 | "label2train":[ 4 | [0, 255], 5 | [1, 255], 6 | [2, 255], 7 | [3, 255], 8 | [4, 255], 9 | [5, 255], 10 | [6, 255], 11 | [7, 0], 12 | [8, 1], 13 | [9, 255], 14 | [10, 255], 15 | [11, 2], 16 | [12, 3], 17 | [13, 4], 18 | [14, 255], 19 | [15, 255], 20 | [16, 255], 21 | [17, 5], 22 | [18, 255], 23 | [19, 6], 24 | [20, 7], 25 | [21, 8], 26 | [22, 9], 27 | [23, 10], 28 | [24, 11], 29 | [25, 12], 30 | [26, 13], 31 | [27, 14], 32 | [28, 15], 33 | [29, 255], 34 | [30, 255], 35 | [31, 16], 36 | [32, 17], 37 | [33, 18], 38 | [-1, 255]], 39 | "label":[ 40 | "road", 41 | "sidewalk", 42 | "building", 43 | "wall", 44 | "fence", 45 | "pole", 46 | "light", 47 | "sign", 48 | "vegetation", 49 | "terrain", 50 | "sky", 51 | "person", 52 | "rider", 53 | "car", 54 | "truck", 55 | "bus", 56 | "train", 57 | "motocycle", 58 | "bicycle"], 59 | "palette":[ 60 | [128,64,128], 61 | [244,35,232], 62 | [70,70,70], 63 | [102,102,156], 64 | [190,153,153], 65 | [153,153,153], 66 | [250,170,30], 67 | [220,220,0], 68 | [107,142,35], 69 | [152,251,152], 70 | [70,130,180], 71 | [220,20,60], 72 | [255,0,0], 73 | [0,0,142], 74 | [0,0,70], 75 | [0,60,100], 76 | [0,80,100], 77 | [0,0,230], 78 | [119,11,32], 79 | [0,0,0]], 80 | "mean":[ 81 | 73.158359210711552, 82 | 82.908917542625858, 83 | 72.392398761941593], 84 | "std":[ 85 | 47.675755341814678, 86 | 48.494214368814916, 87 | 47.736546325441594] 88 | } 89 | -------------------------------------------------------------------------------- /dataset/cityscapes_pseudo_dataset.py: -------------------------------------------------------------------------------- 1 | import os 2 | import os.path as osp 3 | import numpy as np 4 | import random 5 | import matplotlib 6 | matplotlib.use('agg') 7 | import matplotlib.pyplot as plt 8 | import collections 9 | import torch 10 | import torchvision 11 | from torch.utils import data 12 | from PIL import Image,ImageFile 13 | from dataset.autoaugment import ImageNetPolicy 14 | 15 | ImageFile.LOAD_TRUNCATED_IMAGES = True 16 | 17 | class cityscapes_pseudo_DataSet(data.Dataset): 18 | def __init__(self, root, list_path, max_iters=None, resize_size=(1024, 512), crop_size=(512, 1024), mean=(128, 128, 128), scale=False, mirror=True, ignore_label=255, set='val', autoaug=False, synthia=False, threshold = 1.0): 19 | self.root = root 20 | self.threshold = threshold 21 | self.list_path = list_path 22 | self.crop_size = crop_size 23 | self.scale = scale 24 | self.ignore_label = ignore_label 25 | self.mean = mean 26 | self.is_mirror = mirror 27 | self.resize_size = resize_size 28 | self.autoaug = autoaug 29 | self.h = crop_size[0] 30 | self.w = crop_size[1] 31 | # self.mean_bgr = np.array([104.00698793, 116.66876762, 122.67891434]) 32 | self.img_ids = [i_id.strip() for i_id in open(list_path)] 33 | if not max_iters==None: 34 | self.img_ids = self.img_ids * int(np.ceil(float(max_iters) / len(self.img_ids))) 35 | self.files = [] 36 | self.set = set 37 | # for split in ["train", "trainval", "val"]: 38 | 39 | #https://github.com/mcordts/cityscapesScripts/blob/master/cityscapesscripts/helpers/labels.py 40 | self.id_to_trainid = {7: 0, 8: 1, 11: 2, 12: 3, 13: 4, 17: 5, 41 | 19: 6, 20: 7, 21: 8, 22: 9, 23: 10, 24: 11, 25: 12, 42 | 26: 13, 27: 14, 28: 15, 31: 16, 32: 17, 33: 18} 43 | 44 | for name in self.img_ids: 45 | img_file = osp.join(self.root, "leftImg8bit/%s/%s" % (self.set, name)) 46 | label_file = osp.join(self.root, "pseudo_FULL/%s/%s" % (self.set, name )) 47 | score_file = osp.join(self.root, "pseudo/%s/%s_score.png" % (self.set, name )) 48 | if synthia: 49 | label_file = osp.join(self.root, "pseudo_SYNTHIA/%s/%s" % (self.set, name )) 50 | score_file = osp.join(self.root, "pseudo_SYNTHIA/%s/%s_score.png" % (self.set, name )) 51 | self.files.append({ 52 | "img": img_file, 53 | "label": label_file, 54 | "score": score_file, 55 | "name": name 56 | }) 57 | 58 | def __len__(self): 59 | return len(self.files) 60 | 61 | def __getitem__(self, index): 62 | datafiles = self.files[index] 63 | 64 | image = Image.open(datafiles["img"]).convert('RGB') 65 | label = Image.open(datafiles["label"]) 66 | score = Image.open(datafiles["score"]) 67 | name = datafiles["name"] 68 | 69 | 70 | # resize 71 | if self.scale: 72 | random_scale = 0.8 + random.random()*0.4 # 0.8 - 1.2 73 | image = image.resize( ( round(self.resize_size[0] * random_scale), round(self.resize_size[1] * random_scale)) , Image.BICUBIC) 74 | label = label.resize( ( round(self.resize_size[0] * random_scale), round(self.resize_size[1] * random_scale)) , Image.NEAREST) 75 | score = score.resize( ( round(self.resize_size[0] * random_scale), round(self.resize_size[1] * random_scale)) , Image.NEAREST) 76 | else: 77 | image = image.resize( ( self.resize_size[0], self.resize_size[1] ) , Image.BICUBIC) 78 | label = label.resize( ( self.resize_size[0], self.resize_size[1] ) , Image.NEAREST) 79 | score = score.resize( ( self.resize_size[0], self.resize_size[1] ) , Image.NEAREST) 80 | 81 | if self.autoaug: 82 | policy = ImageNetPolicy() 83 | image = policy(image) 84 | 85 | image = np.array(image, np.float32) 86 | label = np.array(label, np.uint8) 87 | score = np.array(score, np.uint8) 88 | 89 | # re-assign labels to match the format of Cityscapes 90 | #label_copy = 255 * np.ones(label.shape, dtype=np.float32) 91 | #for k, v in list(self.id_to_trainid.items()): 92 | # label_copy[label == k] = v 93 | label_copy = label 94 | # threshold 95 | if self.threshold<1.0: 96 | label_copy[score<(self.threshold*100)] = 255 97 | 98 | size = image.shape 99 | image = image[:, :, ::-1] # change to BGR 100 | image -= self.mean 101 | image = image.transpose((2, 0, 1)) 102 | #print(image.shape, label.shape) 103 | for i in range(10): #find hard samples 104 | x1 = random.randint(0, image.shape[1] - self.h) 105 | y1 = random.randint(0, image.shape[2] - self.w) 106 | tmp_label_copy = label_copy[x1:x1+self.h, y1:y1+self.w] 107 | tmp_image = image[:, x1:x1+self.h, y1:y1+self.w] 108 | u = np.unique(tmp_label_copy) 109 | if len(u) > 10: 110 | break 111 | #else: 112 | #print('Cityscape-Pseudo: Too young too naive for %d times!'%i) 113 | image = tmp_image 114 | label_copy = tmp_label_copy 115 | 116 | if self.is_mirror and random.random() < 0.5: 117 | image = np.flip(image, axis = 2) 118 | label_copy = np.flip(label_copy, axis = 1) 119 | 120 | return image.copy(), label_copy.copy(), np.array(size), name 121 | 122 | 123 | if __name__ == '__main__': 124 | dst = cityscapes_pseudo_DataSet('./data/Cityscapes/data', './dataset/cityscapes_list/train.txt', mean=(0,0,0), set = 'train', autoaug=True) 125 | trainloader = data.DataLoader(dst, batch_size=4) 126 | for i, data in enumerate(trainloader): 127 | imgs, _, _,_ = data 128 | if i == 0: 129 | img = torchvision.utils.make_grid(imgs).numpy() 130 | img = np.transpose(img, (1, 2, 0)) 131 | img = img[:, :, ::-1] 132 | img = Image.fromarray(np.uint8(img) ) 133 | img.save('Cityscape_Demo.jpg') 134 | break 135 | -------------------------------------------------------------------------------- /dataset/cityscapes_train_dataset.py: -------------------------------------------------------------------------------- 1 | import os 2 | import os.path as osp 3 | import numpy as np 4 | import random 5 | import matplotlib 6 | matplotlib.use('agg') 7 | import matplotlib.pyplot as plt 8 | import collections 9 | import torch 10 | import torchvision 11 | from torch.utils import data 12 | from PIL import Image 13 | from dataset.autoaugment import ImageNetPolicy 14 | import time 15 | 16 | class cityscapesDataSet(data.Dataset): 17 | def __init__(self, root, list_path, max_iters=None, resize_size=(1024, 512), crop_size=(512, 1024), mean=(128, 128, 128), scale=False, mirror=True, ignore_label=255, set='train', autoaug=False): 18 | self.root = root 19 | self.list_path = list_path 20 | self.crop_size = crop_size 21 | self.scale = scale 22 | self.ignore_label = ignore_label 23 | self.mean = mean 24 | self.is_mirror = mirror 25 | self.resize_size = resize_size 26 | self.autoaug = autoaug 27 | self.h = crop_size[0] 28 | self.w = crop_size[1] 29 | # self.mean_bgr = np.array([104.00698793, 116.66876762, 122.67891434]) 30 | self.img_ids = [i_id.strip() for i_id in open(list_path)] 31 | if not max_iters==None: 32 | self.img_ids = self.img_ids * int(np.ceil(float(max_iters) / len(self.img_ids))) 33 | self.files = [] 34 | self.set = set 35 | # for split in ["train", "trainval", "val"]: 36 | 37 | #https://github.com/mcordts/cityscapesScripts/blob/master/cityscapesscripts/helpers/labels.py 38 | ''' 39 | project Cityscapes to Oxford Robot 40 | 7 road -> 8; 8 sidewalk -> 7; building 11 -> 6; wall 12 -> 255; 41 | fence 13 -> 255; pole 17-> 255: light 19 -> 5; sign 20->4; 42 | vegetation -> 255; terrain -> 255; sky 23 -> 0; person 24 -> 1 ; 43 | rider 25 -> 1 ; car 26 -> 3; truck 27 ->3; bus 28 ->3; train 31->255; 44 | motorcycle 32->2 ; bike 33 -> 2; 45 | 46 | ''' 47 | self.id_to_trainid = {7: 8, 8: 7, 11: 6, 48 | 19: 5, 20: 4, 23: 0, 24: 1, 25: 1, 49 | 26: 3, 27: 3, 28: 3, 32: 2, 33: 2} 50 | 51 | for name in self.img_ids: 52 | img_file = osp.join(self.root, "leftImg8bit/%s/%s" % (self.set, name)) 53 | label_file = osp.join(self.root, "gtFine/%s/%s" % (self.set, name.replace('leftImg8bit', 'gtFine_labelIds') )) 54 | self.files.append({ 55 | "img": img_file, 56 | "label": label_file, 57 | "name": name 58 | }) 59 | 60 | def __len__(self): 61 | return len(self.files) 62 | 63 | def __getitem__(self, index): 64 | #tt = time.time() 65 | datafiles = self.files[index] 66 | name = datafiles["name"] 67 | 68 | image, label = Image.open(datafiles["img"]).convert('RGB'), Image.open(datafiles["label"]) 69 | # resize 70 | if self.scale: 71 | random_scale = 0.8 + random.random()*0.4 # 0.8 - 1.2 72 | image = image.resize( ( round(self.resize_size[0] * random_scale), round(self.resize_size[1] * random_scale)) , Image.BICUBIC) 73 | label = label.resize( ( round(self.resize_size[0] * random_scale), round(self.resize_size[1] * random_scale)) , Image.NEAREST) 74 | else: 75 | image, label = image.resize(self.resize_size, Image.BICUBIC), label.resize(self.resize_size, Image.NEAREST) 76 | 77 | if self.autoaug: 78 | policy = ImageNetPolicy() 79 | image = policy(image) 80 | 81 | image, label = np.asarray(image, np.float32), np.asarray(label, np.uint8) 82 | 83 | # re-assign labels to match the format of Cityscapes 84 | label_copy = 255 * np.ones(label.shape, dtype=np.uint8) 85 | for k, v in list(self.id_to_trainid.items()): 86 | label_copy[label == k] = v 87 | 88 | size = image.shape 89 | image = image[:, :, ::-1] # change to BGR 90 | image -= self.mean 91 | image = image.transpose((2, 0, 1)) 92 | x1 = random.randint(0, image.shape[1] - self.h) 93 | y1 = random.randint(0, image.shape[2] - self.w) 94 | image = image[:, x1:x1+self.h, y1:y1+self.w] 95 | label_copy = label_copy[x1:x1+self.h, y1:y1+self.w] 96 | 97 | if self.is_mirror and random.random() < 0.5: 98 | image = np.flip(image, axis = 2) 99 | label_copy = np.flip(label_copy, axis = 1) 100 | #print('Time used: {} sec'.format(time.time()-tt)) 101 | return image.copy(), label_copy.copy(), np.array(size), name 102 | 103 | 104 | if __name__ == '__main__': 105 | dst = cityscapesDataSet('./data/Cityscapes/data', './dataset/cityscapes_list/train.txt', mean=(0,0,0), set = 'train') 106 | trainloader = data.DataLoader(dst, batch_size=4) 107 | for i, data in enumerate(trainloader): 108 | imgs, _, _, _ = data 109 | if i == 0: 110 | img = torchvision.utils.make_grid(imgs).numpy() 111 | img = np.transpose(img, (1, 2, 0)) 112 | img = img[:, :, ::-1] 113 | img = Image.fromarray(np.uint8(img) ) 114 | img.save('Cityscape_Demo.jpg') 115 | break 116 | -------------------------------------------------------------------------------- /dataset/drone_dataset.py: -------------------------------------------------------------------------------- 1 | import os 2 | import os.path as osp 3 | import numpy as np 4 | import random 5 | import matplotlib.pyplot as plt 6 | import collections 7 | import torch 8 | import torchvision 9 | from torch.utils import data 10 | from PIL import Image, ImageFile 11 | from dataset.autoaugment import ImageNetPolicy 12 | import imageio 13 | ImageFile.LOAD_TRUNCATED_IMAGES = True 14 | 15 | 16 | class DroneDataSet(data.Dataset): 17 | def __init__(self, root, list_path, max_iters=None, resize_size=(1024, 512), crop_size=(512, 1024), mean=(128, 128, 128), scale=False, mirror=True, ignore_label=255, autoaug = False): 18 | self.root = root 19 | self.list_path = list_path 20 | self.crop_size = crop_size 21 | self.scale = scale 22 | self.ignore_label = ignore_label 23 | self.mean = mean 24 | self.is_mirror = mirror 25 | self.resize_size = resize_size 26 | self.autoaug = autoaug 27 | self.h = crop_size[0] 28 | self.w = crop_size[1] 29 | # self.mean_bgr = np.array([104.00698793, 116.66876762, 122.67891434]) 30 | self.img_ids = [i_id.strip() for i_id in open(list_path)] 31 | if not max_iters==None: 32 | self.img_ids = self.img_ids * int(np.ceil(float(max_iters) / len(self.img_ids))) 33 | self.files = [] 34 | 35 | self.id_to_trainid = {0:0, 1:1, 2:2, 3:3, 4:4, 5:5, 6:6, 7:7, 8:8, 9:9, 10:10, 11:11, 12:12} 36 | # for split in ["train", "trainval", "val"]: 37 | for name in self.img_ids: 38 | img_file = osp.join(self.root, "JPEGImages/%s" % name) 39 | label_file = osp.join(self.root, "SegmentationClass/%s" % name.replace('jpg','png')) 40 | self.files.append({ 41 | "img": img_file, 42 | "label": label_file, 43 | "name": name 44 | }) 45 | 46 | def __len__(self): 47 | return len(self.files) 48 | 49 | 50 | def __getitem__(self, index): 51 | datafiles = self.files[index] 52 | 53 | image = Image.open(datafiles["img"]).convert('RGB') 54 | #label = Image.open(datafiles["label"]) 55 | label = np.asarray(imageio.imread(datafiles["label"], format='PNG-FI')) # uint16 56 | label = Image.fromarray(label) 57 | name = datafiles["name"] 58 | 59 | # resize 60 | if self.scale: 61 | random_scale = 0.8 + random.random()*0.4 # 0.8 - 1.2 62 | image = image.resize( ( round(self.resize_size[0] * random_scale), round(self.resize_size[1] * random_scale)) , Image.BICUBIC) 63 | label = label.resize( ( round(self.resize_size[0] * random_scale), round(self.resize_size[1] * random_scale)) , Image.NEAREST) 64 | else: 65 | image = image.resize( ( self.resize_size[0], self.resize_size[1] ) , Image.BICUBIC) 66 | label = label.resize( ( self.resize_size[0], self.resize_size[1] ) , Image.NEAREST) 67 | 68 | if self.autoaug: 69 | policy = ImageNetPolicy() 70 | image = policy(image) 71 | 72 | image = np.asarray(image, np.float32) 73 | label = np.asarray(label, np.uint8) 74 | 75 | # re-assign labels to match the format of Cityscapes 76 | label_copy = 255 * np.ones(label.shape, dtype=np.uint8) 77 | for k, v in list(self.id_to_trainid.items()): 78 | label_copy[label == k] = v 79 | 80 | size = image.shape 81 | image = image[:, :, ::-1] # change to BGR 82 | image -= self.mean 83 | image = image.transpose((2, 0, 1)) 84 | print(image.shape, label.shape) 85 | for i in range(10): #find hard samples 86 | x1 = random.randint(0, image.shape[1] - self.h) 87 | y1 = random.randint(0, image.shape[2] - self.w) 88 | tmp_label_copy = label_copy[x1:x1+self.h, y1:y1+self.w] 89 | tmp_image = image[:, x1:x1+self.h, y1:y1+self.w] 90 | u = np.unique(tmp_label_copy) 91 | if len(u) > 10: 92 | break 93 | else: 94 | continue 95 | #print('GTA5: Too young too naive for %d times!'%i) 96 | 97 | image = tmp_image 98 | label_copy = tmp_label_copy 99 | 100 | if self.is_mirror and random.random() < 0.5: 101 | image = np.flip(image, axis = 2) 102 | label_copy = np.flip(label_copy, axis = 1) 103 | 104 | return image.copy(), label_copy.copy(), np.array(size), name 105 | 106 | 107 | if __name__ == '__main__': 108 | dst = DroneDataSet('./data/drone/', './dataset/drone_list/train.txt', mean=(0,0,0), autoaug=True) 109 | trainloader = data.DataLoader(dst, batch_size=4) 110 | for i, data in enumerate(trainloader): 111 | imgs, _, _, _ = data 112 | if i == 0: 113 | img = torchvision.utils.make_grid(imgs).numpy() 114 | img = np.transpose(img, (1, 2, 0)) 115 | img = img[:, :, ::-1] 116 | img = Image.fromarray(np.uint8(img) ) 117 | img.save('Drone_Demo.jpg') 118 | break 119 | -------------------------------------------------------------------------------- /dataset/gta5_dataset.py: -------------------------------------------------------------------------------- 1 | import os 2 | import os.path as osp 3 | import numpy as np 4 | import random 5 | import matplotlib.pyplot as plt 6 | import collections 7 | import torch 8 | import torchvision 9 | from torch.utils import data 10 | from PIL import Image, ImageFile 11 | from dataset.autoaugment import ImageNetPolicy 12 | 13 | ImageFile.LOAD_TRUNCATED_IMAGES = True 14 | 15 | 16 | class GTA5DataSet(data.Dataset): 17 | def __init__(self, root, list_path, max_iters=None, resize_size=(1024, 512), crop_size=(512, 1024), mean=(128, 128, 128), scale=False, mirror=True, ignore_label=255, autoaug = False): 18 | self.root = root 19 | self.list_path = list_path 20 | self.crop_size = crop_size 21 | self.scale = scale 22 | self.ignore_label = ignore_label 23 | self.mean = mean 24 | self.is_mirror = mirror 25 | self.resize_size = resize_size 26 | self.autoaug = autoaug 27 | self.h = crop_size[0] 28 | self.w = crop_size[1] 29 | # self.mean_bgr = np.array([104.00698793, 116.66876762, 122.67891434]) 30 | self.img_ids = [i_id.strip() for i_id in open(list_path)] 31 | if not max_iters==None: 32 | self.img_ids = self.img_ids * int(np.ceil(float(max_iters) / len(self.img_ids))) 33 | self.files = [] 34 | 35 | self.id_to_trainid = {7: 0, 8: 1, 11: 2, 12: 3, 13: 4, 17: 5, 36 | 19: 6, 20: 7, 21: 8, 22: 9, 23: 10, 24: 11, 25: 12, 37 | 26: 13, 27: 14, 28: 15, 31: 16, 32: 17, 33: 18} 38 | 39 | # for split in ["train", "trainval", "val"]: 40 | for name in self.img_ids: 41 | img_file = osp.join(self.root, "images/%s" % name) 42 | label_file = osp.join(self.root, "labels/%s" % name) 43 | self.files.append({ 44 | "img": img_file, 45 | "label": label_file, 46 | "name": name 47 | }) 48 | 49 | def __len__(self): 50 | return len(self.files) 51 | 52 | 53 | def __getitem__(self, index): 54 | datafiles = self.files[index] 55 | 56 | image = Image.open(datafiles["img"]).convert('RGB') 57 | label = Image.open(datafiles["label"]) 58 | name = datafiles["name"] 59 | 60 | # resize 61 | if self.scale: 62 | random_scale = 0.8 + random.random()*0.4 # 0.8 - 1.2 63 | image = image.resize( ( round(self.resize_size[0] * random_scale), round(self.resize_size[1] * random_scale)) , Image.BICUBIC) 64 | label = label.resize( ( round(self.resize_size[0] * random_scale), round(self.resize_size[1] * random_scale)) , Image.NEAREST) 65 | else: 66 | image = image.resize( ( self.resize_size[0], self.resize_size[1] ) , Image.BICUBIC) 67 | label = label.resize( ( self.resize_size[0], self.resize_size[1] ) , Image.NEAREST) 68 | 69 | if self.autoaug: 70 | policy = ImageNetPolicy() 71 | image = policy(image) 72 | 73 | image = np.asarray(image, np.float32) 74 | label = np.asarray(label, np.uint8) 75 | 76 | # re-assign labels to match the format of Cityscapes 77 | label_copy = 255 * np.ones(label.shape, dtype=np.uint8) 78 | for k, v in list(self.id_to_trainid.items()): 79 | label_copy[label == k] = v 80 | 81 | size = image.shape 82 | image = image[:, :, ::-1] # change to BGR 83 | image -= self.mean 84 | image = image.transpose((2, 0, 1)) 85 | #print(image.shape, label.shape) 86 | for i in range(10): #find hard samples 87 | x1 = random.randint(0, image.shape[1] - self.h) 88 | y1 = random.randint(0, image.shape[2] - self.w) 89 | tmp_label_copy = label_copy[x1:x1+self.h, y1:y1+self.w] 90 | tmp_image = image[:, x1:x1+self.h, y1:y1+self.w] 91 | u = np.unique(tmp_label_copy) 92 | if len(u) > 10: 93 | break 94 | #else: 95 | # print('GTA5: Too young too naive for %d times!'%i) 96 | 97 | image = tmp_image 98 | label_copy = tmp_label_copy 99 | 100 | if self.is_mirror and random.random() < 0.5: 101 | image = np.flip(image, axis = 2) 102 | label_copy = np.flip(label_copy, axis = 1) 103 | 104 | return image.copy(), label_copy.copy(), np.array(size), name 105 | 106 | 107 | if __name__ == '__main__': 108 | dst = GTA5DataSet('./data/GTA5/', './dataset/gta5_list/train.txt', mean=(0,0,0), autoaug=True) 109 | trainloader = data.DataLoader(dst, batch_size=4) 110 | for i, data in enumerate(trainloader): 111 | imgs, _, _, _ = data 112 | if i == 0: 113 | img = torchvision.utils.make_grid(imgs).numpy() 114 | img = np.transpose(img, (1, 2, 0)) 115 | img = img[:, :, ::-1] 116 | img = Image.fromarray(np.uint8(img) ) 117 | img.save('GTA5_Demo.jpg') 118 | break 119 | -------------------------------------------------------------------------------- /dataset/gta5_list/gta5_short.py: -------------------------------------------------------------------------------- 1 | fp = open('train_short.txt', 'w') 2 | index = 0 3 | with open('train.txt') as f: 4 | for line in f: 5 | index = index + 1 6 | if index%100 == 0: 7 | fp.write(line) 8 | -------------------------------------------------------------------------------- /dataset/gta5_list/info.json: -------------------------------------------------------------------------------- 1 | { 2 | "classes":19, 3 | "label2train":[ 4 | [0, 255], 5 | [1, 255], 6 | [2, 255], 7 | [3, 255], 8 | [4, 255], 9 | [5, 255], 10 | [6, 255], 11 | [7, 0], 12 | [8, 1], 13 | [9, 255], 14 | [10, 255], 15 | [11, 2], 16 | [12, 3], 17 | [13, 4], 18 | [14, 255], 19 | [15, 255], 20 | [16, 255], 21 | [17, 5], 22 | [18, 255], 23 | [19, 6], 24 | [20, 7], 25 | [21, 8], 26 | [22, 9], 27 | [23, 10], 28 | [24, 11], 29 | [25, 12], 30 | [26, 13], 31 | [27, 14], 32 | [28, 15], 33 | [29, 255], 34 | [30, 255], 35 | [31, 16], 36 | [32, 17], 37 | [33, 18], 38 | [-1, 255]], 39 | "label":[ 40 | "road", 41 | "sidewalk", 42 | "building", 43 | "wall", 44 | "fence", 45 | "pole", 46 | "light", 47 | "sign", 48 | "vegetation", 49 | "terrain", 50 | "sky", 51 | "person", 52 | "rider", 53 | "car", 54 | "truck", 55 | "bus", 56 | "train", 57 | "motocycle", 58 | "bicycle"], 59 | "palette":[ 60 | [128,64,128], 61 | [244,35,232], 62 | [70,70,70], 63 | [102,102,156], 64 | [190,153,153], 65 | [153,153,153], 66 | [250,170,30], 67 | [220,220,0], 68 | [107,142,35], 69 | [152,251,152], 70 | [70,130,180], 71 | [220,20,60], 72 | [255,0,0], 73 | [0,0,142], 74 | [0,0,70], 75 | [0,60,100], 76 | [0,80,100], 77 | [0,0,230], 78 | [119,11,32], 79 | [0,0,0]], 80 | "mean":[ 81 | 73.158359210711552, 82 | 82.908917542625858, 83 | 72.392398761941593], 84 | "std":[ 85 | 47.675755341814678, 86 | 48.494214368814916, 87 | 47.736546325441594] 88 | } 89 | -------------------------------------------------------------------------------- /dataset/gta5_list/train_short.txt: -------------------------------------------------------------------------------- 1 | 00100.png 2 | 00200.png 3 | 00300.png 4 | 00400.png 5 | 00500.png 6 | 00600.png 7 | 00700.png 8 | 00800.png 9 | 00900.png 10 | 01000.png 11 | 01100.png 12 | 01200.png 13 | 01300.png 14 | 01400.png 15 | 01500.png 16 | 01600.png 17 | 01700.png 18 | 01800.png 19 | 01900.png 20 | 02000.png 21 | 02100.png 22 | 02200.png 23 | 02300.png 24 | 02400.png 25 | 02500.png 26 | 02600.png 27 | 02700.png 28 | 02800.png 29 | 02900.png 30 | 03000.png 31 | 03100.png 32 | 03200.png 33 | 03300.png 34 | 03400.png 35 | 03500.png 36 | 03600.png 37 | 03700.png 38 | 03800.png 39 | 03900.png 40 | 04000.png 41 | 04100.png 42 | 04200.png 43 | 04300.png 44 | 04400.png 45 | 04500.png 46 | 04600.png 47 | 04700.png 48 | 04800.png 49 | 04900.png 50 | 05000.png 51 | 05100.png 52 | 05200.png 53 | 05300.png 54 | 05400.png 55 | 05500.png 56 | 05600.png 57 | 05700.png 58 | 05800.png 59 | 05900.png 60 | 06000.png 61 | 06100.png 62 | 06200.png 63 | 06300.png 64 | 06400.png 65 | 06500.png 66 | 06600.png 67 | 06700.png 68 | 06800.png 69 | 06900.png 70 | 07000.png 71 | 07100.png 72 | 07200.png 73 | 07300.png 74 | 07400.png 75 | 07500.png 76 | 07600.png 77 | 07700.png 78 | 07800.png 79 | 07900.png 80 | 08000.png 81 | 08100.png 82 | 08200.png 83 | 08300.png 84 | 08400.png 85 | 08500.png 86 | 08600.png 87 | 08700.png 88 | 08800.png 89 | 08900.png 90 | 09000.png 91 | 09100.png 92 | 09200.png 93 | 09300.png 94 | 09400.png 95 | 09500.png 96 | 09600.png 97 | 09700.png 98 | 09800.png 99 | 09900.png 100 | 10000.png 101 | 10100.png 102 | 10200.png 103 | 10300.png 104 | 10400.png 105 | 10500.png 106 | 10600.png 107 | 10700.png 108 | 10800.png 109 | 10900.png 110 | 11000.png 111 | 11100.png 112 | 11200.png 113 | 11300.png 114 | 11400.png 115 | 11500.png 116 | 11600.png 117 | 11700.png 118 | 11800.png 119 | 11900.png 120 | 12000.png 121 | 12100.png 122 | 12200.png 123 | 12300.png 124 | 12400.png 125 | 12500.png 126 | 12600.png 127 | 12700.png 128 | 12800.png 129 | 12900.png 130 | 13000.png 131 | 13100.png 132 | 13200.png 133 | 13300.png 134 | 13400.png 135 | 13500.png 136 | 13600.png 137 | 13700.png 138 | 13800.png 139 | 13900.png 140 | 14000.png 141 | 14100.png 142 | 14200.png 143 | 14300.png 144 | 14400.png 145 | 14500.png 146 | 14600.png 147 | 14700.png 148 | 14800.png 149 | 14900.png 150 | 15000.png 151 | 15100.png 152 | 15200.png 153 | 15300.png 154 | 15400.png 155 | 15500.png 156 | 15600.png 157 | 15700.png 158 | 15800.png 159 | 15900.png 160 | 16000.png 161 | 16100.png 162 | 16200.png 163 | 16300.png 164 | 16400.png 165 | 16500.png 166 | 16600.png 167 | 16700.png 168 | 16800.png 169 | 16900.png 170 | 17000.png 171 | 17100.png 172 | 17200.png 173 | 17300.png 174 | 17400.png 175 | 17500.png 176 | 17600.png 177 | 17700.png 178 | 17800.png 179 | 17900.png 180 | 18000.png 181 | 18100.png 182 | 18200.png 183 | 18300.png 184 | 18400.png 185 | 18500.png 186 | 18600.png 187 | 18700.png 188 | 18800.png 189 | 18900.png 190 | 19000.png 191 | 19100.png 192 | 19200.png 193 | 19300.png 194 | 19400.png 195 | 19500.png 196 | 19600.png 197 | 19700.png 198 | 19800.png 199 | 19900.png 200 | 20000.png 201 | 20100.png 202 | 20200.png 203 | 20300.png 204 | 20400.png 205 | 20500.png 206 | 20600.png 207 | 20700.png 208 | 20800.png 209 | 20900.png 210 | 21000.png 211 | 21100.png 212 | 21200.png 213 | 21300.png 214 | 21400.png 215 | 21500.png 216 | 21600.png 217 | 21700.png 218 | 21800.png 219 | 21900.png 220 | 22000.png 221 | 22100.png 222 | 22200.png 223 | 22300.png 224 | 22400.png 225 | 22500.png 226 | 22600.png 227 | 22700.png 228 | 22800.png 229 | 22900.png 230 | 23000.png 231 | 23100.png 232 | 23200.png 233 | 23300.png 234 | 23400.png 235 | 23500.png 236 | 23600.png 237 | 23700.png 238 | 23800.png 239 | 23900.png 240 | 24000.png 241 | 24100.png 242 | 24200.png 243 | 24300.png 244 | 24400.png 245 | 24500.png 246 | 24600.png 247 | 24700.png 248 | 24800.png 249 | 24900.png 250 | -------------------------------------------------------------------------------- /dataset/robot_dataset.py: -------------------------------------------------------------------------------- 1 | import os 2 | import os.path as osp 3 | import numpy as np 4 | import random 5 | import matplotlib 6 | matplotlib.use('agg') 7 | import matplotlib.pyplot as plt 8 | import collections 9 | import torch 10 | import torchvision 11 | from torch.utils import data 12 | from PIL import Image,ImageFile 13 | from dataset.autoaugment import ImageNetPolicy 14 | import time 15 | ImageFile.LOAD_TRUNCATED_IMAGES = True 16 | 17 | 18 | class robotDataSet(data.Dataset): 19 | def __init__(self, root, list_path, max_iters=None, resize_size=(1024, 512), crop_size=(512, 1024), mean=(128, 128, 128), scale=False, mirror=True, ignore_label=255, set='val', autoaug=False): 20 | self.root = root 21 | self.list_path = list_path 22 | self.crop_size = crop_size 23 | self.scale = scale 24 | self.ignore_label = ignore_label 25 | self.mean = mean 26 | self.is_mirror = mirror 27 | self.resize_size = resize_size 28 | self.autoaug = autoaug 29 | self.h = crop_size[0] 30 | self.w = crop_size[1] 31 | # self.mean_bgr = np.array([104.00698793, 116.66876762, 122.67891434]) 32 | self.img_ids = [i_id.strip() for i_id in open(list_path)] 33 | if not max_iters==None: 34 | self.img_ids = self.img_ids * int(np.ceil(float(max_iters) / len(self.img_ids))) 35 | self.files = [] 36 | self.set = set 37 | # for split in ["train", "trainval", "val"]: 38 | 39 | #https://github.com/mcordts/cityscapesScripts/blob/master/cityscapesscripts/helpers/labels.py 40 | ''' 41 | 0 sky; 1 person; 2 two-wheel; 3 automobile; 4 sign 42 | 5 light 6 building 7 sidewalk 8 road 43 | ''' 44 | self.id_to_trainid = {1:0, 2:1, 3:2, 4:3, 5:4, 6:5, 7:6, 10:7, 11:8, 12:8, 13:8, 14:8, 17:8} 45 | 46 | for name in self.img_ids: 47 | img_file = osp.join(self.root, "%s/%s" % (self.set, name)) 48 | if set == 'val': 49 | label_file = osp.join(self.root, "anno/%s" %name ) 50 | else: 51 | label_file = '' 52 | self.files.append({ 53 | "img": img_file, 54 | "label": label_file, 55 | "name": name 56 | }) 57 | 58 | def __len__(self): 59 | return len(self.files) 60 | 61 | def __getitem__(self, index): 62 | #tt = time.time() 63 | datafiles = self.files[index] 64 | name = datafiles["name"] 65 | 66 | image = Image.open(datafiles["img"]).convert('RGB') 67 | image= image.resize(self.resize_size, Image.BICUBIC) 68 | 69 | if self.set == 'val': 70 | label = Image.open(datafiles["label"]) 71 | label = label.resize(self.resize_size, Image.NEAREST) 72 | label = np.asarray(label, np.uint8) 73 | # re-assign labels to match the format of Cityscapes 74 | label_copy = 255 * np.ones(label.shape, dtype=np.uint8) 75 | for k, v in list(self.id_to_trainid.items()): 76 | label_copy[label == k] = v 77 | 78 | if self.autoaug: 79 | policy = ImageNetPolicy() 80 | image = policy(image) 81 | image = np.asarray(image, np.float32) 82 | size = image.shape 83 | image = image[:, :, ::-1] # change to BGR 84 | image -= self.mean 85 | image = image.transpose((2, 0, 1)) 86 | x1 = random.randint(0, image.shape[1] - self.h) 87 | y1 = random.randint(0, image.shape[2] - self.w) 88 | image = image[:, x1:x1+self.h, y1:y1+self.w] 89 | 90 | if self.set == 'val': 91 | label_copy = label_copy[x1:x1+self.h, y1:y1+self.w] 92 | else: 93 | label_copy = np.ones(image.shape[1:3])*255 94 | 95 | if self.is_mirror and random.random() < 0.5: 96 | image = np.flip(image, axis = 2) 97 | if self.set == 'val': 98 | label_copy = np.flip(label_copy, axis = 1) 99 | 100 | 101 | return image.copy(), label_copy.copy(), np.array(size), name 102 | 103 | 104 | if __name__ == '__main__': 105 | dst = robotDataSet('./data/Oxford_Robot_ICCV19', './dataset/robot_list/train.txt', mean=(0,0,0), set = 'train') 106 | trainloader = data.DataLoader(dst, batch_size=4) 107 | for i, data in enumerate(trainloader): 108 | imgs, _, _, _ = data 109 | if i == 0: 110 | img = torchvision.utils.make_grid(imgs).numpy() 111 | img = np.transpose(img, (1, 2, 0)) 112 | img = img[:, :, ::-1] 113 | img = Image.fromarray(np.uint8(img) ) 114 | img.save('Robot_Demo.jpg') 115 | break 116 | -------------------------------------------------------------------------------- /dataset/robot_list/info.json: -------------------------------------------------------------------------------- 1 | { 2 | "classes":9, 3 | "label2train":[ 4 | [0, 255], 5 | [1, 0], 6 | [2, 1], 7 | [3, 2], 8 | [4, 3], 9 | [5, 4], 10 | [6, 5], 11 | [7, 6], 12 | [8, 255], 13 | [9, 255], 14 | [10, 7], 15 | [11, 8], 16 | [12, 8], 17 | [13, 8], 18 | [14, 8], 19 | [15, 255], 20 | [16, 255], 21 | [17, 8], 22 | [18, 255], 23 | [19, 255], 24 | [20, 255], 25 | [21, 255], 26 | [22, 255], 27 | [23, 255], 28 | [24, 255], 29 | [25, 255], 30 | [26, 255], 31 | [27, 255], 32 | [28, 255], 33 | [29, 255], 34 | [30, 255], 35 | [31, 255], 36 | [32, 255], 37 | [33, 255], 38 | [-1, 255]], 39 | "label":[ 40 | "sky", 41 | "person", 42 | "two-wheel", 43 | "automobile", 44 | "sign", 45 | "light", 46 | "building", 47 | "sidewalk", 48 | "road"], 49 | "palette":[ 50 | [70,130,180], 51 | [220,20,60], 52 | [119,11,32], 53 | [0,0,142], 54 | [220,220,0], 55 | [250,170,30], 56 | [70,70,70], 57 | [244,35,232], 58 | [128,64,128], 59 | [0,0,0]], 60 | "mean":[ 61 | 73.158359210711552, 62 | 82.908917542625858, 63 | 72.392398761941593], 64 | "std":[ 65 | 47.675755341814678, 66 | 48.494214368814916, 67 | 47.736546325441594] 68 | } 69 | -------------------------------------------------------------------------------- /dataset/robot_list/label.txt: -------------------------------------------------------------------------------- 1 | 2014-11-25-09-18-32_01_000038.png 2 | 2014-11-25-09-18-32_01_000039.png 3 | 2014-11-25-09-18-32_01_000040.png 4 | 2014-11-25-09-18-32_01_000041.png 5 | 2014-11-25-09-18-32_01_000042.png 6 | 2014-11-25-09-18-32_01_000043.png 7 | 2014-11-25-09-18-32_01_000044.png 8 | 2014-11-25-09-18-32_01_000045.png 9 | 2014-11-25-09-18-32_01_000046.png 10 | 2014-11-25-09-18-32_01_000047.png 11 | 2014-11-25-09-18-32_01_000048.png 12 | 2014-11-25-09-18-32_01_000049.png 13 | 2014-11-25-09-18-32_01_000050.png 14 | 2014-11-25-09-18-32_01_000051.png 15 | 2014-11-25-09-18-32_01_000052.png 16 | 2014-11-25-09-18-32_01_000053.png 17 | 2014-11-25-09-18-32_01_000054.png 18 | 2014-11-25-09-18-32_01_000055.png 19 | 2014-11-25-09-18-32_01_000056.png 20 | 2014-11-25-09-18-32_01_000057.png 21 | 2014-11-25-09-18-32_01_000058.png 22 | 2014-11-25-09-18-32_01_000059.png 23 | 2014-11-25-09-18-32_01_000060.png 24 | 2014-11-25-09-18-32_01_000061.png 25 | 2014-11-25-09-18-32_01_000062.png 26 | 2014-11-25-09-18-32_01_000063.png 27 | 2014-11-25-09-18-32_01_000064.png 28 | 2014-11-25-09-18-32_01_000065.png 29 | 2014-11-25-09-18-32_01_000066.png 30 | 2014-11-25-09-18-32_01_000067.png 31 | 2014-11-25-09-18-32_01_000068.png 32 | 2014-11-25-09-18-32_01_000069.png 33 | 2014-11-25-09-18-32_01_000070.png 34 | 2014-11-25-09-18-32_01_000071.png 35 | 2014-11-25-09-18-32_01_000072.png 36 | 2014-11-25-09-18-32_01_000073.png 37 | 2014-11-25-09-18-32_01_000074.png 38 | 2014-11-25-09-18-32_01_000075.png 39 | 2014-11-25-09-18-32_01_000076.png 40 | 2014-11-25-09-18-32_01_000077.png 41 | 2014-11-25-09-18-32_01_000078.png 42 | 2014-11-25-09-18-32_01_000079.png 43 | 2014-11-25-09-18-32_01_000080.png 44 | 2014-11-25-09-18-32_01_000081.png 45 | 2014-11-25-09-18-32_01_000082.png 46 | 2014-11-25-09-18-32_01_000083.png 47 | 2014-11-25-09-18-32_01_000084.png 48 | 2014-11-25-09-18-32_01_000085.png 49 | 2014-11-25-09-18-32_01_000086.png 50 | 2014-11-25-09-18-32_01_000087.png 51 | 2014-11-25-09-18-32_01_000088.png 52 | 2014-11-25-09-18-32_01_000089.png 53 | 2014-11-25-09-18-32_01_000090.png 54 | 2014-11-25-09-18-32_01_000128.png 55 | 2014-11-25-09-18-32_01_000129.png 56 | 2014-11-25-09-18-32_01_000130.png 57 | 2014-11-25-09-18-32_01_000131.png 58 | 2014-11-25-09-18-32_01_000132.png 59 | 2014-11-25-09-18-32_01_000133.png 60 | 2014-11-25-09-18-32_01_000134.png 61 | 2014-11-25-09-18-32_01_000135.png 62 | 2014-11-25-09-18-32_01_000136.png 63 | 2014-11-25-09-18-32_01_000137.png 64 | 2014-11-25-09-18-32_01_000138.png 65 | 2014-11-25-09-18-32_01_000139.png 66 | 2014-11-25-09-18-32_01_000140.png 67 | 2014-11-25-09-18-32_01_000141.png 68 | 2014-11-25-09-18-32_01_000142.png 69 | 2014-11-25-09-18-32_01_000143.png 70 | 2014-11-25-09-18-32_01_000144.png 71 | 2014-11-25-09-18-32_01_000145.png 72 | 2014-11-25-09-18-32_01_000146.png 73 | 2014-11-25-09-18-32_01_000147.png 74 | 2014-11-25-09-18-32_01_000148.png 75 | 2014-11-25-09-18-32_01_000149.png 76 | 2014-11-25-09-18-32_01_000150.png 77 | 2014-11-25-09-18-32_01_000151.png 78 | 2014-11-25-09-18-32_01_000152.png 79 | 2014-11-25-09-18-32_01_000153.png 80 | 2014-11-25-09-18-32_01_000154.png 81 | 2014-11-25-09-18-32_01_000155.png 82 | 2014-11-25-09-18-32_01_000156.png 83 | 2014-11-25-09-18-32_01_000157.png 84 | 2014-11-25-09-18-32_01_000158.png 85 | 2014-11-25-09-18-32_01_000159.png 86 | 2014-11-25-09-18-32_01_000160.png 87 | 2014-11-25-09-18-32_01_000161.png 88 | 2014-11-25-09-18-32_01_000162.png 89 | 2014-11-25-09-18-32_01_000163.png 90 | 2014-11-25-09-18-32_01_000164.png 91 | 2014-11-25-09-18-32_01_000165.png 92 | 2014-11-25-09-18-32_01_000166.png 93 | 2014-11-25-09-18-32_01_000167.png 94 | 2014-11-25-09-18-32_01_000168.png 95 | 2014-11-25-09-18-32_01_000169.png 96 | 2014-11-25-09-18-32_01_000170.png 97 | 2014-11-25-09-18-32_01_000171.png 98 | 2014-11-25-09-18-32_01_000172.png 99 | 2014-11-25-09-18-32_01_000173.png 100 | 2014-11-25-09-18-32_01_000174.png 101 | 2014-11-25-09-18-32_01_000175.png 102 | 2014-11-25-09-18-32_01_000176.png 103 | 2014-11-25-09-18-32_01_000177.png 104 | 2014-11-25-09-18-32_01_000178.png 105 | 2014-11-25-09-18-32_01_000179.png 106 | 2014-11-25-09-18-32_01_000180.png 107 | 2014-11-25-09-18-32_01_000181.png 108 | 2014-11-25-09-18-32_01_000182.png 109 | 2014-11-25-09-18-32_01_000183.png 110 | 2014-11-25-09-18-32_01_000184.png 111 | 2014-11-25-09-18-32_01_000185.png 112 | 2014-11-25-09-18-32_01_000186.png 113 | 2014-11-25-09-18-32_01_000187.png 114 | 2014-11-25-09-18-32_01_000188.png 115 | 2014-11-25-09-18-32_01_000189.png 116 | 2014-11-25-09-18-32_01_000190.png 117 | 2014-11-25-09-18-32_01_000191.png 118 | 2014-11-25-09-18-32_01_000192.png 119 | 2014-11-25-09-18-32_01_000193.png 120 | 2014-11-25-09-18-32_01_000194.png 121 | 2014-11-25-09-18-32_01_000195.png 122 | 2014-11-25-09-18-32_01_000196.png 123 | 2014-11-25-09-18-32_01_000197.png 124 | 2014-11-25-09-18-32_01_000198.png 125 | 2014-11-25-09-18-32_01_000199.png 126 | 2015-10-29-12-18-17_07_000000.png 127 | 2015-10-29-12-18-17_07_000001.png 128 | 2015-10-29-12-18-17_07_000002.png 129 | 2015-10-29-12-18-17_07_000003.png 130 | 2015-10-29-12-18-17_07_000004.png 131 | 2015-10-29-12-18-17_07_000005.png 132 | 2015-10-29-12-18-17_07_000006.png 133 | 2015-10-29-12-18-17_07_000007.png 134 | 2015-10-29-12-18-17_07_000008.png 135 | 2015-10-29-12-18-17_07_000009.png 136 | 2015-10-29-12-18-17_07_000010.png 137 | 2015-10-29-12-18-17_07_000011.png 138 | 2015-10-29-12-18-17_07_000012.png 139 | 2015-10-29-12-18-17_07_000013.png 140 | 2015-10-29-12-18-17_07_000014.png 141 | 2015-10-29-12-18-17_07_000015.png 142 | 2015-10-29-12-18-17_07_000016.png 143 | 2015-10-29-12-18-17_07_000017.png 144 | 2015-10-29-12-18-17_07_000018.png 145 | 2015-10-29-12-18-17_07_000019.png 146 | 2015-10-29-12-18-17_07_000020.png 147 | 2015-10-29-12-18-17_07_000021.png 148 | 2015-10-29-12-18-17_07_000022.png 149 | 2015-10-29-12-18-17_07_000023.png 150 | 2015-10-29-12-18-17_07_000024.png 151 | 2015-10-29-12-18-17_07_000025.png 152 | 2015-10-29-12-18-17_07_000026.png 153 | 2015-10-29-12-18-17_07_000027.png 154 | 2015-10-29-12-18-17_07_000028.png 155 | 2015-10-29-12-18-17_07_000029.png 156 | 2015-10-29-12-18-17_07_000030.png 157 | 2015-10-29-12-18-17_07_000031.png 158 | 2015-10-29-12-18-17_07_000032.png 159 | 2015-10-29-12-18-17_07_000033.png 160 | 2015-10-29-12-18-17_07_000034.png 161 | 2015-10-29-12-18-17_07_000035.png 162 | 2015-10-29-12-18-17_07_000036.png 163 | 2015-10-29-12-18-17_07_000037.png 164 | 2015-10-29-12-18-17_07_000038.png 165 | 2015-10-29-12-18-17_07_000039.png 166 | 2015-10-29-12-18-17_07_000040.png 167 | 2015-10-29-12-18-17_07_000041.png 168 | 2015-11-06-11-21-12_05_000000.png 169 | 2015-11-06-11-21-12_05_000001.png 170 | 2015-11-06-11-21-12_05_000002.png 171 | 2015-11-06-11-21-12_05_000003.png 172 | 2015-11-06-11-21-12_05_000004.png 173 | 2015-11-06-11-21-12_05_000005.png 174 | 2015-11-06-11-21-12_05_000006.png 175 | 2015-11-06-11-21-12_05_000007.png 176 | 2015-11-06-11-21-12_05_000008.png 177 | 2015-11-06-11-21-12_05_000009.png 178 | 2015-11-06-11-21-12_05_000010.png 179 | 2015-11-06-11-21-12_05_000011.png 180 | 2015-11-06-11-21-12_05_000012.png 181 | 2015-11-06-11-21-12_05_000013.png 182 | 2015-11-06-11-21-12_05_000014.png 183 | 2015-11-06-11-21-12_05_000015.png 184 | 2015-11-06-11-21-12_05_000016.png 185 | 2015-11-06-11-21-12_05_000017.png 186 | 2015-11-06-11-21-12_05_000018.png 187 | 2015-11-06-11-21-12_05_000019.png 188 | 2015-11-06-11-21-12_05_000020.png 189 | 2015-11-06-11-21-12_05_000021.png 190 | 2015-11-06-11-21-12_05_000022.png 191 | 2015-11-06-11-21-12_05_000023.png 192 | 2015-11-06-11-21-12_05_000024.png 193 | 2015-11-06-11-21-12_05_000025.png 194 | 2015-11-06-11-21-12_05_000026.png 195 | 2015-11-06-11-21-12_05_000027.png 196 | 2015-11-06-11-21-12_05_000028.png 197 | 2015-11-06-11-21-12_05_000029.png 198 | 2015-11-06-11-21-12_05_000030.png 199 | 2015-11-06-11-21-12_05_000031.png 200 | 2015-11-06-11-21-12_05_000032.png 201 | 2015-11-06-11-21-12_05_000033.png 202 | 2015-11-06-11-21-12_05_000034.png 203 | 2015-11-06-11-21-12_05_000035.png 204 | 2015-11-06-11-21-12_05_000036.png 205 | 2015-11-06-11-21-12_05_000037.png 206 | 2015-11-06-11-21-12_05_000038.png 207 | 2015-11-06-11-21-12_05_000039.png 208 | 2015-11-06-11-21-12_05_000040.png 209 | 2015-11-06-11-21-12_05_000041.png 210 | 2015-11-06-11-21-12_05_000042.png 211 | 2015-11-06-11-21-12_05_000043.png 212 | 2015-11-06-11-21-12_05_000044.png 213 | 2015-11-06-11-21-12_05_000045.png 214 | 2015-11-06-11-21-12_05_000046.png 215 | 2015-11-06-11-21-12_05_000047.png 216 | 2015-11-06-11-21-12_05_000049.png 217 | 2015-11-06-11-21-12_05_000050.png 218 | 2015-11-06-11-21-12_05_000051.png 219 | 2015-11-06-11-21-12_05_000052.png 220 | 2015-11-06-11-21-12_05_000053.png 221 | 2015-11-06-11-21-12_05_000054.png 222 | 2015-11-06-11-21-12_05_000055.png 223 | 2015-11-06-11-21-12_05_000056.png 224 | 2015-11-06-11-21-12_05_000057.png 225 | 2015-11-06-11-21-12_05_000058.png 226 | 2015-11-06-11-21-12_05_000059.png 227 | 2015-11-06-11-21-12_05_000060.png 228 | 2015-11-06-11-21-12_05_000061.png 229 | 2015-11-06-11-21-12_05_000062.png 230 | 2015-11-06-11-21-12_05_000063.png 231 | 2015-11-06-11-21-12_05_000064.png 232 | 2015-11-06-11-21-12_05_000065.png 233 | 2015-11-06-11-21-12_05_000066.png 234 | 2015-11-06-11-21-12_05_000067.png 235 | 2015-11-06-11-21-12_05_000068.png 236 | 2015-11-06-11-21-12_05_000069.png 237 | 2015-11-06-11-21-12_05_000071.png 238 | 2015-11-06-11-21-12_05_000072.png 239 | 2015-11-06-11-21-12_05_000073.png 240 | 2015-11-06-11-21-12_05_000074.png 241 | 2015-11-06-11-21-12_05_000075.png 242 | 2015-11-06-11-21-12_05_000076.png 243 | 2015-11-06-11-21-12_05_000078.png 244 | 2015-11-06-11-21-12_05_000079.png 245 | 2015-11-06-11-21-12_05_000080.png 246 | 2015-11-06-11-21-12_05_000081.png 247 | 2015-11-06-11-21-12_05_000082.png 248 | 2015-11-06-11-21-12_05_000083.png 249 | 2015-11-06-11-21-12_05_000084.png 250 | 2015-11-06-11-21-12_05_000085.png 251 | 2015-11-06-11-21-12_05_000086.png 252 | 2015-11-06-11-21-12_05_000087.png 253 | 2015-11-06-11-21-12_05_000088.png 254 | 2015-11-06-11-21-12_05_000089.png 255 | 2015-11-06-11-21-12_05_000090.png 256 | 2015-11-06-11-21-12_05_000091.png 257 | 2015-11-06-11-21-12_05_000092.png 258 | 2015-11-06-11-21-12_05_000095.png 259 | 2015-11-06-11-21-12_05_000096.png 260 | 2015-11-06-11-21-12_05_000097.png 261 | 2015-11-06-11-21-12_05_000099.png 262 | 2015-11-06-11-21-12_05_000100.png 263 | 2015-11-06-11-21-12_05_000101.png 264 | 2015-11-06-11-21-12_05_000102.png 265 | 2015-11-06-11-21-12_05_000103.png 266 | 2015-11-06-11-21-12_05_000104.png 267 | 2015-11-06-11-21-12_05_000105.png 268 | 2015-11-06-11-21-12_05_000106.png 269 | 2015-11-06-11-21-12_05_000107.png 270 | 2015-11-06-11-21-12_05_000108.png 271 | 2015-11-06-11-21-12_05_000109.png 272 | -------------------------------------------------------------------------------- /dataset/robot_list/val.txt: -------------------------------------------------------------------------------- 1 | 2014-11-25-09-18-32_01_000038.png 2 | 2014-11-25-09-18-32_01_000039.png 3 | 2014-11-25-09-18-32_01_000040.png 4 | 2014-11-25-09-18-32_01_000041.png 5 | 2014-11-25-09-18-32_01_000042.png 6 | 2014-11-25-09-18-32_01_000043.png 7 | 2014-11-25-09-18-32_01_000044.png 8 | 2014-11-25-09-18-32_01_000045.png 9 | 2014-11-25-09-18-32_01_000046.png 10 | 2014-11-25-09-18-32_01_000047.png 11 | 2014-11-25-09-18-32_01_000048.png 12 | 2014-11-25-09-18-32_01_000049.png 13 | 2014-11-25-09-18-32_01_000050.png 14 | 2014-11-25-09-18-32_01_000051.png 15 | 2014-11-25-09-18-32_01_000052.png 16 | 2014-11-25-09-18-32_01_000053.png 17 | 2014-11-25-09-18-32_01_000054.png 18 | 2014-11-25-09-18-32_01_000055.png 19 | 2014-11-25-09-18-32_01_000056.png 20 | 2014-11-25-09-18-32_01_000057.png 21 | 2014-11-25-09-18-32_01_000058.png 22 | 2014-11-25-09-18-32_01_000059.png 23 | 2014-11-25-09-18-32_01_000060.png 24 | 2014-11-25-09-18-32_01_000061.png 25 | 2014-11-25-09-18-32_01_000062.png 26 | 2014-11-25-09-18-32_01_000063.png 27 | 2014-11-25-09-18-32_01_000064.png 28 | 2014-11-25-09-18-32_01_000065.png 29 | 2014-11-25-09-18-32_01_000066.png 30 | 2014-11-25-09-18-32_01_000067.png 31 | 2014-11-25-09-18-32_01_000068.png 32 | 2014-11-25-09-18-32_01_000069.png 33 | 2014-11-25-09-18-32_01_000070.png 34 | 2014-11-25-09-18-32_01_000071.png 35 | 2014-11-25-09-18-32_01_000072.png 36 | 2014-11-25-09-18-32_01_000073.png 37 | 2014-11-25-09-18-32_01_000074.png 38 | 2014-11-25-09-18-32_01_000075.png 39 | 2014-11-25-09-18-32_01_000076.png 40 | 2014-11-25-09-18-32_01_000077.png 41 | 2014-11-25-09-18-32_01_000078.png 42 | 2014-11-25-09-18-32_01_000079.png 43 | 2014-11-25-09-18-32_01_000080.png 44 | 2014-11-25-09-18-32_01_000081.png 45 | 2014-11-25-09-18-32_01_000082.png 46 | 2014-11-25-09-18-32_01_000083.png 47 | 2014-11-25-09-18-32_01_000084.png 48 | 2014-11-25-09-18-32_01_000085.png 49 | 2014-11-25-09-18-32_01_000086.png 50 | 2014-11-25-09-18-32_01_000087.png 51 | 2014-11-25-09-18-32_01_000088.png 52 | 2014-11-25-09-18-32_01_000089.png 53 | 2014-11-25-09-18-32_01_000090.png 54 | 2014-11-25-09-18-32_01_000128.png 55 | 2014-11-25-09-18-32_01_000129.png 56 | 2014-11-25-09-18-32_01_000130.png 57 | 2014-11-25-09-18-32_01_000131.png 58 | 2014-11-25-09-18-32_01_000132.png 59 | 2014-11-25-09-18-32_01_000133.png 60 | 2014-11-25-09-18-32_01_000134.png 61 | 2014-11-25-09-18-32_01_000135.png 62 | 2014-11-25-09-18-32_01_000136.png 63 | 2014-11-25-09-18-32_01_000137.png 64 | 2014-11-25-09-18-32_01_000138.png 65 | 2014-11-25-09-18-32_01_000139.png 66 | 2014-11-25-09-18-32_01_000140.png 67 | 2014-11-25-09-18-32_01_000141.png 68 | 2014-11-25-09-18-32_01_000142.png 69 | 2014-11-25-09-18-32_01_000143.png 70 | 2014-11-25-09-18-32_01_000144.png 71 | 2014-11-25-09-18-32_01_000145.png 72 | 2014-11-25-09-18-32_01_000146.png 73 | 2014-11-25-09-18-32_01_000147.png 74 | 2014-11-25-09-18-32_01_000148.png 75 | 2014-11-25-09-18-32_01_000149.png 76 | 2014-11-25-09-18-32_01_000150.png 77 | 2014-11-25-09-18-32_01_000151.png 78 | 2014-11-25-09-18-32_01_000152.png 79 | 2014-11-25-09-18-32_01_000153.png 80 | 2014-11-25-09-18-32_01_000154.png 81 | 2014-11-25-09-18-32_01_000155.png 82 | 2014-11-25-09-18-32_01_000156.png 83 | 2014-11-25-09-18-32_01_000157.png 84 | 2014-11-25-09-18-32_01_000158.png 85 | 2014-11-25-09-18-32_01_000159.png 86 | 2014-11-25-09-18-32_01_000160.png 87 | 2014-11-25-09-18-32_01_000161.png 88 | 2014-11-25-09-18-32_01_000162.png 89 | 2014-11-25-09-18-32_01_000163.png 90 | 2014-11-25-09-18-32_01_000164.png 91 | 2014-11-25-09-18-32_01_000165.png 92 | 2014-11-25-09-18-32_01_000166.png 93 | 2014-11-25-09-18-32_01_000167.png 94 | 2014-11-25-09-18-32_01_000168.png 95 | 2014-11-25-09-18-32_01_000169.png 96 | 2014-11-25-09-18-32_01_000170.png 97 | 2014-11-25-09-18-32_01_000171.png 98 | 2014-11-25-09-18-32_01_000172.png 99 | 2014-11-25-09-18-32_01_000173.png 100 | 2014-11-25-09-18-32_01_000174.png 101 | 2014-11-25-09-18-32_01_000175.png 102 | 2014-11-25-09-18-32_01_000176.png 103 | 2014-11-25-09-18-32_01_000177.png 104 | 2014-11-25-09-18-32_01_000178.png 105 | 2014-11-25-09-18-32_01_000179.png 106 | 2014-11-25-09-18-32_01_000180.png 107 | 2014-11-25-09-18-32_01_000181.png 108 | 2014-11-25-09-18-32_01_000182.png 109 | 2014-11-25-09-18-32_01_000183.png 110 | 2014-11-25-09-18-32_01_000184.png 111 | 2014-11-25-09-18-32_01_000185.png 112 | 2014-11-25-09-18-32_01_000186.png 113 | 2014-11-25-09-18-32_01_000187.png 114 | 2014-11-25-09-18-32_01_000188.png 115 | 2014-11-25-09-18-32_01_000189.png 116 | 2014-11-25-09-18-32_01_000190.png 117 | 2014-11-25-09-18-32_01_000191.png 118 | 2014-11-25-09-18-32_01_000192.png 119 | 2014-11-25-09-18-32_01_000193.png 120 | 2014-11-25-09-18-32_01_000194.png 121 | 2014-11-25-09-18-32_01_000195.png 122 | 2014-11-25-09-18-32_01_000196.png 123 | 2014-11-25-09-18-32_01_000197.png 124 | 2014-11-25-09-18-32_01_000198.png 125 | 2014-11-25-09-18-32_01_000199.png 126 | 2015-10-29-12-18-17_07_000000.png 127 | 2015-10-29-12-18-17_07_000001.png 128 | 2015-10-29-12-18-17_07_000002.png 129 | 2015-10-29-12-18-17_07_000003.png 130 | 2015-10-29-12-18-17_07_000004.png 131 | 2015-10-29-12-18-17_07_000005.png 132 | 2015-10-29-12-18-17_07_000006.png 133 | 2015-10-29-12-18-17_07_000007.png 134 | 2015-10-29-12-18-17_07_000008.png 135 | 2015-10-29-12-18-17_07_000009.png 136 | 2015-10-29-12-18-17_07_000010.png 137 | 2015-10-29-12-18-17_07_000011.png 138 | 2015-10-29-12-18-17_07_000012.png 139 | 2015-10-29-12-18-17_07_000013.png 140 | 2015-10-29-12-18-17_07_000014.png 141 | 2015-10-29-12-18-17_07_000015.png 142 | 2015-10-29-12-18-17_07_000016.png 143 | 2015-10-29-12-18-17_07_000017.png 144 | 2015-10-29-12-18-17_07_000018.png 145 | 2015-10-29-12-18-17_07_000019.png 146 | 2015-10-29-12-18-17_07_000020.png 147 | 2015-10-29-12-18-17_07_000021.png 148 | 2015-10-29-12-18-17_07_000022.png 149 | 2015-10-29-12-18-17_07_000023.png 150 | 2015-10-29-12-18-17_07_000024.png 151 | 2015-10-29-12-18-17_07_000025.png 152 | 2015-10-29-12-18-17_07_000026.png 153 | 2015-10-29-12-18-17_07_000027.png 154 | 2015-10-29-12-18-17_07_000028.png 155 | 2015-10-29-12-18-17_07_000029.png 156 | 2015-10-29-12-18-17_07_000030.png 157 | 2015-10-29-12-18-17_07_000031.png 158 | 2015-10-29-12-18-17_07_000032.png 159 | 2015-10-29-12-18-17_07_000033.png 160 | 2015-10-29-12-18-17_07_000034.png 161 | 2015-10-29-12-18-17_07_000035.png 162 | 2015-10-29-12-18-17_07_000036.png 163 | 2015-10-29-12-18-17_07_000037.png 164 | 2015-10-29-12-18-17_07_000038.png 165 | 2015-10-29-12-18-17_07_000039.png 166 | 2015-10-29-12-18-17_07_000040.png 167 | 2015-10-29-12-18-17_07_000041.png 168 | 2015-11-06-11-21-12_05_000000.png 169 | 2015-11-06-11-21-12_05_000001.png 170 | 2015-11-06-11-21-12_05_000002.png 171 | 2015-11-06-11-21-12_05_000003.png 172 | 2015-11-06-11-21-12_05_000004.png 173 | 2015-11-06-11-21-12_05_000005.png 174 | 2015-11-06-11-21-12_05_000006.png 175 | 2015-11-06-11-21-12_05_000007.png 176 | 2015-11-06-11-21-12_05_000008.png 177 | 2015-11-06-11-21-12_05_000009.png 178 | 2015-11-06-11-21-12_05_000010.png 179 | 2015-11-06-11-21-12_05_000011.png 180 | 2015-11-06-11-21-12_05_000012.png 181 | 2015-11-06-11-21-12_05_000013.png 182 | 2015-11-06-11-21-12_05_000014.png 183 | 2015-11-06-11-21-12_05_000015.png 184 | 2015-11-06-11-21-12_05_000016.png 185 | 2015-11-06-11-21-12_05_000017.png 186 | 2015-11-06-11-21-12_05_000018.png 187 | 2015-11-06-11-21-12_05_000019.png 188 | 2015-11-06-11-21-12_05_000020.png 189 | 2015-11-06-11-21-12_05_000021.png 190 | 2015-11-06-11-21-12_05_000022.png 191 | 2015-11-06-11-21-12_05_000023.png 192 | 2015-11-06-11-21-12_05_000024.png 193 | 2015-11-06-11-21-12_05_000025.png 194 | 2015-11-06-11-21-12_05_000026.png 195 | 2015-11-06-11-21-12_05_000027.png 196 | 2015-11-06-11-21-12_05_000028.png 197 | 2015-11-06-11-21-12_05_000029.png 198 | 2015-11-06-11-21-12_05_000030.png 199 | 2015-11-06-11-21-12_05_000031.png 200 | 2015-11-06-11-21-12_05_000032.png 201 | 2015-11-06-11-21-12_05_000033.png 202 | 2015-11-06-11-21-12_05_000034.png 203 | 2015-11-06-11-21-12_05_000035.png 204 | 2015-11-06-11-21-12_05_000036.png 205 | 2015-11-06-11-21-12_05_000037.png 206 | 2015-11-06-11-21-12_05_000038.png 207 | 2015-11-06-11-21-12_05_000039.png 208 | 2015-11-06-11-21-12_05_000040.png 209 | 2015-11-06-11-21-12_05_000041.png 210 | 2015-11-06-11-21-12_05_000042.png 211 | 2015-11-06-11-21-12_05_000043.png 212 | 2015-11-06-11-21-12_05_000044.png 213 | 2015-11-06-11-21-12_05_000045.png 214 | 2015-11-06-11-21-12_05_000046.png 215 | 2015-11-06-11-21-12_05_000047.png 216 | 2015-11-06-11-21-12_05_000049.png 217 | 2015-11-06-11-21-12_05_000050.png 218 | 2015-11-06-11-21-12_05_000051.png 219 | 2015-11-06-11-21-12_05_000052.png 220 | 2015-11-06-11-21-12_05_000053.png 221 | 2015-11-06-11-21-12_05_000054.png 222 | 2015-11-06-11-21-12_05_000055.png 223 | 2015-11-06-11-21-12_05_000056.png 224 | 2015-11-06-11-21-12_05_000057.png 225 | 2015-11-06-11-21-12_05_000058.png 226 | 2015-11-06-11-21-12_05_000059.png 227 | 2015-11-06-11-21-12_05_000060.png 228 | 2015-11-06-11-21-12_05_000061.png 229 | 2015-11-06-11-21-12_05_000062.png 230 | 2015-11-06-11-21-12_05_000063.png 231 | 2015-11-06-11-21-12_05_000064.png 232 | 2015-11-06-11-21-12_05_000065.png 233 | 2015-11-06-11-21-12_05_000066.png 234 | 2015-11-06-11-21-12_05_000067.png 235 | 2015-11-06-11-21-12_05_000068.png 236 | 2015-11-06-11-21-12_05_000069.png 237 | 2015-11-06-11-21-12_05_000071.png 238 | 2015-11-06-11-21-12_05_000072.png 239 | 2015-11-06-11-21-12_05_000073.png 240 | 2015-11-06-11-21-12_05_000074.png 241 | 2015-11-06-11-21-12_05_000075.png 242 | 2015-11-06-11-21-12_05_000076.png 243 | 2015-11-06-11-21-12_05_000078.png 244 | 2015-11-06-11-21-12_05_000079.png 245 | 2015-11-06-11-21-12_05_000080.png 246 | 2015-11-06-11-21-12_05_000081.png 247 | 2015-11-06-11-21-12_05_000082.png 248 | 2015-11-06-11-21-12_05_000083.png 249 | 2015-11-06-11-21-12_05_000084.png 250 | 2015-11-06-11-21-12_05_000085.png 251 | 2015-11-06-11-21-12_05_000086.png 252 | 2015-11-06-11-21-12_05_000087.png 253 | 2015-11-06-11-21-12_05_000088.png 254 | 2015-11-06-11-21-12_05_000089.png 255 | 2015-11-06-11-21-12_05_000090.png 256 | 2015-11-06-11-21-12_05_000091.png 257 | 2015-11-06-11-21-12_05_000092.png 258 | 2015-11-06-11-21-12_05_000095.png 259 | 2015-11-06-11-21-12_05_000096.png 260 | 2015-11-06-11-21-12_05_000097.png 261 | 2015-11-06-11-21-12_05_000099.png 262 | 2015-11-06-11-21-12_05_000100.png 263 | 2015-11-06-11-21-12_05_000101.png 264 | 2015-11-06-11-21-12_05_000102.png 265 | 2015-11-06-11-21-12_05_000103.png 266 | 2015-11-06-11-21-12_05_000104.png 267 | 2015-11-06-11-21-12_05_000105.png 268 | 2015-11-06-11-21-12_05_000106.png 269 | 2015-11-06-11-21-12_05_000107.png 270 | 2015-11-06-11-21-12_05_000108.png 271 | 2015-11-06-11-21-12_05_000109.png 272 | -------------------------------------------------------------------------------- /dataset/robot_pseudo_dataset.py: -------------------------------------------------------------------------------- 1 | import os 2 | import os.path as osp 3 | import numpy as np 4 | import random 5 | import matplotlib 6 | matplotlib.use('agg') 7 | import matplotlib.pyplot as plt 8 | import collections 9 | import torch 10 | import torchvision 11 | from torch.utils import data 12 | from PIL import Image, ImageFile 13 | from dataset.autoaugment import ImageNetPolicy 14 | import time 15 | 16 | ImageFile.LOAD_TRUNCATED_IMAGES = True 17 | 18 | class robot_pseudo_DataSet(data.Dataset): 19 | def __init__(self, root, list_path, max_iters=None, resize_size=(1280, 960), crop_size=(512, 1024), mean=(128, 128, 128), scale=True, mirror=True, ignore_label=255, set='train', autoaug=False, threshold = 1.0): 20 | self.root = root 21 | self.threshold = threshold 22 | self.list_path = list_path 23 | self.crop_size = crop_size 24 | self.scale = scale 25 | self.ignore_label = ignore_label 26 | self.mean = mean 27 | self.is_mirror = mirror 28 | self.resize_size = resize_size 29 | self.autoaug = autoaug 30 | self.h = crop_size[0] 31 | self.w = crop_size[1] 32 | # self.mean_bgr = np.array([104.00698793, 116.66876762, 122.67891434]) 33 | self.img_ids = [i_id.strip() for i_id in open(list_path)] 34 | if not max_iters==None: 35 | self.img_ids = self.img_ids * int(np.ceil(float(max_iters) / len(self.img_ids))) 36 | self.files = [] 37 | self.set = set 38 | # for split in ["train", "trainval", "val"]: 39 | 40 | #https://github.com/mcordts/cityscapesScripts/blob/master/cityscapesscripts/helpers/labels.py 41 | ''' 42 | 0 sky; 1 person; 2 two-wheel; 3 automobile; 4 sign 43 | 5 light 6 building 7 sidewalk 8 road 44 | ''' 45 | self.id_to_trainid = {0:0, 1:1, 2:2, 3:3, 4:4, 5:5, 6:6, 7:7, 8:8} 46 | 47 | for name in self.img_ids: 48 | img_file = osp.join(self.root, "%s/%s" % (self.set, name)) 49 | if set == 'val': 50 | label_file = osp.join(self.root, "anno/%s" %name ) 51 | else: 52 | label_file = osp.join(self.root, "pseudo_train/%s" %name ) 53 | score_file = osp.join(self.root, "pseudo_train/%s_score.png" % name) 54 | self.files.append({ 55 | "img": img_file, 56 | "label": label_file, 57 | "score": score_file, 58 | "name": name 59 | }) 60 | 61 | def __len__(self): 62 | return len(self.files) 63 | 64 | def __getitem__(self, index): 65 | #tt = time.time() 66 | datafiles = self.files[index] 67 | name = datafiles["name"] 68 | 69 | image = Image.open(datafiles["img"]).convert('RGB') 70 | label = Image.open(datafiles["label"]) 71 | score = Image.open(datafiles["score"]) 72 | 73 | if self.scale: 74 | random_scale = 0.8 + random.random()*0.4 # 0.8 - 1.2 75 | image = image.resize( ( round(self.resize_size[0] * random_scale), round(self.resize_size[1] * random_scale)) , Image.BICUBIC) 76 | label = label.resize( ( round(self.resize_size[0] * random_scale), round(self.resize_size[1] * random_scale)) , Image.NEAREST) 77 | score = score.resize( ( round(self.resize_size[0] * random_scale), round(self.resize_size[1] * random_scale)) , Image.NEAREST) 78 | else: 79 | image = image.resize( ( self.resize_size[0], self.resize_size[1] ) , Image.BICUBIC) 80 | label = label.resize( ( self.resize_size[0], self.resize_size[1] ) , Image.NEAREST) 81 | score = score.resize( ( self.resize_size[0], self.resize_size[1] ) , Image.NEAREST) 82 | 83 | label = np.asarray(label, np.uint8) 84 | score = np.asarray(score, np.uint8) 85 | 86 | # re-assign labels to match the format of Cityscapes 87 | label_copy = 255 * np.ones(label.shape, dtype=np.uint8) 88 | for k, v in list(self.id_to_trainid.items()): 89 | label_copy[label == k] = v 90 | # threshold 91 | if self.threshold<1.0: 92 | label_copy[score<(self.threshold*100)] = 255 93 | 94 | if self.autoaug: 95 | policy = ImageNetPolicy() 96 | image = policy(image) 97 | image = np.asarray(image, np.float32) 98 | size = image.shape 99 | image = image[:, :, ::-1] # change to BGR 100 | image -= self.mean 101 | image = image.transpose((2, 0, 1)) 102 | if self.set == 'train': 103 | for i in range(10): #find hard samples 104 | x1 = random.randint(0, image.shape[1] - self.h) 105 | y1 = random.randint(0, image.shape[2] - self.w) 106 | tmp_label_copy = label_copy[x1:x1+self.h, y1:y1+self.w] 107 | tmp_image = image[:, x1:x1+self.h, y1:y1+self.w] 108 | u = np.unique(tmp_label_copy) 109 | if len(u) > 4: 110 | break 111 | else: 112 | print('RB: Too young too naive for %d times!'%i) 113 | else: 114 | x1 = random.randint(0, image.shape[1] - self.h) 115 | y1 = random.randint(0, image.shape[2] - self.w) 116 | tmp_image = image[:, x1:x1+self.h, y1:y1+self.w] 117 | tmp_label_copy = label_copy[x1:x1+self.h, y1:y1+self.w] 118 | 119 | image = tmp_image 120 | label_copy = tmp_label_copy 121 | 122 | if self.is_mirror and random.random() < 0.5: 123 | image = np.flip(image, axis = 2) 124 | label_copy = np.flip(label_copy, axis = 1) 125 | 126 | return image.copy(), label_copy.copy(), np.array(size), name 127 | 128 | 129 | if __name__ == '__main__': 130 | dst = robot_pseudo_DataSet('./data/Oxford_Robot_ICCV19', './dataset/robot_list/train.txt', mean=(0,0,0), set = 'train', autoaug=True) 131 | trainloader = data.DataLoader(dst, batch_size=4) 132 | for i, data in enumerate(trainloader): 133 | imgs, _, _, _ = data 134 | if i == 0: 135 | img = torchvision.utils.make_grid(imgs).numpy() 136 | img = np.transpose(img, (1, 2, 0)) 137 | img = img[:, :, ::-1] 138 | img = Image.fromarray(np.uint8(img) ) 139 | img.save('Robot_pseudo_Demo.jpg') 140 | break 141 | -------------------------------------------------------------------------------- /dataset/synthia_dataset.py: -------------------------------------------------------------------------------- 1 | import os 2 | import os.path as osp 3 | import numpy as np 4 | import random 5 | import matplotlib.pyplot as plt 6 | import collections 7 | import torch 8 | import torchvision 9 | from torch.utils import data 10 | from PIL import Image, ImageFile 11 | from dataset.autoaugment import ImageNetPolicy 12 | import imageio 13 | ImageFile.LOAD_TRUNCATED_IMAGES = True 14 | 15 | 16 | class SynthiaDataSet(data.Dataset): 17 | def __init__(self, root, list_path, max_iters=None, resize_size=(1024, 512), crop_size=(512, 1024), mean=(128, 128, 128), scale=False, mirror=True, ignore_label=255, autoaug = False): 18 | self.root = root 19 | self.list_path = list_path 20 | self.crop_size = crop_size 21 | self.scale = scale 22 | self.ignore_label = ignore_label 23 | self.mean = mean 24 | self.is_mirror = mirror 25 | self.resize_size = resize_size 26 | self.autoaug = autoaug 27 | self.h = crop_size[0] 28 | self.w = crop_size[1] 29 | # self.mean_bgr = np.array([104.00698793, 116.66876762, 122.67891434]) 30 | self.img_ids = [i_id.strip() for i_id in open(list_path)] 31 | if not max_iters==None: 32 | self.img_ids = self.img_ids * int(np.ceil(float(max_iters) / len(self.img_ids))) 33 | self.files = [] 34 | 35 | self.id_to_trainid = {3: 0, 4: 1, 2: 2, 21: 3, 5: 4, 7: 5, 36 | 15: 6, 9: 7, 6: 8, 16: 9, 1: 10, 10: 11, 17: 12, 37 | 8: 13, 18: 14, 19: 15, 20: 16, 12: 17, 11: 18} 38 | # for split in ["train", "trainval", "val"]: 39 | for name in self.img_ids: 40 | img_file = osp.join(self.root, "RGB/%s" % name) 41 | label_file = osp.join(self.root, "GT/LABELS/%s" % name) 42 | self.files.append({ 43 | "img": img_file, 44 | "label": label_file, 45 | "name": name 46 | }) 47 | 48 | def __len__(self): 49 | return len(self.files) 50 | 51 | 52 | def __getitem__(self, index): 53 | datafiles = self.files[index] 54 | 55 | image = Image.open(datafiles["img"]).convert('RGB') 56 | #label = Image.open(datafiles["label"]) 57 | label = np.asarray(imageio.imread(datafiles["label"], format='PNG-FI'))[:,:,0] # uint16 58 | label = Image.fromarray(label) 59 | name = datafiles["name"] 60 | 61 | # resize 62 | if self.scale: 63 | random_scale = 0.8 + random.random()*0.4 # 0.8 - 1.2 64 | image = image.resize( ( round(self.resize_size[0] * random_scale), round(self.resize_size[1] * random_scale)) , Image.BICUBIC) 65 | label = label.resize( ( round(self.resize_size[0] * random_scale), round(self.resize_size[1] * random_scale)) , Image.NEAREST) 66 | else: 67 | image = image.resize( ( self.resize_size[0], self.resize_size[1] ) , Image.BICUBIC) 68 | label = label.resize( ( self.resize_size[0], self.resize_size[1] ) , Image.NEAREST) 69 | 70 | if self.autoaug: 71 | policy = ImageNetPolicy() 72 | image = policy(image) 73 | 74 | image = np.asarray(image, np.float32) 75 | label = np.asarray(label, np.uint8) 76 | 77 | # re-assign labels to match the format of Cityscapes 78 | label_copy = 255 * np.ones(label.shape, dtype=np.uint8) 79 | for k, v in list(self.id_to_trainid.items()): 80 | label_copy[label == k] = v 81 | 82 | size = image.shape 83 | image = image[:, :, ::-1] # change to BGR 84 | image -= self.mean 85 | image = image.transpose((2, 0, 1)) 86 | print(image.shape, label.shape) 87 | for i in range(10): #find hard samples 88 | x1 = random.randint(0, image.shape[1] - self.h) 89 | y1 = random.randint(0, image.shape[2] - self.w) 90 | tmp_label_copy = label_copy[x1:x1+self.h, y1:y1+self.w] 91 | tmp_image = image[:, x1:x1+self.h, y1:y1+self.w] 92 | u = np.unique(tmp_label_copy) 93 | if len(u) > 10: 94 | break 95 | else: 96 | continue 97 | #print('GTA5: Too young too naive for %d times!'%i) 98 | 99 | image = tmp_image 100 | label_copy = tmp_label_copy 101 | 102 | if self.is_mirror and random.random() < 0.5: 103 | image = np.flip(image, axis = 2) 104 | label_copy = np.flip(label_copy, axis = 1) 105 | 106 | return image.copy(), label_copy.copy(), np.array(size), name 107 | 108 | 109 | if __name__ == '__main__': 110 | dst = SynthiaDataSet('./data/synthia/', './dataset/synthia_list/train.txt', mean=(0,0,0), autoaug=True) 111 | trainloader = data.DataLoader(dst, batch_size=4) 112 | for i, data in enumerate(trainloader): 113 | imgs, _, _, _ = data 114 | if i == 0: 115 | img = torchvision.utils.make_grid(imgs).numpy() 116 | img = np.transpose(img, (1, 2, 0)) 117 | img = img[:, :, ::-1] 118 | img = Image.fromarray(np.uint8(img) ) 119 | img.save('Synthia_Demo.jpg') 120 | break 121 | -------------------------------------------------------------------------------- /discuss_plabel.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import scipy 3 | from scipy import ndimage 4 | import numpy as np 5 | import sys 6 | import re 7 | from packaging import version 8 | 9 | import torch 10 | from torch.autograd import Variable 11 | import torchvision.models as models 12 | import torch.nn.functional as F 13 | from torch.utils import data, model_zoo 14 | from model.deeplab import Res_Deeplab 15 | from model.deeplab_multi import DeeplabMulti 16 | from model.deeplab_vgg import DeeplabVGG 17 | from dataset.cityscapes_dataset import cityscapesDataSet 18 | from collections import OrderedDict 19 | import os 20 | from PIL import Image 21 | from utils.tool import fliplr 22 | import matplotlib.pyplot as plt 23 | import torch.nn as nn 24 | import yaml 25 | 26 | torch.backends.cudnn.benchmark=True 27 | 28 | IMG_MEAN = np.array((104.00698793,116.66876762,122.67891434), dtype=np.float32) 29 | 30 | DATA_DIRECTORY = './data/Cityscapes/data' 31 | DATA_LIST_PATH = './dataset/cityscapes_list/train.txt' 32 | SAVE_PATH = './data/Cityscapes/data/pseudo/train' 33 | 34 | if not os.path.isdir('./data/Cityscapes/data/pseudo/'): 35 | os.mkdir('./data/Cityscapes/data/pseudo/') 36 | os.mkdir(SAVE_PATH) 37 | 38 | IGNORE_LABEL = 255 39 | NUM_CLASSES = 19 40 | NUM_STEPS = 2975 # Number of images in the validation set. 41 | RESTORE_FROM = 'http://vllab.ucmerced.edu/ytsai/CVPR18/GTA2Cityscapes_multi-ed35151c.pth' 42 | RESTORE_FROM_VGG = 'http://vllab.ucmerced.edu/ytsai/CVPR18/GTA2Cityscapes_vgg-ac4ac9f6.pth' 43 | RESTORE_FROM_ORC = 'http://vllab1.ucmerced.edu/~whung/adaptSeg/cityscapes_oracle-b7b9934.pth' 44 | SET = 'train' # We generate pseudo label for training set 45 | 46 | MODEL = 'DeeplabMulti' 47 | 48 | palette = [128, 64, 128, 244, 35, 232, 70, 70, 70, 102, 102, 156, 190, 153, 153, 153, 153, 153, 250, 170, 30, 49 | 220, 220, 0, 107, 142, 35, 152, 251, 152, 70, 130, 180, 220, 20, 60, 255, 0, 0, 0, 0, 142, 0, 0, 70, 50 | 0, 60, 100, 0, 80, 100, 0, 0, 230, 119, 11, 32] 51 | zero_pad = 256 * 3 - len(palette) 52 | for i in range(zero_pad): 53 | palette.append(0) 54 | 55 | def draw_hist(x, y, xx, yy, name = 'Var'): 56 | fig = plt.figure() 57 | bins = np.linspace(0, 1, 50) 58 | x = x.flatten() 59 | y = y.flatten() 60 | xx = np.append(xx,x) 61 | yy = np.append(yy,y) 62 | print(len(xx)) 63 | #plt.hist(xx, bins, alpha=0.5, label='Positive') 64 | plt.hist(yy, bins, alpha=0.5, label='Negative') 65 | plt.legend(loc='upper right') 66 | fig.savefig('%s_hist.png' % name) 67 | plt.clf() 68 | return xx, yy 69 | 70 | def colorize_mask(mask): 71 | # mask: numpy array of the mask 72 | new_mask = Image.fromarray(mask.astype(np.uint8)).convert('P') 73 | new_mask.putpalette(palette) 74 | 75 | return new_mask 76 | 77 | def get_arguments(): 78 | """Parse all the arguments provided from the CLI. 79 | 80 | Returns: 81 | A list of parsed arguments. 82 | """ 83 | parser = argparse.ArgumentParser(description="DeepLab-ResNet Network") 84 | parser.add_argument("--model", type=str, default=MODEL, 85 | help="Model Choice (DeeplabMulti/DeeplabVGG/Oracle).") 86 | parser.add_argument("--data-dir", type=str, default=DATA_DIRECTORY, 87 | help="Path to the directory containing the Cityscapes dataset.") 88 | parser.add_argument("--data-list", type=str, default=DATA_LIST_PATH, 89 | help="Path to the file listing the images in the dataset.") 90 | parser.add_argument("--ignore-label", type=int, default=IGNORE_LABEL, 91 | help="The index of the label to ignore during the training.") 92 | parser.add_argument("--num-classes", type=int, default=NUM_CLASSES, 93 | help="Number of classes to predict (including background).") 94 | parser.add_argument("--restore-from", type=str, default=RESTORE_FROM, 95 | help="Where restore model parameters from.") 96 | parser.add_argument("--gpu", type=int, default=0, 97 | help="choose gpu device.") 98 | parser.add_argument("--batchsize", type=int, default=12, 99 | help="choose gpu device.") 100 | parser.add_argument("--set", type=str, default=SET, 101 | help="choose evaluation set.") 102 | parser.add_argument("--save", type=str, default=SAVE_PATH, 103 | help="Path to save result.") 104 | return parser.parse_args() 105 | 106 | def save_heatmap(output_name): 107 | output, name = output_name 108 | fig = plt.figure() 109 | plt.axis('off') 110 | heatmap = plt.imshow(output, cmap='viridis') 111 | fig.colorbar(heatmap) 112 | fig.savefig('%s_heatmap.png' % (name.split('.jpg')[0])) 113 | return 114 | 115 | def main(): 116 | """Create the model and start the evaluation process.""" 117 | x0 = [] 118 | y0 = [] 119 | x1 = [] 120 | y1 = [] 121 | count = 0 122 | right_var = 0 123 | wrong_var = 0 124 | 125 | args = get_arguments() 126 | 127 | config_path = os.path.join(os.path.dirname(args.restore_from),'opts.yaml') 128 | with open(config_path, 'r') as stream: 129 | config = yaml.load(stream) 130 | 131 | args.model = config['model'] 132 | print('ModelType:%s'%args.model) 133 | print('NormType:%s'%config['norm_style']) 134 | gpu0 = args.gpu 135 | batchsize = args.batchsize 136 | 137 | model_name = os.path.basename( os.path.dirname(args.restore_from) ) 138 | #args.save += model_name 139 | 140 | if not os.path.exists(args.save): 141 | os.makedirs(args.save) 142 | 143 | if args.model == 'DeepLab': 144 | model = DeeplabMulti(num_classes=args.num_classes, use_se = config['use_se'], train_bn = False, norm_style = config['norm_style']) 145 | elif args.model == 'Oracle': 146 | model = Res_Deeplab(num_classes=args.num_classes) 147 | if args.restore_from == RESTORE_FROM: 148 | args.restore_from = RESTORE_FROM_ORC 149 | elif args.model == 'DeeplabVGG': 150 | model = DeeplabVGG(num_classes=args.num_classes) 151 | if args.restore_from == RESTORE_FROM: 152 | args.restore_from = RESTORE_FROM_VGG 153 | 154 | if args.restore_from[:4] == 'http' : 155 | saved_state_dict = model_zoo.load_url(args.restore_from) 156 | else: 157 | saved_state_dict = torch.load(args.restore_from) 158 | 159 | try: 160 | model.load_state_dict(saved_state_dict) 161 | except: 162 | model = torch.nn.DataParallel(model) 163 | model.load_state_dict(saved_state_dict) 164 | model.eval() 165 | model.cuda(gpu0) 166 | 167 | testloader = data.DataLoader(cityscapesDataSet(args.data_dir, args.data_list, crop_size=(512, 1024), resize_size=(1024, 512), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 168 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 169 | 170 | scale = 1.25 171 | testloader2 = data.DataLoader(cityscapesDataSet(args.data_dir, args.data_list, crop_size=(round(512*scale), round(1024*scale) ), resize_size=( round(1024*scale), round(512*scale)), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 172 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 173 | 174 | 175 | gtloader = data.DataLoader(cityscapesDataSet(args.data_dir, args.data_list, crop_size=(65, 129), resize_size=(129, 65), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 176 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 177 | if version.parse(torch.__version__) >= version.parse('0.4.0'): 178 | interp = nn.Upsample(size=(1024, 2048), mode='bilinear', align_corners=True) 179 | interp2 = nn.Upsample(size=(64, 128), mode='nearest') 180 | else: 181 | interp = nn.Upsample(size=(1024, 2048), mode='bilinear') 182 | 183 | sm = torch.nn.Softmax(dim = 1) 184 | log_sm = torch.nn.LogSoftmax(dim = 1) 185 | kl_distance = nn.KLDivLoss( reduction = 'none') 186 | 187 | for index, img_data in enumerate(zip(testloader, testloader2, gtloader) ): 188 | batch, batch2, gt = img_data 189 | image, _, _, name = batch 190 | image2, _, _, name2 = batch2 191 | _, gt_label, _, _ = gt 192 | 193 | inputs = image.cuda() 194 | inputs2 = image2.cuda() 195 | print('\r>>>>Extracting feature...%04d/%04d'%(index*batchsize, NUM_STEPS), end='') 196 | if args.model == 'DeepLab': 197 | with torch.no_grad(): 198 | output1, output2 = model(inputs) 199 | output_batch = sm(0.5* output1 + output2) 200 | print(output_batch.shape) 201 | 202 | heatmap_batch = torch.sum(kl_distance(log_sm(output1), sm(output2)), dim=1) 203 | heatmap_batch = torch.exp(-heatmap_batch) 204 | #output1, output2 = model(fliplr(inputs)) 205 | #output1, output2 = fliplr(output1), fliplr(output2) 206 | #output_batch += interp(sm(0.5 * output1 + output2)) 207 | #del output1, output2, inputs 208 | 209 | #output1, output2 = model(inputs2) 210 | #output_batch += interp(sm(0.5* output1 + output2)) 211 | #output1, output2 = model(fliplr(inputs2)) 212 | #output1, output2 = fliplr(output1), fliplr(output2) 213 | #output_batch += interp(sm(0.5 * output1 + output2)) 214 | #del output1, output2, inputs2 215 | output_batch = output_batch.cpu().data.numpy() 216 | heatmap_batch = heatmap_batch.cpu().data.numpy() 217 | elif args.model == 'DeeplabVGG' or args.model == 'Oracle': 218 | output_batch = model(Variable(image).cuda()) 219 | output_batch = interp(output_batch).cpu().data.numpy() 220 | 221 | #output_batch = output_batch.transpose(0,2,3,1) 222 | #output_batch = np.asarray(np.argmax(output_batch, axis=3), dtype=np.uint8) 223 | output_batch = output_batch.transpose(0,2,3,1) 224 | score_batch = np.max(output_batch, axis=3) 225 | output_batch = np.asarray(np.argmax(output_batch, axis=3), dtype=np.uint8) 226 | #output_batch[score_batch<3.2] = 255 #3.2 = 4*0.8 227 | for i in range(output_batch.shape[0]): 228 | output = output_batch[i,:,:] 229 | name_tmp = name[i].split('/')[-1] 230 | dir_name = name[i].split('/')[-2] 231 | save_path = args.save + '/' + dir_name 232 | print('%s/%s' % (save_path, name_tmp)) 233 | 234 | # resize to 64*128 235 | variance_tmp = heatmap_batch[i,:,:]/np.max(heatmap_batch[i,:,:]) 236 | score_tmp = score_batch[i,:,:]/np.max(score_batch[i,:,:]) 237 | prediction = output 238 | ground_truth = gt_label[i,:,:].numpy() 239 | right_mask = prediction==ground_truth 240 | ignore_mask = ground_truth==255 241 | wrong_mask = np.logical_and( (~right_mask), (~ignore_mask)) 242 | # Use High-confidence or not 243 | #high_mask = score_tmp > 0.95 244 | #wrong_mask = np.logical_and( wrong_mask, high_mask) 245 | #right_mask = np.logical_and( right_mask, high_mask) 246 | right_var += np.mean( variance_tmp[right_mask]) 247 | wrong_var += np.mean( variance_tmp[wrong_mask]) 248 | count += 1 249 | print( right_var/count, wrong_var/count) 250 | x0, y0 = draw_hist(variance_tmp[right_mask],variance_tmp[wrong_mask], x0, y0, name='Var') 251 | #x1, y1 = draw_hist(score_tmp[right_mask],score_tmp[wrong_mask], x1, y1, name='Bias') 252 | return args.save 253 | 254 | if __name__ == '__main__': 255 | with torch.no_grad(): 256 | save_path = main() 257 | #os.system('python compute_iou.py ./data/Cityscapes/data/gtFine/train %s'%save_path) 258 | -------------------------------------------------------------------------------- /discuss_plabel_MC_dropout.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import scipy 3 | from scipy import ndimage 4 | import numpy as np 5 | import sys 6 | import re 7 | from packaging import version 8 | 9 | import torch 10 | from torch.autograd import Variable 11 | import torchvision.models as models 12 | import torch.nn.functional as F 13 | from torch.utils import data, model_zoo 14 | from model.deeplab import Res_Deeplab 15 | from model.deeplab_multi import DeeplabMulti 16 | from model.deeplab_vgg import DeeplabVGG 17 | from dataset.cityscapes_dataset import cityscapesDataSet 18 | from collections import OrderedDict 19 | import os 20 | from PIL import Image 21 | from utils.tool import fliplr 22 | import matplotlib.pyplot as plt 23 | import torch.nn as nn 24 | import yaml 25 | import copy 26 | 27 | torch.backends.cudnn.benchmark=True 28 | 29 | IMG_MEAN = np.array((104.00698793,116.66876762,122.67891434), dtype=np.float32) 30 | 31 | DATA_DIRECTORY = './data/Cityscapes/data' 32 | DATA_LIST_PATH = './dataset/cityscapes_list/train.txt' 33 | SAVE_PATH = './data/Cityscapes/data/pseudo/train' 34 | 35 | if not os.path.isdir('./data/Cityscapes/data/pseudo/'): 36 | os.mkdir('./data/Cityscapes/data/pseudo/') 37 | os.mkdir(SAVE_PATH) 38 | 39 | IGNORE_LABEL = 255 40 | NUM_CLASSES = 19 41 | NUM_STEPS = 2975 # Number of images in the validation set. 42 | RESTORE_FROM = 'http://vllab.ucmerced.edu/ytsai/CVPR18/GTA2Cityscapes_multi-ed35151c.pth' 43 | RESTORE_FROM_VGG = 'http://vllab.ucmerced.edu/ytsai/CVPR18/GTA2Cityscapes_vgg-ac4ac9f6.pth' 44 | RESTORE_FROM_ORC = 'http://vllab1.ucmerced.edu/~whung/adaptSeg/cityscapes_oracle-b7b9934.pth' 45 | SET = 'train' # We generate pseudo label for training set 46 | 47 | MODEL = 'DeeplabMulti' 48 | 49 | palette = [128, 64, 128, 244, 35, 232, 70, 70, 70, 102, 102, 156, 190, 153, 153, 153, 153, 153, 250, 170, 30, 50 | 220, 220, 0, 107, 142, 35, 152, 251, 152, 70, 130, 180, 220, 20, 60, 255, 0, 0, 0, 0, 142, 0, 0, 70, 51 | 0, 60, 100, 0, 80, 100, 0, 0, 230, 119, 11, 32] 52 | zero_pad = 256 * 3 - len(palette) 53 | for i in range(zero_pad): 54 | palette.append(0) 55 | 56 | def draw_hist(x, y, xx, yy, name = 'Var'): 57 | fig = plt.figure() 58 | bins = np.linspace(0, 1, 50) 59 | x = x.flatten() 60 | y = y.flatten() 61 | xx = np.append(xx,x) 62 | yy = np.append(yy,y) 63 | print(len(xx)) 64 | #plt.hist(xx, bins, alpha=0.5, label='Positive') 65 | plt.hist(yy, bins, alpha=0.5, label='Negative') 66 | plt.legend(loc='upper right') 67 | fig.savefig('%s_hist.png' % name) 68 | plt.clf() 69 | return xx, yy 70 | 71 | def colorize_mask(mask): 72 | # mask: numpy array of the mask 73 | new_mask = Image.fromarray(mask.astype(np.uint8)).convert('P') 74 | new_mask.putpalette(palette) 75 | 76 | return new_mask 77 | 78 | def get_arguments(): 79 | """Parse all the arguments provided from the CLI. 80 | 81 | Returns: 82 | A list of parsed arguments. 83 | """ 84 | parser = argparse.ArgumentParser(description="DeepLab-ResNet Network") 85 | parser.add_argument("--model", type=str, default=MODEL, 86 | help="Model Choice (DeeplabMulti/DeeplabVGG/Oracle).") 87 | parser.add_argument("--data-dir", type=str, default=DATA_DIRECTORY, 88 | help="Path to the directory containing the Cityscapes dataset.") 89 | parser.add_argument("--data-list", type=str, default=DATA_LIST_PATH, 90 | help="Path to the file listing the images in the dataset.") 91 | parser.add_argument("--ignore-label", type=int, default=IGNORE_LABEL, 92 | help="The index of the label to ignore during the training.") 93 | parser.add_argument("--num-classes", type=int, default=NUM_CLASSES, 94 | help="Number of classes to predict (including background).") 95 | parser.add_argument("--restore-from", type=str, default=RESTORE_FROM, 96 | help="Where restore model parameters from.") 97 | parser.add_argument("--gpu", type=int, default=0, 98 | help="choose gpu device.") 99 | parser.add_argument("--batchsize", type=int, default=12, 100 | help="choose gpu device.") 101 | parser.add_argument("--set", type=str, default=SET, 102 | help="choose evaluation set.") 103 | parser.add_argument("--save", type=str, default=SAVE_PATH, 104 | help="Path to save result.") 105 | return parser.parse_args() 106 | 107 | def save_heatmap(output_name): 108 | output, name = output_name 109 | fig = plt.figure() 110 | plt.axis('off') 111 | heatmap = plt.imshow(output, cmap='viridis') 112 | fig.colorbar(heatmap) 113 | fig.savefig('%s_heatmap.png' % (name.split('.jpg')[0])) 114 | return 115 | 116 | def activate_drop(m, drop = 0.5): 117 | classname = m.__class__.__name__ 118 | if classname.find('Drop') != -1: 119 | m.p = drop 120 | m.train() 121 | 122 | def main(): 123 | """Create the model and start the evaluation process.""" 124 | x0 = [] 125 | y0 = [] 126 | x1 = [] 127 | y1 = [] 128 | count = 0 129 | right_var = 0 130 | wrong_var = 0 131 | 132 | args = get_arguments() 133 | 134 | config_path = os.path.join(os.path.dirname(args.restore_from),'opts.yaml') 135 | with open(config_path, 'r') as stream: 136 | config = yaml.load(stream) 137 | 138 | args.model = config['model'] 139 | print('ModelType:%s'%args.model) 140 | print('NormType:%s'%config['norm_style']) 141 | gpu0 = args.gpu 142 | batchsize = args.batchsize 143 | 144 | model_name = os.path.basename( os.path.dirname(args.restore_from) ) 145 | #args.save += model_name 146 | 147 | if not os.path.exists(args.save): 148 | os.makedirs(args.save) 149 | 150 | if args.model == 'DeepLab': 151 | model = DeeplabMulti(num_classes=args.num_classes, use_se = config['use_se'], train_bn = False, norm_style = config['norm_style']) 152 | elif args.model == 'Oracle': 153 | model = Res_Deeplab(num_classes=args.num_classes) 154 | if args.restore_from == RESTORE_FROM: 155 | args.restore_from = RESTORE_FROM_ORC 156 | elif args.model == 'DeeplabVGG': 157 | model = DeeplabVGG(num_classes=args.num_classes) 158 | if args.restore_from == RESTORE_FROM: 159 | args.restore_from = RESTORE_FROM_VGG 160 | 161 | if args.restore_from[:4] == 'http' : 162 | saved_state_dict = model_zoo.load_url(args.restore_from) 163 | else: 164 | saved_state_dict = torch.load(args.restore_from) 165 | 166 | try: 167 | model.load_state_dict(saved_state_dict) 168 | except: 169 | model = torch.nn.DataParallel(model) 170 | model.load_state_dict(saved_state_dict) 171 | model.eval() 172 | model.cuda(gpu0) 173 | model_drop = copy.deepcopy(model) 174 | model_drop.apply(activate_drop) 175 | print(model_drop) 176 | 177 | testloader = data.DataLoader(cityscapesDataSet(args.data_dir, args.data_list, crop_size=(512, 1024), resize_size=(1024, 512), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 178 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 179 | 180 | scale = 1.25 181 | testloader2 = data.DataLoader(cityscapesDataSet(args.data_dir, args.data_list, crop_size=(round(512*scale), round(1024*scale) ), resize_size=( round(1024*scale), round(512*scale)), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 182 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 183 | 184 | 185 | gtloader = data.DataLoader(cityscapesDataSet(args.data_dir, args.data_list, crop_size=(65, 129), resize_size=(129, 65), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 186 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 187 | if version.parse(torch.__version__) >= version.parse('0.4.0'): 188 | interp = nn.Upsample(size=(1024, 2048), mode='bilinear', align_corners=True) 189 | interp2 = nn.Upsample(size=(64, 128), mode='nearest') 190 | else: 191 | interp = nn.Upsample(size=(1024, 2048), mode='bilinear') 192 | 193 | sm = torch.nn.Softmax(dim = 1) 194 | log_sm = torch.nn.LogSoftmax(dim = 1) 195 | kl_distance = nn.KLDivLoss( reduction = 'none') 196 | 197 | for index, img_data in enumerate(zip(testloader, testloader2, gtloader) ): 198 | batch, batch2, gt = img_data 199 | image, _, _, name = batch 200 | image2, _, _, name2 = batch2 201 | _, gt_label, _, _ = gt 202 | 203 | inputs = image.cuda() 204 | inputs2 = image2.cuda() 205 | print('\r>>>>Extracting feature...%04d/%04d'%(index*batchsize, NUM_STEPS), end='') 206 | if args.model == 'DeepLab': 207 | with torch.no_grad(): 208 | output1, output2 = model(inputs) 209 | output1_drop , output2_drop = model_drop(inputs) 210 | output_batch = sm(0.5* output1 + output2) 211 | print(output_batch.shape) 212 | 213 | #heatmap_batch = torch.sum(kl_distance(log_sm(output2_drop), sm(output2)), dim=1) 214 | heatmap_batch = torch.sum(kl_distance(log_sm(output1_drop), sm(output2_drop)), dim=1) 215 | heatmap_batch = torch.exp(-heatmap_batch) 216 | #output1, output2 = model(fliplr(inputs)) 217 | #output1, output2 = fliplr(output1), fliplr(output2) 218 | #output_batch += interp(sm(0.5 * output1 + output2)) 219 | #del output1, output2, inputs 220 | 221 | #output1, output2 = model(inputs2) 222 | #output_batch += interp(sm(0.5* output1 + output2)) 223 | #output1, output2 = model(fliplr(inputs2)) 224 | #output1, output2 = fliplr(output1), fliplr(output2) 225 | #output_batch += interp(sm(0.5 * output1 + output2)) 226 | #del output1, output2, inputs2 227 | output_batch = output_batch.cpu().data.numpy() 228 | heatmap_batch = heatmap_batch.cpu().data.numpy() 229 | elif args.model == 'DeeplabVGG' or args.model == 'Oracle': 230 | output_batch = model(Variable(image).cuda()) 231 | output_batch = interp(output_batch).cpu().data.numpy() 232 | 233 | #output_batch = output_batch.transpose(0,2,3,1) 234 | #output_batch = np.asarray(np.argmax(output_batch, axis=3), dtype=np.uint8) 235 | output_batch = output_batch.transpose(0,2,3,1) 236 | score_batch = np.max(output_batch, axis=3) 237 | output_batch = np.asarray(np.argmax(output_batch, axis=3), dtype=np.uint8) 238 | #output_batch[score_batch<3.2] = 255 #3.2 = 4*0.8 239 | for i in range(output_batch.shape[0]): 240 | output = output_batch[i,:,:] 241 | name_tmp = name[i].split('/')[-1] 242 | dir_name = name[i].split('/')[-2] 243 | save_path = args.save + '/' + dir_name 244 | print('%s/%s' % (save_path, name_tmp)) 245 | 246 | # resize to 64*128 247 | variance_tmp = heatmap_batch[i,:,:]/np.max(heatmap_batch[i,:,:]) 248 | score_tmp = score_batch[i,:,:]/np.max(score_batch[i,:,:]) 249 | prediction = output 250 | ground_truth = gt_label[i,:,:].numpy() 251 | right_mask = prediction==ground_truth 252 | ignore_mask = ground_truth==255 253 | wrong_mask = np.logical_and( (~right_mask), (~ignore_mask)) 254 | # Use High-confidence or not 255 | #high_mask = score_tmp > 0.95 256 | #wrong_mask = np.logical_and( wrong_mask, high_mask) 257 | #right_mask = np.logical_and( right_mask, high_mask) 258 | right_var += np.mean( variance_tmp[right_mask]) 259 | wrong_var += np.mean( variance_tmp[wrong_mask]) 260 | count += 1 261 | print( right_var/count, wrong_var/count) 262 | x0, y0 = draw_hist(variance_tmp[right_mask],variance_tmp[wrong_mask], x0, y0, name='Var') 263 | #x1, y1 = draw_hist(score_tmp[right_mask],score_tmp[wrong_mask], x1, y1, name='Bias') 264 | return args.save 265 | 266 | if __name__ == '__main__': 267 | with torch.no_grad(): 268 | save_path = main() 269 | #os.system('python compute_iou.py ./data/Cityscapes/data/gtFine/train %s'%save_path) 270 | -------------------------------------------------------------------------------- /evaluate_cityscapes_folder.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import argparse 3 | import os 4 | parser = argparse.ArgumentParser(description='Test') 5 | parser.add_argument('--name', default='SWA_MRNet', type=str, help='save model path') 6 | parser.add_argument('--average', action='store_true', help='using average model') 7 | opt = parser.parse_args() 8 | result = './snapshots/%s/result.txt'%opt.name 9 | 10 | for iter in np.arange(1,11): 11 | if opt.average: 12 | os.system('echo %d+%s | tee -a %s'%(iter*10000, 'average', result)) 13 | os.system('python evaluate_cityscapes.py --restore ./snapshots/%s/GTA5_%d_average.pth --batchsize 6 | tee -a %s'%(opt.name,iter*10000, result)) 14 | else: 15 | os.system('echo %d | tee -a %s'%(iter*10000, result)) 16 | os.system('python evaluate_cityscapes.py --restore ./snapshots/%s/GTA5_%d.pth --batchsize 6 | tee -a %s'%(opt.name,iter*10000, result)) 17 | -------------------------------------------------------------------------------- /evaluate_cityscapes_train.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import scipy 3 | from scipy import ndimage 4 | import numpy as np 5 | import sys 6 | from packaging import version 7 | from multiprocessing import Pool 8 | import torch 9 | from torch.autograd import Variable 10 | import torchvision.models as models 11 | import torch.nn.functional as F 12 | from torch.utils import data, model_zoo 13 | from model.deeplab import Res_Deeplab 14 | from model.deeplab_multi import DeeplabMulti 15 | from model.deeplab_vgg import DeeplabVGG 16 | from dataset.cityscapes_dataset import cityscapesDataSet 17 | from collections import OrderedDict 18 | import os 19 | from PIL import Image 20 | from utils.tool import fliplr 21 | import matplotlib.pyplot as plt 22 | import torch.nn as nn 23 | import yaml 24 | import time 25 | 26 | torch.backends.cudnn.benchmark=True 27 | 28 | IMG_MEAN = np.array((104.00698793,116.66876762,122.67891434), dtype=np.float32) 29 | 30 | DATA_DIRECTORY = './data/Cityscapes/data' 31 | DATA_LIST_PATH = './dataset/cityscapes_list/train.txt' 32 | SAVE_PATH = './result/cityscapes' 33 | 34 | IGNORE_LABEL = 255 35 | NUM_CLASSES = 19 36 | NUM_STEPS = 2975 # Number of images in the train set. 37 | RESTORE_FROM = 'http://vllab.ucmerced.edu/ytsai/CVPR18/GTA2Cityscapes_multi-ed35151c.pth' 38 | RESTORE_FROM_VGG = 'http://vllab.ucmerced.edu/ytsai/CVPR18/GTA2Cityscapes_vgg-ac4ac9f6.pth' 39 | RESTORE_FROM_ORC = 'http://vllab1.ucmerced.edu/~whung/adaptSeg/cityscapes_oracle-b7b9934.pth' 40 | SET = 'train' 41 | 42 | MODEL = 'DeeplabMulti' 43 | 44 | palette = [128, 64, 128, 244, 35, 232, 70, 70, 70, 102, 102, 156, 190, 153, 153, 153, 153, 153, 250, 170, 30, 45 | 220, 220, 0, 107, 142, 35, 152, 251, 152, 70, 130, 180, 220, 20, 60, 255, 0, 0, 0, 0, 142, 0, 0, 70, 46 | 0, 60, 100, 0, 80, 100, 0, 0, 230, 119, 11, 32] 47 | zero_pad = 256 * 3 - len(palette) 48 | for i in range(zero_pad): 49 | palette.append(0) 50 | 51 | 52 | def colorize_mask(mask): 53 | # mask: numpy array of the mask 54 | new_mask = Image.fromarray(mask.astype(np.uint8)).convert('P') 55 | new_mask.putpalette(palette) 56 | 57 | return new_mask 58 | 59 | def get_arguments(): 60 | """Parse all the arguments provided from the CLI. 61 | 62 | Returns: 63 | A list of parsed arguments. 64 | """ 65 | parser = argparse.ArgumentParser(description="DeepLab-ResNet Network") 66 | parser.add_argument("--model", type=str, default=MODEL, 67 | help="Model Choice (DeeplabMulti/DeeplabVGG/Oracle).") 68 | parser.add_argument("--data-dir", type=str, default=DATA_DIRECTORY, 69 | help="Path to the directory containing the Cityscapes dataset.") 70 | parser.add_argument("--data-list", type=str, default=DATA_LIST_PATH, 71 | help="Path to the file listing the images in the dataset.") 72 | parser.add_argument("--ignore-label", type=int, default=IGNORE_LABEL, 73 | help="The index of the label to ignore during the training.") 74 | parser.add_argument("--num-classes", type=int, default=NUM_CLASSES, 75 | help="Number of classes to predict (including background).") 76 | parser.add_argument("--restore-from", type=str, default=RESTORE_FROM, 77 | help="Where restore model parameters from.") 78 | parser.add_argument("--gpu", type=int, default=0, 79 | help="choose gpu device.") 80 | parser.add_argument("--batchsize", type=int, default=16, 81 | help="choose gpu device.") 82 | parser.add_argument("--set", type=str, default=SET, 83 | help="choose evaluation set.") 84 | parser.add_argument("--save", type=str, default=SAVE_PATH, 85 | help="Path to save result.") 86 | return parser.parse_args() 87 | 88 | def save(output_name): 89 | output, name = output_name 90 | output_col = colorize_mask(output) 91 | output = Image.fromarray(output) 92 | 93 | output.save('%s' % (name)) 94 | output_col.save('%s_color.png' % (name.split('.jpg')[0])) 95 | return 96 | 97 | def save_heatmap(output_name): 98 | output, name = output_name 99 | #output.save('%s_h.png' % (name)) 100 | fig = plt.figure() 101 | plt.axis('off') 102 | heatmap = plt.imshow(output, cmap='viridis') 103 | #fig.colorbar(heatmap) 104 | fig.savefig('%s_heatmap.png' % (name.split('.jpg')[0])) 105 | return 106 | 107 | def save_scoremap(output_name): 108 | output, name = output_name 109 | #output.save('%s_s.png' % (name)) 110 | fig = plt.figure() 111 | plt.axis('off') 112 | heatmap = plt.imshow(output, cmap='viridis') 113 | #fig.colorbar(heatmap) 114 | fig.savefig('%s_scoremap.png' % (name.split('.jpg')[0])) 115 | return 116 | 117 | def main(): 118 | """Create the model and start the evaluation process.""" 119 | args = get_arguments() 120 | 121 | config_path = os.path.join(os.path.dirname(args.restore_from),'opts.yaml') 122 | with open(config_path, 'r') as stream: 123 | config = yaml.load(stream) 124 | 125 | if not 'use_blur' in config: 126 | config['use_blur'] = False 127 | 128 | args.model = config['model'] 129 | print('ModelType:%s'%args.model) 130 | print('NormType:%s'%config['norm_style']) 131 | gpu0 = args.gpu 132 | batchsize = args.batchsize 133 | 134 | model_name = os.path.basename( os.path.dirname(args.restore_from) ) 135 | args.save += model_name 136 | 137 | if not os.path.exists(args.save): 138 | os.makedirs(args.save) 139 | 140 | if args.model == 'DeepLab': 141 | model = DeeplabMulti(num_classes=args.num_classes, use_se = config['use_se'], train_bn = False, norm_style = config['norm_style'], use_blur = config['use_blur']) 142 | elif args.model == 'Oracle': 143 | model = Res_Deeplab(num_classes=args.num_classes) 144 | if args.restore_from == RESTORE_FROM: 145 | args.restore_from = RESTORE_FROM_ORC 146 | elif args.model == 'DeeplabVGG': 147 | model = DeeplabVGG(num_classes=args.num_classes) 148 | if args.restore_from == RESTORE_FROM: 149 | args.restore_from = RESTORE_FROM_VGG 150 | 151 | if args.restore_from[:4] == 'http' : 152 | saved_state_dict = model_zoo.load_url(args.restore_from) 153 | else: 154 | saved_state_dict = torch.load(args.restore_from) 155 | 156 | try: 157 | model.load_state_dict(saved_state_dict) 158 | except: 159 | model = torch.nn.DataParallel(model) 160 | model.load_state_dict(saved_state_dict) 161 | #model = torch.nn.DataParallel(model) 162 | model.eval() 163 | model.cuda(gpu0) 164 | 165 | testloader = data.DataLoader(cityscapesDataSet(args.data_dir, args.data_list, crop_size=(512, 1024), resize_size=(1024, 512), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 166 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 167 | 168 | scale = 1.25 169 | testloader2 = data.DataLoader(cityscapesDataSet(args.data_dir, args.data_list, crop_size=(round(512*scale), round(1024*scale) ), resize_size=( round(1024*scale), round(512*scale)), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 170 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 171 | scale = 0.9 172 | testloader3 = data.DataLoader(cityscapesDataSet(args.data_dir, args.data_list, crop_size=(round(512*scale), round(1024*scale) ), resize_size=( round(1024*scale), round(512*scale)), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 173 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 174 | 175 | 176 | if version.parse(torch.__version__) >= version.parse('0.4.0'): 177 | interp = nn.Upsample(size=(1024, 2048), mode='bilinear', align_corners=True) 178 | else: 179 | interp = nn.Upsample(size=(1024, 2048), mode='bilinear') 180 | 181 | sm = torch.nn.Softmax(dim = 1) 182 | log_sm = torch.nn.LogSoftmax(dim = 1) 183 | kl_distance = nn.KLDivLoss( reduction = 'none') 184 | 185 | for index, img_data in enumerate(zip(testloader, testloader2, testloader3) ): 186 | batch, batch2, batch3 = img_data 187 | image, _, _, name = batch 188 | image2, _, _, name2 = batch2 189 | #image3, _, _, name3 = batch3 190 | 191 | inputs = image.cuda() 192 | inputs2 = image2.cuda() 193 | #inputs3 = Variable(image3).cuda() 194 | print('\r>>>>Extracting feature...%03d/%03d'%(index*batchsize, NUM_STEPS), end='') 195 | if args.model == 'DeepLab': 196 | with torch.no_grad(): 197 | output1, output2 = model(inputs) 198 | output_batch = interp(sm(0.5* output1 + output2)) 199 | heatmap_output1, heatmap_output2 = output1, output2 200 | #output_batch = interp(sm(output1)) 201 | #output_batch = interp(sm(output2)) 202 | output1, output2 = model(fliplr(inputs)) 203 | output1, output2 = fliplr(output1), fliplr(output2) 204 | output_batch += interp(sm(0.5 * output1 + output2)) 205 | heatmap_output1, heatmap_output2 = heatmap_output1+output1, heatmap_output2+output2 206 | #output_batch += interp(sm(output1)) 207 | #output_batch += interp(sm(output2)) 208 | del output1, output2, inputs 209 | 210 | output1, output2 = model(inputs2) 211 | output_batch += interp(sm(0.5* output1 + output2)) 212 | #output_batch += interp(sm(output1)) 213 | #output_batch += interp(sm(output2)) 214 | output1, output2 = model(fliplr(inputs2)) 215 | output1, output2 = fliplr(output1), fliplr(output2) 216 | output_batch += interp(sm(0.5 * output1 + output2)) 217 | #output_batch += interp(sm(output1)) 218 | #output_batch += interp(sm(output2)) 219 | del output1, output2, inputs2 220 | output_batch = output_batch.cpu().data.numpy() 221 | heatmap_batch = torch.sum(kl_distance(log_sm(heatmap_output1), sm(heatmap_output2)), dim=1) 222 | heatmap_batch = torch.log(1 + 10*heatmap_batch) # for visualization 223 | heatmap_batch = heatmap_batch.cpu().data.numpy() 224 | 225 | #output1, output2 = model(inputs3) 226 | #output_batch += interp(sm(0.5* output1 + output2)).cpu().data.numpy() 227 | #output1, output2 = model(fliplr(inputs3)) 228 | #output1, output2 = fliplr(output1), fliplr(output2) 229 | #output_batch += interp(sm(0.5 * output1 + output2)).cpu().data.numpy() 230 | #del output1, output2, inputs3 231 | elif args.model == 'DeeplabVGG' or args.model == 'Oracle': 232 | output_batch = model(Variable(image).cuda()) 233 | output_batch = interp(output_batch).cpu().data.numpy() 234 | 235 | output_batch = output_batch.transpose(0,2,3,1) 236 | scoremap_batch = np.asarray(np.max(output_batch, axis=3)) 237 | output_batch = np.asarray(np.argmax(output_batch, axis=3), dtype=np.uint8) 238 | output_iterator = [] 239 | heatmap_iterator = [] 240 | scoremap_iterator = [] 241 | 242 | for i in range(output_batch.shape[0]): 243 | output_iterator.append(output_batch[i,:,:]) 244 | heatmap_iterator.append(heatmap_batch[i,:,:]/np.max(heatmap_batch[i,:,:])) 245 | scoremap_iterator.append(1-scoremap_batch[i,:,:]/np.max(scoremap_batch[i,:,:])) 246 | name_tmp = name[i].split('/')[-1] 247 | name[i] = '%s/%s' % (args.save, name_tmp) 248 | with Pool(4) as p: 249 | p.map(save, zip(output_iterator, name) ) 250 | p.map(save_heatmap, zip(heatmap_iterator, name) ) 251 | p.map(save_scoremap, zip(scoremap_iterator, name) ) 252 | 253 | del output_batch 254 | 255 | 256 | return args.save 257 | 258 | if __name__ == '__main__': 259 | tt = time.time() 260 | with torch.no_grad(): 261 | save_path = main() 262 | print('Time used: {} sec'.format(time.time()-tt)) 263 | os.system('python compute_iou.py ./data/Cityscapes/data/gtFine/train %s'%save_path) 264 | -------------------------------------------------------------------------------- /evaluate_gta5.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import scipy 3 | from scipy import ndimage 4 | import numpy as np 5 | import sys 6 | from packaging import version 7 | 8 | import torch 9 | from torch.autograd import Variable 10 | import torchvision.models as models 11 | import torch.nn.functional as F 12 | from torch.utils import data, model_zoo 13 | from model.deeplab import Res_Deeplab 14 | from model.deeplab_multi import DeeplabMulti 15 | from model.deeplab_vgg import DeeplabVGG 16 | from dataset.gta5_dataset import GTA5DataSet 17 | from collections import OrderedDict 18 | import os 19 | from PIL import Image 20 | from utils.tool import fliplr 21 | import matplotlib.pyplot as plt 22 | import torch.nn as nn 23 | import yaml 24 | IMG_MEAN = np.array((104.00698793,116.66876762,122.67891434), dtype=np.float32) 25 | 26 | # We just use this file to evaluate the perfromance on the training set 27 | DATA_DIRECTORY = './data/GTA5' 28 | DATA_LIST_PATH = './dataset/gta5_list/train_short.txt' 29 | SAVE_PATH = './result/GTA5' 30 | 31 | IGNORE_LABEL = 255 32 | NUM_CLASSES = 19 33 | NUM_STEPS = 500 # Number of images in the validation set. 34 | RESTORE_FROM = 'http://vllab.ucmerced.edu/ytsai/CVPR18/GTA2Cityscapes_multi-ed35151c.pth' 35 | RESTORE_FROM_VGG = 'http://vllab.ucmerced.edu/ytsai/CVPR18/GTA2Cityscapes_vgg-ac4ac9f6.pth' 36 | RESTORE_FROM_ORC = 'http://vllab1.ucmerced.edu/~whung/adaptSeg/cityscapes_oracle-b7b9934.pth' 37 | SET = 'val' 38 | 39 | MODEL = 'DeeplabMulti' 40 | 41 | palette = [128, 64, 128, 244, 35, 232, 70, 70, 70, 102, 102, 156, 190, 153, 153, 153, 153, 153, 250, 170, 30, 42 | 220, 220, 0, 107, 142, 35, 152, 251, 152, 70, 130, 180, 220, 20, 60, 255, 0, 0, 0, 0, 142, 0, 0, 70, 43 | 0, 60, 100, 0, 80, 100, 0, 0, 230, 119, 11, 32] 44 | zero_pad = 256 * 3 - len(palette) 45 | for i in range(zero_pad): 46 | palette.append(0) 47 | 48 | 49 | def colorize_mask(mask): 50 | # mask: numpy array of the mask 51 | new_mask = Image.fromarray(mask.astype(np.uint8)).convert('P') 52 | new_mask.putpalette(palette) 53 | 54 | return new_mask 55 | 56 | def get_arguments(): 57 | """Parse all the arguments provided from the CLI. 58 | 59 | Returns: 60 | A list of parsed arguments. 61 | """ 62 | parser = argparse.ArgumentParser(description="DeepLab-ResNet Network") 63 | parser.add_argument("--model", type=str, default=MODEL, 64 | help="Model Choice (DeeplabMulti/DeeplabVGG/Oracle).") 65 | parser.add_argument("--data-dir", type=str, default=DATA_DIRECTORY, 66 | help="Path to the directory containing the Cityscapes dataset.") 67 | parser.add_argument("--data-list", type=str, default=DATA_LIST_PATH, 68 | help="Path to the file listing the images in the dataset.") 69 | parser.add_argument("--ignore-label", type=int, default=IGNORE_LABEL, 70 | help="The index of the label to ignore during the training.") 71 | parser.add_argument("--num-classes", type=int, default=NUM_CLASSES, 72 | help="Number of classes to predict (including background).") 73 | parser.add_argument("--restore-from", type=str, default=RESTORE_FROM, 74 | help="Where restore model parameters from.") 75 | parser.add_argument("--gpu", type=int, default=0, 76 | help="choose gpu device.") 77 | parser.add_argument("--batchsize", type=int, default=10, 78 | help="choose gpu device.") 79 | parser.add_argument("--set", type=str, default=SET, 80 | help="choose evaluation set.") 81 | parser.add_argument("--save", type=str, default=SAVE_PATH, 82 | help="Path to save result.") 83 | return parser.parse_args() 84 | 85 | 86 | def main(): 87 | """Create the model and start the evaluation process.""" 88 | args = get_arguments() 89 | 90 | config_path = os.path.join(os.path.dirname(args.restore_from),'opts.yaml') 91 | with open(config_path, 'r') as stream: 92 | config = yaml.safe_load(stream) 93 | 94 | if not 'use_blur' in config: 95 | config['use_blur'] = False 96 | 97 | args.model = config['model'] 98 | print('ModelType:%s NormType:%s'% (args.model, config['norm_style'])) 99 | gpu0 = args.gpu 100 | batchsize = args.batchsize 101 | 102 | model_name = os.path.basename( os.path.dirname(args.restore_from) ) 103 | args.save += model_name 104 | 105 | if not os.path.exists(args.save): 106 | os.makedirs(args.save) 107 | os.makedirs(args.save+'_a') 108 | os.makedirs(args.save+'_p') 109 | 110 | if args.model == 'DeepLab': 111 | model = DeeplabMulti(num_classes=args.num_classes, use_se = config['use_se'], train_bn = False, norm_style = config['norm_style'], use_blur = config['use_blur']) 112 | elif args.model == 'Oracle': 113 | model = Res_Deeplab(num_classes=args.num_classes) 114 | if args.restore_from == RESTORE_FROM: 115 | args.restore_from = RESTORE_FROM_ORC 116 | elif args.model == 'DeepVGG': 117 | model = DeeplabVGG(num_classes=args.num_classes) 118 | #if args.restore_from == RESTORE_FROM: 119 | # args.restore_from = RESTORE_FROM_VGG 120 | 121 | if args.restore_from[:4] == 'http' : 122 | saved_state_dict = model_zoo.load_url(args.restore_from) 123 | else: 124 | saved_state_dict = torch.load(args.restore_from) 125 | 126 | try: 127 | model.load_state_dict(saved_state_dict) 128 | print('single GPU model') 129 | model = torch.nn.DataParallel(model) 130 | except: 131 | model = torch.nn.DataParallel(model) 132 | print('multiple GPU model') 133 | model.load_state_dict(saved_state_dict) 134 | model.eval() 135 | model.cuda(gpu0) 136 | 137 | testloader = data.DataLoader(GTA5DataSet(args.data_dir, args.data_list, crop_size=(640, 1280), resize_size=(1280, 640), mean=IMG_MEAN, scale=False, mirror=False), 138 | batch_size=batchsize, shuffle=False, pin_memory=True) 139 | 140 | 141 | if version.parse(torch.__version__) >= version.parse('0.4.0'): 142 | interp = nn.Upsample(size=(640, 1280 ), mode='bilinear', align_corners=True) 143 | else: 144 | interp = nn.Upsample(size=(640, 1280 ), mode='bilinear') 145 | 146 | sm = torch.nn.Softmax(dim = 1) 147 | for index, batch in enumerate(testloader): 148 | if (index*batchsize) % 100 == 0: 149 | print('%d processd' % (index*batchsize)) 150 | image, _, _, name = batch 151 | #print(image.shape) 152 | 153 | inputs = Variable(image).cuda() 154 | if args.model == 'DeepLab' or args.model == 'DeepVGG': 155 | with torch.no_grad(): 156 | output1, output2 = model(inputs) 157 | output_batch = interp(sm(0.5* output1 + output2)).cpu().data.numpy() 158 | #output1, output2 = model(fliplr(inputs)) 159 | #output2 = fliplr(output2) 160 | #output_batch += interp(output2).cpu().data.numpy() 161 | elif args.model == 'Oracle': 162 | output_batch = model(Variable(image).cuda()) 163 | output_batch = interp(output_batch).cpu().data.numpy() 164 | 165 | output_batch = output_batch.transpose(0,2,3,1) 166 | output_batch = np.asarray(np.argmax(output_batch, axis=3), dtype=np.uint8) 167 | 168 | for i in range(output_batch.shape[0]): 169 | output = output_batch[i,:,:] 170 | output_col = colorize_mask(output) 171 | output = Image.fromarray(output) 172 | 173 | name_tmp = name[i].split('/')[-1] 174 | output.save('%s/%s' % (args.save, name_tmp)) 175 | output_col.save('%s/%s_color.png' % (args.save, name_tmp.split('.')[0])) 176 | 177 | return args.save 178 | 179 | if __name__ == '__main__': 180 | with torch.no_grad(): 181 | save_path = main() 182 | devkit_path='dataset/gta5_list' 183 | os.system('python compute_iou_train.py ./data/GTA5/labels/ %s --devkit_dir %s'%(save_path, devkit_path)) 184 | -------------------------------------------------------------------------------- /evaluate_gta5_folder.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import argparse 3 | import os 4 | parser = argparse.ArgumentParser(description='Test') 5 | parser.add_argument('--name', default='SWA_MRNet', type=str, help='save model path') 6 | parser.add_argument('--average', action='store_true', help='using average model') 7 | opt = parser.parse_args() 8 | result = './snapshots/%s/result.txt'%opt.name 9 | 10 | for iter in np.arange(1,11): 11 | if opt.average: 12 | os.system('echo %d+%s | tee -a %s'%(iter*10000, 'average', result)) 13 | os.system('python evaluate_gta5.py --restore ./snapshots/%s/GTA5_%d_average.pth --batchsize 6 | tee -a %s'%(opt.name,iter*10000, result)) 14 | else: 15 | os.system('echo %d | tee -a %s'%(iter*10000, result)) 16 | os.system('python evaluate_gta5.py --restore ./snapshots/%s/GTA5_%d.pth --batchsize 6 | tee -a %s'%(opt.name,iter*10000, result)) 17 | -------------------------------------------------------------------------------- /evaluate_robot_train.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import scipy 3 | from scipy import ndimage 4 | import numpy as np 5 | import sys 6 | from packaging import version 7 | from multiprocessing import Pool 8 | import torch 9 | from torch.autograd import Variable 10 | import torchvision.models as models 11 | import torch.nn.functional as F 12 | from torch.utils import data, model_zoo 13 | from model.deeplab import Res_Deeplab 14 | from model.deeplab_multi import DeeplabMulti 15 | from model.deeplab_vgg import DeeplabVGG 16 | from dataset.robot_dataset import robotDataSet 17 | from collections import OrderedDict 18 | import os 19 | from PIL import Image 20 | from utils.tool import fliplr 21 | import matplotlib.pyplot as plt 22 | import torch.nn as nn 23 | import yaml 24 | import time 25 | 26 | torch.backends.cudnn.benchmark=True 27 | 28 | IMG_MEAN = np.array((104.00698793,116.66876762,122.67891434), dtype=np.float32) 29 | 30 | DATA_DIRECTORY = './data/Oxford_Robot_ICCV19' 31 | DATA_LIST_PATH = './dataset/robot_list/train.txt' 32 | SAVE_PATH = './result/robot' 33 | 34 | IGNORE_LABEL = 255 35 | NUM_CLASSES = 9 36 | NUM_STEPS = 894 # Number of images in the validation set. 37 | RESTORE_FROM = 'http://vllab.ucmerced.edu/ytsai/CVPR18/GTA2Cityscapes_multi-ed35151c.pth' 38 | RESTORE_FROM_VGG = 'http://vllab.ucmerced.edu/ytsai/CVPR18/GTA2Cityscapes_vgg-ac4ac9f6.pth' 39 | RESTORE_FROM_ORC = 'http://vllab1.ucmerced.edu/~whung/adaptSeg/cityscapes_oracle-b7b9934.pth' 40 | SET = 'train' 41 | 42 | MODEL = 'DeeplabMulti' 43 | 44 | palette = [ 45 | [70,130,180], 46 | [220,20,60], 47 | [119,11,32], 48 | [0,0,142], 49 | [220,220,0], 50 | [250,170,30], 51 | [70,70,70], 52 | [244,35,232], 53 | [128,64,128], 54 | ] 55 | palette = [item for sublist in palette for item in sublist] 56 | zero_pad = 256 * 3 - len(palette) 57 | for i in range(zero_pad): 58 | palette.append(0) 59 | 60 | def colorize_mask(mask): 61 | # mask: numpy array of the mask 62 | new_mask = Image.fromarray(mask.astype(np.uint8)).convert('P') 63 | new_mask.putpalette(palette) 64 | 65 | return new_mask 66 | 67 | def get_arguments(): 68 | """Parse all the arguments provided from the CLI. 69 | 70 | Returns: 71 | A list of parsed arguments. 72 | """ 73 | parser = argparse.ArgumentParser(description="DeepLab-ResNet Network") 74 | parser.add_argument("--model", type=str, default=MODEL, 75 | help="Model Choice (DeeplabMulti/DeeplabVGG/Oracle).") 76 | parser.add_argument("--data-dir", type=str, default=DATA_DIRECTORY, 77 | help="Path to the directory containing the Cityscapes dataset.") 78 | parser.add_argument("--data-list", type=str, default=DATA_LIST_PATH, 79 | help="Path to the file listing the images in the dataset.") 80 | parser.add_argument("--ignore-label", type=int, default=IGNORE_LABEL, 81 | help="The index of the label to ignore during the training.") 82 | parser.add_argument("--num-classes", type=int, default=NUM_CLASSES, 83 | help="Number of classes to predict (including background).") 84 | parser.add_argument("--restore-from", type=str, default=RESTORE_FROM, 85 | help="Where restore model parameters from.") 86 | parser.add_argument("--gpu", type=int, default=0, 87 | help="choose gpu device.") 88 | parser.add_argument("--batchsize", type=int, default=12, 89 | help="choose gpu device.") 90 | parser.add_argument("--set", type=str, default=SET, 91 | help="choose evaluation set.") 92 | parser.add_argument("--save", type=str, default=SAVE_PATH, 93 | help="Path to save result.") 94 | return parser.parse_args() 95 | 96 | def save(output_name): 97 | output, name = output_name 98 | output_col = colorize_mask(output) 99 | output = Image.fromarray(output) 100 | 101 | output.save('%s' % (name)) 102 | output_col.save('%s_color.png' % (name.split('.jpg')[0])) 103 | return 104 | 105 | 106 | def main(): 107 | """Create the model and start the evaluation process.""" 108 | args = get_arguments() 109 | 110 | config_path = os.path.join(os.path.dirname(args.restore_from),'opts.yaml') 111 | with open(config_path, 'r') as stream: 112 | config = yaml.load(stream) 113 | 114 | args.model = config['model'] 115 | print('ModelType:%s'%args.model) 116 | print('NormType:%s'%config['norm_style']) 117 | gpu0 = args.gpu 118 | batchsize = args.batchsize 119 | 120 | model_name = os.path.basename( os.path.dirname(args.restore_from) ) 121 | args.save += model_name 122 | 123 | if not os.path.exists(args.save): 124 | os.makedirs(args.save) 125 | 126 | if args.model == 'DeepLab': 127 | model = DeeplabMulti(num_classes=args.num_classes, use_se = config['use_se'], train_bn = False, norm_style = config['norm_style']) 128 | elif args.model == 'Oracle': 129 | model = Res_Deeplab(num_classes=args.num_classes) 130 | if args.restore_from == RESTORE_FROM: 131 | args.restore_from = RESTORE_FROM_ORC 132 | elif args.model == 'DeeplabVGG': 133 | model = DeeplabVGG(num_classes=args.num_classes) 134 | if args.restore_from == RESTORE_FROM: 135 | args.restore_from = RESTORE_FROM_VGG 136 | 137 | if args.restore_from[:4] == 'http' : 138 | saved_state_dict = model_zoo.load_url(args.restore_from) 139 | else: 140 | saved_state_dict = torch.load(args.restore_from) 141 | 142 | try: 143 | model.load_state_dict(saved_state_dict) 144 | except: 145 | model = torch.nn.DataParallel(model) 146 | model.load_state_dict(saved_state_dict) 147 | model = torch.nn.DataParallel(model) 148 | model.eval() 149 | model.cuda(gpu0) 150 | 151 | th = 960 152 | tw = 1280 153 | 154 | testloader = data.DataLoader(robotDataSet(args.data_dir, args.data_list, crop_size=(th, tw), resize_size=(tw, th), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 155 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 156 | 157 | scale = 0.8 158 | testloader2 = data.DataLoader(robotDataSet(args.data_dir, args.data_list, crop_size=(round(th*scale), round(tw*scale) ), resize_size=( round(tw*scale), round(th*scale)), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 159 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 160 | scale = 0.9 161 | testloader3 = data.DataLoader(robotDataSet(args.data_dir, args.data_list, crop_size=(round(th*scale), round(tw*scale) ), resize_size=( round(tw*scale), round(th*scale)), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 162 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 163 | 164 | 165 | if version.parse(torch.__version__) >= version.parse('0.4.0'): 166 | interp = nn.Upsample(size=(960, 1280), mode='bilinear', align_corners=True) 167 | else: 168 | interp = nn.Upsample(size=(960, 1280), mode='bilinear') 169 | 170 | sm = torch.nn.Softmax(dim = 1) 171 | for index, img_data in enumerate(zip(testloader, testloader2, testloader3) ): 172 | batch, batch2, batch3 = img_data 173 | image, _, _, name = batch 174 | image2, _, _, name2 = batch2 175 | image3, _, _, name3 = batch3 176 | 177 | inputs = image.cuda() 178 | inputs2 = image2.cuda() 179 | inputs3 = Variable(image3).cuda() 180 | print('\r>>>>Extracting feature...%03d/%03d'%(index*batchsize, NUM_STEPS), end='') 181 | if args.model == 'DeepLab': 182 | with torch.no_grad(): 183 | output1, output2 = model(inputs) 184 | output_batch = interp(sm(0.5* output1 + output2)) 185 | #output_batch = interp(sm(output1)) 186 | #output_batch = interp(sm(output2)) 187 | output1, output2 = model(fliplr(inputs)) 188 | output1, output2 = fliplr(output1), fliplr(output2) 189 | output_batch += interp(sm(0.5 * output1 + output2)) 190 | #output_batch += interp(sm(output1)) 191 | #output_batch += interp(sm(output2)) 192 | del output1, output2, inputs 193 | 194 | output1, output2 = model(inputs2) 195 | output_batch += interp(sm(0.5* output1 + output2)) 196 | #output_batch += interp(sm(output1)) 197 | #output_batch += interp(sm(output2)) 198 | output1, output2 = model(fliplr(inputs2)) 199 | output1, output2 = fliplr(output1), fliplr(output2) 200 | output_batch += interp(sm(0.5 * output1 + output2)) 201 | #output_batch += interp(sm(output1)) 202 | #output_batch += interp(sm(output2)) 203 | del output1, output2, inputs2 204 | 205 | #output1, output2 = model(inputs3) 206 | #output_batch += interp(sm(0.5* output1 + output2)) 207 | #output1, output2 = model(fliplr(inputs3)) 208 | #output1, output2 = fliplr(output1), fliplr(output2) 209 | #output_batch += interp(sm(0.5 * output1 + output2)) 210 | #del output1, output2, inputs3 211 | 212 | output_batch = output_batch.cpu().data.numpy() 213 | elif args.model == 'DeeplabVGG' or args.model == 'Oracle': 214 | output_batch = model(Variable(image).cuda()) 215 | output_batch = interp(output_batch).cpu().data.numpy() 216 | 217 | output_batch = output_batch.transpose(0,2,3,1) 218 | output_batch = np.asarray(np.argmax(output_batch, axis=3), dtype=np.uint8) 219 | output_iterator = [] 220 | for i in range(output_batch.shape[0]): 221 | output_iterator.append(output_batch[i,:,:]) 222 | name_tmp = name[i].split('/')[-1] 223 | name[i] = '%s/%s' % (args.save, name_tmp) 224 | with Pool(4) as p: 225 | p.map(save, zip(output_iterator, name) ) 226 | del output_batch 227 | 228 | 229 | return args.save 230 | 231 | if __name__ == '__main__': 232 | tt = time.time() 233 | with torch.no_grad(): 234 | save_path = main() 235 | print('Time used: {} sec'.format(time.time()-tt)) 236 | devkit_path='dataset/robot_list' 237 | os.system('python compute_iou_train.py ./data/Oxford_Robot_ICCV19/anno %s --devkit_dir %s'%(save_path, devkit_path)) 238 | -------------------------------------------------------------------------------- /focalloss.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import torch.nn.functional as F 4 | 5 | class FocalLoss(nn.Module): 6 | 7 | def __init__(self, weight=None, 8 | gamma=2., reduction='mean', ignore_index = 255): 9 | nn.Module.__init__(self) 10 | self.weight = weight 11 | self.gamma = gamma 12 | self.reduction = reduction 13 | self.ignore_index = ignore_index 14 | 15 | def forward(self, input_tensor, target_tensor): 16 | log_prob = F.log_softmax(input_tensor, dim=1) 17 | prob = torch.exp(log_prob) 18 | return F.nll_loss( 19 | ((1 - prob) ** self.gamma) * log_prob, 20 | target_tensor, 21 | weight=self.weight, 22 | reduction = self.reduction, 23 | ignore_index = self.ignore_index 24 | ) 25 | -------------------------------------------------------------------------------- /generate_plabel_cityscapes.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import scipy 3 | from scipy import ndimage 4 | import numpy as np 5 | import sys 6 | import re 7 | from packaging import version 8 | 9 | import torch 10 | from torch.autograd import Variable 11 | import torchvision.models as models 12 | import torch.nn.functional as F 13 | from torch.utils import data, model_zoo 14 | from model.deeplab import Res_Deeplab 15 | from model.deeplab_multi import DeeplabMulti 16 | from model.deeplab_vgg import DeeplabVGG 17 | from dataset.cityscapes_dataset import cityscapesDataSet 18 | from collections import OrderedDict 19 | import os 20 | from PIL import Image 21 | from utils.tool import fliplr 22 | import matplotlib.pyplot as plt 23 | import torch.nn as nn 24 | import yaml 25 | 26 | torch.backends.cudnn.benchmark=True 27 | 28 | IMG_MEAN = np.array((104.00698793,116.66876762,122.67891434), dtype=np.float32) 29 | 30 | DATA_DIRECTORY = './data/Cityscapes/data' 31 | DATA_LIST_PATH = './dataset/cityscapes_list/train.txt' 32 | SAVE_PATH = './data/Cityscapes/data/pseudo/train' 33 | 34 | if not os.path.isdir('./data/Cityscapes/data/pseudo/'): 35 | os.mkdir('./data/Cityscapes/data/pseudo/') 36 | os.mkdir(SAVE_PATH) 37 | 38 | IGNORE_LABEL = 255 39 | NUM_CLASSES = 19 40 | NUM_STEPS = 2975 # Number of images in the validation set. 41 | RESTORE_FROM = 'http://vllab.ucmerced.edu/ytsai/CVPR18/GTA2Cityscapes_multi-ed35151c.pth' 42 | RESTORE_FROM_VGG = 'http://vllab.ucmerced.edu/ytsai/CVPR18/GTA2Cityscapes_vgg-ac4ac9f6.pth' 43 | RESTORE_FROM_ORC = 'http://vllab1.ucmerced.edu/~whung/adaptSeg/cityscapes_oracle-b7b9934.pth' 44 | SET = 'train' # We generate pseudo label for training set 45 | 46 | MODEL = 'DeeplabMulti' 47 | 48 | palette = [128, 64, 128, 244, 35, 232, 70, 70, 70, 102, 102, 156, 190, 153, 153, 153, 153, 153, 250, 170, 30, 49 | 220, 220, 0, 107, 142, 35, 152, 251, 152, 70, 130, 180, 220, 20, 60, 255, 0, 0, 0, 0, 142, 0, 0, 70, 50 | 0, 60, 100, 0, 80, 100, 0, 0, 230, 119, 11, 32] 51 | zero_pad = 256 * 3 - len(palette) 52 | for i in range(zero_pad): 53 | palette.append(0) 54 | 55 | 56 | def colorize_mask(mask): 57 | # mask: numpy array of the mask 58 | new_mask = Image.fromarray(mask.astype(np.uint8)).convert('P') 59 | new_mask.putpalette(palette) 60 | 61 | return new_mask 62 | 63 | def get_arguments(): 64 | """Parse all the arguments provided from the CLI. 65 | 66 | Returns: 67 | A list of parsed arguments. 68 | """ 69 | parser = argparse.ArgumentParser(description="DeepLab-ResNet Network") 70 | parser.add_argument("--model", type=str, default=MODEL, 71 | help="Model Choice (DeeplabMulti/DeeplabVGG/Oracle).") 72 | parser.add_argument("--data-dir", type=str, default=DATA_DIRECTORY, 73 | help="Path to the directory containing the Cityscapes dataset.") 74 | parser.add_argument("--data-list", type=str, default=DATA_LIST_PATH, 75 | help="Path to the file listing the images in the dataset.") 76 | parser.add_argument("--ignore-label", type=int, default=IGNORE_LABEL, 77 | help="The index of the label to ignore during the training.") 78 | parser.add_argument("--num-classes", type=int, default=NUM_CLASSES, 79 | help="Number of classes to predict (including background).") 80 | parser.add_argument("--restore-from", type=str, default=RESTORE_FROM, 81 | help="Where restore model parameters from.") 82 | parser.add_argument("--gpu", type=int, default=0, 83 | help="choose gpu device.") 84 | parser.add_argument("--batchsize", type=int, default=12, 85 | help="choose gpu device.") 86 | parser.add_argument("--set", type=str, default=SET, 87 | help="choose evaluation set.") 88 | parser.add_argument("--save", type=str, default=SAVE_PATH, 89 | help="Path to save result.") 90 | return parser.parse_args() 91 | 92 | def save_heatmap(output_name): 93 | output, name = output_name 94 | fig = plt.figure() 95 | plt.axis('off') 96 | heatmap = plt.imshow(output, cmap='viridis') 97 | fig.colorbar(heatmap) 98 | fig.savefig('%s_heatmap.png' % (name.split('.jpg')[0])) 99 | return 100 | 101 | def main(): 102 | """Create the model and start the evaluation process.""" 103 | 104 | args = get_arguments() 105 | 106 | config_path = os.path.join(os.path.dirname(args.restore_from),'opts.yaml') 107 | with open(config_path, 'r') as stream: 108 | config = yaml.load(stream) 109 | 110 | args.model = config['model'] 111 | print('ModelType:%s'%args.model) 112 | print('NormType:%s'%config['norm_style']) 113 | gpu0 = args.gpu 114 | batchsize = args.batchsize 115 | 116 | model_name = os.path.basename( os.path.dirname(args.restore_from) ) 117 | #args.save += model_name 118 | 119 | if not os.path.exists(args.save): 120 | os.makedirs(args.save) 121 | 122 | if args.model == 'DeepLab': 123 | model = DeeplabMulti(num_classes=args.num_classes, use_se = config['use_se'], train_bn = False, norm_style = config['norm_style']) 124 | elif args.model == 'Oracle': 125 | model = Res_Deeplab(num_classes=args.num_classes) 126 | if args.restore_from == RESTORE_FROM: 127 | args.restore_from = RESTORE_FROM_ORC 128 | elif args.model == 'DeeplabVGG': 129 | model = DeeplabVGG(num_classes=args.num_classes) 130 | if args.restore_from == RESTORE_FROM: 131 | args.restore_from = RESTORE_FROM_VGG 132 | 133 | if args.restore_from[:4] == 'http' : 134 | saved_state_dict = model_zoo.load_url(args.restore_from) 135 | else: 136 | saved_state_dict = torch.load(args.restore_from) 137 | 138 | try: 139 | model.load_state_dict(saved_state_dict) 140 | print('single GPU model') 141 | model = torch.nn.DataParallel(model) 142 | except: 143 | model = torch.nn.DataParallel(model) 144 | print('multiple GPU model') 145 | model.load_state_dict(saved_state_dict) 146 | model.eval() 147 | model.cuda(gpu0) 148 | 149 | testloader = data.DataLoader(cityscapesDataSet(args.data_dir, args.data_list, crop_size=(512, 1024), resize_size=(1024, 512), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 150 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 151 | 152 | scale = 1.25 153 | testloader2 = data.DataLoader(cityscapesDataSet(args.data_dir, args.data_list, crop_size=(round(512*scale), round(1024*scale) ), resize_size=( round(1024*scale), round(512*scale)), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 154 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 155 | 156 | 157 | if version.parse(torch.__version__) >= version.parse('0.4.0'): 158 | interp = nn.Upsample(size=(1024, 2048), mode='bilinear', align_corners=True) 159 | else: 160 | interp = nn.Upsample(size=(1024, 2048), mode='bilinear') 161 | 162 | sm = torch.nn.Softmax(dim = 1) 163 | log_sm = torch.nn.LogSoftmax(dim = 1) 164 | kl_distance = nn.KLDivLoss( reduction = 'none') 165 | 166 | for index, img_data in enumerate(zip(testloader, testloader2) ): 167 | batch, batch2 = img_data 168 | image, _, _, name = batch 169 | image2, _, _, name2 = batch2 170 | print(image.shape) 171 | 172 | inputs = image.cuda() 173 | inputs2 = image2.cuda() 174 | print('\r>>>>Extracting feature...%04d/%04d'%(index*batchsize, NUM_STEPS), end='') 175 | if args.model == 'DeepLab': 176 | with torch.no_grad(): 177 | output1, output2 = model(inputs) 178 | output_batch = interp(sm(0.5* output1 + output2)) 179 | 180 | heatmap_batch = torch.sum(kl_distance(log_sm(output1), sm(output2)), dim=1) 181 | 182 | output1, output2 = model(fliplr(inputs)) 183 | output1, output2 = fliplr(output1), fliplr(output2) 184 | output_batch += interp(sm(0.5 * output1 + output2)) 185 | del output1, output2, inputs 186 | 187 | output1, output2 = model(inputs2) 188 | output_batch += interp(sm(0.5* output1 + output2)) 189 | output1, output2 = model(fliplr(inputs2)) 190 | output1, output2 = fliplr(output1), fliplr(output2) 191 | output_batch += interp(sm(0.5 * output1 + output2)) 192 | del output1, output2, inputs2 193 | output_batch = sm(output_batch) #new add 194 | output_batch = output_batch.cpu().data.numpy() 195 | heatmap_batch = heatmap_batch.cpu().data.numpy() 196 | elif args.model == 'DeeplabVGG' or args.model == 'Oracle': 197 | output_batch = model(Variable(image).cuda()) 198 | output_batch = interp(output_batch).cpu().data.numpy() 199 | 200 | #output_batch = output_batch.transpose(0,2,3,1) 201 | #output_batch = np.asarray(np.argmax(output_batch, axis=3), dtype=np.uint8) 202 | output_batch = output_batch.transpose(0,2,3,1) 203 | score_batch = np.max(output_batch, axis=3) 204 | score_batch = np.asarray(np.round(score_batch*100), dtype=np.uint8) 205 | output_batch = np.asarray(np.argmax(output_batch, axis=3), dtype=np.uint8) 206 | #output_batch[score_batch<3.2] = 255 #3.2 = 4*0.8 207 | for i in range(output_batch.shape[0]): 208 | output = output_batch[i,:,:] 209 | score = score_batch[i,:,:] 210 | output_col = colorize_mask(output) 211 | output = Image.fromarray(output) 212 | score = Image.fromarray(score) 213 | 214 | name_tmp = name[i].split('/')[-1] 215 | dir_name = name[i].split('/')[-2] 216 | save_path = args.save + '/' + dir_name 217 | #save_path = re.replace(save_path, 'leftImg8bit', 'pseudo') 218 | #print(save_path) 219 | if not os.path.isdir(save_path): 220 | os.mkdir(save_path) 221 | output.save('%s/%s' % (save_path, name_tmp)) 222 | score.save('%s/%s_score.png' % (save_path, name_tmp)) 223 | print('%s/%s' % (save_path, name_tmp)) 224 | output_col.save('%s/%s_color.png' % (save_path, name_tmp.split('.')[0])) 225 | 226 | heatmap_tmp = heatmap_batch[i,:,:]/np.max(heatmap_batch[i,:,:]) 227 | fig = plt.figure() 228 | plt.axis('off') 229 | heatmap = plt.imshow(heatmap_tmp, cmap='viridis') 230 | fig.colorbar(heatmap) 231 | fig.savefig('%s/%s_heatmap.png' % (save_path, name_tmp.split('.')[0])) 232 | 233 | return args.save 234 | 235 | if __name__ == '__main__': 236 | with torch.no_grad(): 237 | save_path = main() 238 | #os.system('python compute_iou.py ./data/Cityscapes/data/gtFine/train %s'%save_path) 239 | -------------------------------------------------------------------------------- /generate_plabel_cityscapes_99.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import scipy 3 | from scipy import ndimage 4 | import numpy as np 5 | import sys 6 | import re 7 | from packaging import version 8 | 9 | import torch 10 | from torch.autograd import Variable 11 | import torchvision.models as models 12 | import torch.nn.functional as F 13 | from torch.utils import data, model_zoo 14 | from model.deeplab import Res_Deeplab 15 | from model.deeplab_multi import DeeplabMulti 16 | from model.deeplab_vgg import DeeplabVGG 17 | from dataset.cityscapes_dataset import cityscapesDataSet 18 | from collections import OrderedDict 19 | import os 20 | from PIL import Image 21 | from utils.tool import fliplr 22 | import matplotlib.pyplot as plt 23 | import torch.nn as nn 24 | import yaml 25 | 26 | torch.backends.cudnn.benchmark=True 27 | 28 | IMG_MEAN = np.array((104.00698793,116.66876762,122.67891434), dtype=np.float32) 29 | 30 | DATA_DIRECTORY = './data/Cityscapes/data' 31 | DATA_LIST_PATH = './dataset/cityscapes_list/train.txt' 32 | SAVE_PATH = './data/Cityscapes/data/pseudo_9.5/train' 33 | 34 | if not os.path.isdir('./data/Cityscapes/data/pseudo_9.5/'): 35 | os.mkdir('./data/Cityscapes/data/pseudo_9.5/') 36 | os.mkdir(SAVE_PATH) 37 | 38 | IGNORE_LABEL = 255 39 | NUM_CLASSES = 19 40 | NUM_STEPS = 2975 # Number of images in the validation set. 41 | RESTORE_FROM = 'http://vllab.ucmerced.edu/ytsai/CVPR18/GTA2Cityscapes_multi-ed35151c.pth' 42 | RESTORE_FROM_VGG = 'http://vllab.ucmerced.edu/ytsai/CVPR18/GTA2Cityscapes_vgg-ac4ac9f6.pth' 43 | RESTORE_FROM_ORC = 'http://vllab1.ucmerced.edu/~whung/adaptSeg/cityscapes_oracle-b7b9934.pth' 44 | SET = 'train' # We generate pseudo label for training set 45 | 46 | MODEL = 'DeeplabMulti' 47 | 48 | palette = [128, 64, 128, 244, 35, 232, 70, 70, 70, 102, 102, 156, 190, 153, 153, 153, 153, 153, 250, 170, 30, 49 | 220, 220, 0, 107, 142, 35, 152, 251, 152, 70, 130, 180, 220, 20, 60, 255, 0, 0, 0, 0, 142, 0, 0, 70, 50 | 0, 60, 100, 0, 80, 100, 0, 0, 230, 119, 11, 32] 51 | zero_pad = 256 * 3 - len(palette) 52 | for i in range(zero_pad): 53 | palette.append(0) 54 | 55 | 56 | def colorize_mask(mask): 57 | # mask: numpy array of the mask 58 | new_mask = Image.fromarray(mask.astype(np.uint8)).convert('P') 59 | new_mask.putpalette(palette) 60 | 61 | return new_mask 62 | 63 | def get_arguments(): 64 | """Parse all the arguments provided from the CLI. 65 | 66 | Returns: 67 | A list of parsed arguments. 68 | """ 69 | parser = argparse.ArgumentParser(description="DeepLab-ResNet Network") 70 | parser.add_argument("--model", type=str, default=MODEL, 71 | help="Model Choice (DeeplabMulti/DeeplabVGG/Oracle).") 72 | parser.add_argument("--data-dir", type=str, default=DATA_DIRECTORY, 73 | help="Path to the directory containing the Cityscapes dataset.") 74 | parser.add_argument("--data-list", type=str, default=DATA_LIST_PATH, 75 | help="Path to the file listing the images in the dataset.") 76 | parser.add_argument("--ignore-label", type=int, default=IGNORE_LABEL, 77 | help="The index of the label to ignore during the training.") 78 | parser.add_argument("--num-classes", type=int, default=NUM_CLASSES, 79 | help="Number of classes to predict (including background).") 80 | parser.add_argument("--restore-from", type=str, default=RESTORE_FROM, 81 | help="Where restore model parameters from.") 82 | parser.add_argument("--gpu", type=int, default=0, 83 | help="choose gpu device.") 84 | parser.add_argument("--batchsize", type=int, default=12, 85 | help="choose gpu device.") 86 | parser.add_argument("--set", type=str, default=SET, 87 | help="choose evaluation set.") 88 | parser.add_argument("--save", type=str, default=SAVE_PATH, 89 | help="Path to save result.") 90 | return parser.parse_args() 91 | 92 | def save_heatmap(output_name): 93 | output, name = output_name 94 | fig = plt.figure() 95 | plt.axis('off') 96 | heatmap = plt.imshow(output, cmap='viridis') 97 | fig.colorbar(heatmap) 98 | fig.savefig('%s_heatmap.png' % (name.split('.jpg')[0])) 99 | return 100 | 101 | def main(): 102 | """Create the model and start the evaluation process.""" 103 | 104 | args = get_arguments() 105 | 106 | config_path = os.path.join(os.path.dirname(args.restore_from),'opts.yaml') 107 | with open(config_path, 'r') as stream: 108 | config = yaml.load(stream) 109 | 110 | args.model = config['model'] 111 | print('ModelType:%s'%args.model) 112 | print('NormType:%s'%config['norm_style']) 113 | gpu0 = args.gpu 114 | batchsize = args.batchsize 115 | 116 | model_name = os.path.basename( os.path.dirname(args.restore_from) ) 117 | #args.save += model_name 118 | 119 | if not os.path.exists(args.save): 120 | os.makedirs(args.save) 121 | 122 | if args.model == 'DeepLab': 123 | model = DeeplabMulti(num_classes=args.num_classes, use_se = config['use_se'], train_bn = False, norm_style = config['norm_style']) 124 | elif args.model == 'Oracle': 125 | model = Res_Deeplab(num_classes=args.num_classes) 126 | if args.restore_from == RESTORE_FROM: 127 | args.restore_from = RESTORE_FROM_ORC 128 | elif args.model == 'DeeplabVGG': 129 | model = DeeplabVGG(num_classes=args.num_classes) 130 | if args.restore_from == RESTORE_FROM: 131 | args.restore_from = RESTORE_FROM_VGG 132 | 133 | if args.restore_from[:4] == 'http' : 134 | saved_state_dict = model_zoo.load_url(args.restore_from) 135 | else: 136 | saved_state_dict = torch.load(args.restore_from) 137 | 138 | try: 139 | model.load_state_dict(saved_state_dict) 140 | except: 141 | model = torch.nn.DataParallel(model) 142 | model.load_state_dict(saved_state_dict) 143 | model.eval() 144 | model.cuda(gpu0) 145 | 146 | testloader = data.DataLoader(cityscapesDataSet(args.data_dir, args.data_list, crop_size=(512, 1024), resize_size=(1024, 512), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 147 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 148 | 149 | scale = 1.25 150 | testloader2 = data.DataLoader(cityscapesDataSet(args.data_dir, args.data_list, crop_size=(round(512*scale), round(1024*scale) ), resize_size=( round(1024*scale), round(512*scale)), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 151 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 152 | 153 | 154 | if version.parse(torch.__version__) >= version.parse('0.4.0'): 155 | interp = nn.Upsample(size=(1024, 2048), mode='bilinear', align_corners=True) 156 | else: 157 | interp = nn.Upsample(size=(1024, 2048), mode='bilinear') 158 | 159 | sm = torch.nn.Softmax(dim = 1) 160 | log_sm = torch.nn.LogSoftmax(dim = 1) 161 | kl_distance = nn.KLDivLoss( reduction = 'none') 162 | 163 | for index, img_data in enumerate(zip(testloader, testloader2) ): 164 | batch, batch2 = img_data 165 | image, _, _, name = batch 166 | image2, _, _, name2 = batch2 167 | print(image.shape) 168 | 169 | inputs = image.cuda() 170 | inputs2 = image2.cuda() 171 | print('\r>>>>Extracting feature...%04d/%04d'%(index*batchsize, NUM_STEPS), end='') 172 | if args.model == 'DeepLab': 173 | with torch.no_grad(): 174 | output1, output2 = model(inputs) 175 | output_batch = interp(sm(0.5* output1 + output2)) 176 | 177 | heatmap_batch = torch.sum(kl_distance(log_sm(output1), sm(output2)), dim=1) 178 | 179 | output1, output2 = model(fliplr(inputs)) 180 | output1, output2 = fliplr(output1), fliplr(output2) 181 | output_batch += interp(sm(0.5 * output1 + output2)) 182 | del output1, output2, inputs 183 | 184 | output1, output2 = model(inputs2) 185 | output_batch += interp(sm(0.5* output1 + output2)) 186 | output1, output2 = model(fliplr(inputs2)) 187 | output1, output2 = fliplr(output1), fliplr(output2) 188 | output_batch += interp(sm(0.5 * output1 + output2)) 189 | del output1, output2, inputs2 190 | output_batch = output_batch.cpu().data.numpy() 191 | heatmap_batch = heatmap_batch.cpu().data.numpy() 192 | elif args.model == 'DeeplabVGG' or args.model == 'Oracle': 193 | output_batch = model(Variable(image).cuda()) 194 | output_batch = interp(output_batch).cpu().data.numpy() 195 | 196 | #output_batch = output_batch.transpose(0,2,3,1) 197 | #output_batch = np.asarray(np.argmax(output_batch, axis=3), dtype=np.uint8) 198 | output_batch = output_batch.transpose(0,2,3,1) 199 | score_batch = np.max(output_batch, axis=3) 200 | output_batch = np.asarray(np.argmax(output_batch, axis=3), dtype=np.uint8) 201 | output_batch[score_batch< 4*0.95] = 255 #3.2 = 4*0.8 202 | for i in range(output_batch.shape[0]): 203 | output = output_batch[i,:,:] 204 | output_col = colorize_mask(output) 205 | output = Image.fromarray(output) 206 | 207 | name_tmp = name[i].split('/')[-1] 208 | dir_name = name[i].split('/')[-2] 209 | save_path = args.save + '/' + dir_name 210 | #save_path = re.replace(save_path, 'leftImg8bit', 'pseudo') 211 | #print(save_path) 212 | if not os.path.isdir(save_path): 213 | os.mkdir(save_path) 214 | output.save('%s/%s' % (save_path, name_tmp)) 215 | print('%s/%s' % (save_path, name_tmp)) 216 | output_col.save('%s/%s_color.png' % (save_path, name_tmp.split('.')[0])) 217 | 218 | heatmap_tmp = heatmap_batch[i,:,:]/np.max(heatmap_batch[i,:,:]) 219 | fig = plt.figure() 220 | plt.axis('off') 221 | heatmap = plt.imshow(heatmap_tmp, cmap='viridis') 222 | fig.colorbar(heatmap) 223 | fig.savefig('%s/%s_heatmap.png' % (save_path, name_tmp.split('.')[0])) 224 | 225 | return args.save 226 | 227 | if __name__ == '__main__': 228 | with torch.no_grad(): 229 | save_path = main() 230 | #os.system('python compute_iou.py ./data/Cityscapes/data/gtFine/train %s'%save_path) 231 | -------------------------------------------------------------------------------- /generate_plabel_cityscapes_SYNTHIA.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import scipy 3 | from scipy import ndimage 4 | import numpy as np 5 | import sys 6 | import re 7 | from packaging import version 8 | 9 | import torch 10 | from torch.autograd import Variable 11 | import torchvision.models as models 12 | import torch.nn.functional as F 13 | from torch.utils import data, model_zoo 14 | from model.deeplab import Res_Deeplab 15 | from model.deeplab_multi import DeeplabMulti 16 | from model.deeplab_vgg import DeeplabVGG 17 | from dataset.cityscapes_dataset import cityscapesDataSet 18 | from collections import OrderedDict 19 | import os 20 | from PIL import Image 21 | from utils.tool import fliplr 22 | import matplotlib.pyplot as plt 23 | import torch.nn as nn 24 | import yaml 25 | 26 | torch.backends.cudnn.benchmark=True 27 | 28 | IMG_MEAN = np.array((104.00698793,116.66876762,122.67891434), dtype=np.float32) 29 | 30 | DATA_DIRECTORY = './data/Cityscapes/data' 31 | DATA_LIST_PATH = './dataset/cityscapes_list/train.txt' 32 | SAVE_PATH = './data/Cityscapes/data/pseudo_SYNTHIA/train' 33 | 34 | if not os.path.isdir('./data/Cityscapes/data/pseudo_SYNTHIA/'): 35 | os.mkdir('./data/Cityscapes/data/pseudo_SYNTHIA/') 36 | os.mkdir(SAVE_PATH) 37 | 38 | IGNORE_LABEL = 255 39 | NUM_CLASSES = 19 40 | NUM_STEPS = 2975 # Number of images in the validation set. 41 | RESTORE_FROM = 'http://vllab.ucmerced.edu/ytsai/CVPR18/GTA2Cityscapes_multi-ed35151c.pth' 42 | RESTORE_FROM_VGG = 'http://vllab.ucmerced.edu/ytsai/CVPR18/GTA2Cityscapes_vgg-ac4ac9f6.pth' 43 | RESTORE_FROM_ORC = 'http://vllab1.ucmerced.edu/~whung/adaptSeg/cityscapes_oracle-b7b9934.pth' 44 | SET = 'train' # We generate pseudo label for training set 45 | 46 | MODEL = 'DeeplabMulti' 47 | 48 | palette = [128, 64, 128, 244, 35, 232, 70, 70, 70, 102, 102, 156, 190, 153, 153, 153, 153, 153, 250, 170, 30, 49 | 220, 220, 0, 107, 142, 35, 152, 251, 152, 70, 130, 180, 220, 20, 60, 255, 0, 0, 0, 0, 142, 0, 0, 70, 50 | 0, 60, 100, 0, 80, 100, 0, 0, 230, 119, 11, 32] 51 | zero_pad = 256 * 3 - len(palette) 52 | for i in range(zero_pad): 53 | palette.append(0) 54 | 55 | 56 | def colorize_mask(mask): 57 | # mask: numpy array of the mask 58 | new_mask = Image.fromarray(mask.astype(np.uint8)).convert('P') 59 | new_mask.putpalette(palette) 60 | 61 | return new_mask 62 | 63 | def get_arguments(): 64 | """Parse all the arguments provided from the CLI. 65 | 66 | Returns: 67 | A list of parsed arguments. 68 | """ 69 | parser = argparse.ArgumentParser(description="DeepLab-ResNet Network") 70 | parser.add_argument("--model", type=str, default=MODEL, 71 | help="Model Choice (DeeplabMulti/DeeplabVGG/Oracle).") 72 | parser.add_argument("--data-dir", type=str, default=DATA_DIRECTORY, 73 | help="Path to the directory containing the Cityscapes dataset.") 74 | parser.add_argument("--data-list", type=str, default=DATA_LIST_PATH, 75 | help="Path to the file listing the images in the dataset.") 76 | parser.add_argument("--ignore-label", type=int, default=IGNORE_LABEL, 77 | help="The index of the label to ignore during the training.") 78 | parser.add_argument("--num-classes", type=int, default=NUM_CLASSES, 79 | help="Number of classes to predict (including background).") 80 | parser.add_argument("--restore-from", type=str, default=RESTORE_FROM, 81 | help="Where restore model parameters from.") 82 | parser.add_argument("--gpu", type=int, default=0, 83 | help="choose gpu device.") 84 | parser.add_argument("--batchsize", type=int, default=12, 85 | help="choose gpu device.") 86 | parser.add_argument("--set", type=str, default=SET, 87 | help="choose evaluation set.") 88 | parser.add_argument("--save", type=str, default=SAVE_PATH, 89 | help="Path to save result.") 90 | return parser.parse_args() 91 | 92 | 93 | def main(): 94 | """Create the model and start the evaluation process.""" 95 | 96 | args = get_arguments() 97 | 98 | config_path = os.path.join(os.path.dirname(args.restore_from),'opts.yaml') 99 | with open(config_path, 'r') as stream: 100 | config = yaml.load(stream) 101 | 102 | args.model = config['model'] 103 | print('ModelType:%s'%args.model) 104 | print('NormType:%s'%config['norm_style']) 105 | gpu0 = args.gpu 106 | batchsize = args.batchsize 107 | 108 | model_name = os.path.basename( os.path.dirname(args.restore_from) ) 109 | #args.save += model_name 110 | 111 | if not os.path.exists(args.save): 112 | os.makedirs(args.save) 113 | 114 | if args.model == 'DeepLab': 115 | model = DeeplabMulti(num_classes=args.num_classes, use_se = config['use_se'], train_bn = False, norm_style = config['norm_style']) 116 | elif args.model == 'Oracle': 117 | model = Res_Deeplab(num_classes=args.num_classes) 118 | if args.restore_from == RESTORE_FROM: 119 | args.restore_from = RESTORE_FROM_ORC 120 | elif args.model == 'DeeplabVGG': 121 | model = DeeplabVGG(num_classes=args.num_classes) 122 | if args.restore_from == RESTORE_FROM: 123 | args.restore_from = RESTORE_FROM_VGG 124 | 125 | if args.restore_from[:4] == 'http' : 126 | saved_state_dict = model_zoo.load_url(args.restore_from) 127 | else: 128 | saved_state_dict = torch.load(args.restore_from) 129 | 130 | try: 131 | model.load_state_dict(saved_state_dict) 132 | except: 133 | model = torch.nn.DataParallel(model) 134 | model.load_state_dict(saved_state_dict) 135 | model = torch.nn.DataParallel(model) 136 | model.eval() 137 | model.cuda(gpu0) 138 | 139 | testloader = data.DataLoader(cityscapesDataSet(args.data_dir, args.data_list, crop_size=(512, 1024), resize_size=(1024, 512), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 140 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 141 | 142 | scale = 1.25 143 | testloader2 = data.DataLoader(cityscapesDataSet(args.data_dir, args.data_list, crop_size=(round(512*scale), round(1024*scale) ), resize_size=( round(1024*scale), round(512*scale)), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 144 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 145 | 146 | 147 | if version.parse(torch.__version__) >= version.parse('0.4.0'): 148 | interp = nn.Upsample(size=(1024, 2048), mode='bilinear', align_corners=True) 149 | else: 150 | interp = nn.Upsample(size=(1024, 2048), mode='bilinear') 151 | 152 | sm = torch.nn.Softmax(dim = 1) 153 | for index, img_data in enumerate(zip(testloader, testloader2) ): 154 | batch, batch2 = img_data 155 | image, _, _, name = batch 156 | image2, _, _, name2 = batch2 157 | print(image.shape) 158 | 159 | inputs = image.cuda() 160 | inputs2 = image2.cuda() 161 | print('\r>>>>Extracting feature...%04d/%04d'%(index*batchsize, NUM_STEPS), end='') 162 | if args.model == 'DeepLab': 163 | with torch.no_grad(): 164 | output1, output2 = model(inputs) 165 | output_batch = interp(sm(0.5* output1 + output2)) 166 | output1, output2 = model(fliplr(inputs)) 167 | output1, output2 = fliplr(output1), fliplr(output2) 168 | output_batch += interp(sm(0.5 * output1 + output2)) 169 | del output1, output2, inputs 170 | 171 | output1, output2 = model(inputs2) 172 | output_batch += interp(sm(0.5* output1 + output2)) 173 | output1, output2 = model(fliplr(inputs2)) 174 | output1, output2 = fliplr(output1), fliplr(output2) 175 | output_batch += interp(sm(0.5 * output1 + output2)) 176 | del output1, output2, inputs2 177 | output_batch = sm(output_batch) #new add 178 | output_batch = output_batch.cpu().data.numpy() 179 | elif args.model == 'DeeplabVGG' or args.model == 'Oracle': 180 | output_batch = model(Variable(image).cuda()) 181 | output_batch = interp(output_batch).cpu().data.numpy() 182 | 183 | #output_batch = output_batch.transpose(0,2,3,1) 184 | #output_batch = np.asarray(np.argmax(output_batch, axis=3), dtype=np.uint8) 185 | output_batch = output_batch.transpose(0,2,3,1) 186 | score_batch = np.max(output_batch, axis=3) 187 | score_batch = np.asarray(np.round(score_batch*100), dtype=np.uint8) 188 | output_batch = np.asarray(np.argmax(output_batch, axis=3), dtype=np.uint8) 189 | output_batch[score_batch<3.6] = 255 #3.6 = 4*0.9 190 | 191 | 192 | for i in range(output_batch.shape[0]): 193 | output = output_batch[i,:,:] 194 | score = score_batch[i,:,:] 195 | output_col = colorize_mask(output) 196 | output = Image.fromarray(output) 197 | score = Image.fromarray(score) 198 | 199 | name_tmp = name[i].split('/')[-1] 200 | dir_name = name[i].split('/')[-2] 201 | save_path = args.save + '/' + dir_name 202 | #save_path = re.replace(save_path, 'leftImg8bit', 'pseudo') 203 | #print(save_path) 204 | if not os.path.isdir(save_path): 205 | os.mkdir(save_path) 206 | output.save('%s/%s' % (save_path, name_tmp)) 207 | score.save('%s/%s_score.png' % (save_path, name_tmp)) 208 | print('%s/%s' % (save_path, name_tmp)) 209 | output_col.save('%s/%s_color.png' % (save_path, name_tmp.split('.')[0])) 210 | 211 | return args.save 212 | 213 | if __name__ == '__main__': 214 | with torch.no_grad(): 215 | save_path = main() 216 | #os.system('python compute_iou.py ./data/Cityscapes/data/gtFine/train %s'%save_path) 217 | -------------------------------------------------------------------------------- /generate_plabel_robot.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import scipy 3 | from scipy import ndimage 4 | import numpy as np 5 | import sys 6 | import re 7 | from packaging import version 8 | 9 | import torch 10 | from torch.autograd import Variable 11 | import torchvision.models as models 12 | import torch.nn.functional as F 13 | from torch.utils import data, model_zoo 14 | from model.deeplab import Res_Deeplab 15 | from model.deeplab_multi import DeeplabMulti 16 | from model.deeplab_vgg import DeeplabVGG 17 | from dataset.robot_dataset import robotDataSet 18 | from collections import OrderedDict 19 | import os 20 | from PIL import Image 21 | from utils.tool import fliplr 22 | import matplotlib.pyplot as plt 23 | import torch.nn as nn 24 | import yaml 25 | 26 | torch.backends.cudnn.benchmark=True 27 | 28 | IMG_MEAN = np.array((104.00698793,116.66876762,122.67891434), dtype=np.float32) 29 | 30 | DATA_DIRECTORY = './data/Oxford_Robot_ICCV19/' 31 | DATA_LIST_PATH = './dataset/robot_list/train.txt' 32 | SAVE_PATH = './data/Oxford_Robot_ICCV19/pseudo_train' 33 | 34 | if not os.path.isdir(SAVE_PATH): 35 | os.mkdir(SAVE_PATH) 36 | 37 | IGNORE_LABEL = 255 38 | NUM_CLASSES = 9 39 | NUM_STEPS = 894 # Number of images in the validation set. 40 | RESTORE_FROM = 'http://vllab.ucmerced.edu/ytsai/CVPR18/GTA2Cityscapes_multi-ed35151c.pth' 41 | RESTORE_FROM_VGG = 'http://vllab.ucmerced.edu/ytsai/CVPR18/GTA2Cityscapes_vgg-ac4ac9f6.pth' 42 | RESTORE_FROM_ORC = 'http://vllab1.ucmerced.edu/~whung/adaptSeg/cityscapes_oracle-b7b9934.pth' 43 | SET = 'train' # We generate pseudo label for training set 44 | 45 | MODEL = 'DeeplabMulti' 46 | 47 | palette = [ 48 | [70,130,180], 49 | [220,20,60], 50 | [119,11,32], 51 | [0,0,142], 52 | [220,220,0], 53 | [250,170,30], 54 | [70,70,70], 55 | [244,35,232], 56 | [128,64,128], 57 | ] 58 | palette = [item for sublist in palette for item in sublist] 59 | 60 | zero_pad = 256 * 3 - len(palette) 61 | for i in range(zero_pad): 62 | palette.append(0) 63 | 64 | 65 | def colorize_mask(mask): 66 | # mask: numpy array of the mask 67 | new_mask = Image.fromarray(mask.astype(np.uint8)).convert('P') 68 | new_mask.putpalette(palette) 69 | 70 | return new_mask 71 | 72 | def get_arguments(): 73 | """Parse all the arguments provided from the CLI. 74 | 75 | Returns: 76 | A list of parsed arguments. 77 | """ 78 | parser = argparse.ArgumentParser(description="DeepLab-ResNet Network") 79 | parser.add_argument("--model", type=str, default=MODEL, 80 | help="Model Choice (DeeplabMulti/DeeplabVGG/Oracle).") 81 | parser.add_argument("--data-dir", type=str, default=DATA_DIRECTORY, 82 | help="Path to the directory containing the Cityscapes dataset.") 83 | parser.add_argument("--data-list", type=str, default=DATA_LIST_PATH, 84 | help="Path to the file listing the images in the dataset.") 85 | parser.add_argument("--ignore-label", type=int, default=IGNORE_LABEL, 86 | help="The index of the label to ignore during the training.") 87 | parser.add_argument("--num-classes", type=int, default=NUM_CLASSES, 88 | help="Number of classes to predict (including background).") 89 | parser.add_argument("--restore-from", type=str, default=RESTORE_FROM, 90 | help="Where restore model parameters from.") 91 | parser.add_argument("--gpu", type=int, default=0, 92 | help="choose gpu device.") 93 | parser.add_argument("--batchsize", type=int, default=12, 94 | help="choose gpu device.") 95 | parser.add_argument("--set", type=str, default=SET, 96 | help="choose evaluation set.") 97 | parser.add_argument("--save", type=str, default=SAVE_PATH, 98 | help="Path to save result.") 99 | return parser.parse_args() 100 | 101 | 102 | def main(): 103 | """Create the model and start the evaluation process.""" 104 | 105 | args = get_arguments() 106 | 107 | config_path = os.path.join(os.path.dirname(args.restore_from),'opts.yaml') 108 | with open(config_path, 'r') as stream: 109 | config = yaml.load(stream) 110 | 111 | args.model = config['model'] 112 | print('ModelType:%s'%args.model) 113 | print('NormType:%s'%config['norm_style']) 114 | gpu0 = args.gpu 115 | batchsize = args.batchsize 116 | 117 | model_name = os.path.basename( os.path.dirname(args.restore_from) ) 118 | #args.save += model_name 119 | 120 | if not os.path.exists(args.save): 121 | os.makedirs(args.save) 122 | 123 | if args.model == 'DeepLab': 124 | model = DeeplabMulti(num_classes=args.num_classes, use_se = config['use_se'], train_bn = False, norm_style = config['norm_style']) 125 | elif args.model == 'Oracle': 126 | model = Res_Deeplab(num_classes=args.num_classes) 127 | if args.restore_from == RESTORE_FROM: 128 | args.restore_from = RESTORE_FROM_ORC 129 | elif args.model == 'DeeplabVGG': 130 | model = DeeplabVGG(num_classes=args.num_classes) 131 | if args.restore_from == RESTORE_FROM: 132 | args.restore_from = RESTORE_FROM_VGG 133 | 134 | if args.restore_from[:4] == 'http' : 135 | saved_state_dict = model_zoo.load_url(args.restore_from) 136 | else: 137 | saved_state_dict = torch.load(args.restore_from) 138 | 139 | try: 140 | model.load_state_dict(saved_state_dict) 141 | print('single GPU model') 142 | model = torch.nn.DataParallel(model) 143 | except: 144 | model = torch.nn.DataParallel(model) 145 | print('multiple GPU model') 146 | model.load_state_dict(saved_state_dict) 147 | model.eval() 148 | model.cuda(gpu0) 149 | 150 | testloader = data.DataLoader(robotDataSet(args.data_dir, args.data_list, crop_size=(960, 1280), resize_size=(1280, 960), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 151 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 152 | 153 | scale = 0.8 154 | testloader2 = data.DataLoader(robotDataSet(args.data_dir, args.data_list, crop_size=(round(960*scale), round(1280*scale) ), resize_size=( round(1280*scale), round(960*scale)), mean=IMG_MEAN, scale=False, mirror=False, set=args.set), 155 | batch_size=batchsize, shuffle=False, pin_memory=True, num_workers=4) 156 | 157 | 158 | if version.parse(torch.__version__) >= version.parse('0.4.0'): 159 | interp = nn.Upsample(size=(960, 1280), mode='bilinear', align_corners=True) 160 | else: 161 | interp = nn.Upsample(size=(960, 1280), mode='bilinear') 162 | 163 | sm = torch.nn.Softmax(dim = 1) 164 | for index, img_data in enumerate(zip(testloader, testloader2) ): 165 | batch, batch2 = img_data 166 | image, _, _, name = batch 167 | image2, _, _, name2 = batch2 168 | print(image.shape) 169 | 170 | inputs = image.cuda() 171 | inputs2 = image2.cuda() 172 | print('\r>>>>Extracting feature...%04d/%04d'%(index*batchsize, NUM_STEPS), end='') 173 | if args.model == 'DeepLab': 174 | with torch.no_grad(): 175 | output1, output2 = model(inputs) 176 | output_batch = interp(sm(0.5* output1 + output2)) 177 | output1, output2 = model(fliplr(inputs)) 178 | output1, output2 = fliplr(output1), fliplr(output2) 179 | output_batch += interp(sm(0.5 * output1 + output2)) 180 | del output1, output2, inputs 181 | 182 | output1, output2 = model(inputs2) 183 | output_batch += interp(sm(0.5* output1 + output2)) 184 | output1, output2 = model(fliplr(inputs2)) 185 | output1, output2 = fliplr(output1), fliplr(output2) 186 | output_batch += interp(sm(0.5 * output1 + output2)) 187 | del output1, output2, inputs2 188 | output_batch = sm(output_batch) #new add 189 | output_batch = output_batch.cpu().data.numpy() 190 | elif args.model == 'DeeplabVGG' or args.model == 'Oracle': 191 | output_batch = model(Variable(image).cuda()) 192 | output_batch = interp(output_batch).cpu().data.numpy() 193 | 194 | output_batch = output_batch.transpose(0,2,3,1) 195 | score_batch = np.max(output_batch, axis=3) 196 | score_batch = np.asarray(np.round(score_batch*100), dtype=np.uint8) 197 | output_batch = np.asarray(np.argmax(output_batch, axis=3), dtype=np.uint8) 198 | #output_batch[score_batch<3.6] = 255 #3.6 = 4*0.9 199 | 200 | for i in range(output_batch.shape[0]): 201 | output = output_batch[i,:,:] 202 | score = score_batch[i,:,:] 203 | output_col = colorize_mask(output) 204 | output = Image.fromarray(output) 205 | score = Image.fromarray(score) 206 | 207 | name_tmp = name[i].split('/')[-1] 208 | dir_name = name[i].split('/')[-2] 209 | save_path = args.save + '/' + dir_name 210 | #save_path = re.replace(save_path, 'leftImg8bit', 'pseudo') 211 | #print(save_path) 212 | if not os.path.isdir(save_path): 213 | os.mkdir(save_path) 214 | output.save('%s/%s' % (save_path, name_tmp)) 215 | score.save('%s/%s_score.png' % (save_path, name_tmp)) 216 | #savemat('%s/%s_score.mat' % (save_path, name_tmp), {'score_fp16':score}) 217 | print('%s/%s' % (save_path, name_tmp)) 218 | output_col.save('%s/%s_color.png' % (save_path, name_tmp.split('.')[0])) 219 | 220 | return args.save 221 | 222 | if __name__ == '__main__': 223 | with torch.no_grad(): 224 | save_path = main() 225 | #os.system('python compute_iou.py ./data/Cityscapes/data/gtFine/train %s'%save_path) 226 | -------------------------------------------------------------------------------- /model/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/layumi/AdaBoost_Seg/2624fe9c7e0c877097248b7fa9d3cf3c367f73c3/model/__init__.py -------------------------------------------------------------------------------- /model/blurpool.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2019, Adobe Inc. All rights reserved. 2 | # 3 | # This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4 | # 4.0 International Public License. To view a copy of this license, visit 5 | # https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode. 6 | 7 | import torch 8 | import torch.nn.parallel 9 | import numpy as np 10 | import torch.nn as nn 11 | import torch.nn.functional as F 12 | 13 | class BlurPool(nn.Module): 14 | def __init__(self, channels, pad_type='reflect', filt_size=4, stride=2, pad_off=0): 15 | super(BlurPool, self).__init__() 16 | self.filt_size = filt_size 17 | self.pad_off = pad_off 18 | self.pad_sizes = [int(1.*(filt_size-1)/2), int(np.ceil(1.*(filt_size-1)/2)), int(1.*(filt_size-1)/2), int(np.ceil(1.*(filt_size-1)/2))] 19 | self.pad_sizes = [pad_size+pad_off for pad_size in self.pad_sizes] 20 | self.stride = stride 21 | self.off = int((self.stride-1)/2.) 22 | self.channels = channels 23 | 24 | if(self.filt_size==1): 25 | a = np.array([1.,]) 26 | elif(self.filt_size==2): 27 | a = np.array([1., 1.]) 28 | elif(self.filt_size==3): 29 | a = np.array([1., 2., 1.]) 30 | elif(self.filt_size==4): 31 | a = np.array([1., 3., 3., 1.]) 32 | elif(self.filt_size==5): 33 | a = np.array([1., 4., 6., 4., 1.]) 34 | elif(self.filt_size==6): 35 | a = np.array([1., 5., 10., 10., 5., 1.]) 36 | elif(self.filt_size==7): 37 | a = np.array([1., 6., 15., 20., 15., 6., 1.]) 38 | 39 | filt = torch.Tensor(a[:,None]*a[None,:]) 40 | filt = filt/torch.sum(filt) 41 | self.register_buffer('filt', filt[None,None,:,:].repeat((self.channels,1,1,1))) 42 | 43 | self.pad = get_pad_layer(pad_type)(self.pad_sizes) 44 | 45 | def forward(self, inp): 46 | if(self.filt_size==1): 47 | if(self.pad_off==0): 48 | return inp[:,:,::self.stride,::self.stride] 49 | else: 50 | return self.pad(inp)[:,:,::self.stride,::self.stride] 51 | else: 52 | return F.conv2d(self.pad(inp), self.filt, stride=self.stride, groups=inp.shape[1]) 53 | 54 | def get_pad_layer(pad_type): 55 | if(pad_type in ['refl','reflect']): 56 | PadLayer = nn.ReflectionPad2d 57 | elif(pad_type in ['repl','replicate']): 58 | PadLayer = nn.ReplicationPad2d 59 | elif(pad_type=='zero'): 60 | PadLayer = nn.ZeroPad2d 61 | else: 62 | print('Pad type [%s] not recognized'%pad_type) 63 | return PadLayer 64 | 65 | class BlurPool1D(nn.Module): 66 | def __init__(self, channels, pad_type='reflect', filt_size=3, stride=2, pad_off=0): 67 | super(BlurPool1D, self).__init__() 68 | self.filt_size = filt_size 69 | self.pad_off = pad_off 70 | self.pad_sizes = [int(1. * (filt_size - 1) / 2), int(np.ceil(1. * (filt_size - 1) / 2))] 71 | self.pad_sizes = [pad_size + pad_off for pad_size in self.pad_sizes] 72 | self.stride = stride 73 | self.off = int((self.stride - 1) / 2.) 74 | self.channels = channels 75 | 76 | # print('Filter size [%i]' % filt_size) 77 | if(self.filt_size == 1): 78 | a = np.array([1., ]) 79 | elif(self.filt_size == 2): 80 | a = np.array([1., 1.]) 81 | elif(self.filt_size == 3): 82 | a = np.array([1., 2., 1.]) 83 | elif(self.filt_size == 4): 84 | a = np.array([1., 3., 3., 1.]) 85 | elif(self.filt_size == 5): 86 | a = np.array([1., 4., 6., 4., 1.]) 87 | elif(self.filt_size == 6): 88 | a = np.array([1., 5., 10., 10., 5., 1.]) 89 | elif(self.filt_size == 7): 90 | a = np.array([1., 6., 15., 20., 15., 6., 1.]) 91 | 92 | filt = torch.Tensor(a) 93 | filt = filt / torch.sum(filt) 94 | self.register_buffer('filt', filt[None, None, :].repeat((self.channels, 1, 1))) 95 | 96 | self.pad = get_pad_layer_1d(pad_type)(self.pad_sizes) 97 | 98 | def forward(self, inp): 99 | if(self.filt_size == 1): 100 | if(self.pad_off == 0): 101 | return inp[:, :, ::self.stride] 102 | else: 103 | return self.pad(inp)[:, :, ::self.stride] 104 | else: 105 | return F.conv1d(self.pad(inp), self.filt, stride=self.stride, groups=inp.shape[1]) 106 | 107 | def get_pad_layer_1d(pad_type): 108 | if(pad_type in ['refl', 'reflect']): 109 | PadLayer = nn.ReflectionPad1d 110 | elif(pad_type in ['repl', 'replicate']): 111 | PadLayer = nn.ReplicationPad1d 112 | elif(pad_type == 'zero'): 113 | PadLayer = nn.ZeroPad1d 114 | else: 115 | print('Pad type [%s] not recognized' % pad_type) 116 | return PadLayer 117 | -------------------------------------------------------------------------------- /model/deeplab.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import math 3 | import torch.utils.model_zoo as model_zoo 4 | import torch 5 | import numpy as np 6 | affine_par = True 7 | 8 | 9 | def outS(i): 10 | i = int(i) 11 | i = (i+1)/2 12 | i = int(np.ceil((i+1)/2.0)) 13 | i = (i+1)/2 14 | return i 15 | 16 | def conv3x3(in_planes, out_planes, stride=1): 17 | "3x3 convolution with padding" 18 | return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride, 19 | padding=1, bias=False) 20 | 21 | 22 | class BasicBlock(nn.Module): 23 | expansion = 1 24 | 25 | def __init__(self, inplanes, planes, stride=1, downsample=None): 26 | super(BasicBlock, self).__init__() 27 | self.conv1 = conv3x3(inplanes, planes, stride) 28 | self.bn1 = nn.BatchNorm2d(planes, affine = affine_par) 29 | self.relu = nn.ReLU(inplace=True) 30 | self.conv2 = conv3x3(planes, planes) 31 | self.bn2 = nn.BatchNorm2d(planes, affine = affine_par) 32 | self.downsample = downsample 33 | self.stride = stride 34 | 35 | def forward(self, x): 36 | residual = x 37 | 38 | out = self.conv1(x) 39 | out = self.bn1(out) 40 | out = self.relu(out) 41 | 42 | out = self.conv2(out) 43 | out = self.bn2(out) 44 | 45 | if self.downsample is not None: 46 | residual = self.downsample(x) 47 | 48 | out += residual 49 | out = self.relu(out) 50 | 51 | return out 52 | 53 | 54 | class Bottleneck(nn.Module): 55 | expansion = 4 56 | 57 | def __init__(self, inplanes, planes, stride=1, dilation=1, downsample=None): 58 | super(Bottleneck, self).__init__() 59 | self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=1, stride=stride, bias=False) # change 60 | self.bn1 = nn.BatchNorm2d(planes,affine = affine_par) 61 | for i in self.bn1.parameters(): 62 | i.requires_grad = False 63 | 64 | padding = dilation 65 | self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=1, # change 66 | padding=padding, bias=False, dilation = dilation) 67 | self.bn2 = nn.BatchNorm2d(planes,affine = affine_par) 68 | for i in self.bn2.parameters(): 69 | i.requires_grad = False 70 | self.conv3 = nn.Conv2d(planes, planes * 4, kernel_size=1, bias=False) 71 | self.bn3 = nn.BatchNorm2d(planes * 4, affine = affine_par) 72 | for i in self.bn3.parameters(): 73 | i.requires_grad = False 74 | self.relu = nn.ReLU(inplace=True) 75 | self.downsample = downsample 76 | self.stride = stride 77 | 78 | 79 | def forward(self, x): 80 | residual = x 81 | 82 | out = self.conv1(x) 83 | out = self.bn1(out) 84 | out = self.relu(out) 85 | 86 | out = self.conv2(out) 87 | out = self.bn2(out) 88 | out = self.relu(out) 89 | 90 | out = self.conv3(out) 91 | out = self.bn3(out) 92 | 93 | if self.downsample is not None: 94 | residual = self.downsample(x) 95 | 96 | out += residual 97 | out = self.relu(out) 98 | 99 | return out 100 | 101 | class Classifier_Module(nn.Module): 102 | 103 | def __init__(self, dilation_series, padding_series, num_classes): 104 | super(Classifier_Module, self).__init__() 105 | self.conv2d_list = nn.ModuleList() 106 | for dilation, padding in zip(dilation_series, padding_series): 107 | self.conv2d_list.append(nn.Conv2d(2048, num_classes, kernel_size=3, stride=1, padding=padding, dilation=dilation, bias = True)) 108 | 109 | for m in self.conv2d_list: 110 | m.weight.data.normal_(0, 0.01) 111 | 112 | def forward(self, x): 113 | out = self.conv2d_list[0](x) 114 | for i in range(len(self.conv2d_list)-1): 115 | out += self.conv2d_list[i+1](x) 116 | return out 117 | 118 | 119 | 120 | class ResNet(nn.Module): 121 | def __init__(self, block, layers, num_classes): 122 | self.inplanes = 64 123 | super(ResNet, self).__init__() 124 | self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, 125 | bias=False) 126 | self.bn1 = nn.BatchNorm2d(64, affine = affine_par) 127 | for i in self.bn1.parameters(): 128 | i.requires_grad = False 129 | self.relu = nn.ReLU(inplace=True) 130 | self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1, ceil_mode=True) # change 131 | self.layer1 = self._make_layer(block, 64, layers[0]) 132 | self.layer2 = self._make_layer(block, 128, layers[1], stride=2) 133 | self.layer3 = self._make_layer(block, 256, layers[2], stride=1, dilation=2) 134 | self.layer4 = self._make_layer(block, 512, layers[3], stride=1, dilation=4) 135 | self.layer5 = self._make_pred_layer(Classifier_Module, [6,12,18,24],[6,12,18,24],num_classes) 136 | 137 | for m in self.modules(): 138 | if isinstance(m, nn.Conv2d): 139 | n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels 140 | m.weight.data.normal_(0, 0.01) 141 | elif isinstance(m, nn.BatchNorm2d): 142 | m.weight.data.fill_(1) 143 | m.bias.data.zero_() 144 | # for i in m.parameters(): 145 | # i.requires_grad = False 146 | 147 | def _make_layer(self, block, planes, blocks, stride=1, dilation=1): 148 | downsample = None 149 | if stride != 1 or self.inplanes != planes * block.expansion or dilation == 2 or dilation == 4: 150 | downsample = nn.Sequential( 151 | nn.Conv2d(self.inplanes, planes * block.expansion, 152 | kernel_size=1, stride=stride, bias=False), 153 | nn.BatchNorm2d(planes * block.expansion,affine = affine_par)) 154 | for i in downsample._modules['1'].parameters(): 155 | i.requires_grad = False 156 | layers = [] 157 | layers.append(block(self.inplanes, planes, stride,dilation=dilation, downsample=downsample)) 158 | self.inplanes = planes * block.expansion 159 | for i in range(1, blocks): 160 | layers.append(block(self.inplanes, planes, dilation=dilation)) 161 | 162 | return nn.Sequential(*layers) 163 | def _make_pred_layer(self,block, dilation_series, padding_series,num_classes): 164 | return block(dilation_series,padding_series,num_classes) 165 | 166 | def forward(self, x): 167 | x = self.conv1(x) 168 | x = self.bn1(x) 169 | x = self.relu(x) 170 | x = self.maxpool(x) 171 | x = self.layer1(x) 172 | x = self.layer2(x) 173 | x = self.layer3(x) 174 | x = self.layer4(x) 175 | x = self.layer5(x) 176 | 177 | return x 178 | 179 | def get_1x_lr_params_NOscale(self): 180 | """ 181 | This generator returns all the parameters of the net except for 182 | the last classification layer. Note that for each batchnorm layer, 183 | requires_grad is set to False in deeplab_resnet.py, therefore this function does not return 184 | any batchnorm parameter 185 | """ 186 | b = [] 187 | 188 | b.append(self.conv1) 189 | b.append(self.bn1) 190 | b.append(self.layer1) 191 | b.append(self.layer2) 192 | b.append(self.layer3) 193 | b.append(self.layer4) 194 | 195 | 196 | for i in range(len(b)): 197 | for j in b[i].modules(): 198 | jj = 0 199 | for k in j.parameters(): 200 | jj+=1 201 | if k.requires_grad: 202 | yield k 203 | 204 | def get_10x_lr_params(self): 205 | """ 206 | This generator returns all the parameters for the last layer of the net, 207 | which does the classification of pixel into classes 208 | """ 209 | b = [] 210 | b.append(self.layer5.parameters()) 211 | 212 | for j in range(len(b)): 213 | for i in b[j]: 214 | yield i 215 | 216 | 217 | 218 | def optim_parameters(self, args): 219 | return [{'params': self.get_1x_lr_params_NOscale(), 'lr': args.learning_rate}, 220 | {'params': self.get_10x_lr_params(), 'lr': 10*args.learning_rate}] 221 | 222 | 223 | def Res_Deeplab(num_classes=21): 224 | model = ResNet(Bottleneck,[3, 4, 23, 3], num_classes) 225 | return model 226 | 227 | -------------------------------------------------------------------------------- /model/deeplab_vgg.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import torch 3 | from torch import nn 4 | from torchvision import models 5 | 6 | 7 | def NormLayer(norm_dim, norm_style = 'gn'): 8 | if norm_style == 'bn': 9 | norm_layer = nn.BatchNorm2d(norm_dim) 10 | elif norm_style == 'in': 11 | norm_layer = nn.InstanceNorm2d(norm_dim, affine = True) 12 | elif norm_style == 'ln': 13 | norm_layer = nn.LayerNorm(norm_dim, elementwise_affine=True) 14 | elif norm_style == 'gn': 15 | norm_layer = nn.GroupNorm(num_groups=32, num_channels=norm_dim, affine = True) 16 | return norm_layer 17 | 18 | class SEBlock(nn.Module): 19 | def __init__(self, inplanes, r = 16): 20 | super(SEBlock, self).__init__() 21 | self.global_pool = nn.AdaptiveAvgPool2d((1,1)) 22 | self.se = nn.Sequential( 23 | nn.Linear(inplanes, inplanes//r), 24 | nn.ReLU(inplace=True), 25 | nn.Linear(inplanes//r, inplanes), 26 | nn.Sigmoid() 27 | ) 28 | def forward(self, x): 29 | xx = self.global_pool(x) 30 | xx = xx.view(xx.size(0), xx.size(1)) 31 | se_weight = self.se(xx).unsqueeze(-1).unsqueeze(-1) 32 | return x.mul(se_weight) 33 | 34 | class Classifier_Module(nn.Module): 35 | 36 | def __init__(self, dims_in, dilation_series, padding_series, num_classes, norm_style = 'gn'): 37 | super(Classifier_Module, self).__init__() 38 | self.conv2d_list = nn.ModuleList() 39 | self.conv2d_list.append( 40 | nn.Sequential(*[ 41 | nn.Conv2d(dims_in, 256, kernel_size=1, stride=1, padding=0, dilation=1, bias=True), 42 | NormLayer(256, norm_style), 43 | nn.ReLU(inplace=True) ])) 44 | for dilation, padding in zip(dilation_series, padding_series): 45 | self.conv2d_list.append( 46 | nn.Sequential(*[ 47 | nn.Conv2d(dims_in, 256, kernel_size=3, stride=1, padding=padding, dilation=dilation, bias=True), 48 | NormLayer(256, norm_style), 49 | nn.ReLU(inplace=True) ])) 50 | 51 | 52 | self.bottleneck = nn.Sequential(*[SEBlock(256 * (len(dilation_series) + 1)), 53 | nn.Conv2d(256 * (len(dilation_series) + 1), 512, kernel_size=3, stride=1, padding=1, dilation=1, bias=True) , 54 | NormLayer(512, norm_style) ]) 55 | 56 | self.head = nn.Sequential(*[nn.Dropout2d(0.2), 57 | nn.Conv2d(512, num_classes, kernel_size=1, padding=0, dilation=1, bias=False) ]) 58 | 59 | ##########init####### 60 | for m in self.conv2d_list: 61 | if isinstance(m, nn.Conv2d): 62 | torch.nn.init.kaiming_normal_(m.weight.data, a=0, mode='fan_in') 63 | m.bias.data.zero_() 64 | elif isinstance(m, nn.BatchNorm2d) or isinstance(m, nn.InstanceNorm2d) or isinstance(m, nn.GroupNorm): 65 | m.weight.data.fill_(1) 66 | m.bias.data.zero_() 67 | 68 | for m in self.bottleneck: 69 | if isinstance(m, nn.Conv2d): 70 | torch.nn.init.kaiming_normal_(m.weight.data, a=0, mode='fan_in') 71 | m.bias.data.zero_() 72 | elif isinstance(m, nn.Linear): 73 | torch.nn.init.kaiming_normal_(m.weight.data, a=0, mode='fan_out') 74 | m.bias.data.zero_() 75 | elif isinstance(m, nn.BatchNorm2d) or isinstance(m, nn.InstanceNorm2d) or isinstance(m, nn.GroupNorm) or isinstance(m, nn.LayerNorm): 76 | m.weight.data.fill_(1) 77 | m.bias.data.zero_() 78 | 79 | for m in self.head: 80 | if isinstance(m, nn.Conv2d): 81 | m.weight.data.normal_(0, 0.001) 82 | 83 | def forward(self, x): 84 | out = self.conv2d_list[0](x) 85 | for i in range(len(self.conv2d_list) - 1): 86 | out = torch.cat( (out, self.conv2d_list[i+1](x)), 1) 87 | out = self.bottleneck(out) 88 | out = self.head(out) 89 | return out 90 | 91 | 92 | class DeeplabVGG(nn.Module): 93 | def __init__(self, num_classes, vgg16_caffe_path=None, pretrained=False, vggbn = False): 94 | super(DeeplabVGG, self).__init__() 95 | if vggbn: 96 | vgg = models.vgg16_bn(pretrained=True) 97 | else: 98 | vgg = models.vgg16(pretrained=True) 99 | 100 | if pretrained: 101 | vgg.load_state_dict(torch.load(vgg16_caffe_path)) 102 | 103 | features, classifier = list(vgg.features.children()), list(vgg.classifier.children()) 104 | 105 | #remove pool4/pool5 106 | features = nn.Sequential(*(features[i] for i in list(range(23))+list(range(24,30)))) 107 | 108 | for i in [23,25,27]: 109 | features[i].dilation = (2,2) 110 | features[i].padding = (2,2) 111 | 112 | fc6 = nn.Conv2d(512, 1024, kernel_size=3, padding=4, dilation=4) 113 | fc7 = nn.Conv2d(1024, 1024, kernel_size=3, padding=4, dilation=4) 114 | 115 | self.features1 = nn.Sequential(*[features[i] for i in range(len(features))]) 116 | self.features2 = nn.Sequential(*[ fc6, nn.ReLU(inplace=True), fc7, nn.ReLU(inplace=True)]) 117 | 118 | self.classifier1 = Classifier_Module(512, [6,12,18,24],[6,12,18,24],num_classes) 119 | self.classifier2 = Classifier_Module(1024, [6,12,18,24],[6,12,18,24],num_classes) 120 | 121 | 122 | def forward(self, x): 123 | x = self.features1(x) 124 | x1 = self.classifier1(x) 125 | x = self.features2(x) 126 | x2 = self.classifier2(x) 127 | return x1, x2 128 | 129 | def optim_parameters(self, args): 130 | return self.parameters() 131 | -------------------------------------------------------------------------------- /model/discriminator.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import torch.nn.functional as F 3 | 4 | 5 | class FCDiscriminator(nn.Module): 6 | 7 | def __init__(self, num_classes, ndf = 64): 8 | super(FCDiscriminator, self).__init__() 9 | 10 | self.conv1 = nn.Conv2d(num_classes, ndf, kernel_size=4, stride=2, padding=1) 11 | self.conv2 = nn.Conv2d(ndf, ndf*2, kernel_size=4, stride=2, padding=1) 12 | self.conv3 = nn.Conv2d(ndf*2, ndf*4, kernel_size=4, stride=2, padding=1) 13 | self.conv4 = nn.Conv2d(ndf*4, ndf*8, kernel_size=4, stride=2, padding=1) 14 | self.classifier = nn.Conv2d(ndf*8, 1, kernel_size=4, stride=2, padding=1) 15 | 16 | self.leaky_relu = nn.LeakyReLU(negative_slope=0.2, inplace=True) 17 | #self.up_sample = nn.Upsample(scale_factor=32, mode='bilinear') 18 | #self.sigmoid = nn.Sigmoid() 19 | 20 | 21 | def forward(self, x): 22 | x = self.conv1(x) 23 | x = self.leaky_relu(x) 24 | x = self.conv2(x) 25 | x = self.leaky_relu(x) 26 | x = self.conv3(x) 27 | x = self.leaky_relu(x) 28 | x = self.conv4(x) 29 | x = self.leaky_relu(x) 30 | x = self.classifier(x) 31 | #x = self.up_sample(x) 32 | #x = self.sigmoid(x) 33 | 34 | return x 35 | -------------------------------------------------------------------------------- /pipeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/layumi/AdaBoost_Seg/2624fe9c7e0c877097248b7fa9d3cf3c367f73c3/pipeline.png -------------------------------------------------------------------------------- /sam.py: -------------------------------------------------------------------------------- 1 | import torch 2 | 3 | 4 | class SAM(torch.optim.Optimizer): 5 | def __init__(self, params, base_optimizer=torch.optim.SGD, rho=0.05, **kwargs): 6 | assert rho >= 0.0, f"Invalid rho, should be non-negative: {rho}" 7 | 8 | defaults = dict(rho=rho, **kwargs) 9 | super(SAM, self).__init__(params, defaults) 10 | 11 | self.base_optimizer = base_optimizer(self.param_groups, **kwargs) 12 | self.param_groups = self.base_optimizer.param_groups 13 | 14 | @torch.no_grad() 15 | def first_step(self, zero_grad=False): 16 | grad_norm = self._grad_norm() 17 | for group in self.param_groups: 18 | scale = group["rho"] / (grad_norm + 1e-12) 19 | 20 | for p in group["params"]: 21 | if p.grad is None: continue 22 | e_w = p.grad * scale.to(p) 23 | p.add_(e_w) # climb to the local maximum "w + e(w)" 24 | self.state[p]["e_w"] = e_w 25 | 26 | if zero_grad: self.zero_grad() 27 | 28 | @torch.no_grad() 29 | def second_step(self, zero_grad=False): 30 | for group in self.param_groups: 31 | for p in group["params"]: 32 | if p.grad is None: continue 33 | p.sub_(self.state[p]["e_w"]) # get back to "w" from "w + e(w)" 34 | 35 | self.base_optimizer.step() # do the actual "sharpness-aware" update 36 | 37 | if zero_grad: self.zero_grad() 38 | 39 | @torch.no_grad() 40 | def step(self, closure=None): 41 | assert closure is not None, "Sharpness Aware Minimization requires closure, but it was not provided" 42 | closure = torch.enable_grad()(closure) # the closure should do a full forward-backward pass 43 | 44 | self.first_step(zero_grad=True) 45 | closure() 46 | self.second_step() 47 | 48 | def _grad_norm(self): 49 | shared_device = self.param_groups[0]["params"][0].device # put everything on the same device, in case of model parallelism 50 | norm = torch.norm( 51 | torch.stack([ 52 | p.grad.norm(p=2).to(shared_device) 53 | for group in self.param_groups for p in group["params"] 54 | if p.grad is not None 55 | ]), 56 | p=2 57 | ) 58 | return norm 59 | -------------------------------------------------------------------------------- /test.py: -------------------------------------------------------------------------------- 1 | # -*- coding=utf-8 -*- 2 | # @Mail: beanocean@outlook.com 3 | # @D&T: Sat 07 Dec 2019 22:04:31 AEDT 4 | 5 | import os 6 | 7 | while(1): 8 | try: 9 | os.system('python train_ms.py --snapshot-dir ./snapshots/SE_GN_batchsize2_1024x512_pp_ms_me0_classbalance7_kl0.1_lr2_drop0.1_seg0.5_aug_fp16 --drop 0.1 --warm-up 5000 --batch-size 2 --learning-rate 2e-4 --crop-size 1024,512 --lambda-seg 0.5 --lambda-adv-target1 0.0002 --lambda-adv-target2 0.001 --lambda-me-target 0 --lambda-kl-target 0.1 --norm-style gn --class-balance --only-hard-label 80 --max-value 7 --gpu-ids 0,1 --often-balance --use-se --autoaug') 10 | except: 11 | continue 12 | -------------------------------------------------------------------------------- /try_run.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | while(True): 4 | os.system('python train_ms.py --snapshot-dir ./snapshots/SE_GN_batchsize2_1024x512_pp_ms_me0_classbalance7_kl0.1_lr2_drop0.1_seg0.5_aug_fp16 --drop 0.1 --warm-up 5000 --batch-size 2 --learning-rate 2e-4 --crop-size 1024,512 --lambda-seg 0.5 --lambda-adv-target1 0.0002 --lambda-adv-target2 0.001 --lambda-me-target 0 --lambda-kl-target 0.1 --norm-style gn --class-balance --only-hard-label 80 --max-value 7 --gpu-ids 0,1 --often-balance --use-se --autoaug --fp16') 5 | -------------------------------------------------------------------------------- /utils/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/layumi/AdaBoost_Seg/2624fe9c7e0c877097248b7fa9d3cf3c367f73c3/utils/__init__.py -------------------------------------------------------------------------------- /utils/clear_model.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | root = '../snapshots/' 4 | nn = [] 5 | for f in os.listdir(root): 6 | for ff in os.listdir(root+f): 7 | dir_name = root+f 8 | if len(os.listdir(dir_name)) == 1: 9 | print(dir_name) 10 | os.system('rm -r %s'%dir_name) 11 | continue 12 | 13 | for fff in os.listdir(dir_name): 14 | if fff =='opts.yaml' or fff=='GTA5_100000.pth': 15 | continue 16 | if fff.endswith('D1.pth') or fff.endswith('D2.pth'): 17 | dst = dir_name+'/'+fff 18 | print(dst) 19 | os.remove(dst) 20 | 21 | try: 22 | if int(fff[5:10])//10000==1 and not fff.endswith('average.pth'): 23 | dst = dir_name+'/'+fff 24 | print(dst) 25 | os.remove(dst) 26 | if not(int(fff[5:10])%10000==0) and not fff.endswith('average.pth'): 27 | dst = dir_name+'/'+fff 28 | print(dst) 29 | os.remove(dst) 30 | except: 31 | continue 32 | 33 | -------------------------------------------------------------------------------- /utils/loss.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn.functional as F 3 | import torch.nn as nn 4 | from torch.autograd import Variable 5 | 6 | 7 | class CrossEntropy2d(nn.Module): 8 | 9 | def __init__(self, size_average=True, ignore_label=255): 10 | super(CrossEntropy2d, self).__init__() 11 | self.size_average = size_average 12 | self.ignore_label = ignore_label 13 | 14 | def forward(self, predict, target, weight=None): 15 | """ 16 | Args: 17 | predict:(n, c, h, w) 18 | target:(n, h, w) 19 | weight (Tensor, optional): a manual rescaling weight given to each class. 20 | If given, has to be a Tensor of size "nclasses" 21 | """ 22 | assert not target.requires_grad 23 | assert predict.dim() == 4 24 | assert target.dim() == 3 25 | assert predict.size(0) == target.size(0), "{0} vs {1} ".format(predict.size(0), target.size(0)) 26 | assert predict.size(2) == target.size(1), "{0} vs {1} ".format(predict.size(2), target.size(1)) 27 | assert predict.size(3) == target.size(2), "{0} vs {1} ".format(predict.size(3), target.size(3)) 28 | n, c, h, w = predict.size() 29 | target_mask = (target >= 0) * (target != self.ignore_label) 30 | target = target[target_mask] 31 | if not target.data.dim(): 32 | return Variable(torch.zeros(1)) 33 | predict = predict.transpose(1, 2).transpose(2, 3).contiguous() 34 | predict = predict[target_mask.view(n, h, w, 1).repeat(1, 1, 1, c)].view(-1, c) 35 | loss = F.cross_entropy(predict, target, weight=weight, size_average=self.size_average) 36 | return loss 37 | -------------------------------------------------------------------------------- /utils/tool.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import time 3 | import numpy as np 4 | 5 | def cosine_rampdown(current, rampdown_length): 6 | """Cosine rampdown from https://arxiv.org/abs/1608.03983""" 7 | assert 0 <= current <= rampdown_length 8 | return max(0.01, float(.5 * (np.cos(np.pi * current / rampdown_length) + 1))) 9 | 10 | def lr_poly(base_lr, iter, max_iter, power): 11 | return base_lr * ((1 - float(iter) / max_iter) ** (power)) 12 | 13 | def lr_step(base_lr, iter): 14 | lr = base_lr 15 | if iter>40000: 16 | lr = base_lr * 0.5 17 | if iter>60000: 18 | lr = base_lr * 0.5 * 0.5 19 | if iter>70000: 20 | lr = base_lr * 0.5 * 0.5 * 0.5 21 | return lr 22 | 23 | def adjust_learning_rate(optimizer, i_iter, args): 24 | if i_iter < args.warm_up: 25 | lr = args.learning_rate * (0.1 + 0.9 * i_iter / args.warm_up) 26 | else: 27 | lr = lr_poly(args.learning_rate, i_iter, args.num_steps, args.power) 28 | #lr = lr_step(args.learning_rate, i_iter) 29 | 30 | if i_iter>=args.swa_start and args.cosine: 31 | lr = lr * cosine_rampdown( (i_iter - args.swa_start)%args.save_pred_every, args.save_pred_every) 32 | 33 | optimizer.param_groups[0]['lr'] = lr 34 | print('-------lr_G: %f-------'%lr) 35 | if len(optimizer.param_groups) > 1: 36 | optimizer.param_groups[1]['lr'] = lr * 10 37 | 38 | 39 | def adjust_learning_rate_D(optimizer, i_iter, args): 40 | lr = lr_poly(args.learning_rate_D, i_iter, args.num_steps, args.power) 41 | optimizer.param_groups[0]['lr'] = lr 42 | #if len(optimizer.param_groups) > 1: 43 | # optimizer.param_groups[1]['lr'] = lr * 10 44 | 45 | 46 | def fliplr(img): 47 | '''flip horizontal''' 48 | inv_idx = torch.arange(img.size(3)-1,-1,-1).long().cuda() # N x C x H x W 49 | img_flip = img.index_select(3,inv_idx) 50 | return img_flip 51 | 52 | class Timer: 53 | def __init__(self, msg): 54 | self.msg = msg 55 | self.start_time = None 56 | 57 | def __enter__(self): 58 | self.start_time = time.time() 59 | 60 | def __exit__(self, exc_type, exc_value, exc_tb): 61 | print(self.msg % (time.time() - self.start_time)) 62 | -------------------------------------------------------------------------------- /visualize_noisy_label.py: -------------------------------------------------------------------------------- 1 | from PIL import Image 2 | import numpy as np 3 | import matplotlib.pyplot as plt 4 | import json 5 | 6 | devkit_dir = 'dataset/cityscapes_list' 7 | with open(devkit_dir+'/info.json', 'r') as fp: 8 | info = json.load(fp) 9 | num_classes = np.int(info['classes']) 10 | name_classes = np.array(info['label'], dtype=np.str) 11 | mapping = np.array(info['label2train'], dtype=np.int) 12 | 13 | def label_mapping(input, mapping): 14 | output = np.copy(input) 15 | for ind in range(len(mapping)): 16 | output[input == mapping[ind][0]] = mapping[ind][1] 17 | return np.array(output, dtype=np.int64) 18 | 19 | img1 = 'result/cityscapesSE_GN_batchsize2_1024x512_pp_ms_me0_classbalance7_kl0.1_lr2_drop0.1_seg0.5/frankfurt_000001_005898_leftImg8bit.png' 20 | img2 = 'data/Cityscapes/data/gtFine/val/frankfurt/frankfurt_000001_005898_gtFine_labelIds.png' 21 | 22 | img1 = np.asarray(Image.open(img1)) 23 | img2 = np.asarray(Image.open(img2)) 24 | img2 = label_mapping(img2, mapping) 25 | 26 | print(img1) 27 | print(img2) 28 | output = np.abs(img1-img2) 29 | output[output>200] = 0 30 | output[output>1] = 1 31 | 32 | fig = plt.figure() 33 | plt.axis('off') 34 | heatmap = plt.imshow(output, cmap='viridis') 35 | fig.colorbar(heatmap) 36 | fig.savefig('label_heatmap.png') 37 | --------------------------------------------------------------------------------