├── .gitignore ├── README.md ├── domain_adaptation ├── README.md ├── source │ ├── core │ │ ├── bdds.py │ │ ├── cityscapes.py │ │ ├── csvs │ │ │ ├── stratified_GTA.csv │ │ │ ├── stratified_bdds.csv │ │ │ └── stratified_mapillary.csv │ │ ├── gta.py │ │ └── mapillary.py │ └── prep_all.sh └── target │ ├── semi-supervised │ └── selected_samples.csv │ └── weakly-supervised │ ├── train1.zip │ ├── train2.zip │ └── val.zip ├── evaluation ├── csHelpers.py ├── evaluate_detection.py ├── evaluate_instance_segmentation.py ├── evaluate_mIoU.py ├── helper_eval_detection.py ├── idd_lite_evaluate_mIoU.py ├── instance.py └── instances2dict.py ├── helpers ├── __pycache__ │ ├── annotation.cpython-35.pyc │ ├── annotation.cpython-37.pyc │ ├── anue_labels.cpython-35.pyc │ └── anue_labels.cpython-37.pyc ├── annotation.py └── anue_labels.py ├── preperation ├── __pycache__ │ ├── json2instanceImg.cpython-37.pyc │ └── json2labelImg.cpython-37.pyc ├── cityscape_panoptic_gt.py ├── createLabels.py ├── json2instanceImg.py └── json2labelImg.py ├── requirements.txt └── viewer ├── viewer.py └── viewer2.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | __pycache__ -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Scene Understanding Challenge for Autonomous Navigation in Unstructured Environments 2 | 3 | Code for working with the dataset used for the [Scene Understanding Challenge for Autonomous Navigation in Unstructured Environments](http://cvit.iiit.ac.in/scene-understanding-challenge-2018/). For details of getting the dataset and updates see: 4 | - https://cvit.iiit.ac.in/autonue2021/ 5 | - https://cvit.iiit.ac.in/autonue2019/ 6 | - http://cvit.iiit.ac.in/autonue2018/ 7 | - http://cvit.iiit.ac.in/scene-understanding-challenge-2018/ 8 | 9 | 10 | # AutoNUE 2021 (Domain Adaptation and Semantic Segmentation) 11 | 12 | This repository contains the datasets related to domain adaptation and segmentation challenges for AutoNEU 2021, CVPR Workshop. For more details, please visit https://cvit.iiit.ac.in/autonue2021/challenge. For the segmentation challenge, please skip "Source datasets" section below. 13 | 14 | 15 | ## Source datasets: 16 | 17 | Participants are requested to download the datasets from original websites, given below for easy reference:- 18 | 1. https://www.mapillary.com/dataset/vistas?pKey=q0GhQpk20wJm1ba1mfwJmw 19 | 2. https://bdd-data.berkeley.edu/ (you might have to click on Advanced tab, and then click on "proceed to bdd-data.berkeley.edu") 20 | 3. https://download.visinf.tu-darmstadt.de/data/from_games/ 21 | 4. https://www.cityscapes-dataset.com/examples/#fine-annotations (only fine-annotations to be used) 22 | 23 | After downloading all the source datasets, move them to folder ./domain_adaptation/source/datasets/. Its folder structure should be as follows: 24 | ``` 25 | datasets 26 | |--mapillary-vistas-dataset_public_v1.1/ 27 | | |--training/ 28 | | | |--images/ 29 | | | |--labels/ 30 | | |--validation/ 31 | | | |--images/ 32 | | | |--labels/ 33 | | |--testing/ 34 | | |--images/ 35 | |--bdd100k/ 36 | | |--seg/ 37 | | |--images/ 38 | | | |--train/ 39 | | | |--val/ 40 | | | |--test/ 41 | | |--labels/ 42 | | |--train/ 43 | | |--val/ 44 | |--gta/ 45 | | |--images/ 46 | | |--labels/ 47 | |--cityscapes/ 48 | |--gtFine/ 49 | | |--train/ 50 | | |--val/ 51 | | |--test/ 52 | |--leftImg8bit/ 53 | |--train/ 54 | |--val/ 55 | |--test/ 56 | ``` 57 | 58 | 59 | Run the following commands: 60 | 61 | ``` 62 | pip3 install requirements.txt 63 | chmod +x domain_adaptation/source/prep_all.sh 64 | ./domain_adaptation/source/prep_all.sh 65 | ``` 66 | 67 | This will create a folder "domain_adaptation/source/source_datasets_dir/" where you will find the images and annotations for the source dataset to be used for any of the domain adaptation challenges. 68 | 69 | ## Target datasets: 70 | 71 | **For using first add helpers/ to $PYTHONPATH** 72 | ``` 73 | export PYTHONPATH="${PYTHONPATH}:helpers/" 74 | ``` 75 | 76 | ### Dataset Structure 77 | 78 | The structure is similar to the cityscapes dataset. That is: 79 | ``` 80 | gtFine/{split}/{drive_no}/{img_id}_gtFine_polygons.json for ground truths 81 | leftImg8bit/{split}/{drive_no}/{img_id}_leftImg8bit.png for image frames 82 | ``` 83 | #### Semantic Segmentation 84 | 85 | Furthermore for training, label masks needs to be generated as described below resulting in the following files: 86 | ``` 87 | gtFine/{split}/{drive_no}/{img_id}_gtFine_labellevel3Ids.png 88 | gtFine/{split}/{drive_no}/{img_id}_gtFine_instancelevel3Ids.png 89 | ``` 90 | ### Labels 91 | 92 | See helpers/anue_labels.py 93 | 94 | #### Generate Label Masks (for training/evaluation) (Semantic/Instance/Panoptic Segmentation) 95 | ```bash 96 | python preperation/createLabels.py --datadir $ANUE --id-type $IDTYPE --color [True|False] --instance [True|False] --num-workers $C 97 | ``` 98 | 99 | - ANUE is the path to the AutoNUE dataset 100 | - IDTYPE can be id, csId, csTrainId, level3Id, level2Id, level1Id. 101 | - color True generates the color masks 102 | - instance True generates the instance masks with the id given by IDTYPE 103 | - panoptic True generates panoptic masks in the format similar to COCO. See the modified evaluation scripts here: https://github.com/AutoNUE/panopticapi 104 | - C is the number of threads to run in parallel 105 | 106 | 107 | For the supervised domain adaptation and semantic segmentation tasks, the masks should be generated using IDTYPE of level3Id and used for training models (similar to trainId in cityscapes). This can be done by the command: 108 | ```bash 109 | python preperation/createLabels.py --datadir $ANUE --id-type level3Id --num-workers $C 110 | ``` 111 | 112 | Following commands are updated for the target labels of other domain adaptation tasks: 113 | 114 | ``` 115 | python3 preperation/createLabels.py --datadir $ANUE --id-type level3Id --num-workers $C --semisup_da True 116 | python3 preperation/createLabels.py --datadir $ANUE --id-type level3Id --num-workers $C --weaksup_da True 117 | python3 preperation/createLabels.py --datadir $ANUE --id-type level3Id --num-workers $C --unsup_da True 118 | 119 | ``` 120 | The bounding box labels for weakly supervised domain adapation can be downloaded from here: https://github.com/AutoNUE/public-code/tree/master/domain_adaptation/target/weakly-supervised 121 | 122 | 123 | The generated files: 124 | 125 | - _gtFine_labelLevel3Ids.png will be used for semantic segmentation 126 | 127 | 128 | 129 | # AutoNUE 2019 130 | 131 | **For using first add helpers/ to $PYTHONPATH** 132 | ``` 133 | export PYTHONPATH="${PYTHONPATH}:helpers/" 134 | ``` 135 | 136 | **The code has been tested on python 3.6.4** 137 | 138 | ## Dataset Structure 139 | 140 | The structure is similar to the cityscapes dataset. That is: 141 | ``` 142 | gtFine/{split}/{drive_no}/{img_id}_gtFine_polygons.json for ground truths 143 | leftImg8bit/{split}/{drive_no}/{img_id}_leftImg8bit.png for image frames 144 | ``` 145 | ### Semantic Segmentation and Instance Segmentation 146 | 147 | Furthermore for training, label masks needs to be generated as described below resulting in the following files: 148 | ``` 149 | gtFine/{split}/{drive_no}/{img_id}_gtFine_labellevel3Ids.png 150 | gtFine/{split}/{drive_no}/{img_id}_gtFine_instancelevel3Ids.png 151 | ``` 152 | 153 | ### Panoptic Challenge 154 | 155 | Furthermore for training, panoptic masks needs to be generated as described below resulting in the following files: 156 | ``` 157 | gtFine/{split}_panoptic/{drive_no}_{img_id}_gtFine_panopticlevel3Ids.png 158 | gtFine/{split}_panoptic.json 159 | ``` 160 | ### Detection 161 | 162 | The structure is slightly similar to Pascal VOC dataset. 163 | - JPEGImages///<>.jpg for images 164 | - Annotations///<>.xml for Annotations 165 | 166 | ## Labels 167 | 168 | See helpers/anue_labels.py 169 | 170 | ### Generate Label Masks (for training/evaluation) (Semantic/Instance/Panoptic Segmentation) 171 | ```bash 172 | python preperation/createLabels.py --datadir $ANUE --id-type $IDTYPE --color [True|False] --instance [True|False] --num-workers $C 173 | ``` 174 | 175 | - ANUE is the path to the AutoNUE dataset 176 | - IDTYPE can be id, csId, csTrainId, level3Id, level2Id, level1Id. 177 | - color True generates the color masks 178 | - instance True generates the instance masks with the id given by IDTYPE 179 | - panoptic True generates panoptic masks in the format similar to COCO. See the modified evaluation scripts here: https://github.com/AutoNUE/panopticapi 180 | - C is the number of threads to run in parallel 181 | 182 | For the semantic segmentation challenge, masks should be generated using IDTYPE of level3Id and used for training models (similar to trainId in cityscapes). This can be done by the command: 183 | ```bash 184 | python preperation/createLabels.py --datadir $ANUE --id-type level3Id --num-workers $C 185 | ``` 186 | For the instance segmentation challenge, instance masks should be generated by the following comand: 187 | ```bash 188 | python preperation/createLabels.py --datadir $ANUE --id-type id --num-workers $C 189 | ``` 190 | 191 | The generated files: 192 | 193 | - _gtFine_labelLevel3Ids.png will be used for semantic segmentation 194 | - _gtFine_instanceids.png will be used for instance segmentation 195 | - _gtFine_panopticLevel3Ids.png will be used for panoptic segmentation under the folder gtFine/{split}_panoptic and the gtFine/{split}_panoptic.json 196 | 197 | ### Detection 198 | 199 | We use subset of labels from helpers/anue_labels.py. 200 | 201 | We have person(level3Id: 4 , Trainable : True), rider (level3Id: 5, Trainable : True), car (level3Id: 9, Trainable : True), truck (level3Id: 10, Trainable : True), bus(level3Id: 11, Trainable : True), motorcycle(level3Id: 6, Trainable : True), bicycle(level3Id: 7, Trainable : True), autorickshaw(level3Id: 8, Trainable : True), animal(level3Id: 4 , Trainable : True), traffic light(level3Id: 18, Trainable : True), traffic sign(level3Id: 19, Trainable : True), vehicle fallback (level3Id: 12, Trainable : False), caravan (level3Id: 12, Trainable : False), trailer (level3Id: 12, Trainable : False), train (level3Id: 12, Trainable : False). 202 | 203 | Note : We train based on level3Id’s and only those labels which are mentioned as trainable and report accuracies on them. 204 | 205 | 206 | ## Viewer 207 | 208 | First generate label masks as described above. To view the ground truths / prediction masks at different levels of heirarchy use: 209 | ```bash 210 | python viewer/viewer.py ---datadir $ANUE 211 | ``` 212 | 213 | - ANUE has the folder path to the dataset or prediction masks with similar file/folder structure as dataset. 214 | 215 | TODO: Make the color map more sensible. 216 | 217 | 218 | ## Evaluation 219 | 220 | ### Semantic Segmentation 221 | 222 | First generate labels masks with level3Ids as described before. Then 223 | ```bash 224 | python evaluate/evaluate_mIoU.py --gts $GT --preds $PRED --num-workers $C 225 | ``` 226 | 227 | - GT is the folder path of ground truths containing /_gtFine_labellevel3Ids.png 228 | - PRED is the folder paths of predictions with the same folder structure and file names. 229 | - C is the number of threads to run in parallel 230 | 231 | 232 | ### Constrained Semantic Segmentation 233 | 234 | First generate labels masks with level1Ids as described before. Then 235 | ```bash 236 | python evaluate/idd_lite_evaluate_mIoU.py --gts $GT --preds $PRED --num-workers $C 237 | ``` 238 | 239 | - GT is the folder path of ground truths containing /_gtFine_labellevel1Ids.png 240 | - PRED is the folder paths of predictions with the same folder structure and file names. 241 | - C is the number of threads to run in parallel 242 | 243 | 244 | ### Instance Segmentation 245 | 246 | 247 | First generate instance label masks with ID_TYPE=id, as described before. Then 248 | ```bash 249 | python evaluate/evaluate_instance_segmentation.py --gts $GT --preds $PRED 250 | ``` 251 | 252 | - GT is the folder path of ground truths containing /_gtFine_labellevel3Ids.png 253 | - PRED is the folder paths of predictions with the same folder structure and file names. The format for predictions is the same as the cityscapes dataset. That is a .txt file where each line is of the form "