├── LICENSE ├── README.md ├── configs ├── a2d2_semantic_kitti │ ├── SHOT.yaml │ ├── baseline.yaml │ ├── xmuda.yaml │ ├── xmuda_SF.yaml │ ├── xmuda_pl.yaml │ └── xmuda_pl_SF.yaml ├── nuscenes │ ├── day_night │ │ ├── SHOT.yaml │ │ ├── baseline.yaml │ │ ├── xmuda.yaml │ │ ├── xmuda_SF.yaml │ │ ├── xmuda_pl.yaml │ │ └── xmuda_pl_SF.yaml │ └── usa_singapore │ │ ├── SHOT_1.yaml │ │ ├── SHOT_2.yaml │ │ ├── baseline.yaml │ │ ├── xmuda.yaml │ │ ├── xmuda_SF.yaml │ │ ├── xmuda_pl.yaml │ │ └── xmuda_pl_SF.yaml └── nuscenes_lidarseg_semantic_kitti │ ├── day_night │ ├── baseline_day.yaml │ ├── baseline_night.yaml │ ├── shot_day.yaml │ ├── shot_night.yaml │ ├── xmuda_day.yaml │ └── xmuda_night.yaml │ └── usa_singapore │ ├── baseline_singapore.yaml │ ├── baseline_usa.yaml │ ├── shot_singapore.yaml │ ├── shot_usa.yaml │ ├── xmuda_singapore.yaml │ └── xmuda_usa.yaml ├── setup.py ├── slurm_pseudolabel.sh ├── slurm_test.sh ├── slurm_train.sh ├── teaser.png └── xmuda ├── check_margin.py ├── common ├── config │ ├── __init__.py │ └── base.py ├── solver │ ├── __init__.py │ ├── build.py │ └── lr_scheduler.py └── utils │ ├── checkpoint.py │ ├── io.py │ ├── logger.py │ ├── metric_logger.py │ ├── sampler.py │ └── torch_util.py ├── config └── xmuda.py ├── data ├── a2d2 │ ├── a2d2_dataloader.py │ ├── preprocess.py │ └── splits.py ├── build.py ├── calc_stats.py ├── check_split_size.py ├── collate.py ├── nuscenes │ ├── nuscenes_dataloader.py │ ├── preprocess.py │ ├── projection.py │ └── splits.py ├── nuscenes_lidarseg │ ├── nuscenes_lidarseg_dataloader.py │ ├── preprocess.py │ ├── projection.py │ └── splits.py ├── semantic_kitti │ ├── preprocess.py │ ├── semantic_kitti_dataloader.py │ └── splits.py ├── stats_a2d2_semantic_kitti.txt ├── stats_day_night.txt ├── stats_usa_singapore.txt └── utils │ ├── augmentation_3d.py │ ├── evaluate.py │ ├── refine_pseudo_labels.py │ ├── turbo_cmap.py │ ├── validate.py │ └── visualize.py ├── draw_images.py ├── models ├── build.py ├── losses.py ├── metric.py ├── resnet34_unet.py ├── scn_unet.py └── xmuda_arch.py ├── refine_pseudolabels.py ├── test.py ├── train_baseline.py ├── train_xmuda.py └── train_xmuda_SF.py /README.md: -------------------------------------------------------------------------------- 1 | # SUMMIT: Source-Free Adaptation of Uni-Modal Models to Multi-Modal Targets 2 | 3 | Official code for the paper. 4 | 5 | ## Paper 6 | ![](./teaser.png) 7 | 8 | [SUMMIt: Source-Free Adaptation fo Uni-Modal Models to Multi-Modal Targets](https://arxiv.org/pdf/2308.11880v1.pdf) 9 | Cody Simons, Dripta Raychaudhuri, Sk Miraj Ahmed, Suya You, Konstantinos Karydis, Amit K. Roy-Chowdhury 10 | University of California, Riverside & Army Research Lab 11 | ICCV 2023 12 | 13 | If you find this code useful for your research, please cite our [paper](https://arxiv.org/pdf/2308.11880v1.pdf): 14 | 15 | ``` 16 | @inproceedings{simons2023summit, 17 | title={SUMMIT: Source-Free Adaptationof Uni-Modal Models to Multi-Modal Targets}, 18 | author={Simons, Cody and Raychaudhuri, Dripta and Ahmed, Sk Miraj and Karydis, Konstantinos and You, Suya and Roy-Chowdhury, Amit}, 19 | booktitle={ICCV}, 20 | year={2023} 21 | } 22 | ``` 23 | 24 | ## Preparation 25 | ### Prerequisites 26 | Tested with 27 | * PyTorch 1.4 28 | * CUDA 10.0 29 | * Python 3.8 30 | * [SparseConvNet](https://github.com/facebookresearch/SparseConvNet) 31 | * [nuscenes-devkit](https://github.com/nutonomy/nuscenes-devkit) 32 | 33 | ### Installation 34 | For installation please follow all instructions to install [xMUDA](https://github.com/valeoai/xmuda) 35 | 36 | ### Datasets 37 | Please refer to [xMUDA](https://github.com/valeoai/xmuda) for instructions on downloading the NuScenes, A2D2, and SemanticKITTI datasets. 38 | #### NuScenes LidarSeg 39 | Please download the Full dataset (v1.0) from the [NuScenes website](https://www.nuscenes.org) and extract it. 40 | 41 | You need to perform preprocessing to generate the data for xMUDA first. 42 | The preprocessing subsamples the 360° LiDAR point cloud to only keep the points that project into 43 | the front camera image. It also generates the point-wise segmentation labels using 44 | the 3D objects by checking which points lie inside the 3D boxes. 45 | All information will be stored in a pickle file (except the images which will be 46 | read frame by frame by the dataloader during training). 47 | 48 | Please edit the script `xmuda/data/nuscenes/preprocess.py` as follows and then run it. 49 | * `root_dir` should point to the root directory of the NuScenes dataset 50 | * `out_dir` should point to the desired output directory to store the pickle files 51 | 52 | ## Training 53 | ### Baseline 54 | Train the baselines (only on source) with: 55 | ``` 56 | $ python xmuda/train_baseline.py --cfg=configs/nuscenes/usa_singapore/baseline.yaml 57 | $ python xmuda/train_baseline.py --cfg=configs/nuscenes/day_night/baseline.yaml 58 | $ python xmuda/train_baseline.py --cfg=configs/a2d2_semantic_kitti/baseline.yaml 59 | ``` 60 | 61 | ### Pseudo-Label Generation 62 | After having trained the xMUDA model, generate the pseudo-labels as follows: 63 | ``` 64 | $ python xmuda/test.py --cfg=configs/nuscenes/usa_singapore/xmuda.yaml --pselab @/model_2d_100000.pth @/model_3d_100000.pth DATASET_TARGET.TEST "('train_singapore',)" 65 | ``` 66 | Note that we use the last model at 100,000 steps to exclude supervision from the validation set by picking the best 67 | weights. The pseudo labels and maximum probabilities are saved as `.npy` file. 68 | 69 | Pseudo-labels are then further refined using a script as follows. 70 | ``` 71 | $ python xmuda/refine_pseudo_labels.py /path/to/pseudo/labels /output/path --AF 72 | $ python xmuda/refine_pseudo_labels.py /path/to/pseudo/labels /output/path --EW --HT --k 0.5 73 | ``` 74 | 75 | Please edit the `pselab_paths` in the config file, e.g. `configs/nuscenes/usa_singapore/xmuda_pl_SF.yaml`, 76 | to match your path of the refined pseudo-lables. 77 | 78 | ### SUMMIT 79 | You can run the training with 80 | ``` 81 | $ python xmuda/train_xmuda_SF.py --cfg=configs/nuscenes/usa_singapore/xmuda_pl_SF.yaml 82 | ``` 83 | 84 | The output will be written to `/home//workspace/outputs/xmuda/` by 85 | default. The `OUTPUT_DIR` can be modified in the config file in 86 | (e.g. `configs/nuscenes/usa_singapore/xmuda.yaml`) or optionally at run time in the 87 | command line (dominates over config file). Note that `@` in the following example will be 88 | automatically replaced with the config path, i.e. with `nuscenes/usa_singapore/xmuda`. 89 | ``` 90 | $ python xmuda/train_xmuda_SF.py --cfg=configs/nuscenes/usa_singapore/xmuda_pl_SF.yaml OUTPUT_DIR path/to/output/directory/@ 91 | ``` 92 | 93 | You can start the trainings on the other UDA scenarios (Day/Night and A2D2/SemanticKITTI) analogously: 94 | ``` 95 | $ python xmuda/train_xmuda_SF.py --cfg=configs/nuscenes/day_night/xmuda_pl_SF.yaml 96 | $ python xmuda/train_xmuda_SF.py --cfg=configs/a2d2_semantic_kitti/xmuda_pl_SF.yaml 97 | ``` 98 | 99 | ## Testing 100 | You can provide which checkpoints you want to use for testing. We used the ones 101 | that performed best on the validation set during training (the best val iteration for 2D and 3D is 102 | shown at the end of each training). Note that `@` will be replaced 103 | by the output directory for that config file. For example: 104 | ``` 105 | $ cd 106 | $ python xmuda/test.py --cfg=configs/nuscenes/usa_singapore/xmuda_pl_SF.yaml @/model_2d_065000.pth @/model_3d_095000.pth 107 | ``` 108 | You can also provide an absolute path without `@`. 109 | 110 | ## Acknowledgements 111 | Note that this code builds on the [xMUDA](https://github.com/valeoai/xmuda) repo. 112 | 113 | ## License 114 | SUMMIT is released under the [Apache 2.0 license](./LICENSE). 115 | -------------------------------------------------------------------------------- /configs/a2d2_semantic_kitti/SHOT.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | NUM_CLASSES: 10 5 | MODEL_3D: 6 | TYPE: "SCN" 7 | DUAL_HEAD: True 8 | NUM_CLASSES: 10 9 | DATASET_SOURCE: 10 | TYPE: "A2D2SCN" 11 | TRAIN: ("train",) 12 | TEST: ("test",) 13 | A2D2SCN: 14 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/a2d2" 15 | DATASET_TARGET: 16 | TYPE: "SemanticKITTISCN" 17 | TRAIN: ("train",) 18 | VAL: ("val",) 19 | TEST: ("test",) 20 | SemanticKITTISCN: 21 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess" 22 | semantic_kitti_dir: "/data/AmitRoyChowdhury/SemanticKITTI" 23 | pselab_paths: "/home/eegrad/csimons/Projects/xmuda/refined_pselab/a2d2_semantic_kitti/unimodal_pselab.npz" 24 | DATALOADER: 25 | NUM_WORKERS: 4 26 | OPTIMIZER: 27 | TYPE: "Adam" 28 | BASE_LR: 0.001 29 | SCHEDULER: 30 | TYPE: "MultiStepLR" 31 | MultiStepLR: 32 | gamma: 0.1 33 | milestones: (80000, 90000) 34 | MAX_ITERATION: 100000 35 | TRAIN: 36 | BATCH_SIZE: 8 37 | SUMMARY_PERIOD: 50 38 | CHECKPOINT_PERIOD: 5000 39 | # CLASS_WEIGHTS: [1.89090012, 2.0585112, 3.1970535, 3.1111633, 1., 2.93751704, 1.92053733, 1.47886874, 1.04654198, 1.78266561] # Calculated on source 40 | CLASS_WEIGHTS: [1.43362546, 1.68430174, 2.54949885, 2.43392463, 1.05573959, 1.96980431, 1.33744715, 1.16531414, 1., 1.37781799] # Estimated from pseudo labels 41 | XMUDA: 42 | lambda_xm_trg: 0. 43 | lambda_ent: 0.1 44 | lambda_div: 0.001 45 | lambda_pl: 1.0 46 | ckpt_2d: '/home/eegrad/csimons/Projects/xmuda/output/76054/a2d2_semantic_kitti/baseline/model_2d_100000.pth' 47 | ckpt_3d: '/home/eegrad/csimons/Projects/xmuda/output/76054/a2d2_semantic_kitti/baseline/model_3d_100000.pth' 48 | VAL: 49 | BATCH_SIZE: 1 50 | PERIOD: 5000 51 | OUTPUT_DIR: "eval_result/@" 52 | -------------------------------------------------------------------------------- /configs/a2d2_semantic_kitti/baseline.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | NUM_CLASSES: 10 4 | MODEL_3D: 5 | TYPE: "SCN" 6 | NUM_CLASSES: 10 7 | DATASET_SOURCE: 8 | TYPE: "A2D2SCN" 9 | TRAIN: ("train",) 10 | TEST: ("test",) 11 | A2D2SCN: 12 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/a2d2" 13 | DATASET_TARGET: 14 | TYPE: "SemanticKITTISCN" 15 | VAL: ("val",) 16 | TEST: ("test",) 17 | SemanticKITTISCN: 18 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess" 19 | semantic_kitti_dir: "/data/AmitRoyChowdhury/SemanticKITTI" 20 | DATALOADER: 21 | NUM_WORKERS: 4 22 | OPTIMIZER: 23 | TYPE: "Adam" 24 | BASE_LR: 0.001 25 | SCHEDULER: 26 | TYPE: "MultiStepLR" 27 | MultiStepLR: 28 | gamma: 0.1 29 | milestones: (80000, 90000) 30 | MAX_ITERATION: 100000 31 | TRAIN: 32 | BATCH_SIZE: 8 33 | SUMMARY_PERIOD: 50 34 | CHECKPOINT_PERIOD: 5000 35 | CLASS_WEIGHTS: [1.89090012, 2.0585112, 3.1970535, 3.1111633, 1., 2.93751704, 1.92053733, 36 | 1.47886874, 1.04654198, 1.78266561] 37 | VAL: 38 | BATCH_SIZE: 1 39 | PERIOD: 5000 40 | OUTPUT_DIR: "/data/AmitRoyChowdhury/xmuda/pselab/@" 41 | #OUTPUT_DIR: "path/to/output/directory/@" # @ will be replaced with config path, e.g. nuscenes/usa_singapore/xmuda 42 | -------------------------------------------------------------------------------- /configs/a2d2_semantic_kitti/xmuda.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | NUM_CLASSES: 10 5 | MODEL_3D: 6 | TYPE: "SCN" 7 | DUAL_HEAD: True 8 | NUM_CLASSES: 10 9 | DATASET_SOURCE: 10 | TYPE: "A2D2SCN" 11 | TRAIN: ("train",) 12 | TEST: ("test",) 13 | A2D2SCN: 14 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/a2d2" 15 | DATASET_TARGET: 16 | TYPE: "SemanticKITTISCN" 17 | TRAIN: ("train",) 18 | VAL: ("val",) 19 | TEST: ("test",) 20 | SemanticKITTISCN: 21 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess" 22 | semantic_kitti_dir: "/data/AmitRoyChowdhury/SemanticKITTI" 23 | DATALOADER: 24 | NUM_WORKERS: 4 25 | OPTIMIZER: 26 | TYPE: "Adam" 27 | BASE_LR: 0.001 28 | SCHEDULER: 29 | TYPE: "MultiStepLR" 30 | MultiStepLR: 31 | gamma: 0.1 32 | milestones: (80000, 90000) 33 | MAX_ITERATION: 100000 34 | TRAIN: 35 | BATCH_SIZE: 8 36 | SUMMARY_PERIOD: 50 37 | CHECKPOINT_PERIOD: 5000 38 | CLASS_WEIGHTS: [1.89090012, 2.0585112, 3.1970535, 3.1111633, 1., 2.93751704, 1.92053733, 39 | 1.47886874, 1.04654198, 1.78266561] 40 | XMUDA: 41 | lambda_xm_src: 0.1 42 | lambda_xm_trg: 0.01 43 | VAL: 44 | BATCH_SIZE: 2 45 | PERIOD: 5000 46 | #OUTPUT_DIR: "path/to/output/directory/@" # @ will be replaced with config path, e.g. nuscenes/usa_singapore/xmuda 47 | -------------------------------------------------------------------------------- /configs/a2d2_semantic_kitti/xmuda_SF.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | NUM_CLASSES: 10 5 | MODEL_3D: 6 | TYPE: "SCN" 7 | DUAL_HEAD: True 8 | NUM_CLASSES: 10 9 | DATASET_SOURCE: 10 | TYPE: "A2D2SCN" 11 | TRAIN: ("train",) 12 | TEST: ("test",) 13 | A2D2SCN: 14 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/a2d2" 15 | DATASET_TARGET: 16 | TYPE: "SemanticKITTISCN" 17 | TRAIN: ("train",) 18 | VAL: ("val",) 19 | TEST: ("test",) 20 | SemanticKITTISCN: 21 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess" 22 | semantic_kitti_dir: "/data/AmitRoyChowdhury/SemanticKITTI" 23 | DATALOADER: 24 | NUM_WORKERS: 4 25 | OPTIMIZER: 26 | TYPE: "Adam" 27 | BASE_LR: 0.001 28 | SCHEDULER: 29 | TYPE: "MultiStepLR" 30 | MultiStepLR: 31 | gamma: 0.1 32 | milestones: (80000, 90000) 33 | MAX_ITERATION: 100000 34 | TRAIN: 35 | BATCH_SIZE: 8 36 | SUMMARY_PERIOD: 50 37 | CHECKPOINT_PERIOD: 5000 38 | CLASS_WEIGHTS: [1.89090012, 2.0585112, 3.1970535, 3.1111633, 1., 2.93751704, 1.92053733, 39 | 1.47886874, 1.04654198, 1.78266561] 40 | XMUDA: 41 | lambda_seg: 0. 42 | lambda_ent: 1.0 43 | lambda_div: 1.0 44 | VAL: 45 | BATCH_SIZE: 1 46 | PERIOD: 5000 47 | OUTPUT_DIR: "eval_result/@" 48 | -------------------------------------------------------------------------------- /configs/a2d2_semantic_kitti/xmuda_pl.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | NUM_CLASSES: 10 5 | MODEL_3D: 6 | TYPE: "SCN" 7 | DUAL_HEAD: True 8 | NUM_CLASSES: 10 9 | DATASET_SOURCE: 10 | TYPE: "A2D2SCN" 11 | TRAIN: ("train",) 12 | TEST: ("test",) 13 | A2D2SCN: 14 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/a2d2" 15 | DATASET_TARGET: 16 | TYPE: "SemanticKITTISCN" 17 | TRAIN: ("train",) 18 | VAL: ("val",) 19 | TEST: ("test",) 20 | SemanticKITTISCN: 21 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess" 22 | semantic_kitti_dir: "/data/AmitRoyChowdhury/SemanticKITTI" 23 | # pselab_paths: ("/home/docker_user/workspace/outputs/xmuda/a2d2_semantic_kitti/xmuda/pselab_data/train.npy",) 24 | DATALOADER: 25 | NUM_WORKERS: 4 26 | OPTIMIZER: 27 | TYPE: "Adam" 28 | BASE_LR: 0.001 29 | SCHEDULER: 30 | TYPE: "MultiStepLR" 31 | MultiStepLR: 32 | gamma: 0.1 33 | milestones: (80000, 90000) 34 | MAX_ITERATION: 100000 35 | TRAIN: 36 | BATCH_SIZE: 8 37 | SUMMARY_PERIOD: 50 38 | CHECKPOINT_PERIOD: 5000 39 | CLASS_WEIGHTS: [1.89090012, 2.0585112, 3.1970535, 3.1111633, 1., 2.93751704, 1.92053733, 40 | 1.47886874, 1.04654198, 1.78266561] 41 | XMUDA: 42 | lambda_xm_src: 0.1 43 | lambda_xm_trg: 0.01 44 | lambda_pl: 1.0 45 | VAL: 46 | BATCH_SIZE: 2 47 | PERIOD: 5000 48 | #OUTPUT_DIR: "path/to/output/directory/@" # @ will be replaced with config path, e.g. nuscenes/usa_singapore/xmuda 49 | -------------------------------------------------------------------------------- /configs/a2d2_semantic_kitti/xmuda_pl_SF.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | NUM_CLASSES: 10 5 | MODEL_3D: 6 | TYPE: "SCN" 7 | DUAL_HEAD: True 8 | NUM_CLASSES: 10 9 | DATASET_SOURCE: 10 | TYPE: "A2D2SCN" 11 | TRAIN: ("train",) 12 | TEST: ("test",) 13 | A2D2SCN: 14 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/a2d2" 15 | DATASET_TARGET: 16 | TYPE: "SemanticKITTISCN" 17 | TRAIN: ("train",) 18 | VAL: ("val",) 19 | TEST: ("test",) 20 | SemanticKITTISCN: 21 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess" 22 | semantic_kitti_dir: "/data/AmitRoyChowdhury/SemanticKITTI" 23 | pselab_paths: "/home/eegrad/csimons/Projects/xmuda/refined_pselab/a2d2_semantic_kitti/unimodal_pselab.npz" 24 | DATALOADER: 25 | NUM_WORKERS: 4 26 | OPTIMIZER: 27 | TYPE: "Adam" 28 | BASE_LR: 0.001 29 | SCHEDULER: 30 | TYPE: "MultiStepLR" 31 | MultiStepLR: 32 | gamma: 0.1 33 | milestones: (80000, 90000) 34 | MAX_ITERATION: 100000 35 | TRAIN: 36 | BATCH_SIZE: 8 37 | SUMMARY_PERIOD: 50 38 | CHECKPOINT_PERIOD: 5000 39 | # CLASS_WEIGHTS: [1.89090012, 2.0585112, 3.1970535, 3.1111633, 1., 2.93751704, 1.92053733, 1.47886874, 1.04654198, 1.78266561] # Calculated on source 40 | CLASS_WEIGHTS: [1.43362546, 1.68430174, 2.54949885, 2.43392463, 1.05573959, 1.96980431, 1.33744715, 1.16531414, 1., 1.37781799] # Estimated from pseudo labels 41 | XMUDA: 42 | lambda_xm_trg: 0.01 43 | lambda_ent: 0. 44 | lambda_div: 0. 45 | lambda_pl: 1.0 46 | ckpt_2d: '/home/eegrad/csimons/Projects/xmuda/output/76054/a2d2_semantic_kitti/baseline/model_2d_100000.pth' 47 | ckpt_3d: '/home/eegrad/csimons/Projects/xmuda/output/76054/a2d2_semantic_kitti/baseline/model_3d_100000.pth' 48 | VAL: 49 | BATCH_SIZE: 1 50 | PERIOD: 5000 51 | OUTPUT_DIR: "eval_result/@" 52 | RNG_SEED: -1 53 | -------------------------------------------------------------------------------- /configs/nuscenes/day_night/SHOT.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | MODEL_3D: 5 | TYPE: "SCN" 6 | DUAL_HEAD: True 7 | DATASET_SOURCE: 8 | TYPE: "NuScenesSCN" 9 | TRAIN: ("train_day",) 10 | TEST: ("test_day",) 11 | NuScenesSCN: 12 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 13 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 14 | DATASET_TARGET: 15 | TYPE: "NuScenesSCN" 16 | TRAIN: ("train_night",) 17 | VAL: ("val_night",) 18 | TEST: ("test_night",) 19 | NuScenesSCN: 20 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 21 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 22 | pselab_paths: "/home/eegrad/csimons/Projects/xmuda/refined_pselab/nuscenes/day_night/unimodal.npz" 23 | DATALOADER: 24 | NUM_WORKERS: 4 25 | OPTIMIZER: 26 | TYPE: "Adam" 27 | BASE_LR: 0.00001 28 | CLASSIFIER_LR: 0. 29 | SCHEDULER: 30 | TYPE: "MultiStepLR" 31 | MultiStepLR: 32 | gamma: 0.1 33 | milestones: (80000, 90000) 34 | MAX_ITERATION: 100000 35 | TRAIN: 36 | BATCH_SIZE: 8 37 | SUMMARY_PERIOD: 50 38 | CHECKPOINT_PERIOD: 5000 39 | # CLASS_WEIGHTS: [2.47956584, 4.26788384, 5.71114131, 3.80241668, 1.] # Calculated on source, not source free 40 | CLASS_WEIGHTS: [2.94507397, 4.21026124, 5.43160993, 3.95353505, 1.] 41 | XMUDA: 42 | lambda_ent: 0.1 43 | lambda_div: 0.001 44 | lambda_pl: 1.0 45 | pseudo_label_period: 0 46 | ckpt_2d: '/home/eegrad/csimons/Projects/xmuda/output/73301/nuscenes/usa_singapore/baseline/model_2d_100000.pth' 47 | ckpt_3d: '/home/eegrad/csimons/Projects/xmuda/output/73301/nuscenes/usa_singapore/baseline/model_3d_100000.pth' 48 | VAL: 49 | BATCH_SIZE: 32 50 | PERIOD: 5000 51 | OUTPUT_DIR: "eval_result/@" # @ will be replaced with config path, e.g. nuscenes/usa_singapore/xmuda 52 | -------------------------------------------------------------------------------- /configs/nuscenes/day_night/baseline.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | MODEL_3D: 4 | TYPE: "SCN" 5 | DATASET_SOURCE: 6 | TYPE: "NuScenesSCN" 7 | TRAIN: ("train_day",) 8 | TEST: ("test_day",) 9 | NuScenesSCN: 10 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 11 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 12 | DATASET_TARGET: 13 | TYPE: "NuScenesSCN" 14 | VAL: ("val_night",) 15 | TEST: ("test_night",) 16 | NuScenesSCN: 17 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 18 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 19 | DATALOADER: 20 | NUM_WORKERS: 4 21 | OPTIMIZER: 22 | TYPE: "Adam" 23 | BASE_LR: 0.001 24 | SCHEDULER: 25 | TYPE: "MultiStepLR" 26 | MultiStepLR: 27 | gamma: 0.1 28 | milestones: (80000, 90000) 29 | MAX_ITERATION: 100000 30 | TRAIN: 31 | BATCH_SIZE: 8 32 | SUMMARY_PERIOD: 50 33 | CHECKPOINT_PERIOD: 5000 34 | CLASS_WEIGHTS: [2.68678412, 4.36182969, 5.47896839, 3.89026883, 1.] 35 | VAL: 36 | BATCH_SIZE: 32 37 | PERIOD: 5000 38 | OUTPUT_DIR: "/data/AmitRoyChowdhury/xmuda/pselab/@" 39 | #OUTPUT_DIR: "eval_result/@" 40 | -------------------------------------------------------------------------------- /configs/nuscenes/day_night/xmuda.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | MODEL_3D: 5 | TYPE: "SCN" 6 | DUAL_HEAD: True 7 | DATASET_SOURCE: 8 | TYPE: "NuScenesSCN" 9 | TRAIN: ("train_day",) 10 | TEST: ("test_day",) 11 | NuScenesSCN: 12 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 13 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 14 | DATASET_TARGET: 15 | TYPE: "NuScenesSCN" 16 | TRAIN: ("train_night",) 17 | VAL: ("val_night",) 18 | TEST: ("test_night",) 19 | NuScenesSCN: 20 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 21 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 22 | DATALOADER: 23 | NUM_WORKERS: 4 24 | OPTIMIZER: 25 | TYPE: "Adam" 26 | BASE_LR: 0.001 27 | SCHEDULER: 28 | TYPE: "MultiStepLR" 29 | MultiStepLR: 30 | gamma: 0.1 31 | milestones: (80000, 90000) 32 | MAX_ITERATION: 100000 33 | TRAIN: 34 | BATCH_SIZE: 8 35 | SUMMARY_PERIOD: 50 36 | CHECKPOINT_PERIOD: 5000 37 | CLASS_WEIGHTS: [2.68678412, 4.36182969, 5.47896839, 3.89026883, 1.] 38 | XMUDA: 39 | lambda_xm_src: 1.0 40 | lambda_xm_trg: 0.1 41 | VAL: 42 | BATCH_SIZE: 32 43 | PERIOD: 5000 44 | #OUTPUT_DIR: "path/to/output/directory/@" # @ will be replaced with config path, e.g. nuscenes/usa_singapore/xmuda 45 | -------------------------------------------------------------------------------- /configs/nuscenes/day_night/xmuda_SF.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | MODEL_3D: 5 | TYPE: "SCN" 6 | DUAL_HEAD: True 7 | DATASET_SOURCE: 8 | TYPE: "NuScenesSCN" 9 | TRAIN: ("train_day",) 10 | TEST: ("test_day",) 11 | NuScenesSCN: 12 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 13 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 14 | DATASET_TARGET: 15 | TYPE: "NuScenesSCN" 16 | TRAIN: ("train_night",) 17 | VAL: ("val_night",) 18 | TEST: ("test_night",) 19 | NuScenesSCN: 20 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 21 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 22 | DATALOADER: 23 | NUM_WORKERS: 4 24 | OPTIMIZER: 25 | TYPE: "Adam" 26 | BASE_LR: 0.00001 27 | SCHEDULER: 28 | TYPE: "MultiStepLR" 29 | MultiStepLR: 30 | gamma: 0.1 31 | milestones: (80000, 90000) 32 | MAX_ITERATION: 100000 33 | TRAIN: 34 | BATCH_SIZE: 8 35 | SUMMARY_PERIOD: 50 36 | CHECKPOINT_PERIOD: 5000 37 | CLASS_WEIGHTS: [2.68678412, 4.36182969, 5.47896839, 3.89026883, 1.] 38 | XMUDA: 39 | lambda_ent: 1.0 40 | lambda_div: 1.0 41 | VAL: 42 | BATCH_SIZE: 32 43 | PERIOD: 5000 44 | OUTPUT_DIR: "eval_result/@" 45 | -------------------------------------------------------------------------------- /configs/nuscenes/day_night/xmuda_pl.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | MODEL_3D: 5 | TYPE: "SCN" 6 | DUAL_HEAD: True 7 | DATASET_SOURCE: 8 | TYPE: "NuScenesSCN" 9 | TRAIN: ("train_day",) 10 | TEST: ("test_day",) 11 | NuScenesSCN: 12 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes_noisy/preprocess" 13 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 14 | DATASET_TARGET: 15 | TYPE: "NuScenesSCN" 16 | TRAIN: ("train_night",) 17 | VAL: ("val_night",) 18 | TEST: ("test_night",) 19 | NuScenesSCN: 20 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes_noisy/preprocess" 21 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 22 | # pselab_paths: ("/home/docker_user/workspace/outputs/xmuda/nuscenes/day_night/xmuda/pselab_data/train_night.npy",) 23 | DATALOADER: 24 | NUM_WORKERS: 4 25 | OPTIMIZER: 26 | TYPE: "Adam" 27 | BASE_LR: 0.001 28 | SCHEDULER: 29 | TYPE: "MultiStepLR" 30 | MultiStepLR: 31 | gamma: 0.1 32 | milestones: (80000, 90000) 33 | MAX_ITERATION: 100000 34 | TRAIN: 35 | BATCH_SIZE: 8 36 | SUMMARY_PERIOD: 50 37 | CHECKPOINT_PERIOD: 5000 38 | CLASS_WEIGHTS: [2.68678412, 4.36182969, 5.47896839, 3.89026883, 1.] 39 | XMUDA: 40 | lambda_xm_src: 1.0 41 | lambda_xm_trg: 0.1 42 | lambda_pl: 1.0 43 | VAL: 44 | BATCH_SIZE: 32 45 | PERIOD: 5000 46 | #OUTPUT_DIR: "path/to/output/directory/@" # @ will be replaced with config path, e.g. nuscenes/usa_singapore/xmuda 47 | -------------------------------------------------------------------------------- /configs/nuscenes/day_night/xmuda_pl_SF.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | MODEL_3D: 5 | TYPE: "SCN" 6 | DUAL_HEAD: True 7 | DATASET_SOURCE: 8 | TYPE: "NuScenesSCN" 9 | TRAIN: ("train_day",) 10 | TEST: ("test_day",) 11 | NuScenesSCN: 12 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 13 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 14 | DATASET_TARGET: 15 | TYPE: "NuScenesSCN" 16 | TRAIN: ("train_night",) 17 | VAL: ("val_night",) 18 | TEST: ("test_night",) 19 | NuScenesSCN: 20 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 21 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 22 | pselab_paths: "/home/eegrad/csimons/Projects/xmuda/refined_pselab/nuscenes/day_night/unimodal.npz" 23 | DATALOADER: 24 | NUM_WORKERS: 4 25 | OPTIMIZER: 26 | TYPE: "Adam" 27 | BASE_LR: 0.00001 28 | SCHEDULER: 29 | TYPE: "MultiStepLR" 30 | MultiStepLR: 31 | gamma: 0.1 32 | milestones: (80000, 90000) 33 | MAX_ITERATION: 100000 34 | TRAIN: 35 | BATCH_SIZE: 8 36 | SUMMARY_PERIOD: 50 37 | CHECKPOINT_PERIOD: 5000 38 | # CLASS_WEIGHTS: [2.68678412, 4.36182969, 5.47896839, 3.89026883, 1.] # Calculated on source 39 | CLASS_WEIGHTS: [3.03536195, 4.31818534, 5.33373101, 4.35830821, 1.] # Estimated with pseudo labels 40 | XMUDA: 41 | lambda_ent: 0. 42 | lambda_div: 0. 43 | lambda_xm_trg: 0.1 44 | lambda_pl: 1.0 45 | ckpt_2d: '/home/eegrad/csimons/Projects/xmuda/output/76015/nuscenes/day_night/baseline/model_2d_100000.pth' 46 | ckpt_3d: '/home/eegrad/csimons/Projects/xmuda/output/76015/nuscenes/day_night/baseline/model_3d_100000.pth' 47 | VAL: 48 | BATCH_SIZE: 32 49 | PERIOD: 5000 50 | OUTPUT_DIR: "eval_result/@" 51 | -------------------------------------------------------------------------------- /configs/nuscenes/usa_singapore/SHOT_1.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | MODEL_3D: 5 | TYPE: "SCN" 6 | DUAL_HEAD: True 7 | DATASET_SOURCE: 8 | TYPE: "NuScenesSCN" 9 | TRAIN: ("train_usa",) 10 | TEST: ("test_usa",) 11 | NuScenesSCN: 12 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 13 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 14 | DATASET_TARGET: 15 | TYPE: "NuScenesSCN" 16 | TRAIN: ("train_singapore",) 17 | VAL: ("val_singapore",) 18 | TEST: ("test_singapore",) 19 | NuScenesSCN: 20 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 21 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 22 | pselab_paths: "/home/eegrad/csimons/Projects/xmuda/refined_pselab/nuscenes/usa_singapore/unimodal.npz" 23 | DATALOADER: 24 | NUM_WORKERS: 4 25 | OPTIMIZER: 26 | TYPE: "Adam" 27 | BASE_LR: 0.00001 28 | CLASSIFIER_LR: 0. 29 | SCHEDULER: 30 | TYPE: "MultiStepLR" 31 | MultiStepLR: 32 | gamma: 0.1 33 | milestones: (80000, 90000) 34 | MAX_ITERATION: 100000 35 | TRAIN: 36 | BATCH_SIZE: 8 37 | SUMMARY_PERIOD: 50 38 | CHECKPOINT_PERIOD: 5000 39 | # CLASS_WEIGHTS: [2.47956584, 4.26788384, 5.71114131, 3.80241668, 1.] # Calculated on source, not source free 40 | CLASS_WEIGHTS: [2.94507397, 4.21026124, 5.43160993, 3.95353505, 1.] 41 | XMUDA: 42 | lambda_ent: 0.1 43 | lambda_div: 0.001 44 | lambda_pl: 1.0 45 | pseudo_label_period: 0 46 | ckpt_2d: '/home/eegrad/csimons/Projects/xmuda/output/73301/nuscenes/usa_singapore/baseline/model_2d_100000.pth' 47 | ckpt_3d: '/home/eegrad/csimons/Projects/xmuda/output/73301/nuscenes/usa_singapore/baseline/model_3d_100000.pth' 48 | VAL: 49 | BATCH_SIZE: 32 50 | PERIOD: 5000 51 | OUTPUT_DIR: "eval_result/@" # @ will be replaced with config path, e.g. nuscenes/usa_singapore/xmuda 52 | -------------------------------------------------------------------------------- /configs/nuscenes/usa_singapore/SHOT_2.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | MODEL_3D: 5 | TYPE: "SCN" 6 | DUAL_HEAD: True 7 | DATASET_SOURCE: 8 | TYPE: "NuScenesSCN" 9 | TRAIN: ("train_usa",) 10 | TEST: ("test_usa",) 11 | NuScenesSCN: 12 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 13 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 14 | DATASET_TARGET: 15 | TYPE: "NuScenesSCN" 16 | TRAIN: ("train_singapore",) 17 | VAL: ("val_singapore",) 18 | TEST: ("test_singapore",) 19 | NuScenesSCN: 20 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 21 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 22 | pselab_paths: ("/home/eegrad/csimons/Projects/xmuda/eval_result/nuscenes/usa_singapore/baseline/pselab_data/train_singapore.npy",) 23 | DATALOADER: 24 | NUM_WORKERS: 4 25 | OPTIMIZER: 26 | TYPE: "Adam" 27 | BASE_LR: 0.00001 28 | CLASSIFIER_LR: 0. 29 | SCHEDULER: 30 | TYPE: "MultiStepLR" 31 | MultiStepLR: 32 | gamma: 0.1 33 | milestones: (80000, 90000) 34 | MAX_ITERATION: 100000 35 | TRAIN: 36 | BATCH_SIZE: 8 37 | SUMMARY_PERIOD: 50 38 | CHECKPOINT_PERIOD: 5000 39 | # CLASS_WEIGHTS: [2.47956584, 4.26788384, 5.71114131, 3.80241668, 1.] # Calculated on source, not source free 40 | CLASS_WEIGHTS: [2.94507397, 4.21026124, 5.43160993, 3.95353505, 1.] 41 | XMUDA: 42 | lambda_curr_ent: 1.0 43 | lambda_weight_div: 0.1 44 | lambda_pl: 1.0 45 | pseudo_label_period: 0 46 | ckpt_2d: '/home/eegrad/csimons/Projects/xmuda/output/73301/nuscenes/usa_singapore/baseline/model_2d_100000.pth' 47 | ckpt_3d: '/home/eegrad/csimons/Projects/xmuda/output/73301/nuscenes/usa_singapore/baseline/model_3d_100000.pth' 48 | VAL: 49 | BATCH_SIZE: 32 50 | PERIOD: 5000 51 | OUTPUT_DIR: "eval_result/@" # @ will be replaced with config path, e.g. nuscenes/usa_singapore/xmuda 52 | -------------------------------------------------------------------------------- /configs/nuscenes/usa_singapore/baseline.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | MODEL_3D: 4 | TYPE: "SCN" 5 | DATASET_SOURCE: 6 | TYPE: "NuScenesSCN" 7 | TRAIN: ("train_usa",) 8 | TEST: ("test_usa",) 9 | NuScenesSCN: 10 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 11 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 12 | DATASET_TARGET: 13 | TYPE: "NuScenesSCN" 14 | TRAIN: ("train_singapore",) 15 | VAL: ("val_singapore",) 16 | TEST: ("test_singapore",) 17 | NuScenesSCN: 18 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 19 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 20 | DATALOADER: 21 | NUM_WORKERS: 4 22 | OPTIMIZER: 23 | TYPE: "Adam" 24 | BASE_LR: 0.001 25 | SCHEDULER: 26 | TYPE: "MultiStepLR" 27 | MultiStepLR: 28 | gamma: 0.1 29 | milestones: (80000, 90000) 30 | MAX_ITERATION: 100000 31 | TRAIN: 32 | BATCH_SIZE: 8 33 | SUMMARY_PERIOD: 50 34 | CHECKPOINT_PERIOD: 5000 35 | CLASS_WEIGHTS: [2.47956584, 4.26788384, 5.71114131, 3.80241668, 1.] 36 | VAL: 37 | BATCH_SIZE: 32 38 | PERIOD: 5000 39 | OUTPUT_DIR: "/data/AmitRoyChowdhury/xmuda/pselab/@" 40 | #OUTPUT_DIR: "eval_result/@" # @ will be replaced with config path, e.g. nuscenes/usa_singapore/xmuda 41 | -------------------------------------------------------------------------------- /configs/nuscenes/usa_singapore/xmuda.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | MODEL_3D: 5 | TYPE: "SCN" 6 | DUAL_HEAD: True 7 | DATASET_SOURCE: 8 | TYPE: "NuScenesSCN" 9 | TRAIN: ("train_usa",) 10 | TEST: ("test_usa",) 11 | NuScenesSCN: 12 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 13 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 14 | DATASET_TARGET: 15 | TYPE: "NuScenesSCN" 16 | TRAIN: ("train_singapore",) 17 | VAL: ("val_singapore",) 18 | TEST: ("test_singapore",) 19 | NuScenesSCN: 20 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 21 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 22 | DATALOADER: 23 | NUM_WORKERS: 4 24 | OPTIMIZER: 25 | TYPE: "Adam" 26 | BASE_LR: 0.00001 27 | SCHEDULER: 28 | TYPE: "MultiStepLR" 29 | MultiStepLR: 30 | gamma: 0.1 31 | milestones: (80000, 90000) 32 | MAX_ITERATION: 100000 33 | TRAIN: 34 | BATCH_SIZE: 8 35 | SUMMARY_PERIOD: 50 36 | CHECKPOINT_PERIOD: 5000 37 | CLASS_WEIGHTS: [2.47956584, 4.26788384, 5.71114131, 3.80241668, 1.] 38 | XMUDA: 39 | lambda_xm_src: 0. 40 | lambda_xm_trg: 0.1 41 | VAL: 42 | BATCH_SIZE: 32 43 | PERIOD: 5000 44 | OUTPUT_DIR: "eval_result/@" # @ will be replaced with config path, e.g. nuscenes/usa_singapore/xmuda 45 | -------------------------------------------------------------------------------- /configs/nuscenes/usa_singapore/xmuda_SF.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | MODEL_3D: 5 | TYPE: "SCN" 6 | DUAL_HEAD: True 7 | DATASET_SOURCE: 8 | TYPE: "NuScenesSCN" 9 | TRAIN: ("train_usa",) 10 | TEST: ("test_usa",) 11 | NuScenesSCN: 12 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 13 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 14 | DATASET_TARGET: 15 | TYPE: "NuScenesSCN" 16 | TRAIN: ("train_singapore",) 17 | VAL: ("val_singapore",) 18 | TEST: ("test_singapore",) 19 | NuScenesSCN: 20 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 21 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 22 | DATALOADER: 23 | NUM_WORKERS: 4 24 | OPTIMIZER: 25 | TYPE: "Adam" 26 | BASE_LR: 0.00001 27 | SCHEDULER: 28 | TYPE: "MultiStepLR" 29 | MultiStepLR: 30 | gamma: 0.1 31 | milestones: (80000, 90000) 32 | MAX_ITERATION: 10000 33 | TRAIN: 34 | BATCH_SIZE: 8 35 | SUMMARY_PERIOD: 50 36 | CHECKPOINT_PERIOD: 5000 37 | CLASS_WEIGHTS: [2.47956584, 4.26788384, 5.71114131, 3.80241668, 1.] 38 | XMUDA: 39 | lambda_curr_ent: 1.0 40 | lambda_wght_div: 1.0 41 | VAL: 42 | BATCH_SIZE: 32 43 | PERIOD: 5000 44 | OUTPUT_DIR: "eval_result/@" # @ will be replaced with config path, e.g. nuscenes/usa_singapore/xmuda 45 | -------------------------------------------------------------------------------- /configs/nuscenes/usa_singapore/xmuda_pl.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | MODEL_3D: 5 | TYPE: "SCN" 6 | DUAL_HEAD: True 7 | DATASET_SOURCE: 8 | TYPE: "NuScenesSCN" 9 | TRAIN: ("train_usa",) 10 | TEST: ("test_usa",) 11 | NuScenesSCN: 12 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 13 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 14 | DATASET_TARGET: 15 | TYPE: "NuScenesSCN" 16 | TRAIN: ("train_singapore",) 17 | VAL: ("val_singapore",) 18 | TEST: ("test_singapore",) 19 | NuScenesSCN: 20 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 21 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 22 | pselab_paths: ("/home/eegrad/csimons/Projects/xmuda/eval_result/nuscenes/usa_singapore/baseline/pselab_data/train_singapore.npy",) 23 | DATALOADER: 24 | NUM_WORKERS: 4 25 | OPTIMIZER: 26 | TYPE: "Adam" 27 | BASE_LR: 0.001 28 | SCHEDULER: 29 | TYPE: "MultiStepLR" 30 | MultiStepLR: 31 | gamma: 0.1 32 | milestones: (80000, 90000) 33 | MAX_ITERATION: 100000 34 | TRAIN: 35 | BATCH_SIZE: 8 36 | SUMMARY_PERIOD: 50 37 | CHECKPOINT_PERIOD: 5000 38 | CLASS_WEIGHTS: [2.47956584, 4.26788384, 5.71114131, 3.80241668, 1.] 39 | XMUDA: 40 | # lambda_xm_src: 1.0 41 | lambda_xm_trg: 0.1 42 | lambda_pl: 1.0 43 | VAL: 44 | BATCH_SIZE: 32 45 | PERIOD: 5000 46 | OUTPUT_DIR: "eval_result/@" # @ will be replaced with config path, e.g. nuscenes/usa_singapore/xmuda 47 | -------------------------------------------------------------------------------- /configs/nuscenes/usa_singapore/xmuda_pl_SF.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | MODEL_3D: 5 | TYPE: "SCN" 6 | DUAL_HEAD: True 7 | DATASET_SOURCE: 8 | TYPE: "NuScenesSCN" 9 | TRAIN: ("train_usa",) 10 | TEST: ("test_usa",) 11 | NuScenesSCN: 12 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 13 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 14 | DATASET_TARGET: 15 | TYPE: "NuScenesSCN" 16 | TRAIN: ("train_singapore",) 17 | VAL: ("val_singapore",) 18 | TEST: ("test_singapore",) 19 | NuScenesSCN: 20 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess" 21 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 22 | pselab_paths: "/home/eegrad/csimons/Projects/xmuda/refined_pselab/nuscenes/usa_singapore/unimodal.npz" 23 | DATALOADER: 24 | NUM_WORKERS: 4 25 | OPTIMIZER: 26 | TYPE: "Adam" 27 | BASE_LR: 0.00001 28 | SCHEDULER: 29 | TYPE: "MultiStepLR" 30 | MultiStepLR: 31 | gamma: 0.1 32 | milestones: (80000, 90000) 33 | MAX_ITERATION: 100000 34 | TRAIN: 35 | BATCH_SIZE: 8 36 | SUMMARY_PERIOD: 50 37 | CHECKPOINT_PERIOD: 5000 38 | # CLASS_WEIGHTS: [2.47956584, 4.26788384, 5.71114131, 3.80241668, 1.] # Calculated on source, not source free 39 | # CLASS_WEIGHTS: [2.94507397, 4.21026124, 5.43160993, 3.95353505, 1.] 40 | CLASS_WEIGHTS: [1., 1., 1., 1., 1.] 41 | XMUDA: 42 | lambda_xm_src: 0.0 43 | lambda_xm_trg: 0.1 44 | lambda_ent: 0. 45 | lambda_div: 0. 46 | lambda_pl: 1.0 47 | pseudo_label_period: 0 48 | ckpt_2d: '/home/eegrad/csimons/Projects/xmuda/output/73301/nuscenes/usa_singapore/baseline/model_2d_100000.pth' 49 | ckpt_3d: '/home/eegrad/csimons/Projects/xmuda/output/73301/nuscenes/usa_singapore/baseline/model_3d_100000.pth' 50 | VAL: 51 | BATCH_SIZE: 32 52 | PERIOD: 5000 53 | OUTPUT_DIR: "eval_result/@" # @ will be replaced with config path, e.g. nuscenes/usa_singapore/xmuda 54 | -------------------------------------------------------------------------------- /configs/nuscenes_lidarseg_semantic_kitti/day_night/baseline_day.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | NUM_CLASSES: 10 4 | MODEL_3D: 5 | TYPE: "SCN" 6 | NUM_CLASSES: 10 7 | DATASET_SOURCE: 8 | TYPE: "NuScenesLidarSegSCN" 9 | TRAIN: ("train_day",) 10 | TEST: ("test_day",) 11 | NuScenesLidarSegSCN: 12 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes_lidarseg/preprocess" 13 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 14 | DATASET_TARGET: 15 | TYPE: "SemanticKITTISCN" 16 | VAL: ("val",) 17 | TEST: ("test",) 18 | SemanticKITTISCN: 19 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess" 20 | semantic_kitti_dir: "/data/AmitRoyChowdhury/SemanticKITTI" 21 | DATALOADER: 22 | NUM_WORKERS: 4 23 | OPTIMIZER: 24 | TYPE: "Adam" 25 | BASE_LR: 0.001 26 | SCHEDULER: 27 | TYPE: "MultiStepLR" 28 | MultiStepLR: 29 | gamma: 0.1 30 | milestones: (80000, 90000) 31 | MAX_ITERATION: 100000 32 | TRAIN: 33 | BATCH_SIZE: 8 34 | SUMMARY_PERIOD: 50 35 | CHECKPOINT_PERIOD: 5000 36 | CLASS_WEIGHTS: [2.27673237, 2.28281817, 4.07063844, 3.19973815, 1., 0., 2.09485643, 1.53682289, 1.53727826, 1.97895212] 37 | VAL: 38 | BATCH_SIZE: 8 39 | PERIOD: 5000 40 | OUTPUT_DIR: "/data/AmitRoyChowdhury/xmuda/pselab/@" 41 | #OUTPUT_DIR: "eval_result/@" 42 | -------------------------------------------------------------------------------- /configs/nuscenes_lidarseg_semantic_kitti/day_night/baseline_night.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | NUM_CLASSES: 10 4 | MODEL_3D: 5 | TYPE: "SCN" 6 | NUM_CLASSES: 10 7 | DATASET_SOURCE: 8 | TYPE: "NuScenesLidarSegSCN" 9 | TRAIN: ("train_night",) 10 | TEST: ("test_night",) 11 | NuScenesLidarSegSCN: 12 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes_lidarseg/preprocess" 13 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 14 | DATASET_TARGET: 15 | TYPE: "SemanticKITTISCN" 16 | VAL: ("val",) 17 | TEST: ("test",) 18 | SemanticKITTISCN: 19 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess" 20 | semantic_kitti_dir: "/data/AmitRoyChowdhury/SemanticKITTI" 21 | DATALOADER: 22 | NUM_WORKERS: 4 23 | OPTIMIZER: 24 | TYPE: "Adam" 25 | BASE_LR: 0.001 26 | SCHEDULER: 27 | TYPE: "MultiStepLR" 28 | MultiStepLR: 29 | gamma: 0.1 30 | milestones: (80000, 90000) 31 | MAX_ITERATION: 100000 32 | TRAIN: 33 | BATCH_SIZE: 8 34 | SUMMARY_PERIOD: 50 35 | CHECKPOINT_PERIOD: 5000 36 | CLASS_WEIGHTS: [2.27673237, 2.28281817, 4.07063844, 3.19973815, 1., 0., 2.09485643, 1.53682289, 1.53727826, 1.97895212] 37 | VAL: 38 | BATCH_SIZE: 8 39 | PERIOD: 5000 40 | OUTPUT_DIR: "/data/AmitRoyChowdhury/xmuda/pselab/@" 41 | #OUTPUT_DIR: "eval_result/@" 42 | -------------------------------------------------------------------------------- /configs/nuscenes_lidarseg_semantic_kitti/day_night/shot_day.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | NUM_CLASSES: 10 5 | MODEL_3D: 6 | TYPE: "SCN" 7 | DUAL_HEAD: True 8 | NUM_CLASSES: 10 9 | DATASET_SOURCE: 10 | TYPE: "NuScenesLidarSegSCN" 11 | TRAIN: ("train_day",) 12 | TEST: ("test_day",) 13 | NuScenesLidarSegSCN: 14 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes_lidarseg/preprocess" 15 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 16 | DATASET_TARGET: 17 | TYPE: "SemanticKITTISCN" 18 | TRAIN: ("train",) 19 | VAL: ("val",) 20 | TEST: ("test",) 21 | SemanticKITTISCN: 22 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess" 23 | semantic_kitti_dir: "/data/AmitRoyChowdhury/SemanticKITTI" 24 | pselab_paths: "/home/eegrad/csimons/Projects/xmuda/refined_pselab/nuscenes_lidarseg_semantic_kitti/day_night/baseline_day/unfiltered_pselab.npz" 25 | DATALOADER: 26 | NUM_WORKERS: 4 27 | OPTIMIZER: 28 | TYPE: "Adam" 29 | BASE_LR: 0.001 30 | SCHEDULER: 31 | TYPE: "MultiStepLR" 32 | MultiStepLR: 33 | gamma: 0.1 34 | milestones: (80000, 90000) 35 | MAX_ITERATION: 100000 36 | TRAIN: 37 | BATCH_SIZE: 8 38 | SUMMARY_PERIOD: 50 39 | CHECKPOINT_PERIOD: 5000 40 | CLASS_WEIGHTS: [2.27673237, 2.28281817, 4.07063844, 3.19973815, 1., 0., 2.09485643, 1.53682289, 1.53727826, 1.97895212] 41 | XMUDA: 42 | lambda_xm_trg: 0. 43 | lambda_ent: 0.1 44 | lambda_div: 0.001 45 | lambda_pl: 1.0 46 | ckpt_2d: '/home/eegrad/csimons/Projects/xmuda/output/109325/nuscenes_lidarseg_semantic_kitti/day_night/baseline_day/model_2d_100000.pth' 47 | ckpt_3d: '/home/eegrad/csimons/Projects/xmuda/output/109329/nuscenes_lidarseg_semantic_kitti/day_night/baseline_night/model_3d_100000.pth' 48 | VAL: 49 | BATCH_SIZE: 8 50 | PERIOD: 5000 51 | OUTPUT_DIR: "/data/AmitRoyChowdhury/xmuda/pselab/@" 52 | #OUTPUT_DIR: "eval_result/@" 53 | -------------------------------------------------------------------------------- /configs/nuscenes_lidarseg_semantic_kitti/day_night/shot_night.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | NUM_CLASSES: 10 5 | MODEL_3D: 6 | TYPE: "SCN" 7 | DUAL_HEAD: True 8 | NUM_CLASSES: 10 9 | DATASET_SOURCE: 10 | TYPE: "NuScenesLidarSegSCN" 11 | TRAIN: ("train_day",) 12 | TEST: ("test_day",) 13 | NuScenesLidarSegSCN: 14 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes_lidarseg/preprocess" 15 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 16 | DATASET_TARGET: 17 | TYPE: "SemanticKITTISCN" 18 | TRAIN: ("train",) 19 | VAL: ("val",) 20 | TEST: ("test",) 21 | SemanticKITTISCN: 22 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess" 23 | semantic_kitti_dir: "/data/AmitRoyChowdhury/SemanticKITTI" 24 | pselab_paths: "/home/eegrad/csimons/Projects/xmuda/refined_pselab/nuscenes_lidarseg_semantic_kitti/day_night/baseline_night/unfiltered_pselab.npz" 25 | DATALOADER: 26 | NUM_WORKERS: 4 27 | OPTIMIZER: 28 | TYPE: "Adam" 29 | BASE_LR: 0.001 30 | SCHEDULER: 31 | TYPE: "MultiStepLR" 32 | MultiStepLR: 33 | gamma: 0.1 34 | milestones: (80000, 90000) 35 | MAX_ITERATION: 100000 36 | TRAIN: 37 | BATCH_SIZE: 8 38 | SUMMARY_PERIOD: 50 39 | CHECKPOINT_PERIOD: 5000 40 | CLASS_WEIGHTS: [2.27673237, 2.28281817, 4.07063844, 3.19973815, 1., 0., 2.09485643, 1.53682289, 1.53727826, 1.97895212] 41 | XMUDA: 42 | lambda_xm_trg: 0. 43 | lambda_ent: 0.1 44 | lambda_div: 0.001 45 | lambda_pl: 1.0 46 | ckpt_2d: '/home/eegrad/csimons/Projects/xmuda/output/109329/nuscenes_lidarseg_semantic_kitti/day_night/baseline_night/model_2d_100000.pth' 47 | ckpt_3d: '/home/eegrad/csimons/Projects/xmuda/output/109325/nuscenes_lidarseg_semantic_kitti/day_night/baseline_day/model_3d_100000.pth' 48 | VAL: 49 | BATCH_SIZE: 8 50 | PERIOD: 5000 51 | OUTPUT_DIR: "/data/AmitRoyChowdhury/xmuda/pselab/@" 52 | #OUTPUT_DIR: "eval_result/@" 53 | -------------------------------------------------------------------------------- /configs/nuscenes_lidarseg_semantic_kitti/day_night/xmuda_day.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | NUM_CLASSES: 10 5 | MODEL_3D: 6 | TYPE: "SCN" 7 | DUAL_HEAD: True 8 | NUM_CLASSES: 10 9 | DATASET_SOURCE: 10 | TYPE: "NuScenesLidarSegSCN" 11 | TRAIN: ("train_day",) 12 | TEST: ("test_day",) 13 | NuScenesLidarSegSCN: 14 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes_lidarseg/preprocess" 15 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 16 | DATASET_TARGET: 17 | TYPE: "SemanticKITTISCN" 18 | TRAIN: ("train",) 19 | VAL: ("val",) 20 | TEST: ("test",) 21 | SemanticKITTISCN: 22 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess" 23 | semantic_kitti_dir: "/data/AmitRoyChowdhury/SemanticKITTI" 24 | pselab_paths: "/home/eegrad/csimons/Projects/xmuda/refined_pselab/nuscenes_lidarseg_semantic_kitti/day_night/baseline_day/unfiltered_pselab.npz" 25 | DATALOADER: 26 | NUM_WORKERS: 4 27 | OPTIMIZER: 28 | TYPE: "Adam" 29 | BASE_LR: 0.001 30 | SCHEDULER: 31 | TYPE: "MultiStepLR" 32 | MultiStepLR: 33 | gamma: 0.1 34 | milestones: (80000, 90000) 35 | MAX_ITERATION: 100000 36 | TRAIN: 37 | BATCH_SIZE: 8 38 | SUMMARY_PERIOD: 50 39 | CHECKPOINT_PERIOD: 5000 40 | CLASS_WEIGHTS: [2.27673237, 2.28281817, 4.07063844, 3.19973815, 1., 0., 2.09485643, 1.53682289, 1.53727826, 1.97895212] 41 | XMUDA: 42 | lambda_xm_trg: 0.0 43 | lambda_pl: 1.0 44 | ckpt_2d: '/home/eegrad/csimons/Projects/xmuda/output/109325/nuscenes_lidarseg_semantic_kitti/day_night/baseline_day/model_2d_100000.pth' 45 | ckpt_3d: '/home/eegrad/csimons/Projects/xmuda/output/109329/nuscenes_lidarseg_semantic_kitti/day_night/baseline_night/model_3d_100000.pth' 46 | VAL: 47 | BATCH_SIZE: 8 48 | PERIOD: 5000 49 | OUTPUT_DIR: "/data/AmitRoyChowdhury/xmuda/pselab/@" 50 | #OUTPUT_DIR: "eval_result/@" 51 | RNG_SEED: -1 52 | -------------------------------------------------------------------------------- /configs/nuscenes_lidarseg_semantic_kitti/day_night/xmuda_night.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | NUM_CLASSES: 10 5 | MODEL_3D: 6 | TYPE: "SCN" 7 | DUAL_HEAD: True 8 | NUM_CLASSES: 10 9 | DATASET_SOURCE: 10 | TYPE: "NuScenesLidarSegSCN" 11 | TRAIN: ("train_day",) 12 | TEST: ("test_day",) 13 | NuScenesLidarSegSCN: 14 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes_lidarseg/preprocess" 15 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 16 | DATASET_TARGET: 17 | TYPE: "SemanticKITTISCN" 18 | TRAIN: ("train",) 19 | VAL: ("val",) 20 | TEST: ("test",) 21 | SemanticKITTISCN: 22 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess" 23 | semantic_kitti_dir: "/data/AmitRoyChowdhury/SemanticKITTI" 24 | pselab_paths: "/home/eegrad/csimons/Projects/xmuda/refined_pselab/nuscenes_lidarseg_semantic_kitti/day_night/baseline_night/unfiltered_pselab.npz" 25 | DATALOADER: 26 | NUM_WORKERS: 4 27 | OPTIMIZER: 28 | TYPE: "Adam" 29 | BASE_LR: 0.001 30 | SCHEDULER: 31 | TYPE: "MultiStepLR" 32 | MultiStepLR: 33 | gamma: 0.1 34 | milestones: (80000, 90000) 35 | MAX_ITERATION: 100000 36 | TRAIN: 37 | BATCH_SIZE: 8 38 | SUMMARY_PERIOD: 50 39 | CHECKPOINT_PERIOD: 5000 40 | CLASS_WEIGHTS: [2.27673237, 2.28281817, 4.07063844, 3.19973815, 1., 0., 2.09485643, 1.53682289, 1.53727826, 1.97895212] 41 | XMUDA: 42 | lambda_xm_trg: 0.0 43 | lambda_pl: 1.0 44 | ckpt_2d: '/home/eegrad/csimons/Projects/xmuda/output/109329/nuscenes_lidarseg_semantic_kitti/day_night/baseline_night/model_2d_100000.pth' 45 | ckpt_3d: '/home/eegrad/csimons/Projects/xmuda/output/109325/nuscenes_lidarseg_semantic_kitti/day_night/baseline_day/model_3d_100000.pth' 46 | VAL: 47 | BATCH_SIZE: 8 48 | PERIOD: 5000 49 | OUTPUT_DIR: "/data/AmitRoyChowdhury/xmuda/pselab/@" 50 | #OUTPUT_DIR: "eval_result/@" 51 | -------------------------------------------------------------------------------- /configs/nuscenes_lidarseg_semantic_kitti/usa_singapore/baseline_singapore.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | NUM_CLASSES: 10 4 | DUAL_HEAD: TRUE 5 | MODEL_3D: 6 | TYPE: "SCN" 7 | NUM_CLASSES: 10 8 | DUAL_HEAD: TRUE 9 | DATASET_SOURCE: 10 | TYPE: "NuScenesLidarSegSCN" 11 | TRAIN: ("train_singapore",) 12 | TEST: ("test_singapore",) 13 | NuScenesLidarSegSCN: 14 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes_lidarseg/preprocess" 15 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 16 | DATASET_TARGET: 17 | TYPE: "SemanticKITTISCN" 18 | VAL: ("val",) 19 | TEST: ("test",) 20 | SemanticKITTISCN: 21 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess" 22 | semantic_kitti_dir: "/data/AmitRoyChowdhury/SemanticKITTI" 23 | DATALOADER: 24 | NUM_WORKERS: 4 25 | OPTIMIZER: 26 | TYPE: "Adam" 27 | BASE_LR: 0.001 28 | SCHEDULER: 29 | TYPE: "MultiStepLR" 30 | MultiStepLR: 31 | gamma: 0.1 32 | milestones: (80000, 90000) 33 | MAX_ITERATION: 100000 34 | TRAIN: 35 | BATCH_SIZE: 8 36 | SUMMARY_PERIOD: 50 37 | CHECKPOINT_PERIOD: 5000 38 | CLASS_WEIGHTS: [2.27673237, 2.28281817, 4.07063844, 3.19973815, 1., 0., 2.09485643, 1.53682289, 1.53727826, 1.97895212] 39 | VAL: 40 | BATCH_SIZE: 8 41 | PERIOD: 5000 42 | OUTPUT_DIR: "/data/AmitRoyChowdhury/xmuda/pselab/@" 43 | #OUTPUT_DIR: "eval_result/@" 44 | -------------------------------------------------------------------------------- /configs/nuscenes_lidarseg_semantic_kitti/usa_singapore/baseline_usa.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | NUM_CLASSES: 10 4 | DUAL_HEAD: TRUE 5 | MODEL_3D: 6 | TYPE: "SCN" 7 | NUM_CLASSES: 10 8 | DUAL_HEAD: TRUE 9 | DATASET_SOURCE: 10 | TYPE: "NuScenesLidarSegSCN" 11 | TRAIN: ("train_usa",) 12 | TEST: ("test_usa",) 13 | NuScenesLidarSegSCN: 14 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes_lidarseg/preprocess" 15 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 16 | DATASET_TARGET: 17 | TYPE: "SemanticKITTISCN" 18 | VAL: ("val",) 19 | TEST: ("test",) 20 | SemanticKITTISCN: 21 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess" 22 | semantic_kitti_dir: "/data/AmitRoyChowdhury/SemanticKITTI" 23 | DATALOADER: 24 | NUM_WORKERS: 4 25 | OPTIMIZER: 26 | TYPE: "Adam" 27 | BASE_LR: 0.001 28 | SCHEDULER: 29 | TYPE: "MultiStepLR" 30 | MultiStepLR: 31 | gamma: 0.1 32 | milestones: (80000, 90000) 33 | MAX_ITERATION: 100000 34 | TRAIN: 35 | BATCH_SIZE: 8 36 | SUMMARY_PERIOD: 50 37 | CHECKPOINT_PERIOD: 5000 38 | CLASS_WEIGHTS: [2.27673237, 2.28281817, 4.07063844, 3.19973815, 1., 0., 2.09485643, 1.53682289, 1.53727826, 1.97895212] 39 | VAL: 40 | BATCH_SIZE: 8 41 | PERIOD: 5000 42 | OUTPUT_DIR: "/data/AmitRoyChowdhury/xmuda/pselab/@" 43 | #OUTPUT_DIR: "eval_result/@" 44 | -------------------------------------------------------------------------------- /configs/nuscenes_lidarseg_semantic_kitti/usa_singapore/shot_singapore.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | NUM_CLASSES: 10 5 | MODEL_3D: 6 | TYPE: "SCN" 7 | DUAL_HEAD: True 8 | NUM_CLASSES: 10 9 | DATASET_SOURCE: 10 | TYPE: "NuScenesLidarSegSCN" 11 | TRAIN: ("train_day",) 12 | TEST: ("test_day",) 13 | NuScenesLidarSegSCN: 14 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes_lidarseg/preprocess" 15 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 16 | DATASET_TARGET: 17 | TYPE: "SemanticKITTISCN" 18 | TRAIN: ("train",) 19 | VAL: ("val",) 20 | TEST: ("test",) 21 | SemanticKITTISCN: 22 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess" 23 | semantic_kitti_dir: "/data/AmitRoyChowdhury/SemanticKITTI" 24 | pselab_paths: "/home/eegrad/csimons/Projects/xmuda/refined_pselab/nuscenes_lidarseg_semantic_kitti/usa_singapore/baseline_singapore/unfiltered_pselab.npz" 25 | DATALOADER: 26 | NUM_WORKERS: 4 27 | OPTIMIZER: 28 | TYPE: "Adam" 29 | BASE_LR: 0.001 30 | SCHEDULER: 31 | TYPE: "MultiStepLR" 32 | MultiStepLR: 33 | gamma: 0.1 34 | milestones: (80000, 90000) 35 | MAX_ITERATION: 100000 36 | TRAIN: 37 | BATCH_SIZE: 8 38 | SUMMARY_PERIOD: 50 39 | CHECKPOINT_PERIOD: 5000 40 | CLASS_WEIGHTS: [2.27673237, 2.28281817, 4.07063844, 3.19973815, 1., 0., 2.09485643, 1.53682289, 1.53727826, 1.97895212] 41 | XMUDA: 42 | lambda_xm_trg: 0. 43 | lambda_ent: 0.1 44 | lambda_div: 0.001 45 | lambda_pl: 1.0 46 | ckpt_2d: '/home/eegrad/csimons/Projects/xmuda/output/109331/nuscenes_lidarseg_semantic_kitti/usa_singapore/baseline_singapore/model_2d_100000.pth' 47 | ckpt_3d: '/home/eegrad/csimons/Projects/xmuda/output/109330/nuscenes_lidarseg_semantic_kitti/usa_singapore/baseline_usa/model_3d_100000.pth' 48 | VAL: 49 | BATCH_SIZE: 8 50 | PERIOD: 5000 51 | OUTPUT_DIR: "/data/AmitRoyChowdhury/xmuda/pselab/@" 52 | #OUTPUT_DIR: "eval_result/@" 53 | -------------------------------------------------------------------------------- /configs/nuscenes_lidarseg_semantic_kitti/usa_singapore/shot_usa.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | NUM_CLASSES: 10 5 | MODEL_3D: 6 | TYPE: "SCN" 7 | DUAL_HEAD: True 8 | NUM_CLASSES: 10 9 | DATASET_SOURCE: 10 | TYPE: "NuScenesLidarSegSCN" 11 | TRAIN: ("train_day",) 12 | TEST: ("test_day",) 13 | NuScenesLidarSegSCN: 14 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes_lidarseg/preprocess" 15 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 16 | DATASET_TARGET: 17 | TYPE: "SemanticKITTISCN" 18 | TRAIN: ("train",) 19 | VAL: ("val",) 20 | TEST: ("test",) 21 | SemanticKITTISCN: 22 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess" 23 | semantic_kitti_dir: "/data/AmitRoyChowdhury/SemanticKITTI" 24 | pselab_paths: "/home/eegrad/csimons/Projects/xmuda/refined_pselab/nuscenes_lidarseg_semantic_kitti/usa_singapore/baseline_usa/unfiltered_pselab.npz" 25 | DATALOADER: 26 | NUM_WORKERS: 4 27 | OPTIMIZER: 28 | TYPE: "Adam" 29 | BASE_LR: 0.001 30 | SCHEDULER: 31 | TYPE: "MultiStepLR" 32 | MultiStepLR: 33 | gamma: 0.1 34 | milestones: (80000, 90000) 35 | MAX_ITERATION: 100000 36 | TRAIN: 37 | BATCH_SIZE: 8 38 | SUMMARY_PERIOD: 50 39 | CHECKPOINT_PERIOD: 5000 40 | CLASS_WEIGHTS: [2.27673237, 2.28281817, 4.07063844, 3.19973815, 1., 0., 2.09485643, 1.53682289, 1.53727826, 1.97895212] 41 | XMUDA: 42 | lambda_xm_trg: 0. 43 | lambda_ent: 0.1 44 | lambda_div: 0.001 45 | lambda_pl: 1.0 46 | ckpt_2d: '/home/eegrad/csimons/Projects/xmuda/output/109330/nuscenes_lidarseg_semantic_kitti/usa_singapore/baseline_usa/model_2d_100000.pth' 47 | ckpt_3d: '/home/eegrad/csimons/Projects/xmuda/output/109331/nuscenes_lidarseg_semantic_kitti/usa_singapore/baseline_singapore/model_3d_100000.pth' 48 | VAL: 49 | BATCH_SIZE: 8 50 | PERIOD: 5000 51 | OUTPUT_DIR: "/data/AmitRoyChowdhury/xmuda/pselab/@" 52 | #OUTPUT_DIR: "eval_result/@" 53 | -------------------------------------------------------------------------------- /configs/nuscenes_lidarseg_semantic_kitti/usa_singapore/xmuda_singapore.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | NUM_CLASSES: 10 5 | MODEL_3D: 6 | TYPE: "SCN" 7 | DUAL_HEAD: True 8 | NUM_CLASSES: 10 9 | DATASET_SOURCE: 10 | TYPE: "NuScenesLidarSegSCN" 11 | TRAIN: ("train_day",) 12 | TEST: ("test_day",) 13 | NuScenesLidarSegSCN: 14 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes_lidarseg/preprocess" 15 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 16 | DATASET_TARGET: 17 | TYPE: "SemanticKITTISCN" 18 | TRAIN: ("train",) 19 | VAL: ("val",) 20 | TEST: ("test",) 21 | SemanticKITTISCN: 22 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess" 23 | semantic_kitti_dir: "/data/AmitRoyChowdhury/SemanticKITTI" 24 | pselab_paths: "/home/eegrad/csimons/Projects/xmuda/refined_pselab/nuscenes_lidarseg_semantic_kitti/usa_singapore/baseline_usa/unfiltered_pselab.npz" 25 | DATALOADER: 26 | NUM_WORKERS: 4 27 | OPTIMIZER: 28 | TYPE: "Adam" 29 | BASE_LR: 0.001 30 | SCHEDULER: 31 | TYPE: "MultiStepLR" 32 | MultiStepLR: 33 | gamma: 0.1 34 | milestones: (80000, 90000) 35 | MAX_ITERATION: 100000 36 | TRAIN: 37 | BATCH_SIZE: 8 38 | SUMMARY_PERIOD: 50 39 | CHECKPOINT_PERIOD: 5000 40 | CLASS_WEIGHTS: [2.27673237, 2.28281817, 4.07063844, 3.19973815, 1., 0., 2.09485643, 1.53682289, 1.53727826, 1.97895212] 41 | XMUDA: 42 | lambda_xm_trg: 0.0 43 | lambda_pl: 1.0 44 | ckpt_2d: '/home/eegrad/csimons/Projects/xmuda/output/109331/nuscenes_lidarseg_semantic_kitti/usa_singapore/baseline_singapore/model_2d_100000.pth' 45 | ckpt_3d: '/home/eegrad/csimons/Projects/xmuda/output/109330/nuscenes_lidarseg_semantic_kitti/usa_singapore/baseline_usa/model_3d_100000.pth' 46 | VAL: 47 | BATCH_SIZE: 8 48 | PERIOD: 5000 49 | OUTPUT_DIR: "/data/AmitRoyChowdhury/xmuda/pselab/@" 50 | #OUTPUT_DIR: "eval_result/@" 51 | -------------------------------------------------------------------------------- /configs/nuscenes_lidarseg_semantic_kitti/usa_singapore/xmuda_usa.yaml: -------------------------------------------------------------------------------- 1 | MODEL_2D: 2 | TYPE: "UNetResNet34" 3 | DUAL_HEAD: True 4 | NUM_CLASSES: 10 5 | MODEL_3D: 6 | TYPE: "SCN" 7 | DUAL_HEAD: True 8 | NUM_CLASSES: 10 9 | DATASET_SOURCE: 10 | TYPE: "NuScenesLidarSegSCN" 11 | TRAIN: ("train_day",) 12 | TEST: ("test_day",) 13 | NuScenesLidarSegSCN: 14 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/nuscenes_lidarseg/preprocess" 15 | nuscenes_dir: "/data/AmitRoyChowdhury/nuscenes" # only front cam images are needed 16 | DATASET_TARGET: 17 | TYPE: "SemanticKITTISCN" 18 | TRAIN: ("train",) 19 | VAL: ("val",) 20 | TEST: ("test",) 21 | SemanticKITTISCN: 22 | preprocess_dir: "/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess" 23 | semantic_kitti_dir: "/data/AmitRoyChowdhury/SemanticKITTI" 24 | pselab_paths: "/home/eegrad/csimons/Projects/xmuda/refined_pselab/nuscenes_lidarseg_semantic_kitti/usa_singapore/baseline_singapore/AF_pselab.npz" 25 | DATALOADER: 26 | NUM_WORKERS: 4 27 | OPTIMIZER: 28 | TYPE: "Adam" 29 | BASE_LR: 0.001 30 | SCHEDULER: 31 | TYPE: "MultiStepLR" 32 | MultiStepLR: 33 | gamma: 0.1 34 | milestones: (80000, 90000) 35 | MAX_ITERATION: 100000 36 | TRAIN: 37 | BATCH_SIZE: 8 38 | SUMMARY_PERIOD: 50 39 | CHECKPOINT_PERIOD: 5000 40 | CLASS_WEIGHTS: [2.27673237, 2.28281817, 4.07063844, 3.19973815, 1., 0., 2.09485643, 1.53682289, 1.53727826, 1.97895212] 41 | XMUDA: 42 | lambda_xm_trg: 0.0 43 | lambda_pl: 1.0 44 | ckpt_2d: '/home/eegrad/csimons/Projects/xmuda/output/109330/nuscenes_lidarseg_semantic_kitti/usa_singapore/baseline_usa/model_2d_100000.pth' 45 | ckpt_3d: '/home/eegrad/csimons/Projects/xmuda/output/109331/nuscenes_lidarseg_semantic_kitti/usa_singapore/baseline_singapore/model_3d_100000.pth' 46 | VAL: 47 | BATCH_SIZE: 8 48 | PERIOD: 5000 49 | OUTPUT_DIR: "/data/AmitRoyChowdhury/xmuda/pselab/@" 50 | #OUTPUT_DIR: "eval_result/@" 51 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup 2 | from setuptools import find_packages 3 | 4 | exclude_dirs = ("configs",) 5 | 6 | # for install, do: pip install -ve . 7 | 8 | setup( 9 | name='xmuda', 10 | version="0.0.1", 11 | url="https://github.com/maxjaritz/xmuda", 12 | description="xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation", 13 | install_requires=['yacs', 'nuscenes-devkit', 'tabulate'], 14 | packages=find_packages(exclude=exclude_dirs), 15 | ) -------------------------------------------------------------------------------- /slurm_pseudolabel.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -l 2 | 3 | #SBATCH --nodes=1 # Allocate *at least* 5 nodes to this job. 4 | #SBATCH --ntasks=1 # Allocate *at most* 5 tasks for job steps in the job 5 | #SBATCH --cpus-per-task=4 # Each task needs only one CPU 6 | #SBATCH --mem=32G # This particular job won't need much memory 7 | #SBATCH --time=3-00:00:00 # 3 days 8 | #SBATCH --mail-user=csimo005@ucr.edu 9 | #SBATCH --mail-type=ALL #SBATCH --job-name="xMUDA Train" 10 | #SBATCH -p vcggpu # You could pick other partitions for other jobs 11 | #SBATCH --gres=gpu:1 12 | #SBATCH --wait-all-nodes=1 # Run once all resources are available 13 | #SBATCH --output=slurm_output/output_%j-%N.txt # logging per job and per host in the current directory. Both stdout and stderr are logged. 14 | 15 | # Place any commands you want to run below 16 | # Activate conda Env 17 | conda activate xmuda2 18 | 19 | # Calculate NuScenes Pseudo Labels 20 | #python3 xmuda/test.py --cfg=configs/nuscenes/usa_singapore/xmuda_pl_SF.yaml --pselab output/83513/nuscenes/usa_singapore/xmuda_pl_SF/model_2d_100000.pth output/83513/nuscenes/usa_singapore/xmuda_pl_SF/model_3d_100000.pth DATASET_TARGET.TEST "('train_singapore',)" OUTPUT_DIR "/data/AmitRoyChowdhury/xmuda_AF/pselab/@" 21 | #python3 xmuda/test.py --cfg=configs/nuscenes/day_night/xmuda_pl_SF.yaml --pselab output/83535/nuscenes/day_night/xmuda_pl_SF/model_2d_100000.pth output/83535/nuscenes/day_night/xmuda_pl_SF/model_3d_100000.pth DATASET_TARGET.TEST "('train_night',)" OUTPUT_DIR "/data/AmitRoyChowdhury/xmuda_AF/pselab/@" 22 | 23 | 24 | # Calculate SemanticKITTI Pseudo Labels 25 | python3 xmuda/test.py --cfg=configs/a2d2_semantic_kitti/xmuda_pl_SF.yaml --pselab output/81601/a2d2_semantic_kitti/xmuda_pl_SF/model_2d_100000.pth output/81601/a2d2_semantic_kitti/xmuda_pl_SF/model_3d_100000.pth DATASET_TARGET.TEST "('train',)" OUTPUT_DIR "/data/AmitRoyChowdhury/xmuda_AF/pselab/@" 26 | 27 | -------------------------------------------------------------------------------- /slurm_train.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -l 2 | 3 | #SBATCH --nodes=1 # Allocate *at least* 5 nodes to this job. 4 | #SBATCH --ntasks=1 # Allocate *at most* 5 tasks for job steps in the job 5 | #SBATCH --cpus-per-task=4 # Each task needs only one CPU 6 | #SBATCH --mem=24G # This particular job won't need much memory 7 | #SBATCH --time=3-00:00:00 # 3 days 8 | #SBATCH --mail-user=csimo005@ucr.edu 9 | #SBATCH --mail-type=ALL 10 | #SBATCH --job-name="nuscenes" 11 | #SBATCH -p vcggpu # You could pick other partitions for other jobs 12 | #SBATCH --gres=gpu:1 13 | #SBATCH --wait-all-nodes=1 # Run once all resources are available 14 | #SBATCH --output=slurm_output/output_%j-%N.txt # logging per job and per host in the current directory. Both stdout and stderr are logged. 15 | 16 | # Place any commands you want to run below 17 | # Activate conda Env 18 | echo $CUDA_VISIBLE_DEVICES 19 | echo $SLURMD_NODENAME 20 | mamba activate xmuda2 21 | python3 xmuda/train_xmuda_SF.py --cfg=configs/nuscenes_lidarseg_semantic_kitti/usa_singapore/xmuda_usa.yaml OUTPUT_DIR output/$SLURM_JOBID/@ 22 | -------------------------------------------------------------------------------- /teaser.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/csimo005/SUMMIT/e6fa287b13436fd11bd4a01de021baa3daf49fb1/teaser.png -------------------------------------------------------------------------------- /xmuda/common/config/__init__.py: -------------------------------------------------------------------------------- 1 | from yacs.config import CfgNode 2 | 3 | 4 | def purge_cfg(cfg: CfgNode): 5 | """Purge configuration for clean logs and logical check. 6 | If a CfgNode has 'TYPE' attribute, its CfgNode children the key of which do not contain 'TYPE' will be removed. 7 | """ 8 | target_key = cfg.get('TYPE', None) 9 | removed_keys = [] 10 | for k, v in cfg.items(): 11 | if isinstance(v, CfgNode): 12 | if target_key is not None and (k != target_key): 13 | removed_keys.append(k) 14 | else: 15 | purge_cfg(v) 16 | for k in removed_keys: 17 | del cfg[k] 18 | -------------------------------------------------------------------------------- /xmuda/common/config/base.py: -------------------------------------------------------------------------------- 1 | """Basic experiments configuration 2 | For different tasks, a specific configuration might be created by importing this basic config. 3 | """ 4 | 5 | from yacs.config import CfgNode as CN 6 | 7 | # ---------------------------------------------------------------------------- # 8 | # Config definition 9 | # ---------------------------------------------------------------------------- # 10 | _C = CN() 11 | 12 | # ---------------------------------------------------------------------------- # 13 | # Resume 14 | # ---------------------------------------------------------------------------- # 15 | # Automatically resume weights from last checkpoints 16 | _C.AUTO_RESUME = True 17 | # Whether to resume the optimizer and the scheduler 18 | _C.RESUME_STATES = True 19 | # Path of weights to resume 20 | _C.RESUME_PATH = '' 21 | 22 | # ---------------------------------------------------------------------------- # 23 | # Model 24 | # ---------------------------------------------------------------------------- # 25 | _C.MODEL = CN() 26 | _C.MODEL.TYPE = '' 27 | 28 | # ---------------------------------------------------------------------------- # 29 | # DataLoader 30 | # ---------------------------------------------------------------------------- # 31 | _C.DATALOADER = CN() 32 | # Number of data loading threads 33 | _C.DATALOADER.NUM_WORKERS = 0 34 | # Whether to drop last 35 | _C.DATALOADER.DROP_LAST = True 36 | 37 | # ---------------------------------------------------------------------------- # 38 | # Optimizer 39 | # ---------------------------------------------------------------------------- # 40 | _C.OPTIMIZER = CN() 41 | _C.OPTIMIZER.TYPE = '' 42 | 43 | # Basic parameters of the optimizer 44 | # Note that the learning rate should be changed according to batch size 45 | _C.OPTIMIZER.BASE_LR = 0.001 46 | _C.OPTIMIZER.WEIGHT_DECAY = 0.0 47 | 48 | # Each of these can be specified seperately, or if left unchanged will default the above value 49 | _C.OPTIMIZER.FEAT_ENCODER_LR = -1. 50 | _C.OPTIMIZER.FEAT_ENCODER_WEIGHT_DECAY = -1. 51 | _C.OPTIMIZER.CLASSIFIER_LR = -1. 52 | _C.OPTIMIZER.CLASSIFIER_WEIGHT_DECAY = -1. 53 | _C.OPTIMIZER.XMUDA_CLASSIFIER_LR = -1. 54 | _C.OPTIMIZER.XMUDA_CLASSIFIER_WEIGHT_DECAY = -1. 55 | 56 | # Specific parameters of optimizers 57 | _C.OPTIMIZER.SGD = CN() 58 | _C.OPTIMIZER.SGD.momentum = 0.9 59 | _C.OPTIMIZER.SGD.dampening = 0.0 60 | 61 | _C.OPTIMIZER.Adam = CN() 62 | _C.OPTIMIZER.Adam.betas = (0.9, 0.999) 63 | 64 | # ---------------------------------------------------------------------------- # 65 | # Scheduler (learning rate schedule) 66 | # ---------------------------------------------------------------------------- # 67 | _C.SCHEDULER = CN() 68 | _C.SCHEDULER.TYPE = '' 69 | 70 | _C.SCHEDULER.MAX_ITERATION = 1 71 | # Minimum learning rate. 0.0 for disable. 72 | _C.SCHEDULER.CLIP_LR = 0.0 73 | 74 | # Specific parameters of schedulers 75 | _C.SCHEDULER.StepLR = CN() 76 | _C.SCHEDULER.StepLR.step_size = 0 77 | _C.SCHEDULER.StepLR.gamma = 0.1 78 | 79 | _C.SCHEDULER.MultiStepLR = CN() 80 | _C.SCHEDULER.MultiStepLR.milestones = () 81 | _C.SCHEDULER.MultiStepLR.gamma = 0.1 82 | 83 | # ---------------------------------------------------------------------------- # 84 | # Specific train options 85 | # ---------------------------------------------------------------------------- # 86 | _C.TRAIN = CN() 87 | 88 | # Batch size 89 | _C.TRAIN.BATCH_SIZE = 1 90 | # Period to save checkpoints. 0 for disable 91 | _C.TRAIN.CHECKPOINT_PERIOD = 0 92 | # Period to log training status. 0 for disable 93 | _C.TRAIN.LOG_PERIOD = 50 94 | # Period to summary training status. 0 for disable 95 | _C.TRAIN.SUMMARY_PERIOD = 0 96 | # Max number of checkpoints to keep 97 | _C.TRAIN.MAX_TO_KEEP = 100 98 | 99 | # Regex patterns of modules and/or parameters to freeze 100 | _C.TRAIN.FROZEN_PATTERNS = () 101 | 102 | # ---------------------------------------------------------------------------- # 103 | # Specific validation options 104 | # ---------------------------------------------------------------------------- # 105 | _C.VAL = CN() 106 | 107 | # Batch size 108 | _C.VAL.BATCH_SIZE = 1 109 | # Period to validate. 0 for disable 110 | _C.VAL.PERIOD = 0 111 | # Period to log validation status. 0 for disable 112 | _C.VAL.LOG_PERIOD = 20 113 | # The metric for best validation performance 114 | _C.VAL.METRIC = '' 115 | 116 | # ---------------------------------------------------------------------------- # 117 | # Misc options 118 | # ---------------------------------------------------------------------------- # 119 | # if set to @, the filename of config will be used by default 120 | _C.OUTPUT_DIR = '@' 121 | 122 | # For reproducibility...but not really because modern fast GPU libraries use 123 | # non-deterministic op implementations 124 | # -1 means use time seed. 125 | _C.RNG_SEED = 1 126 | -------------------------------------------------------------------------------- /xmuda/common/solver/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/csimo005/SUMMIT/e6fa287b13436fd11bd4a01de021baa3daf49fb1/xmuda/common/solver/__init__.py -------------------------------------------------------------------------------- /xmuda/common/solver/build.py: -------------------------------------------------------------------------------- 1 | """Build optimizers and schedulers""" 2 | import warnings 3 | import torch 4 | from .lr_scheduler import ClipLR 5 | 6 | 7 | def build_optimizer(cfg, model): 8 | name = cfg.OPTIMIZER.TYPE 9 | if name == '': 10 | warnings.warn('No optimizer is built.') 11 | return None 12 | elif hasattr(torch.optim, name): 13 | parameters = [{'params': model.feat_encoder_parameters(), 14 | 'lr': cfg.OPTIMIZER.FEAT_ENCODER_LR if cfg.OPTIMIZER.FEAT_ENCODER_LR > -1 else cfg.OPTIMIZER.BASE_LR, 15 | 'weight_decay': cfg.OPTIMIZER.FEAT_ENCODER_WEIGHT_DECAY if cfg.OPTIMIZER.FEAT_ENCODER_WEIGHT_DECAY > -1 else cfg.OPTIMIZER.WEIGHT_DECAY}, 16 | {'params': model.classifier_parameters(), 17 | 'lr': cfg.OPTIMIZER.CLASSIFIER_LR if cfg.OPTIMIZER.CLASSIFIER_LR > -1 else cfg.OPTIMIZER.BASE_LR, 18 | 'weight_decay': cfg.OPTIMIZER.CLASSIFIER_WEIGHT_DECAY if cfg.OPTIMIZER.CLASSIFIER_WEIGHT_DECAY > -1 else cfg.OPTIMIZER.WEIGHT_DECAY}] 19 | xmuda_params = model.xmuda_classifier_parameters() 20 | if not xmuda_params is None: 21 | parameters.append({'params': model.xmuda_classifier_parameters(), 22 | 'lr': cfg.OPTIMIZER.XMUDA_CLASSIFIER_LR if cfg.OPTIMIZER.XMUDA_CLASSIFIER_LR > -1 else cfg.OPTIMIZER.BASE_LR, 23 | 'weight_decay': cfg.OPTIMIZER.XMUDA_CLASSIFIER_WEIGHT_DECAY if cfg.OPTIMIZER.XMUDA_CLASSIFIER_WEIGHT_DECAY > -1 else cfg.OPTIMIZER.WEIGHT_DECAY}) 24 | return getattr(torch.optim, name)( 25 | parameters, 26 | lr=cfg.OPTIMIZER.BASE_LR, 27 | weight_decay=cfg.OPTIMIZER.WEIGHT_DECAY, 28 | **cfg.OPTIMIZER.get(name, dict()), 29 | ) 30 | else: 31 | raise ValueError('Unsupported type of optimizer.') 32 | 33 | def build_scheduler(cfg, optimizer): 34 | name = cfg.SCHEDULER.TYPE 35 | if name == '': 36 | warnings.warn('no scheduler is built.') 37 | return None 38 | elif hasattr(torch.optim.lr_scheduler, name): 39 | scheduler = getattr(torch.optim.lr_scheduler, name)( 40 | optimizer, 41 | **cfg.SCHEDULER.get(name, dict()), 42 | ) 43 | else: 44 | raise valueerror('unsupported type of scheduler.') 45 | 46 | # clip learning rate 47 | if cfg.SCHEDULER.CLIP_LR > 0.0: 48 | print('learning rate is clipped to {}'.format(cfg.SCHEDULER.CLIP_LR)) 49 | scheduler = cliplr(scheduler, min_lr=cfg.SCHEDULER.CLIP_LR) 50 | return scheduler 51 | -------------------------------------------------------------------------------- /xmuda/common/solver/lr_scheduler.py: -------------------------------------------------------------------------------- 1 | from __future__ import division 2 | from bisect import bisect_right 3 | from torch.optim.lr_scheduler import _LRScheduler, MultiStepLR 4 | 5 | 6 | class WarmupMultiStepLR(_LRScheduler): 7 | """https://github.com/facebookresearch/maskrcnn-benchmark/blob/master/maskrcnn_benchmark/solver/lr_scheduler.py""" 8 | 9 | def __init__( 10 | self, 11 | optimizer, 12 | milestones, 13 | gamma=0.1, 14 | warmup_factor=0.1, 15 | warmup_steps=1, 16 | warmup_method="linear", 17 | last_epoch=-1, 18 | ): 19 | if not list(milestones) == sorted(milestones): 20 | raise ValueError( 21 | "Milestones should be a list of" " increasing integers. Got {}", 22 | milestones, 23 | ) 24 | 25 | if warmup_method not in ("constant", "linear"): 26 | raise ValueError( 27 | "Only 'constant' or 'linear' warmup_method accepted" 28 | "got {}".format(warmup_method) 29 | ) 30 | self.milestones = milestones 31 | self.gamma = gamma 32 | self.warmup_factor = warmup_factor 33 | self.warmup_steps = warmup_steps 34 | self.warmup_method = warmup_method 35 | super(WarmupMultiStepLR, self).__init__(optimizer, last_epoch) 36 | 37 | def get_lr(self): 38 | warmup_factor = 1 39 | if self.last_epoch < self.warmup_steps: 40 | if self.warmup_method == "constant": 41 | warmup_factor = self.warmup_factor 42 | elif self.warmup_method == "linear": 43 | alpha = float(self.last_epoch) / self.warmup_steps 44 | warmup_factor = self.warmup_factor * (1 - alpha) + alpha 45 | return [ 46 | base_lr 47 | * warmup_factor 48 | * self.gamma ** bisect_right(self.milestones, self.last_epoch) 49 | for base_lr in self.base_lrs 50 | ] 51 | 52 | 53 | class ClipLR(object): 54 | """Clip the learning rate of a given scheduler. 55 | Same interfaces of _LRScheduler should be implemented. 56 | 57 | Args: 58 | scheduler (_LRScheduler): an instance of _LRScheduler. 59 | min_lr (float): minimum learning rate. 60 | 61 | """ 62 | 63 | def __init__(self, scheduler, min_lr=1e-5): 64 | assert isinstance(scheduler, _LRScheduler) 65 | self.scheduler = scheduler 66 | self.min_lr = min_lr 67 | 68 | def get_lr(self): 69 | return [max(self.min_lr, lr) for lr in self.scheduler.get_lr()] 70 | 71 | def __getattr__(self, item): 72 | if hasattr(self.scheduler, item): 73 | return getattr(self.scheduler, item) 74 | else: 75 | return getattr(self, item) 76 | -------------------------------------------------------------------------------- /xmuda/common/utils/checkpoint.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved. 2 | # Modified by Jiayuan Gu 3 | import os 4 | import logging 5 | 6 | import torch 7 | from torch.nn.parallel import DataParallel, DistributedDataParallel 8 | 9 | from .io import get_md5 10 | 11 | 12 | class Checkpointer(object): 13 | """Checkpoint the model and relevant states. 14 | 15 | Supported features: 16 | 1. Resume optimizer and scheduler 17 | 2. Automatically deal with DataParallel, DistributedDataParallel 18 | 3. Resume last saved checkpoint 19 | 20 | """ 21 | 22 | def __init__(self, 23 | model, 24 | optimizer=None, 25 | scheduler=None, 26 | save_dir='', 27 | logger=None, 28 | postfix='' 29 | ): 30 | self.model = model 31 | self.optimizer = optimizer 32 | self.scheduler = scheduler 33 | self.save_dir = save_dir 34 | # logging 35 | self.logger = logger 36 | self._print = logger.info if logger else print 37 | self.postfix = postfix 38 | 39 | def save(self, name, tag=True, **kwargs): 40 | if not self.save_dir: 41 | return 42 | 43 | data = dict() 44 | if isinstance(self.model, (DataParallel, DistributedDataParallel)): 45 | data['model'] = self.model.module.state_dict() 46 | else: 47 | data['model'] = self.model.state_dict() 48 | if self.optimizer is not None: 49 | data['optimizer'] = self.optimizer.state_dict() 50 | if self.scheduler is not None: 51 | data['scheduler'] = self.scheduler.state_dict() 52 | data.update(kwargs) 53 | 54 | save_file = os.path.join(self.save_dir, '{}.pth'.format(name)) 55 | self._print('Saving checkpoint to {}'.format(os.path.abspath(save_file))) 56 | torch.save(data, save_file) 57 | if tag: 58 | self.tag_last_checkpoint(save_file) 59 | 60 | def load(self, path=None, resume=True, resume_states=True): 61 | if resume and self.has_checkpoint(): 62 | # override argument with existing checkpoint 63 | path = self.get_checkpoint_file() 64 | if not path: 65 | # no checkpoint could be found 66 | self._print('No checkpoint found. Initializing model from scratch') 67 | return {} 68 | 69 | self._print('Loading checkpoint from {}, MD5: {}'.format(path, get_md5(path))) 70 | checkpoint = self._load_file(path) 71 | 72 | if isinstance(self.model, (DataParallel, DistributedDataParallel)): 73 | self.model.module.load_state_dict(checkpoint.pop('model')) 74 | else: 75 | self.model.load_state_dict(checkpoint.pop('model')) 76 | if resume_states: 77 | if 'optimizer' in checkpoint and self.optimizer: 78 | self.logger.info('Loading optimizer from {}'.format(path)) 79 | self.optimizer.load_state_dict(checkpoint.pop('optimizer')) 80 | if 'scheduler' in checkpoint and self.scheduler: 81 | self.logger.info('Loading scheduler from {}'.format(path)) 82 | self.scheduler.load_state_dict(checkpoint.pop('scheduler')) 83 | else: 84 | checkpoint = {} 85 | 86 | # return any further checkpoint data 87 | return checkpoint 88 | 89 | def has_checkpoint(self): 90 | save_file = os.path.join(self.save_dir, 'last_checkpoint' + self.postfix) 91 | return os.path.exists(save_file) 92 | 93 | def get_checkpoint_file(self): 94 | save_file = os.path.join(self.save_dir, 'last_checkpoint' + self.postfix) 95 | try: 96 | with open(save_file, 'r') as f: 97 | last_saved = f.read() 98 | # If not absolute path, add save_dir as prefix 99 | if not os.path.isabs(last_saved): 100 | last_saved = os.path.join(self.save_dir, last_saved) 101 | except IOError: 102 | # If file doesn't exist, maybe because it has just been 103 | # deleted by a separate process 104 | last_saved = '' 105 | return last_saved 106 | 107 | def tag_last_checkpoint(self, last_filename): 108 | save_file = os.path.join(self.save_dir, 'last_checkpoint' + self.postfix) 109 | # If not absolute path, only save basename 110 | if not os.path.isabs(last_filename): 111 | last_filename = os.path.basename(last_filename) 112 | with open(save_file, 'w') as f: 113 | f.write(last_filename) 114 | 115 | def _load_file(self, path): 116 | return torch.load(path, map_location=torch.device('cpu')) 117 | 118 | 119 | class CheckpointerV2(Checkpointer): 120 | """Support max_to_keep like tf.Saver""" 121 | 122 | def __init__(self, *args, max_to_keep=5, **kwargs): 123 | super(CheckpointerV2, self).__init__(*args, **kwargs) 124 | self.max_to_keep = max_to_keep 125 | self._last_checkpoints = [] 126 | 127 | def get_checkpoint_file(self): 128 | save_file = os.path.join(self.save_dir, 'last_checkpoint' + self.postfix) 129 | try: 130 | self._last_checkpoints = self._load_last_checkpoints(save_file) 131 | last_saved = self._last_checkpoints[-1] 132 | except (IOError, IndexError): 133 | # If file doesn't exist, maybe because it has just been 134 | # deleted by a separate process 135 | last_saved = '' 136 | return last_saved 137 | 138 | def tag_last_checkpoint(self, last_filename): 139 | save_file = os.path.join(self.save_dir, 'last_checkpoint' + self.postfix) 140 | # Remove first from list if the same name was used before. 141 | for path in self._last_checkpoints: 142 | if last_filename == path: 143 | self._last_checkpoints.remove(path) 144 | # Append new path to list 145 | self._last_checkpoints.append(last_filename) 146 | # If more than max_to_keep, remove the oldest. 147 | self._delete_old_checkpoint() 148 | # Dump last checkpoints to a file 149 | self._save_checkpoint_file(save_file) 150 | 151 | def _delete_old_checkpoint(self): 152 | if len(self._last_checkpoints) > self.max_to_keep: 153 | path = self._last_checkpoints.pop(0) 154 | try: 155 | os.remove(path) 156 | except Exception as e: 157 | logging.warning("Ignoring: %s", str(e)) 158 | 159 | def _save_checkpoint_file(self, path): 160 | with open(path, 'w') as f: 161 | lines = [] 162 | for p in self._last_checkpoints: 163 | if not os.path.isabs(p): 164 | # If not absolute path, only save basename 165 | p = os.path.basename(p) 166 | lines.append(p) 167 | f.write('\n'.join(lines)) 168 | 169 | def _load_last_checkpoints(self, path): 170 | last_checkpoints = [] 171 | with open(path, 'r') as f: 172 | for p in f.readlines(): 173 | if not os.path.isabs(p): 174 | # If not absolute path, add save_dir as prefix 175 | p = os.path.join(self.save_dir, p) 176 | last_checkpoints.append(p) 177 | return last_checkpoints 178 | -------------------------------------------------------------------------------- /xmuda/common/utils/io.py: -------------------------------------------------------------------------------- 1 | import hashlib 2 | 3 | 4 | def get_md5(filename): 5 | hash_obj = hashlib.md5() 6 | with open(filename, 'rb') as f: 7 | hash_obj.update(f.read()) 8 | return hash_obj.hexdigest() 9 | -------------------------------------------------------------------------------- /xmuda/common/utils/logger.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved. 2 | # Modified by Jiayuan Gu 3 | import logging 4 | import os 5 | import sys 6 | 7 | 8 | def setup_logger(name, save_dir, comment=''): 9 | logger = logging.getLogger(name) 10 | logger.setLevel(logging.DEBUG) 11 | ch = logging.StreamHandler(stream=sys.stdout) 12 | ch.setLevel(logging.DEBUG) 13 | formatter = logging.Formatter('%(asctime)s %(name)s %(levelname)s: %(message)s') 14 | ch.setFormatter(formatter) 15 | logger.addHandler(ch) 16 | 17 | if save_dir: 18 | filename = 'log' 19 | if comment: 20 | filename += '.' + comment 21 | log_file = os.path.join(save_dir, filename + '.txt') 22 | fh = logging.FileHandler(log_file) 23 | fh.setLevel(logging.DEBUG) 24 | fh.setFormatter(formatter) 25 | logger.addHandler(fh) 26 | 27 | return logger 28 | -------------------------------------------------------------------------------- /xmuda/common/utils/metric_logger.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved. 2 | # Modified by Jiayuan Gu 3 | from __future__ import division 4 | from collections import defaultdict 5 | from collections import deque 6 | 7 | import numpy as np 8 | import torch 9 | 10 | 11 | class AverageMeter(object): 12 | """Track a series of values and provide access to smoothed values over a 13 | window or the global series average. 14 | """ 15 | default_fmt = '{avg:.4f} ({global_avg:.4f})' 16 | default_summary_fmt = '{global_avg:.4f}' 17 | 18 | def __init__(self, window_size=20, fmt=None, summary_fmt=None): 19 | self.values = deque(maxlen=window_size) 20 | self.counts = deque(maxlen=window_size) 21 | self.sum = 0.0 22 | self.count = 0 23 | self.fmt = fmt or self.default_fmt 24 | self.summary_fmt = summary_fmt or self.default_summary_fmt 25 | 26 | def update(self, value, count=1): 27 | self.values.append(value) 28 | self.counts.append(count) 29 | self.sum += value 30 | self.count += count 31 | 32 | @property 33 | def avg(self): 34 | if np.sum(self.counts): 35 | return np.sum(self.values) / np.sum(self.counts) 36 | return 0. 37 | 38 | @property 39 | def global_avg(self): 40 | return self.sum / self.count if self.count != 0 else float('nan') 41 | 42 | def reset(self): 43 | self.values.clear() 44 | self.counts.clear() 45 | self.sum = 0.0 46 | self.count = 0 47 | 48 | def __str__(self): 49 | return self.fmt.format(avg=self.avg, global_avg=self.global_avg) 50 | 51 | @property 52 | def summary_str(self): 53 | return self.summary_fmt.format(global_avg=self.global_avg) 54 | 55 | 56 | class MetricLogger(object): 57 | """Metric logger. 58 | All the meters should implement following methods: 59 | __str__, summary_str, reset 60 | """ 61 | 62 | def __init__(self, delimiter='\t'): 63 | self.meters = defaultdict(AverageMeter) 64 | self.delimiter = delimiter 65 | 66 | def update(self, **kwargs): 67 | for k, v in kwargs.items(): 68 | if isinstance(v, torch.Tensor): 69 | count = v.numel() 70 | value = v.item() if count == 1 else v.sum().item() 71 | elif isinstance(v, np.ndarray): 72 | count = v.size 73 | value = v.item() if count == 1 else v.sum().item() 74 | else: 75 | assert isinstance(v, (float, int)) 76 | value = v 77 | count = 1 78 | self.meters[k].update(value, count) 79 | 80 | def add_meter(self, name, meter): 81 | self.meters[name] = meter 82 | 83 | def add_meters(self, meters): 84 | if not isinstance(meters, (list, tuple)): 85 | meters = [meters] 86 | for meter in meters: 87 | self.add_meter(meter.name, meter) 88 | 89 | def __getattr__(self, attr): 90 | if attr in self.meters: 91 | return self.meters[attr] 92 | return getattr(self, attr) 93 | 94 | def __str__(self): 95 | metric_str = [] 96 | for name, meter in self.meters.items(): 97 | metric_str.append('{}: {}'.format(name, str(meter))) 98 | return self.delimiter.join(metric_str) 99 | 100 | @property 101 | def summary_str(self): 102 | metric_str = [] 103 | for name, meter in self.meters.items(): 104 | metric_str.append('{}: {}'.format(name, meter.summary_str)) 105 | return self.delimiter.join(metric_str) 106 | 107 | def reset(self): 108 | for meter in self.meters.values(): 109 | meter.reset() 110 | -------------------------------------------------------------------------------- /xmuda/common/utils/sampler.py: -------------------------------------------------------------------------------- 1 | from torch.utils.data.sampler import Sampler 2 | 3 | 4 | class IterationBasedBatchSampler(Sampler): 5 | """ 6 | Wraps a BatchSampler, resampling from it until a specified number of iterations have been sampled 7 | 8 | References: 9 | https://github.com/facebookresearch/maskrcnn-benchmark/blob/master/maskrcnn_benchmark/data/samplers/iteration_based_batch_sampler.py 10 | """ 11 | 12 | def __init__(self, batch_sampler, num_iterations, start_iter=0): 13 | self.batch_sampler = batch_sampler 14 | self.num_iterations = num_iterations 15 | self.start_iter = start_iter 16 | 17 | def __iter__(self): 18 | iteration = self.start_iter 19 | while iteration < self.num_iterations: 20 | # if the underlying sampler has a set_epoch method, like 21 | # DistributedSampler, used for making each process see 22 | # a different split of the dataset, then set it 23 | if hasattr(self.batch_sampler.sampler, "set_epoch"): 24 | self.batch_sampler.sampler.set_epoch(iteration) 25 | for batch in self.batch_sampler: 26 | yield batch 27 | iteration += 1 28 | if iteration >= self.num_iterations: 29 | break 30 | 31 | def __len__(self): 32 | return self.num_iterations - self.start_iter 33 | 34 | 35 | def test_IterationBasedBatchSampler(): 36 | from torch.utils.data.sampler import SequentialSampler, RandomSampler, BatchSampler 37 | sampler = RandomSampler([i for i in range(9)]) 38 | batch_sampler = BatchSampler(sampler, batch_size=2, drop_last=True) 39 | batch_sampler = IterationBasedBatchSampler(batch_sampler, 6, start_iter=0) 40 | 41 | # check __len__ 42 | # assert len(batch_sampler) == 5 43 | for i, index in enumerate(batch_sampler): 44 | print(i, index) 45 | # assert [i * 2, i * 2 + 1] == index 46 | 47 | # # check start iter 48 | # batch_sampler.start_iter = 2 49 | # assert len(batch_sampler) == 3 50 | 51 | 52 | if __name__ == '__main__': 53 | test_IterationBasedBatchSampler() 54 | -------------------------------------------------------------------------------- /xmuda/common/utils/torch_util.py: -------------------------------------------------------------------------------- 1 | import random 2 | import numpy as np 3 | import torch 4 | 5 | 6 | def set_random_seed(seed): 7 | if seed < 0: 8 | return 9 | random.seed(seed) 10 | np.random.seed(seed) 11 | torch.manual_seed(seed) 12 | # torch.cuda.manual_seed_all(seed) 13 | 14 | 15 | def worker_init_fn(worker_id): 16 | """The function is designed for pytorch multi-process dataloader. 17 | Note that we use the pytorch random generator to generate a base_seed. 18 | Please try to be consistent. 19 | 20 | References: 21 | https://pytorch.org/docs/stable/notes/faq.html#dataloader-workers-random-seed 22 | 23 | """ 24 | base_seed = torch.IntTensor(1).random_().item() 25 | # print(worker_id, base_seed) 26 | np.random.seed(base_seed + worker_id) 27 | -------------------------------------------------------------------------------- /xmuda/config/xmuda.py: -------------------------------------------------------------------------------- 1 | """xMUDA experiments configuration""" 2 | import os.path as osp 3 | 4 | from xmuda.common.config.base import CN, _C 5 | 6 | # public alias 7 | cfg = _C 8 | _C.VAL.METRIC = 'seg_iou' 9 | 10 | # ---------------------------------------------------------------------------- # 11 | # Specific train options 12 | # ---------------------------------------------------------------------------- # 13 | _C.TRAIN.CLASS_WEIGHTS = [] 14 | 15 | # ---------------------------------------------------------------------------- # 16 | # xMUDA options 17 | # ---------------------------------------------------------------------------- # 18 | _C.TRAIN.XMUDA = CN() 19 | _C.TRAIN.XMUDA.lambda_xm_src = 0.0 20 | _C.TRAIN.XMUDA.lambda_xm_trg = 0.0 21 | _C.TRAIN.XMUDA.lambda_seg = 0.0 22 | _C.TRAIN.XMUDA.lambda_pl = 0.0 23 | _C.TRAIN.XMUDA.lambda_minent = 0.0 24 | _C.TRAIN.XMUDA.lambda_logcoral = 0.0 25 | 26 | _C.TRAIN.XMUDA.lambda_ent = 0.0 27 | _C.TRAIN.XMUDA.lambda_div = 0.0 28 | _C.TRAIN.XMUDA.lambda_curr_ent = 0.0 29 | _C.TRAIN.XMUDA.lambda_weight_div = 0.0 30 | _C.TRAIN.XMUDA.pseudo_label_period = 0 31 | 32 | _C.TRAIN.XMUDA.ckpt_2d = '' 33 | _C.TRAIN.XMUDA.ckpt_3d = '' 34 | _C.TRAIN.XMUDA.threhold_path = '' 35 | 36 | _C.TRAIN.XMUDA.LABELING = CN() 37 | _C.TRAIN.XMUDA.LABELING.noisy_label = False 38 | _C.TRAIN.XMUDA.LABELING.inverse_label = False 39 | 40 | # ---------------------------------------------------------------------------- # 41 | # Datasets 42 | # ---------------------------------------------------------------------------- # 43 | _C.DATASET_SOURCE = CN() 44 | _C.DATASET_SOURCE.TYPE = '' 45 | _C.DATASET_SOURCE.TRAIN = tuple() 46 | _C.DATASET_SOURCE.TEST = tuple() 47 | 48 | _C.DATASET_TARGET = CN() 49 | _C.DATASET_TARGET.TYPE = '' 50 | _C.DATASET_TARGET.TRAIN = tuple() 51 | _C.DATASET_TARGET.VAL = tuple() 52 | _C.DATASET_TARGET.TEST = tuple() 53 | 54 | # NuScenesSCN 55 | _C.DATASET_SOURCE.NuScenesSCN = CN() 56 | _C.DATASET_SOURCE.NuScenesSCN.preprocess_dir = '' 57 | _C.DATASET_SOURCE.NuScenesSCN.nuscenes_dir = '' 58 | _C.DATASET_SOURCE.NuScenesSCN.merge_classes = True 59 | # 3D 60 | _C.DATASET_SOURCE.NuScenesSCN.scale = 20 61 | _C.DATASET_SOURCE.NuScenesSCN.full_scale = 4096 62 | # 2D 63 | _C.DATASET_SOURCE.NuScenesSCN.use_image = True 64 | _C.DATASET_SOURCE.NuScenesSCN.resize = (400, 225) 65 | _C.DATASET_SOURCE.NuScenesSCN.image_normalizer = () 66 | # 3D augmentation 67 | _C.DATASET_SOURCE.NuScenesSCN.augmentation = CN() 68 | _C.DATASET_SOURCE.NuScenesSCN.augmentation.noisy_rot = 0.1 69 | _C.DATASET_SOURCE.NuScenesSCN.augmentation.flip_x = 0.5 70 | _C.DATASET_SOURCE.NuScenesSCN.augmentation.rot_z = 6.2831 # 2 * pi 71 | _C.DATASET_SOURCE.NuScenesSCN.augmentation.transl = True 72 | # 2D augmentation 73 | _C.DATASET_SOURCE.NuScenesSCN.augmentation.fliplr = 0.5 74 | _C.DATASET_SOURCE.NuScenesSCN.augmentation.color_jitter = (0.4, 0.4, 0.4) 75 | # copy over the same arguments to target dataset settings 76 | _C.DATASET_TARGET.NuScenesSCN = CN(_C.DATASET_SOURCE.NuScenesSCN) 77 | _C.DATASET_TARGET.NuScenesSCN.pselab_paths = None 78 | 79 | # NuScenesLidarSegSCN 80 | _C.DATASET_SOURCE.NuScenesLidarSegSCN = CN() 81 | _C.DATASET_SOURCE.NuScenesLidarSegSCN.preprocess_dir = '' 82 | _C.DATASET_SOURCE.NuScenesLidarSegSCN.nuscenes_dir = '' 83 | _C.DATASET_SOURCE.NuScenesLidarSegSCN.merge_classes = True 84 | # 3D 85 | _C.DATASET_SOURCE.NuScenesLidarSegSCN.scale = 20 86 | _C.DATASET_SOURCE.NuScenesLidarSegSCN.full_scale = 4096 87 | # 2D 88 | _C.DATASET_SOURCE.NuScenesLidarSegSCN.use_image = True 89 | _C.DATASET_SOURCE.NuScenesLidarSegSCN.resize = (400, 225) 90 | _C.DATASET_SOURCE.NuScenesLidarSegSCN.image_normalizer = () 91 | # 3D augmentation 92 | _C.DATASET_SOURCE.NuScenesLidarSegSCN.augmentation = CN() 93 | _C.DATASET_SOURCE.NuScenesLidarSegSCN.augmentation.noisy_rot = 0.1 94 | _C.DATASET_SOURCE.NuScenesLidarSegSCN.augmentation.flip_x = 0.5 95 | _C.DATASET_SOURCE.NuScenesLidarSegSCN.augmentation.rot_z = 6.2831 # 2 * pi 96 | _C.DATASET_SOURCE.NuScenesLidarSegSCN.augmentation.transl = True 97 | # 2D augmentation 98 | _C.DATASET_SOURCE.NuScenesLidarSegSCN.augmentation.fliplr = 0.5 99 | _C.DATASET_SOURCE.NuScenesLidarSegSCN.augmentation.color_jitter = (0.4, 0.4, 0.4) 100 | # copy over the same arguments to target dataset settings 101 | _C.DATASET_TARGET.NuScenesLidarSegSCN = CN(_C.DATASET_SOURCE.NuScenesLidarSegSCN) 102 | _C.DATASET_TARGET.NuScenesLidarSegSCN.pselab_paths = None 103 | 104 | # A2D2SCN 105 | _C.DATASET_SOURCE.A2D2SCN = CN() 106 | _C.DATASET_SOURCE.A2D2SCN.preprocess_dir = '' 107 | _C.DATASET_SOURCE.A2D2SCN.merge_classes = True 108 | # 3D 109 | _C.DATASET_SOURCE.A2D2SCN.scale = 20 110 | _C.DATASET_SOURCE.A2D2SCN.full_scale = 4096 111 | # 2D 112 | _C.DATASET_SOURCE.A2D2SCN.use_image = True 113 | _C.DATASET_SOURCE.A2D2SCN.resize = (480, 302) 114 | _C.DATASET_SOURCE.A2D2SCN.image_normalizer = () 115 | # 3D augmentation 116 | _C.DATASET_SOURCE.A2D2SCN.augmentation = CN() 117 | _C.DATASET_SOURCE.A2D2SCN.augmentation.noisy_rot = 0.1 118 | _C.DATASET_SOURCE.A2D2SCN.augmentation.flip_y = 0.5 119 | _C.DATASET_SOURCE.A2D2SCN.augmentation.rot_z = 6.2831 # 2 * pi 120 | _C.DATASET_SOURCE.A2D2SCN.augmentation.transl = True 121 | # 2D augmentation 122 | _C.DATASET_SOURCE.A2D2SCN.augmentation.fliplr = 0.5 123 | _C.DATASET_SOURCE.A2D2SCN.augmentation.color_jitter = (0.4, 0.4, 0.4) 124 | 125 | # SemanticKITTISCN 126 | _C.DATASET_SOURCE.SemanticKITTISCN = CN() 127 | _C.DATASET_SOURCE.SemanticKITTISCN.preprocess_dir = '' 128 | _C.DATASET_SOURCE.SemanticKITTISCN.semantic_kitti_dir = '' 129 | _C.DATASET_SOURCE.SemanticKITTISCN.merge_classes = True 130 | # 3D 131 | _C.DATASET_SOURCE.SemanticKITTISCN.scale = 20 132 | _C.DATASET_SOURCE.SemanticKITTISCN.full_scale = 4096 133 | # 2D 134 | _C.DATASET_SOURCE.SemanticKITTISCN.image_normalizer = () 135 | # 3D augmentation 136 | _C.DATASET_SOURCE.SemanticKITTISCN.augmentation = CN() 137 | _C.DATASET_SOURCE.SemanticKITTISCN.augmentation.noisy_rot = 0.1 138 | _C.DATASET_SOURCE.SemanticKITTISCN.augmentation.flip_y = 0.5 139 | _C.DATASET_SOURCE.SemanticKITTISCN.augmentation.rot_z = 6.2831 # 2 * pi 140 | _C.DATASET_SOURCE.SemanticKITTISCN.augmentation.transl = True 141 | # 2D augmentation 142 | _C.DATASET_SOURCE.SemanticKITTISCN.augmentation.bottom_crop = (480, 302) 143 | _C.DATASET_SOURCE.SemanticKITTISCN.augmentation.fliplr = 0.5 144 | _C.DATASET_SOURCE.SemanticKITTISCN.augmentation.color_jitter = (0.4, 0.4, 0.4) 145 | # copy over the same arguments to target dataset settings 146 | _C.DATASET_TARGET.SemanticKITTISCN = CN(_C.DATASET_SOURCE.SemanticKITTISCN) 147 | _C.DATASET_TARGET.SemanticKITTISCN.pselab_paths = None 148 | 149 | # Rellis3DSCN 150 | _C.DATASET_SOURCE.Rellis3DSCN = CN() 151 | _C.DATASET_SOURCE.Rellis3DSCN.preprocess_dir = '' 152 | _C.DATASET_SOURCE.Rellis3DSCN.rellis_3d_dir = '' 153 | _C.DATASET_SOURCE.Rellis3DSCN.merge_classes = True 154 | # 3D 155 | _C.DATASET_SOURCE.Rellis3DSCN.scale = 20 156 | _C.DATASET_SOURCE.Rellis3DSCN.full_scale = 4096 157 | # 2D 158 | _C.DATASET_SOURCE.Rellis3DSCN.image_normalizer = () 159 | # 3D augmentation 160 | _C.DATASET_SOURCE.Rellis3DSCN.augmentation = CN() 161 | _C.DATASET_SOURCE.Rellis3DSCN.augmentation.noisy_rot = 0.1 162 | _C.DATASET_SOURCE.Rellis3DSCN.augmentation.flip_y = 0.5 163 | _C.DATASET_SOURCE.Rellis3DSCN.augmentation.rot_z = 6.2831 # 2 * pi 164 | _C.DATASET_SOURCE.Rellis3DSCN.augmentation.transl = True 165 | # 2D augmentation 166 | _C.DATASET_SOURCE.Rellis3DSCN.augmentation.bottom_crop = (480, 302) 167 | _C.DATASET_SOURCE.Rellis3DSCN.augmentation.fliplr = 0.5 168 | _C.DATASET_SOURCE.Rellis3DSCN.augmentation.color_jitter = (0.4, 0.4, 0.4) 169 | # copy over the same arguments to target dataset settings 170 | _C.DATASET_TARGET.Rellis3DSCN = CN(_C.DATASET_SOURCE.Rellis3DSCN) 171 | _C.DATASET_TARGET.Rellis3DSCN.pselab_paths = None 172 | 173 | # ---------------------------------------------------------------------------- # 174 | # Model 2D 175 | # ---------------------------------------------------------------------------- # 176 | _C.MODEL_2D = CN() 177 | _C.MODEL_2D.TYPE = '' 178 | _C.MODEL_2D.CKPT_PATH = '' 179 | _C.MODEL_2D.NUM_CLASSES = 5 180 | _C.MODEL_2D.DUAL_HEAD = False 181 | _C.MODEL_2D.FEAT_DIM = 64 182 | # ---------------------------------------------------------------------------- # 183 | # UNetResNet34 options 184 | # ---------------------------------------------------------------------------- # 185 | _C.MODEL_2D.UNetResNet34 = CN() 186 | _C.MODEL_2D.UNetResNet34.pretrained = True 187 | 188 | # ---------------------------------------------------------------------------- # 189 | # Model 3D 190 | # ---------------------------------------------------------------------------- # 191 | _C.MODEL_3D = CN() 192 | _C.MODEL_3D.TYPE = '' 193 | _C.MODEL_3D.CKPT_PATH = '' 194 | _C.MODEL_3D.NUM_CLASSES = 5 195 | _C.MODEL_3D.DUAL_HEAD = False 196 | _C.MODEL_3D.FEAT_DIM = 16 197 | # ----------------------------------------------------------------------------- # 198 | # SCN options 199 | # ----------------------------------------------------------------------------- # 200 | _C.MODEL_3D.SCN = CN() 201 | _C.MODEL_3D.SCN.in_channels = 1 202 | _C.MODEL_3D.SCN.m = 16 # number of unet features (multiplied in each layer) 203 | _C.MODEL_3D.SCN.block_reps = 1 # block repetitions 204 | _C.MODEL_3D.SCN.residual_blocks = False # ResNet style basic blocks 205 | _C.MODEL_3D.SCN.full_scale = 4096 206 | _C.MODEL_3D.SCN.num_planes = 7 207 | 208 | # ----------------------------------------------------------------------------- # 209 | # AutoEncoder options 210 | # ----------------------------------------------------------------------------- # 211 | _C.AE = CN() 212 | _C.AE.EMBED_DIM = 64 213 | _C.AE.HIDDEN_LAYERS = 1 214 | 215 | # ----------------------------------------------------------------------------- # 216 | # Genertor options 217 | # ----------------------------------------------------------------------------- # 218 | _C.GENERATOR = CN() 219 | _C.GENERATOR.EMBED_DIM = 64 220 | 221 | # ---------------------------------------------------------------------------- # 222 | # Misc options 223 | # ---------------------------------------------------------------------------- # 224 | # @ will be replaced by config path 225 | _C.OUTPUT_DIR = osp.expanduser('~/workspace/outputs/xmuda/@') 226 | -------------------------------------------------------------------------------- /xmuda/data/a2d2/preprocess.py: -------------------------------------------------------------------------------- 1 | import os 2 | import os.path as osp 3 | import shutil 4 | import numpy as np 5 | import pickle 6 | import json 7 | from PIL import Image 8 | import cv2 9 | import glob 10 | import torch 11 | from torch.utils.data import Dataset 12 | from torch.utils.data.dataloader import DataLoader 13 | 14 | 15 | from xmuda.data.a2d2 import splits 16 | 17 | from xmuda.data.a2d2.a2d2_dataloader import A2D2Base 18 | 19 | # prevent "RuntimeError: received 0 items of ancdata" 20 | torch.multiprocessing.set_sharing_strategy('file_system') 21 | 22 | 23 | class_names_to_id = dict(zip(A2D2Base.class_names, range(len(A2D2Base.class_names)))) 24 | 25 | 26 | def undistort_image(config, image, cam_name): 27 | """copied from https://www.a2d2.audi/a2d2/en/tutorial.html""" 28 | if cam_name in ['front_left', 'front_center', 29 | 'front_right', 'side_left', 30 | 'side_right', 'rear_center']: 31 | # get parameters from config file 32 | intr_mat_undist = np.asarray(config['cameras'][cam_name]['CamMatrix']) 33 | intr_mat_dist = np.asarray(config['cameras'][cam_name]['CamMatrixOriginal']) 34 | dist_parms = np.asarray(config['cameras'][cam_name]['Distortion']) 35 | lens = config['cameras'][cam_name]['Lens'] 36 | 37 | if lens == 'Fisheye': 38 | return cv2.fisheye.undistortImage(image, intr_mat_dist, D=dist_parms, Knew=intr_mat_undist) 39 | elif lens == 'Telecam': 40 | return cv2.undistort(image, intr_mat_dist, distCoeffs=dist_parms, newCameraMatrix=intr_mat_undist) 41 | else: 42 | return image 43 | else: 44 | return image 45 | 46 | 47 | class DummyDataset(Dataset): 48 | """Use torch dataloader for multiprocessing""" 49 | def __init__(self, root_dir, scenes): 50 | self.class_names = A2D2Base.class_names.copy() 51 | self.categories = A2D2Base.categories.copy() 52 | self.root_dir = root_dir 53 | self.data = [] 54 | self.glob_frames(scenes) 55 | 56 | # load config 57 | with open(osp.join(root_dir, 'cams_lidars.json'), 'r') as f: 58 | self.config = json.load(f) 59 | 60 | # load color to class mapping 61 | with open(osp.join(root_dir, 'class_list.json'), 'r') as f: 62 | class_list = json.load(f) 63 | self.rgb_to_class = {} 64 | self.rgb_to_cls_idx = {} 65 | count = 0 66 | for k, v in class_list.items(): 67 | # hex to rgb 68 | rgb_value = tuple(int(k.lstrip('#')[i:i + 2], 16) for i in (0, 2, 4)) 69 | self.rgb_to_class[rgb_value] = v 70 | self.rgb_to_cls_idx[rgb_value] = count 71 | count += 1 72 | 73 | assert list(class_names_to_id.keys()) == list(self.rgb_to_class.values()) 74 | 75 | def glob_frames(self, scenes): 76 | for scene in scenes: 77 | cam_paths = sorted(glob.glob(osp.join(self.root_dir, scene, 'camera', 'cam_front_center', '*.png'))) 78 | for cam_path in cam_paths: 79 | basename = osp.basename(cam_path) 80 | datetime = basename[:14] 81 | assert datetime.isdigit() 82 | frame_id = basename[-13:-4] 83 | assert frame_id.isdigit() 84 | data = { 85 | 'camera_path': cam_path, 86 | 'lidar_path': osp.join(self.root_dir, scene, 'lidar', 'cam_front_center', 87 | datetime + '_lidar_frontcenter_' + frame_id + '.npz'), 88 | 'label_path': osp.join(self.root_dir, scene, 'label', 'cam_front_center', 89 | datetime + '_label_frontcenter_' + frame_id + '.png'), 90 | } 91 | for k, v in data.items(): 92 | if not osp.exists(v): 93 | raise IOError('File not found {}'.format(v)) 94 | self.data.append(data) 95 | 96 | def __getitem__(self, index): 97 | data_dict = self.data[index].copy() 98 | lidar_front_center = np.load(data_dict['lidar_path']) 99 | points = lidar_front_center['points'] 100 | if 'row' not in lidar_front_center.keys(): 101 | print('row not in lidar dict, return None, {}'.format(data_dict['lidar_path'])) 102 | return {} 103 | rows = lidar_front_center['row'].astype(np.int) 104 | cols = lidar_front_center['col'].astype(np.int) 105 | 106 | # extract 3D labels from 2D 107 | label_img = np.array(Image.open(data_dict['label_path'])) 108 | label_img = undistort_image(self.config, label_img, 'front_center') 109 | label_pc = label_img[rows, cols, :] 110 | seg_label = np.full(label_pc.shape[0], fill_value=len(self.rgb_to_cls_idx), dtype=np.int64) 111 | # map RGB label code to index 112 | for rgb_values, cls_idx in self.rgb_to_cls_idx.items(): 113 | idx = (rgb_values == label_pc).all(1) 114 | if idx.any(): 115 | seg_label[idx] = cls_idx 116 | 117 | # load image 118 | image = Image.open(data_dict['camera_path']) 119 | image_size = image.size 120 | assert image_size == (1920, 1208) 121 | # undistort 122 | image = undistort_image(self.config, np.array(image), 'front_center') 123 | # scale image points 124 | points_img = np.stack([lidar_front_center['row'], lidar_front_center['col']], 1).astype(np.float32) 125 | # check if conversion from float64 to float32 has led to image points outside of image 126 | assert np.all(points_img[:, 0] < image_size[1]) 127 | assert np.all(points_img[:, 1] < image_size[0]) 128 | 129 | data_dict['seg_label'] = seg_label.astype(np.uint8) 130 | data_dict['points'] = points.astype(np.float32) 131 | data_dict['points_img'] = points_img # row, col format, shape: (num_points, 2) 132 | data_dict['img'] = image 133 | 134 | return data_dict 135 | 136 | def __len__(self): 137 | return len(self.data) 138 | 139 | 140 | def preprocess(split_name, root_dir, out_dir): 141 | pkl_data = [] 142 | split = getattr(splits, split_name) 143 | 144 | dataloader = DataLoader(DummyDataset(root_dir, split), num_workers=8) 145 | 146 | num_skips = 0 147 | for i, data_dict in enumerate(dataloader): 148 | # data error leads to returning empty dict 149 | if not data_dict: 150 | print('empty dict, continue') 151 | num_skips += 1 152 | continue 153 | for k, v in data_dict.items(): 154 | data_dict[k] = v[0] 155 | print('{}/{} {}'.format(i, len(dataloader), data_dict['lidar_path'])) 156 | 157 | # convert to relative path 158 | lidar_path = data_dict['lidar_path'].replace(root_dir + '/', '') 159 | cam_path = data_dict['camera_path'].replace(root_dir + '/', '') 160 | 161 | # save undistorted image 162 | new_cam_path = osp.join(out_dir, cam_path) 163 | os.makedirs(osp.dirname(new_cam_path), exist_ok=True) 164 | image = Image.fromarray(data_dict['img'].numpy()) 165 | image.save(new_cam_path) 166 | 167 | # append data 168 | out_dict = { 169 | 'points': data_dict['points'].numpy(), 170 | 'seg_labels': data_dict['seg_label'].numpy(), 171 | 'points_img': data_dict['points_img'].numpy(), # row, col format, shape: (num_points, 2) 172 | 'lidar_path': lidar_path, 173 | 'camera_path': cam_path, 174 | } 175 | pkl_data.append(out_dict) 176 | 177 | print('Skipped {} files'.format(num_skips)) 178 | 179 | # save to pickle file 180 | save_dir = osp.join(out_dir, 'preprocess') 181 | os.makedirs(save_dir, exist_ok=True) 182 | save_path = osp.join(save_dir, '{}.pkl'.format(split_name)) 183 | with open(save_path, 'wb') as f: 184 | pickle.dump(pkl_data, f) 185 | print('Wrote preprocessed data to ' + save_path) 186 | 187 | 188 | if __name__ == '__main__': 189 | root_dir = '/data/AmitRoyChowdhury/a2d2/camera_lidar_semantic' 190 | out_dir = '/data/AmitRoyChowdhury/xmuda/a2d2' 191 | preprocess('test', root_dir, out_dir) 192 | # split into train1 and train2 to prevent segmentation fault in torch dataloader 193 | preprocess('train1', root_dir, out_dir) 194 | preprocess('train2', root_dir, out_dir) 195 | # merge train1 and train2 196 | data = [] 197 | for curr_split in ['train1', 'train2']: 198 | with open(osp.join(out_dir, 'preprocess', curr_split + '.pkl'), 'rb') as f: 199 | data.extend(pickle.load(f)) 200 | save_path = osp.join(out_dir, 'preprocess', 'train.pkl') 201 | with open(save_path, 'wb') as f: 202 | pickle.dump(data, f) 203 | print('Wrote preprocessed data to ' + save_path) 204 | for curr_split in ['train1', 'train2']: 205 | os.remove(osp.join(out_dir, 'preprocess', curr_split + '.pkl')) 206 | 207 | # copy cams_lidars.json and class_list.json to out_dir 208 | for filename in ['cams_lidars.json', 'class_list.json']: 209 | shutil.copyfile(osp.join(root_dir, filename), osp.join(out_dir, filename)) 210 | -------------------------------------------------------------------------------- /xmuda/data/a2d2/splits.py: -------------------------------------------------------------------------------- 1 | train = [ 2 | '20180810_142822', 3 | '20180925_101535', 4 | '20180925_112730', 5 | '20180925_124435', 6 | '20180925_135056', 7 | '20181008_095521', 8 | '20181016_082154', 9 | '20181016_125231', 10 | '20181107_132300', 11 | '20181107_132730', 12 | '20181107_133258', 13 | '20181107_133445', 14 | '20181108_084007', 15 | '20181108_091945', 16 | '20181108_103155', 17 | '20181108_123750', 18 | '20181108_141609', 19 | '20181204_135952', 20 | '20181204_154421', 21 | '20181204_170238', 22 | ] 23 | 24 | 25 | train1 = [ 26 | '20180810_142822', 27 | '20180925_101535', 28 | '20180925_112730', 29 | '20180925_124435', 30 | '20180925_135056', 31 | '20181008_095521', 32 | '20181016_082154', 33 | '20181016_125231', 34 | '20181107_132300', 35 | '20181107_132730', 36 | ] 37 | train2 = [ 38 | '20181107_133258', 39 | '20181107_133445', 40 | '20181108_084007', 41 | '20181108_091945', 42 | '20181108_103155', 43 | '20181108_123750', 44 | '20181108_141609', 45 | '20181204_135952', 46 | '20181204_154421', 47 | '20181204_170238', 48 | ] 49 | 50 | test = [ 51 | '20180807_145028' 52 | ] 53 | 54 | all = [ 55 | '20180807_145028', 56 | '20180810_142822', 57 | '20180925_101535', 58 | '20180925_112730', 59 | '20180925_124435', 60 | '20180925_135056', 61 | '20181008_095521', 62 | '20181016_082154', 63 | # '20181016_095036', # no lidar 64 | '20181016_125231', 65 | '20181107_132300', 66 | '20181107_132730', 67 | '20181107_133258', 68 | '20181107_133445', 69 | '20181108_084007', 70 | '20181108_091945', 71 | '20181108_103155', 72 | '20181108_123750', 73 | '20181108_141609', 74 | '20181204_135952', 75 | '20181204_154421', 76 | '20181204_170238', 77 | # '20181204_191844', # no lidar 78 | ] 79 | -------------------------------------------------------------------------------- /xmuda/data/build.py: -------------------------------------------------------------------------------- 1 | from torch.utils.data.sampler import RandomSampler, BatchSampler 2 | from torch.utils.data.dataloader import DataLoader, default_collate 3 | from yacs.config import CfgNode as CN 4 | 5 | from xmuda.common.utils.torch_util import worker_init_fn 6 | from xmuda.data.collate import get_collate_scn 7 | from xmuda.common.utils.sampler import IterationBasedBatchSampler 8 | from xmuda.data.nuscenes.nuscenes_dataloader import NuScenesSCN 9 | from xmuda.data.nuscenes_lidarseg.nuscenes_lidarseg_dataloader import NuScenesLidarSegSCN 10 | from xmuda.data.a2d2.a2d2_dataloader import A2D2SCN 11 | from xmuda.data.semantic_kitti.semantic_kitti_dataloader import SemanticKITTISCN 12 | from xmuda.data.rellis_3d.rellis_3d_dataloader import Rellis3DSCN 13 | 14 | 15 | def build_dataloader(cfg, mode='train', domain='source', start_iteration=0, halve_batch_size=False): 16 | assert mode in ['train', 'val', 'test', 'train_labeled', 'train_unlabeled'] 17 | dataset_cfg = cfg.get('DATASET_' + domain.upper()) 18 | split = dataset_cfg[mode.upper()] 19 | is_train = 'train' in mode 20 | batch_size = cfg['TRAIN'].BATCH_SIZE if is_train else cfg['VAL'].BATCH_SIZE 21 | if halve_batch_size: 22 | batch_size = batch_size // 2 23 | 24 | # build dataset 25 | # Make a copy of dataset_kwargs so that we can pop augmentation afterwards without destroying the cfg. 26 | # Note that the build_dataloader fn is called twice for train and val. 27 | dataset_kwargs = CN(dataset_cfg.get(dataset_cfg.TYPE, dict())) 28 | if 'SCN' in cfg.MODEL_3D.keys(): 29 | assert dataset_kwargs.full_scale == cfg.MODEL_3D.SCN.full_scale 30 | augmentation = dataset_kwargs.pop('augmentation') 31 | augmentation = augmentation if is_train else dict() 32 | # use pselab_paths only when training on target 33 | if domain == 'target' and not is_train: 34 | dataset_kwargs.pop('pselab_paths') 35 | if dataset_cfg.TYPE == 'NuScenesSCN': 36 | dataset = NuScenesSCN(split=split, 37 | output_orig=not is_train, 38 | **dataset_kwargs, 39 | **augmentation) 40 | elif dataset_cfg.TYPE == 'NuScenesLidarSegSCN': 41 | dataset = NuScenesLidarSegSCN(split=split, 42 | output_orig=not is_train, 43 | **dataset_kwargs, 44 | **augmentation) 45 | elif dataset_cfg.TYPE == 'A2D2SCN': 46 | dataset = A2D2SCN(split=split, 47 | output_orig=not is_train, 48 | **dataset_kwargs, 49 | **augmentation) 50 | elif dataset_cfg.TYPE == 'SemanticKITTISCN': 51 | dataset = SemanticKITTISCN(split=split, 52 | output_orig=not is_train, 53 | **dataset_kwargs, 54 | **augmentation) 55 | elif dataset_cfg.TYPE == 'Rellis3DSCN': 56 | dataset = Rellis3DSCN(split=split, 57 | output_orig=not is_train, 58 | **dataset_kwargs, 59 | **augmentation) 60 | else: 61 | raise ValueError('Unsupported type of dataset: {}.'.format(dataset_cfg.TYPE)) 62 | 63 | if 'SCN' in dataset_cfg.TYPE: 64 | collate_fn = get_collate_scn(is_train) 65 | else: 66 | collate_fn = default_collate 67 | 68 | if is_train: 69 | sampler = RandomSampler(dataset) 70 | batch_sampler = BatchSampler(sampler, batch_size=batch_size, drop_last=cfg.DATALOADER.DROP_LAST) 71 | batch_sampler = IterationBasedBatchSampler(batch_sampler, cfg.SCHEDULER.MAX_ITERATION, start_iteration) 72 | dataloader = DataLoader( 73 | dataset, 74 | batch_sampler=batch_sampler, 75 | num_workers=cfg.DATALOADER.NUM_WORKERS, 76 | worker_init_fn=worker_init_fn, 77 | collate_fn=collate_fn 78 | ) 79 | else: 80 | dataloader = DataLoader( 81 | dataset, 82 | batch_size=batch_size, 83 | drop_last=False, 84 | num_workers=cfg.DATALOADER.NUM_WORKERS, 85 | worker_init_fn=worker_init_fn, 86 | collate_fn=collate_fn 87 | ) 88 | 89 | return dataloader 90 | -------------------------------------------------------------------------------- /xmuda/data/calc_stats.py: -------------------------------------------------------------------------------- 1 | from xmuda.data.nuscenes.nuscenes_dataloader import NuScenesSCN 2 | from xmuda.data.nuscenes_lidarseg.nuscenes_lidarseg_dataloader import NuScenesLidarSegSCN 3 | from xmuda.data.a2d2.a2d2_dataloader import A2D2SCN 4 | from xmuda.data.semantic_kitti.semantic_kitti_dataloader import SemanticKITTISCN 5 | 6 | import numpy as np 7 | 8 | def calcStats(dataset): 9 | stats = {} 10 | stats['length'] = len(dataset) 11 | stats['img_sizes'] = {} 12 | stats['num_points'] = {} 13 | stats['total_points'] = 0 14 | stats['class_dist'] = np.zeros((len(dataset.categories),)) 15 | 16 | for i, sample in enumerate(dataset): 17 | im_sz = sample['img'].shape 18 | if im_sz in stats['img_sizes']: 19 | stats['img_sizes'][im_sz] += 1 20 | else: 21 | stats['img_sizes'][im_sz] = 1 22 | 23 | num_points = sample['coords'].shape[0] 24 | if num_points in stats['num_points']: 25 | stats['num_points'][num_points] += 1 26 | else: 27 | stats['num_points'][num_points] = 1 28 | stats['total_points'] += num_points 29 | 30 | val, cnt = np.unique(sample['seg_label'], return_counts=True) 31 | for v, c in zip(val, cnt): 32 | if v != -100: 33 | stats['class_dist'][v] += c 34 | 35 | print('{}/{}'.format(i, stats['length']), end='\r') 36 | 37 | print('{}/{}'.format(stats['length'], stats['length'])) 38 | stats['class_dist'] = stats['class_dist']/np.sum(stats['class_dist']) 39 | return stats 40 | 41 | def printStats(stats): 42 | print('Length: {}'.format(stats['length'])) 43 | print('Image Sizes:') 44 | for sz in stats['img_sizes']: 45 | print('\t{}: {}'.format(sz, stats['img_sizes'][sz])) 46 | 47 | print('Total Points: {}'.format(stats['total_points'])) 48 | print('Average Points: {}'.format(stats['total_points']/float(stats['length']))) 49 | print('Class Counts: {}'.format(stats['class_dist'])) 50 | 51 | dataset = NuScenesLidarSegSCN(split=('test_usa',), 52 | nuscenes_dir='/data/AmitRoyChowdhury/nuscenes/', 53 | preprocess_dir='/data/AmitRoyChowdhury/xmuda/nuscenes_lidarseg/preprocess', 54 | use_image=True, 55 | resize=None, 56 | merge_classes=True) 57 | printStats(calcStats(dataset)) 58 | dataset = NuScenesLidarSegSCN(split=('test_singapore',), 59 | nuscenes_dir='/data/AmitRoyChowdhury/nuscenes/', 60 | preprocess_dir='/data/AmitRoyChowdhury/xmuda/nuscenes_lidarseg/preprocess', 61 | use_image=True, 62 | resize=None, 63 | merge_classes=True) 64 | printStats(calcStats(dataset)) 65 | dataset = NuScenesLidarSegSCN(split=('test_day',), 66 | nuscenes_dir='/data/AmitRoyChowdhury/nuscenes/', 67 | preprocess_dir='/data/AmitRoyChowdhury/xmuda/nuscenes_lidarseg/preprocess', 68 | use_image=True, 69 | resize=None, 70 | merge_classes=True) 71 | printStats(calcStats(dataset)) 72 | dataset = NuScenesLidarSegSCN(split=('test_night',), 73 | nuscenes_dir='/data/AmitRoyChowdhury/nuscenes/', 74 | preprocess_dir='/data/AmitRoyChowdhury/xmuda/nuscenes_lidarseg/preprocess', 75 | use_image=True, 76 | resize=None, 77 | merge_classes=True) 78 | printStats(calcStats(dataset)) 79 | -------------------------------------------------------------------------------- /xmuda/data/check_split_size.py: -------------------------------------------------------------------------------- 1 | from yacs.config import CfgNode as CN 2 | 3 | from xmuda.data.nuscenes.nuscenes_dataloader import NuScenesSCN 4 | from xmuda.data.nuscenes_lidarseg.nuscenes_lidarseg_dataloader import NuScenesLidarSegSCN 5 | from xmuda.data.a2d2.a2d2_dataloader import A2D2SCN 6 | from xmuda.data.semantic_kitti.semantic_kitti_dataloader import SemanticKITTISCN 7 | 8 | def main(): 9 | nuscenes_usa_train = NuScenesSCN(('train_usa',), 10 | preprocess_dir = '/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess', 11 | merge_classes=True) 12 | print('Train USA', len(nuscenes_usa_train)) 13 | del nuscenes_usa_train 14 | nuscenes_usa_test = NuScenesSCN(('test_usa',), 15 | preprocess_dir = '/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess', 16 | merge_classes=True) 17 | print('Test USA', len(nuscenes_usa_test)) 18 | del nuscenes_usa_test 19 | nuscenes_singapore_train = NuScenesSCN(('train_singapore',), 20 | preprocess_dir = '/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess', 21 | merge_classes=True) 22 | print('Train Singapore', len(nuscenes_singapore_train)) 23 | del nuscenes_singapore_train 24 | nuscenes_singapore_val = NuScenesSCN(('val_singapore',), 25 | preprocess_dir = '/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess', 26 | merge_classes=True) 27 | print('Val Singapore', len(nuscenes_singapore_val)) 28 | del nuscenes_singapore_val 29 | nuscenes_singapore_test = NuScenesSCN(('test_singapore',), 30 | preprocess_dir = '/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess', 31 | merge_classes=True) 32 | print('Test Singapore', len(nuscenes_singapore_test)) 33 | del nuscenes_singapore_test 34 | 35 | 36 | nuscenes_day_train = NuScenesSCN(('train_day',), 37 | preprocess_dir = '/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess', 38 | merge_classes=True) 39 | print('Train Day', len(nuscenes_day_train)) 40 | del nuscenes_day_train 41 | nuscenes_day_test = NuScenesSCN(('test_day',), 42 | preprocess_dir = '/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess', 43 | merge_classes=True) 44 | print('Test Day', len(nuscenes_day_test)) 45 | del nuscenes_day_test 46 | nuscenes_night_train = NuScenesSCN(('train_night',), 47 | preprocess_dir = '/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess', 48 | merge_classes=True) 49 | print('Train Night', len(nuscenes_night_train)) 50 | del nuscenes_night_train 51 | nuscenes_night_val = NuScenesSCN(('val_night',), 52 | preprocess_dir = '/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess', 53 | merge_classes=True) 54 | print('Val Night', len(nuscenes_night_val)) 55 | del nuscenes_night_val 56 | nuscenes_night_test = NuScenesSCN(('test_night',), 57 | preprocess_dir = '/data/AmitRoyChowdhury/xmuda/nuscenes/preprocess', 58 | merge_classes=True) 59 | print('Test Night', len(nuscenes_night_test)) 60 | del nuscenes_night_test 61 | 62 | semantic_kitti_train = SemanticKITTISCN(('train',), 63 | preprocess_dir = '/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess', 64 | merge_classes=True) 65 | print('Train', len(semantic_kitti_train)) 66 | del semantic_kitti_train 67 | semantic_kitti_val = SemanticKITTISCN(('val',), 68 | preprocess_dir = '/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess', 69 | merge_classes=True) 70 | print('Val', len(semantic_kitti_val)) 71 | del semantic_kitti_val 72 | semantic_kitti_test = SemanticKITTISCN(('test',), 73 | preprocess_dir = '/data/AmitRoyChowdhury/xmuda/SemanticKITTI/preprocess', 74 | merge_classes=True) 75 | print('Test', len(semantic_kitti_test)) 76 | del semantic_kitti_test 77 | 78 | if __name__ == '__main__': 79 | main() 80 | -------------------------------------------------------------------------------- /xmuda/data/collate.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from functools import partial 3 | 4 | 5 | def collate_scn_base(input_dict_list, output_orig, output_image=True): 6 | """ 7 | Custom collate function for SCN. The batch size is always 1, 8 | but the batch indices are appended to the locations. 9 | :param input_dict_list: a list of dicts from the dataloader 10 | :param output_orig: whether to output original point cloud/labels/indices 11 | :param output_image: whether to output images 12 | :return: Collated data batch as dict 13 | """ 14 | locs=[] 15 | feats=[] 16 | labels=[] 17 | 18 | if output_image: 19 | imgs = [] 20 | img_idxs = [] 21 | 22 | if output_orig: 23 | orig_seg_label = [] 24 | orig_points_idx = [] 25 | 26 | output_pselab = 'pseudo_label_2d' in input_dict_list[0].keys() 27 | if output_pselab: 28 | pseudo_label_2d = [] 29 | pseudo_label_3d = [] 30 | 31 | for idx, input_dict in enumerate(input_dict_list): 32 | coords = torch.from_numpy(input_dict['coords']) 33 | batch_idxs = torch.LongTensor(coords.shape[0], 1).fill_(idx) 34 | locs.append(torch.cat([coords, batch_idxs], 1)) 35 | feats.append(torch.from_numpy(input_dict['feats'])) 36 | if 'seg_label' in input_dict.keys(): 37 | labels.append(torch.from_numpy(input_dict['seg_label'])) 38 | if output_image: 39 | imgs.append(torch.from_numpy(input_dict['img'])) 40 | img_idxs.append(input_dict['img_indices']) 41 | if output_orig: 42 | orig_seg_label.append(input_dict['orig_seg_label']) 43 | orig_points_idx.append(input_dict['orig_points_idx']) 44 | if output_pselab: 45 | pseudo_label_2d.append(torch.from_numpy(input_dict['pseudo_label_2d'])) 46 | if input_dict['pseudo_label_3d'] is not None: 47 | pseudo_label_3d.append(torch.from_numpy(input_dict['pseudo_label_3d'])) 48 | 49 | locs = torch.cat(locs, 0) 50 | feats = torch.cat(feats, 0) 51 | out_dict = {'x': [locs, feats]} 52 | if labels: 53 | labels = torch.cat(labels, 0) 54 | out_dict['seg_label'] = labels 55 | if output_image: 56 | out_dict['img'] = torch.stack(imgs) 57 | out_dict['img_indices'] = img_idxs 58 | if output_orig: 59 | out_dict['orig_seg_label'] = orig_seg_label 60 | out_dict['orig_points_idx'] = orig_points_idx 61 | if output_pselab: 62 | out_dict['pseudo_label_2d'] = torch.cat(pseudo_label_2d, 0) 63 | out_dict['pseudo_label_3d'] = torch.cat(pseudo_label_3d, 0) if pseudo_label_3d else pseudo_label_3d 64 | 65 | out_dict['img_path'] = [input_dict_list[i]['img_path'] for i in range(len(input_dict_list))] 66 | return out_dict 67 | 68 | 69 | def get_collate_scn(is_train): 70 | return partial(collate_scn_base, 71 | output_orig=not is_train, 72 | ) 73 | -------------------------------------------------------------------------------- /xmuda/data/nuscenes/preprocess.py: -------------------------------------------------------------------------------- 1 | import os 2 | import os.path as osp 3 | import numpy as np 4 | import pickle 5 | from pyquaternion import Quaternion 6 | 7 | from nuscenes.nuscenes import NuScenes 8 | from nuscenes.utils.geometry_utils import points_in_box 9 | from nuscenes.eval.detection.utils import category_to_detection_name 10 | 11 | from xmuda.data.nuscenes.nuscenes_dataloader import NuScenesBase 12 | from xmuda.data.nuscenes.projection import map_pointcloud_to_image 13 | from xmuda.data.nuscenes import splits 14 | 15 | class_names_to_id = dict(zip(NuScenesBase.class_names, range(len(NuScenesBase.class_names)))) 16 | if 'background' in class_names_to_id: 17 | del class_names_to_id['background'] 18 | 19 | 20 | def preprocess(nusc, split_names, root_dir, out_dir, 21 | keyword=None, keyword_action=None, subset_name=None, 22 | location=None): 23 | # cannot process day/night and location at the same time 24 | assert not (bool(keyword) and bool(location)) 25 | if keyword: 26 | assert keyword_action in ['filter', 'exclude'] 27 | 28 | # init dict to save 29 | pkl_dict = {} 30 | for split_name in split_names: 31 | pkl_dict[split_name] = [] 32 | 33 | for i, sample in enumerate(nusc.sample): 34 | curr_scene_name = nusc.get('scene', sample['scene_token'])['name'] 35 | 36 | # get if the current scene is in train, val or test 37 | curr_split = None 38 | for split_name in split_names: 39 | if curr_scene_name in getattr(splits, split_name): 40 | curr_split = split_name 41 | break 42 | if curr_split is None: 43 | continue 44 | 45 | if subset_name == 'night': 46 | if curr_split == 'train': 47 | if curr_scene_name in splits.val_night: 48 | curr_split = 'val' 49 | if subset_name == 'singapore': 50 | if curr_split == 'train': 51 | if curr_scene_name in splits.val_singapore: 52 | curr_split = 'val' 53 | 54 | # filter for day/night 55 | if keyword: 56 | scene_description = nusc.get("scene", sample["scene_token"])["description"] 57 | if keyword.lower() in scene_description.lower(): 58 | if keyword_action == 'exclude': 59 | # skip sample 60 | continue 61 | else: 62 | if keyword_action == 'filter': 63 | # skip sample 64 | continue 65 | 66 | if location: 67 | scene = nusc.get("scene", sample["scene_token"]) 68 | if location not in nusc.get("log", scene['log_token'])['location']: 69 | continue 70 | 71 | lidar_token = sample["data"]["LIDAR_TOP"] 72 | cam_front_token = sample["data"]["CAM_FRONT"] 73 | lidar_path, boxes_lidar, _ = nusc.get_sample_data(lidar_token) 74 | cam_path, boxes_front_cam, cam_intrinsic = nusc.get_sample_data(cam_front_token) 75 | 76 | print('{}/{} {} {}'.format(i + 1, len(nusc.sample), curr_scene_name, lidar_path)) 77 | 78 | sd_rec_lidar = nusc.get('sample_data', sample['data']["LIDAR_TOP"]) 79 | cs_record_lidar = nusc.get('calibrated_sensor', 80 | sd_rec_lidar['calibrated_sensor_token']) 81 | pose_record_lidar = nusc.get('ego_pose', sd_rec_lidar['ego_pose_token']) 82 | sd_rec_cam = nusc.get('sample_data', sample['data']["CAM_FRONT"]) 83 | cs_record_cam = nusc.get('calibrated_sensor', 84 | sd_rec_cam['calibrated_sensor_token']) 85 | pose_record_cam = nusc.get('ego_pose', sd_rec_cam['ego_pose_token']) 86 | 87 | calib_infos = { 88 | "lidar2ego_translation": cs_record_lidar['translation'], 89 | "lidar2ego_rotation": cs_record_lidar['rotation'], 90 | "ego2global_translation_lidar": pose_record_lidar['translation'], 91 | "ego2global_rotation_lidar": pose_record_lidar['rotation'], 92 | "ego2global_translation_cam": pose_record_cam['translation'], 93 | "ego2global_rotation_cam": pose_record_cam['rotation'], 94 | "cam2ego_translation": cs_record_cam['translation'], 95 | "cam2ego_rotation": cs_record_cam['rotation'], 96 | "cam_intrinsic": cam_intrinsic, 97 | } 98 | 99 | alpha = np.random.random_sample()*0.18 - 0.09 100 | rotX = np.asarray([[1, 0, 0], 101 | [0, np.cos(alpha), -np.sin(alpha)], 102 | [0, np.sin(alpha), np.cos(alpha)]]) 103 | beta = np.random.random_sample()*0.18 - 0.09 104 | rotY = np.asarray([[np.cos(beta), 0, np.sin(beta)], 105 | [0, 1, 0], 106 | [-np.sin(beta), 0, np.cos(beta)]]) 107 | gamma = np.random.random_sample()*0.18 - 0.09 108 | rotZ = np.asarray([[np.cos(gamma), -np.sin(gamma), 0], 109 | [np.sin(gamma), np.cos(gamma), 0], 110 | [0, 0, 1]]) 111 | calib_infos['cam2ego_rotation'] = (Quaternion(calib_infos['cam2ego_rotation'])*Quaternion(matrix=rotX)*Quaternion(matrix=rotY)*Quaternion(matrix=rotZ)).elements 112 | 113 | # load lidar points 114 | pts = np.fromfile(lidar_path, dtype=np.float32, count=-1).reshape([-1, 5])[:, :3].T 115 | 116 | # map point cloud into front camera image 117 | pts_valid_flag, pts_cam_coord, pts_img = map_pointcloud_to_image(pts, (900, 1600, 3), calib_infos) 118 | # fliplr so that indexing is row, col and not col, row 119 | pts_img = np.ascontiguousarray(np.fliplr(pts_img)) 120 | 121 | # only use lidar points in the front camera image 122 | pts = pts[:, pts_valid_flag] 123 | 124 | num_pts = pts.shape[1] 125 | seg_labels = np.full(num_pts, fill_value=len(class_names_to_id), dtype=np.uint8) 126 | # only use boxes that are visible in camera 127 | valid_box_tokens = [box.token for box in boxes_front_cam] 128 | boxes = [box for box in boxes_lidar if box.token in valid_box_tokens] 129 | for box in boxes: 130 | # get points that lie inside of the box 131 | fg_mask = points_in_box(box, pts) 132 | det_class = category_to_detection_name(box.name) 133 | if det_class is not None: 134 | seg_labels[fg_mask] = class_names_to_id[det_class] 135 | 136 | # convert to relative path 137 | lidar_path = lidar_path.replace(root_dir + '/', '') 138 | cam_path = cam_path.replace(root_dir + '/', '') 139 | 140 | # transpose to yield shape (num_points, 3) 141 | pts = pts.T 142 | 143 | # append data to train, val or test list in pkl_dict 144 | data_dict = { 145 | 'points': pts, 146 | 'seg_labels': seg_labels, 147 | 'points_img': pts_img, # row, col format, shape: (num_points, 2) 148 | 'lidar_path': lidar_path, 149 | 'camera_path': cam_path, 150 | 'boxes': boxes_lidar, 151 | "sample_token": sample["token"], 152 | "scene_name": curr_scene_name, 153 | "calib": calib_infos 154 | } 155 | pkl_dict[curr_split].append(data_dict) 156 | 157 | # save to pickle file 158 | save_dir = osp.join(out_dir, 'preprocess') 159 | os.makedirs(save_dir, exist_ok=True) 160 | for split_name in split_names: 161 | save_path = osp.join(save_dir, '{}{}.pkl'.format(split_name, '_' + subset_name if subset_name else '')) 162 | with open(save_path, 'wb') as f: 163 | pickle.dump(pkl_dict[split_name], f) 164 | print('Wrote preprocessed data to ' + save_path) 165 | 166 | 167 | if __name__ == '__main__': 168 | root_dir = '/data/AmitRoyChowdhury/nuscenes' 169 | out_dir = '/data/AmitRoyChowdhury/xmuda/nuscenes_noisy_5' 170 | nusc = NuScenes(version='v1.0-trainval', dataroot=root_dir, verbose=True) 171 | # for faster debugging, the script can be run using the mini dataset 172 | # nusc = NuScenes(version='v1.0-mini', dataroot=root_dir, verbose=True) 173 | # We construct the splits by using the meta data of NuScenes: 174 | # USA/Singapore: We check if the location is Boston or Singapore. 175 | # Day/Night: We detect if "night" occurs in the scene description string. 176 | preprocess(nusc, ['train', 'test'], root_dir, out_dir, location='boston', subset_name='usa') 177 | preprocess(nusc, ['train', 'val', 'test'], root_dir, out_dir, location='singapore', subset_name='singapore') 178 | preprocess(nusc, ['train', 'test'], root_dir, out_dir, keyword='night', keyword_action='exclude', subset_name='day') 179 | preprocess(nusc, ['train', 'val', 'test'], root_dir, out_dir, keyword='night', keyword_action='filter', subset_name='night') 180 | -------------------------------------------------------------------------------- /xmuda/data/nuscenes/projection.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from pyquaternion import Quaternion 3 | from nuscenes.utils.geometry_utils import view_points 4 | 5 | import matplotlib.pyplot as plt 6 | 7 | 8 | # modified from https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/nuscenes.py 9 | def map_pointcloud_to_image(pc, im_shape, info, im=None): 10 | """ 11 | Maps the lidar point cloud to the image. 12 | :param pc: (3, N) 13 | :param im_shape: image to check size and debug 14 | :param info: dict with calibration infos 15 | :param im: image, only for visualization 16 | :return: 17 | """ 18 | pc = pc.copy() 19 | 20 | # Points live in the point sensor frame. So they need to be transformed via global to the image plane. 21 | # First step: transform the point-cloud to the ego vehicle frame for the timestamp of the sweep. 22 | pc = Quaternion(info['lidar2ego_rotation']).rotation_matrix @ pc 23 | pc = pc + np.array(info['lidar2ego_translation'])[:, np.newaxis] 24 | 25 | # Second step: transform to the global frame. 26 | pc = Quaternion(info['ego2global_rotation_lidar']).rotation_matrix @ pc 27 | pc = pc + np.array(info['ego2global_translation_lidar'])[:, np.newaxis] 28 | 29 | # Third step: transform into the ego vehicle frame for the timestamp of the image. 30 | pc = pc - np.array(info['ego2global_translation_cam'])[:, np.newaxis] 31 | pc = Quaternion(info['ego2global_rotation_cam']).rotation_matrix.T @ pc 32 | 33 | # Fourth step: transform into the camera. 34 | pc = pc - np.array(info['cam2ego_translation'])[:, np.newaxis] 35 | pc = Quaternion(info['cam2ego_rotation']).rotation_matrix.T @ pc 36 | 37 | # Fifth step: actually take a "picture" of the point cloud. 38 | # Grab the depths (camera frame z axis points away from the camera). 39 | depths = pc[2, :] 40 | 41 | # Take the actual picture (matrix multiplication with camera-matrix + renormalization). 42 | points = view_points(pc, np.array(info['cam_intrinsic']), normalize=True) 43 | 44 | # Cast to float32 to prevent later rounding errors 45 | points = points.astype(np.float32) 46 | 47 | # Remove points that are either outside or behind the camera. 48 | mask = np.ones(depths.shape[0], dtype=bool) 49 | mask = np.logical_and(mask, depths > 0) 50 | mask = np.logical_and(mask, points[0, :] > 0) 51 | mask = np.logical_and(mask, points[0, :] < im_shape[1]) 52 | mask = np.logical_and(mask, points[1, :] > 0) 53 | mask = np.logical_and(mask, points[1, :] < im_shape[0]) 54 | points = points[:, mask] 55 | 56 | # debug 57 | if im is not None: 58 | # Retrieve the color from the depth. 59 | coloring = depths 60 | coloring = coloring[mask] 61 | 62 | plt.figure(figsize=(9, 16)) 63 | plt.imshow(im) 64 | plt.scatter(points[0, :], points[1, :], c=coloring, s=2) 65 | plt.axis('off') 66 | 67 | # plt.show() 68 | 69 | return mask, pc.T, points.T[:, :2] 70 | -------------------------------------------------------------------------------- /xmuda/data/nuscenes_lidarseg/preprocess.py: -------------------------------------------------------------------------------- 1 | import os 2 | import os.path as osp 3 | import numpy as np 4 | import pickle 5 | from pyquaternion import Quaternion 6 | 7 | from nuscenes.nuscenes import NuScenes 8 | from nuscenes.utils.geometry_utils import points_in_box 9 | from nuscenes.eval.detection.utils import category_to_detection_name 10 | 11 | from xmuda.data.nuscenes_lidarseg.projection import map_pointcloud_to_image 12 | from xmuda.data.nuscenes_lidarseg import splits 13 | 14 | def preprocess(nusc, split_names, root_dir, out_dir, 15 | keyword=None, keyword_action=None, subset_name=None, 16 | location=None): 17 | # cannot process day/night and location at the same time 18 | assert not (bool(keyword) and bool(location)) 19 | if keyword: 20 | assert keyword_action in ['filter', 'exclude'] 21 | 22 | # init dict to save 23 | pkl_dict = {} 24 | for split_name in split_names: 25 | pkl_dict[split_name] = [] 26 | 27 | for i, sample in enumerate(nusc.sample): 28 | curr_scene_name = nusc.get('scene', sample['scene_token'])['name'] 29 | 30 | # get if the current scene is in train, val or test 31 | curr_split = None 32 | for split_name in split_names: 33 | if curr_scene_name in getattr(splits, split_name): 34 | curr_split = split_name 35 | break 36 | if curr_split is None: 37 | continue 38 | 39 | if subset_name == 'night': 40 | if curr_split == 'train': 41 | if curr_scene_name in splits.val_night: 42 | curr_split = 'val' 43 | if subset_name == 'singapore': 44 | if curr_split == 'train': 45 | if curr_scene_name in splits.val_singapore: 46 | curr_split = 'val' 47 | 48 | # filter for day/night 49 | if keyword: 50 | scene_description = nusc.get("scene", sample["scene_token"])["description"] 51 | if keyword.lower() in scene_description.lower(): 52 | if keyword_action == 'exclude': 53 | # skip sample 54 | continue 55 | else: 56 | if keyword_action == 'filter': 57 | # skip sample 58 | continue 59 | 60 | if location: 61 | scene = nusc.get("scene", sample["scene_token"]) 62 | if location not in nusc.get("log", scene['log_token'])['location']: 63 | continue 64 | 65 | lidar_token = sample["data"]["LIDAR_TOP"] 66 | cam_front_token = sample["data"]["CAM_FRONT"] 67 | calib_sen_token = nusc.get('sample_data', cam_front_token)['calibrated_sensor_token'] 68 | 69 | lidar_path = osp.join(nusc.dataroot, nusc.get('sample_data', lidar_token)['filename']) 70 | label_path = osp.join(nusc.dataroot, nusc.get('lidarseg', lidar_token)['filename']) 71 | cam_path = osp.join(nusc.dataroot, nusc.get('sample_data', cam_front_token)['filename']) 72 | cam_intrinsic = np.array(nusc.get('calibrated_sensor', calib_sen_token)['camera_intrinsic']) 73 | 74 | print('{}/{} {} {}'.format(i + 1, len(nusc.sample), curr_scene_name, lidar_path)) 75 | 76 | sd_rec_lidar = nusc.get('sample_data', sample['data']["LIDAR_TOP"]) 77 | cs_record_lidar = nusc.get('calibrated_sensor', 78 | sd_rec_lidar['calibrated_sensor_token']) 79 | pose_record_lidar = nusc.get('ego_pose', sd_rec_lidar['ego_pose_token']) 80 | sd_rec_cam = nusc.get('sample_data', sample['data']["CAM_FRONT"]) 81 | cs_record_cam = nusc.get('calibrated_sensor', 82 | sd_rec_cam['calibrated_sensor_token']) 83 | pose_record_cam = nusc.get('ego_pose', sd_rec_cam['ego_pose_token']) 84 | 85 | calib_infos = { 86 | "lidar2ego_translation": cs_record_lidar['translation'], 87 | "lidar2ego_rotation": cs_record_lidar['rotation'], 88 | "ego2global_translation_lidar": pose_record_lidar['translation'], 89 | "ego2global_rotation_lidar": pose_record_lidar['rotation'], 90 | "ego2global_translation_cam": pose_record_cam['translation'], 91 | "ego2global_rotation_cam": pose_record_cam['rotation'], 92 | "cam2ego_translation": cs_record_cam['translation'], 93 | "cam2ego_rotation": cs_record_cam['rotation'], 94 | "cam_intrinsic": cam_intrinsic, 95 | } 96 | 97 | # load lidar points 98 | pts = np.fromfile(lidar_path, dtype=np.float32, count=-1).reshape([-1, 5])[:, :3].T 99 | lbl = np.fromfile(label_path, dtype=np.uint8, count=-1) 100 | 101 | # map point cloud into front camera image 102 | pts_valid_flag, pts_cam_coord, pts_img = map_pointcloud_to_image(pts, (900, 1600, 3), calib_infos) 103 | # fliplr so that indexing is row, col and not col, row 104 | pts_img = np.ascontiguousarray(np.fliplr(pts_img)) 105 | 106 | # only use lidar points in the front camera image 107 | pts = pts[:, pts_valid_flag] 108 | lbl = lbl[pts_valid_flag] 109 | 110 | # convert to relative path 111 | lidar_path = lidar_path.replace(root_dir + '/', '') 112 | cam_path = cam_path.replace(root_dir + '/', '') 113 | 114 | # transpose to yield shape (num_points, 3) 115 | pts = pts.T 116 | 117 | # append data to train, val or test list in pkl_dict 118 | data_dict = { 119 | 'points': pts, 120 | 'seg_labels': lbl, 121 | 'points_img': pts_img, # row, col format, shape: (num_points, 2) 122 | 'lidar_path': lidar_path, 123 | 'label_path': label_path, 124 | 'camera_path': cam_path, 125 | "sample_token": sample["token"], 126 | "scene_name": curr_scene_name, 127 | "calib": calib_infos 128 | } 129 | pkl_dict[curr_split].append(data_dict) 130 | 131 | # save to pickle file 132 | save_dir = osp.join(out_dir, 'preprocess') 133 | os.makedirs(save_dir, exist_ok=True) 134 | for split_name in split_names: 135 | save_path = osp.join(save_dir, '{}{}.pkl'.format(split_name, '_' + subset_name if subset_name else '')) 136 | with open(save_path, 'wb') as f: 137 | pickle.dump(pkl_dict[split_name], f) 138 | print('Wrote preprocessed data to ' + save_path) 139 | 140 | 141 | if __name__ == '__main__': 142 | root_dir = '/data/AmitRoyChowdhury/nuscenes' 143 | out_dir = '/data/AmitRoyChowdhury/xmuda/nuscenes_lidarseg' 144 | nusc = NuScenes(version='v1.0-trainval', dataroot=root_dir, verbose=True) 145 | # for faster debugging, the script can be run using the mini dataset 146 | # nusc = NuScenes(version='v1.0-mini', dataroot=root_dir, verbose=True) 147 | # We construct the splits by using the meta data of NuScenes: 148 | # USA/Singapore: We check if the location is Boston or Singapore. 149 | # Day/Night: We detect if "night" occurs in the scene description string. 150 | preprocess(nusc, ['train', 'test'], root_dir, out_dir, location='boston', subset_name='usa') 151 | preprocess(nusc, ['train', 'val', 'test'], root_dir, out_dir, location='singapore', subset_name='singapore') 152 | preprocess(nusc, ['train', 'test'], root_dir, out_dir, keyword='night', keyword_action='exclude', subset_name='day') 153 | preprocess(nusc, ['train', 'val', 'test'], root_dir, out_dir, keyword='night', keyword_action='filter', subset_name='night') 154 | -------------------------------------------------------------------------------- /xmuda/data/nuscenes_lidarseg/projection.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from pyquaternion import Quaternion 3 | from nuscenes.utils.geometry_utils import view_points 4 | 5 | import matplotlib.pyplot as plt 6 | 7 | 8 | # modified from https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/nuscenes.py 9 | def map_pointcloud_to_image(pc, im_shape, info, im=None): 10 | """ 11 | Maps the lidar point cloud to the image. 12 | :param pc: (3, N) 13 | :param im_shape: image to check size and debug 14 | :param info: dict with calibration infos 15 | :param im: image, only for visualization 16 | :return: 17 | """ 18 | pc = pc.copy() 19 | 20 | # Points live in the point sensor frame. So they need to be transformed via global to the image plane. 21 | # First step: transform the point-cloud to the ego vehicle frame for the timestamp of the sweep. 22 | pc = Quaternion(info['lidar2ego_rotation']).rotation_matrix @ pc 23 | pc = pc + np.array(info['lidar2ego_translation'])[:, np.newaxis] 24 | 25 | # Second step: transform to the global frame. 26 | pc = Quaternion(info['ego2global_rotation_lidar']).rotation_matrix @ pc 27 | pc = pc + np.array(info['ego2global_translation_lidar'])[:, np.newaxis] 28 | 29 | # Third step: transform into the ego vehicle frame for the timestamp of the image. 30 | pc = pc - np.array(info['ego2global_translation_cam'])[:, np.newaxis] 31 | pc = Quaternion(info['ego2global_rotation_cam']).rotation_matrix.T @ pc 32 | 33 | # Fourth step: transform into the camera. 34 | pc = pc - np.array(info['cam2ego_translation'])[:, np.newaxis] 35 | pc = Quaternion(info['cam2ego_rotation']).rotation_matrix.T @ pc 36 | 37 | # Fifth step: actually take a "picture" of the point cloud. 38 | # Grab the depths (camera frame z axis points away from the camera). 39 | depths = pc[2, :] 40 | 41 | # Take the actual picture (matrix multiplication with camera-matrix + renormalization). 42 | points = view_points(pc, np.array(info['cam_intrinsic']), normalize=True) 43 | 44 | # Cast to float32 to prevent later rounding errors 45 | points = points.astype(np.float32) 46 | 47 | # Remove points that are either outside or behind the camera. 48 | mask = np.ones(depths.shape[0], dtype=bool) 49 | mask = np.logical_and(mask, depths > 0) 50 | mask = np.logical_and(mask, points[0, :] > 0) 51 | mask = np.logical_and(mask, points[0, :] < im_shape[1]) 52 | mask = np.logical_and(mask, points[1, :] > 0) 53 | mask = np.logical_and(mask, points[1, :] < im_shape[0]) 54 | points = points[:, mask] 55 | 56 | # debug 57 | if im is not None: 58 | # Retrieve the color from the depth. 59 | coloring = depths 60 | coloring = coloring[mask] 61 | 62 | plt.figure(figsize=(9, 16)) 63 | plt.imshow(im) 64 | plt.scatter(points[0, :], points[1, :], c=coloring, s=2) 65 | plt.axis('off') 66 | 67 | # plt.show() 68 | 69 | return mask, pc.T, points.T[:, :2] 70 | -------------------------------------------------------------------------------- /xmuda/data/semantic_kitti/preprocess.py: -------------------------------------------------------------------------------- 1 | import os 2 | import os.path as osp 3 | import numpy as np 4 | import numpy.random as nr 5 | import pickle 6 | from PIL import Image 7 | import glob 8 | import torch 9 | from torch.utils.data import Dataset 10 | from torch.utils.data.dataloader import DataLoader 11 | 12 | print(os.listdir()) 13 | import xmuda 14 | import xmuda.data 15 | import xmuda.data.semantic_kitti 16 | 17 | from xmuda.data.semantic_kitti import splits 18 | 19 | # prevent "RuntimeError: received 0 items of ancdata" 20 | torch.multiprocessing.set_sharing_strategy('file_system') 21 | 22 | 23 | class DummyDataset(Dataset): 24 | """Use torch dataloader for multiprocessing""" 25 | def __init__(self, root_dir, scenes): 26 | self.root_dir = root_dir 27 | self.data = [] 28 | self.glob_frames(scenes) 29 | 30 | def glob_frames(self, scenes): 31 | for scene in scenes: 32 | glob_path = osp.join(self.root_dir, 'dataset', 'sequences', scene, 'image_2', '*.png') 33 | cam_paths = sorted(glob.glob(glob_path)) 34 | # load calibration 35 | calib = self.read_calib(osp.join(self.root_dir, 'dataset', 'sequences', scene, 'calib.txt')) 36 | proj_matrix = calib['P2'] @ calib['Tr'] 37 | proj_matrix = proj_matrix.astype(np.float32) 38 | 39 | for cam_path in cam_paths: 40 | basename = osp.basename(cam_path) 41 | frame_id = osp.splitext(basename)[0] 42 | assert frame_id.isdigit() 43 | data = { 44 | 'camera_path': cam_path, 45 | 'lidar_path': osp.join(self.root_dir, 'dataset', 'sequences', scene, 'velodyne', 46 | frame_id + '.bin'), 47 | 'label_path': osp.join(self.root_dir, 'dataset', 'sequences', scene, 'labels', 48 | frame_id + '.label'), 49 | 'proj_matrix': proj_matrix 50 | } 51 | for k, v in data.items(): 52 | if isinstance(v, str): 53 | if not osp.exists(v): 54 | raise IOError('File not found {}'.format(v)) 55 | self.data.append(data) 56 | 57 | @staticmethod 58 | def read_calib(calib_path): 59 | """ 60 | :param calib_path: Path to a calibration text file. 61 | :return: dict with calibration matrices. 62 | """ 63 | calib_all = {} 64 | with open(calib_path, 'r') as f: 65 | for line in f.readlines(): 66 | if line == '\n': 67 | break 68 | key, value = line.split(':', 1) 69 | calib_all[key] = np.array([float(x) for x in value.split()]) 70 | 71 | # reshape matrices 72 | calib_out = {} 73 | calib_out['P2'] = calib_all['P2'].reshape(3, 4) # 3x4 projection matrix for left camera 74 | calib_out['Tr'] = np.identity(4) # 4x4 matrix 75 | calib_out['Tr'][:3, :4] = calib_all['Tr'].reshape(3, 4) 76 | return calib_out 77 | 78 | @staticmethod 79 | def select_points_in_frustum(points_2d, x1, y1, x2, y2): 80 | """ 81 | Select points in a 2D frustum parametrized by x1, y1, x2, y2 in image coordinates 82 | :param points_2d: point cloud projected into 2D 83 | :param points_3d: point cloud 84 | :param x1: left bound 85 | :param y1: upper bound 86 | :param x2: right bound 87 | :param y2: lower bound 88 | :return: points (2D and 3D) that are in the frustum 89 | """ 90 | keep_ind = (points_2d[:, 0] > x1) * \ 91 | (points_2d[:, 1] > y1) * \ 92 | (points_2d[:, 0] < x2) * \ 93 | (points_2d[:, 1] < y2) 94 | 95 | return keep_ind 96 | 97 | def __getitem__(self, index): 98 | data_dict = self.data[index].copy() 99 | scan = np.fromfile(data_dict['lidar_path'], dtype=np.float32) 100 | scan = scan.reshape((-1, 4)) 101 | points = scan[:, :3] 102 | label = np.fromfile(data_dict['label_path'], dtype=np.uint32) 103 | label = label.reshape((-1)) 104 | label = label & 0xFFFF # get lower half for semantics 105 | 106 | # load image 107 | image = Image.open(data_dict['camera_path']) 108 | image_size = image.size 109 | 110 | # project points into image 111 | keep_idx = points[:, 0] > 0 # only keep point in front of the vehicle 112 | points_hcoords = np.concatenate([points[keep_idx], np.ones([keep_idx.sum(), 1], dtype=np.float32)], axis=1) 113 | img_points = (data_dict['proj_matrix'] @ points_hcoords.T).T 114 | img_points = img_points[:, :2] / np.expand_dims(img_points[:, 2], axis=1) # scale 2D points 115 | keep_idx_img_pts = self.select_points_in_frustum(img_points, 0, 0, *image_size) 116 | keep_idx[keep_idx] = keep_idx_img_pts 117 | # fliplr so that indexing is row, col and not col, row 118 | img_points = np.fliplr(img_points) 119 | # debug 120 | # from xmuda.data.utils.visualize import draw_points_image, draw_bird_eye_view 121 | # draw_points_image(np.array(image), img_points[keep_idx_img_pts].astype(int), label[keep_idx], 122 | # color_palette_type='SemanticKITTI_long') 123 | 124 | data_dict['seg_label'] = label[keep_idx].astype(np.int16) 125 | data_dict['points'] = points[keep_idx] 126 | data_dict['points_img'] = img_points[keep_idx_img_pts] 127 | data_dict['image_size'] = np.array(image_size) 128 | 129 | return data_dict 130 | 131 | def __len__(self): 132 | return len(self.data) 133 | 134 | 135 | def preprocess(split_name, root_dir, out_dir): 136 | pkl_data = [] 137 | split = getattr(splits, split_name) 138 | 139 | dataloader = DataLoader(DummyDataset(root_dir, split), num_workers=8) 140 | 141 | num_skips = 0 142 | for i, data_dict in enumerate(dataloader): 143 | # data error leads to returning empty dict 144 | if not data_dict: 145 | print('empty dict, continue') 146 | num_skips += 1 147 | continue 148 | for k, v in data_dict.items(): 149 | data_dict[k] = v[0] 150 | print('{}/{} {}'.format(i, len(dataloader), data_dict['lidar_path'])) 151 | 152 | # convert to relative path 153 | lidar_path = data_dict['lidar_path'].replace(root_dir + '/', '') 154 | cam_path = data_dict['camera_path'].replace(root_dir + '/', '') 155 | 156 | # append data 157 | out_dict = { 158 | 'points': data_dict['points'].numpy(), 159 | 'seg_labels': data_dict['seg_label'].numpy(), 160 | 'points_img': data_dict['points_img'].numpy(), # row, col format, shape: (num_points, 2) 161 | 'lidar_path': lidar_path, 162 | 'camera_path': cam_path, 163 | 'image_size': tuple(data_dict['image_size'].numpy()) 164 | } 165 | pkl_data.append(out_dict) 166 | 167 | print('Skipped {} files'.format(num_skips)) 168 | 169 | # save to pickle file 170 | save_dir = osp.join(out_dir, 'preprocess') 171 | os.makedirs(save_dir, exist_ok=True) 172 | save_path = osp.join(save_dir, '{}.pkl'.format(split_name)) 173 | with open(save_path, 'wb') as f: 174 | pickle.dump(pkl_data, f) 175 | print('Wrote preprocessed data to ' + save_path) 176 | 177 | 178 | if __name__ == '__main__': 179 | root_dir = '/data/AmitRoyChowdhury/SemanticKITTI' 180 | out_dir = '/data/AmitRoyChowdhury/xmuda/SemanticKITTI_1' 181 | preprocess('val', root_dir, out_dir) 182 | preprocess('train', root_dir, out_dir) 183 | preprocess('test', root_dir, out_dir) 184 | -------------------------------------------------------------------------------- /xmuda/data/semantic_kitti/splits.py: -------------------------------------------------------------------------------- 1 | # official split defined in https://github.com/PRBonn/semantic-kitti-api/blob/master/config/semantic-kitti.yaml 2 | 3 | train = [ 4 | '00', 5 | '01', 6 | '02', 7 | '03', 8 | '04', 9 | '05', 10 | '06', 11 | '09', 12 | '10', 13 | ] 14 | 15 | val = [ 16 | '07' 17 | ] 18 | 19 | test = [ 20 | '08' 21 | ] 22 | 23 | # not used 24 | hidden_test = [ 25 | '11', 26 | '12', 27 | '13', 28 | '14', 29 | '15', 30 | '16', 31 | '17', 32 | '18', 33 | '19', 34 | '20', 35 | '21', 36 | ] 37 | -------------------------------------------------------------------------------- /xmuda/data/stats_a2d2_semantic_kitti.txt: -------------------------------------------------------------------------------- 1 | Initialize A2D2 dataloader 2 | Load ('train',) 3 | Length: 27695 4 | Image Sizes: 5 | (3, 1208, 1920): 27695 6 | Total Points: 160055380 7 | Average Points: 5779.215742913883 8 | Class Counts: [0.03871744 0.02643469 0.00132319 0.00171755 0.39008063 0.00262995 9 | 0.03508818 0.10681312 0.34532283 0.0518724 ] 10 | Initialize A2D2 dataloader 11 | Load ('test',) 12 | Length: 942 13 | Image Sizes: 14 | (3, 1208, 1920): 942 15 | Total Points: 8379396 16 | Average Points: 8895.324840764331 17 | Class Counts: [0.05098763 0.00970691 0.00254466 0.00186729 0.30064895 0.00386612 18 | 0.06242143 0.2314673 0.2840368 0.0524529 ] 19 | Initialize SemanticKITTI dataloader 20 | Load ('train',) 21 | Length: 18029 22 | Image Sizes: 23 | (3, 376, 1241): 10303 24 | (3, 375, 1242): 801 25 | (3, 370, 1226): 6925 26 | Total Points: 356065437 27 | Average Points: 19749.594375727993 28 | Class Counts: [0.05129574 0.00232826 0.00100088 0.0003912 0.29876709 0.01258095 29 | 0.12535387 0.09147289 0.32876327 0.08804587] 30 | Initialize SemanticKITTI dataloader 31 | Load ('test',) 32 | Length: 4071 33 | Image Sizes: 34 | (3, 370, 1226): 4071 35 | Total Points: 80006777 36 | Average Points: 19652.856055023334 37 | Class Counts: [0.08314751 0.00131546 0.00298265 0.00164804 0.27914575 0.01268129 38 | 0.10612269 0.08209795 0.38286103 0.04799763] 39 | Initialize SemanticKITTI dataloader 40 | Load ('val',) 41 | Length: 1101 42 | Image Sizes: 43 | (3, 370, 1226): 1101 44 | Total Points: 21481760 45 | Average Points: 19511.1353315168 46 | Class Counts: [0.13065671 0.01093972 0.00257748 0.00168982 0.27974412 0.01343546 47 | 0.10203536 0.17166999 0.23103983 0.05621152] 48 | -------------------------------------------------------------------------------- /xmuda/data/stats_day_night.txt: -------------------------------------------------------------------------------- 1 | Initialize Nuscenes dataloader 2 | Load ('train_day',) 3 | Length: 24745 4 | Image Sizes: 5 | (3, 900, 1600): 24745 6 | Total Points: 68165982 7 | Average Points: 2754.737603556274 8 | Class Counts: [5.37324321e-02 3.40967728e-03 5.02655415e-04 7.39585913e-03 9 | 9.34959376e-01] 10 | Initialize Nuscenes dataloader 11 | Load ('test_day',) 12 | Length: 5417 13 | Image Sizes: 14 | (3, 900, 1600): 5417 15 | Total Points: 15015292 16 | Average Points: 2771.8833302565995 17 | Class Counts: [6.12308439e-02 2.88132925e-03 5.35121129e-04 6.93080095e-03 18 | 9.28421905e-01] 19 | Initialize Nuscenes dataloader 20 | Load ('train_night',) 21 | Length: 2779 22 | Image Sizes: 23 | (3, 900, 1600): 2779 24 | Total Points: 8498393 25 | Average Points: 3058.0759265922993 26 | Class Counts: [3.22983416e-02 2.57742846e-03 3.99604961e-04 1.08844107e-04 27 | 9.64615781e-01] 28 | Initialize Nuscenes dataloader 29 | Load ('test_night',) 30 | Length: 602 31 | Image Sizes: 32 | (3, 900, 1600): 602 33 | Total Points: 1869793 34 | Average Points: 3105.968438538206 35 | Class Counts: [3.93551586e-02 1.38518007e-04 5.70116585e-04 8.53035603e-04 36 | 9.59083171e-01] 37 | Initialize Nuscenes dataloader 38 | Load ('val_night',) 39 | Length: 606 40 | Image Sizes: 41 | (3, 900, 1600): 606 42 | Total Points: 1727174 43 | Average Points: 2850.122112211221 44 | Class Counts: [6.49801352e-02 2.98174938e-04 2.68067954e-04 7.16777812e-04 45 | 9.33736844e-01] 46 | -------------------------------------------------------------------------------- /xmuda/data/stats_usa_singapore.txt: -------------------------------------------------------------------------------- 1 | Initialize Nuscenes dataloader 2 | Load ('train_usa',) 3 | Length: 15695 4 | Image Sizes: 5 | (3, 900, 1600): 15695 6 | Total Points: 39757468 7 | Average Points: 2533.129531697993 8 | Class Counts: [7.14538713e-02 3.83930385e-03 3.32943738e-04 7.60566543e-03 9 | 9.16768216e-01] 10 | Initialize Nuscenes dataloader 11 | Load ('test_usa',) 12 | Length: 3090 13 | Image Sizes: 14 | (3, 900, 1600): 3090 15 | Total Points: 7799290 16 | Average Points: 2524.042071197411 17 | Class Counts: [8.71048519e-02 2.05352026e-03 1.64886804e-04 8.91735017e-03 18 | 9.01759391e-01] 19 | Initialize Nuscenes dataloader 20 | Load ('train_singapore',) 21 | Length: 9665 22 | Image Sizes: 23 | (3, 900, 1600): 9665 24 | Total Points: 30152220 25 | Average Points: 3119.733057423694 26 | Class Counts: [3.00955618e-02 2.45305321e-03 6.75273661e-04 2.95540428e-03 27 | 9.63820707e-01] 28 | Initialize Nuscenes dataloader 29 | Load ('test_singapore',) 30 | Length: 2929 31 | Image Sizes: 32 | (3, 900, 1600): 2929 33 | Total Points: 9085795 34 | Average Points: 3102.0126322977126 35 | Class Counts: [3.45186084e-02 3.02747310e-03 8.60133868e-04 3.97477601e-03 36 | 9.57619009e-01] 37 | Initialize Nuscenes dataloader 38 | Load ('val_singapore',) 39 | Length: 2770 40 | Image Sizes: 41 | (3, 900, 1600): 2770 42 | Total Points: 8481861 43 | Average Points: 3062.043682310469 44 | Class Counts: [3.55071841e-02 3.32910431e-03 5.33491412e-04 1.35365340e-02 45 | 9.47093686e-01] 46 | -------------------------------------------------------------------------------- /xmuda/data/utils/augmentation_3d.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | 4 | def augment_and_scale_3d(points, scale, full_scale, 5 | noisy_rot=0.0, 6 | flip_x=0.0, 7 | flip_y=0.0, 8 | rot_z=0.0, 9 | transl=False): 10 | """ 11 | 3D point cloud augmentation and scaling from points (in meters) to voxels 12 | :param points: 3D points in meters 13 | :param scale: voxel scale in 1 / m, e.g. 20 corresponds to 5cm voxels 14 | :param full_scale: size of the receptive field of SparseConvNet 15 | :param noisy_rot: scale of random noise added to all elements of a rotation matrix 16 | :param flip_x: probability of flipping the x-axis (left-right in nuScenes LiDAR coordinate system) 17 | :param flip_y: probability of flipping the y-axis (left-right in Kitti LiDAR coordinate system) 18 | :param rot_z: angle in rad around the z-axis (up-axis) 19 | :param transl: True or False, random translation inside the receptive field of the SCN, defined by full_scale 20 | :return coords: the coordinates that are given as input to SparseConvNet 21 | """ 22 | if noisy_rot > 0 or flip_x > 0 or flip_y > 0 or rot_z > 0: 23 | rot_matrix = np.eye(3, dtype=np.float32) 24 | if noisy_rot > 0: 25 | # add noise to rotation matrix 26 | rot_matrix += np.random.randn(3, 3) * noisy_rot 27 | if flip_x > 0: 28 | # flip x axis: multiply element at (0, 0) with 1 or -1 29 | rot_matrix[0][0] *= np.random.randint(0, 2) * 2 - 1 30 | if flip_y > 0: 31 | # flip y axis: multiply element at (1, 1) with 1 or -1 32 | rot_matrix[1][1] *= np.random.randint(0, 2) * 2 - 1 33 | if rot_z > 0: 34 | # rotate around z-axis (up-axis) 35 | theta = np.random.rand() * rot_z 36 | z_rot_matrix = np.array([[np.cos(theta), -np.sin(theta), 0], 37 | [np.sin(theta), np.cos(theta), 0], 38 | [0, 0, 1]], dtype=np.float32) 39 | rot_matrix = rot_matrix.dot(z_rot_matrix) 40 | points = points.dot(rot_matrix) 41 | 42 | # scale with inverse voxel size (e.g. 20 corresponds to 5cm) 43 | coords = points * scale 44 | # translate points to positive octant (receptive field of SCN in x, y, z coords is in interval [0, full_scale]) 45 | coords -= coords.min(0) 46 | 47 | if transl: 48 | # random translation inside receptive field of SCN 49 | offset = np.clip(full_scale - coords.max(0) - 0.001, a_min=0, a_max=None) * np.random.rand(3) 50 | coords += offset 51 | 52 | return coords 53 | -------------------------------------------------------------------------------- /xmuda/data/utils/evaluate.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from sklearn.metrics import confusion_matrix as CM 3 | 4 | class Evaluator(object): 5 | def __init__(self, class_names, labels=None): 6 | self.class_names = tuple(class_names) 7 | self.num_classes = len(class_names) 8 | self.labels = np.arange(self.num_classes) if labels is None else np.array(labels) 9 | assert self.labels.shape[0] == self.num_classes 10 | self.confusion_matrix = np.zeros((self.num_classes, self.num_classes)) 11 | 12 | def update(self, pred_label, gt_label): 13 | """Update per instance 14 | 15 | Args: 16 | pred_label (np.ndarray): (num_points) 17 | gt_label (np.ndarray): (num_points,) 18 | 19 | """ 20 | # convert ignore_label to num_classes 21 | # refer to sklearn.metrics.confusion_matrix 22 | gt_label[gt_label == -100] = self.num_classes 23 | confusion_matrix = CM(gt_label.flatten(), 24 | pred_label.flatten(), 25 | labels=self.labels) 26 | self.confusion_matrix += confusion_matrix 27 | 28 | def batch_update(self, pred_labels, gt_labels): 29 | assert len(pred_labels) == len(gt_labels) 30 | for pred_label, gt_label in zip(pred_labels, gt_labels): 31 | self.update(pred_label, gt_label) 32 | 33 | @property 34 | def overall_acc(self): 35 | return np.sum(np.diag(self.confusion_matrix)) / np.sum(self.confusion_matrix) 36 | 37 | @property 38 | def overall_iou(self): 39 | class_iou = np.array(self.class_iou.copy()) 40 | class_iou[np.isnan(class_iou)] = 0 41 | return np.mean(class_iou) 42 | 43 | @property 44 | def class_seg_acc(self): 45 | return [self.confusion_matrix[i, i] / np.sum(self.confusion_matrix[i]) if np.sum(self.confusion_matrix[i]) else 0. 46 | for i in range(self.num_classes)] 47 | 48 | @property 49 | def class_iou(self): 50 | iou_list = [] 51 | for i in range(self.num_classes): 52 | tp = self.confusion_matrix[i, i] 53 | p = self.confusion_matrix[:, i].sum() 54 | g = self.confusion_matrix[i, :].sum() 55 | union = p + g - tp 56 | if union == 0: 57 | iou = float('nan') 58 | else: 59 | iou = tp / union 60 | iou_list.append(iou) 61 | return iou_list 62 | 63 | def print_table(self): 64 | from tabulate import tabulate 65 | header = ['Class', 'Accuracy', 'IOU', 'Total'] 66 | seg_acc_per_class = self.class_seg_acc 67 | iou_per_class = self.class_iou 68 | table = [] 69 | for ind, class_name in enumerate(self.class_names): 70 | table.append([class_name, 71 | seg_acc_per_class[ind] * 100, 72 | iou_per_class[ind] * 100, 73 | int(self.confusion_matrix[ind].sum()), 74 | ]) 75 | return tabulate(table, headers=header, tablefmt='psql', floatfmt='.2f') 76 | 77 | def save_table(self, filename): 78 | from tabulate import tabulate 79 | header = ('overall acc', 'overall iou') + self.class_names 80 | table = [[self.overall_acc, self.overall_iou] + self.class_iou] 81 | with open(filename, 'w') as f: 82 | # In order to unify format, remove all the alignments. 83 | f.write(tabulate(table, headers=header, tablefmt='tsv', floatfmt='.5f', 84 | numalign=None, stralign=None)) 85 | -------------------------------------------------------------------------------- /xmuda/data/utils/turbo_cmap.py: -------------------------------------------------------------------------------- 1 | # Reference: https://gist.github.com/mikhailov-work/ee72ba4191942acecc03fe6da94fc73f 2 | 3 | # Copyright 2019 Google LLC. 4 | # SPDX-License-Identifier: Apache-2.0 5 | 6 | # Author: Anton Mikhailov 7 | 8 | turbo_colormap_data = [[0.18995,0.07176,0.23217],[0.19483,0.08339,0.26149],[0.19956,0.09498,0.29024],[0.20415,0.10652,0.31844],[0.20860,0.11802,0.34607],[0.21291,0.12947,0.37314],[0.21708,0.14087,0.39964],[0.22111,0.15223,0.42558],[0.22500,0.16354,0.45096],[0.22875,0.17481,0.47578],[0.23236,0.18603,0.50004],[0.23582,0.19720,0.52373],[0.23915,0.20833,0.54686],[0.24234,0.21941,0.56942],[0.24539,0.23044,0.59142],[0.24830,0.24143,0.61286],[0.25107,0.25237,0.63374],[0.25369,0.26327,0.65406],[0.25618,0.27412,0.67381],[0.25853,0.28492,0.69300],[0.26074,0.29568,0.71162],[0.26280,0.30639,0.72968],[0.26473,0.31706,0.74718],[0.26652,0.32768,0.76412],[0.26816,0.33825,0.78050],[0.26967,0.34878,0.79631],[0.27103,0.35926,0.81156],[0.27226,0.36970,0.82624],[0.27334,0.38008,0.84037],[0.27429,0.39043,0.85393],[0.27509,0.40072,0.86692],[0.27576,0.41097,0.87936],[0.27628,0.42118,0.89123],[0.27667,0.43134,0.90254],[0.27691,0.44145,0.91328],[0.27701,0.45152,0.92347],[0.27698,0.46153,0.93309],[0.27680,0.47151,0.94214],[0.27648,0.48144,0.95064],[0.27603,0.49132,0.95857],[0.27543,0.50115,0.96594],[0.27469,0.51094,0.97275],[0.27381,0.52069,0.97899],[0.27273,0.53040,0.98461],[0.27106,0.54015,0.98930],[0.26878,0.54995,0.99303],[0.26592,0.55979,0.99583],[0.26252,0.56967,0.99773],[0.25862,0.57958,0.99876],[0.25425,0.58950,0.99896],[0.24946,0.59943,0.99835],[0.24427,0.60937,0.99697],[0.23874,0.61931,0.99485],[0.23288,0.62923,0.99202],[0.22676,0.63913,0.98851],[0.22039,0.64901,0.98436],[0.21382,0.65886,0.97959],[0.20708,0.66866,0.97423],[0.20021,0.67842,0.96833],[0.19326,0.68812,0.96190],[0.18625,0.69775,0.95498],[0.17923,0.70732,0.94761],[0.17223,0.71680,0.93981],[0.16529,0.72620,0.93161],[0.15844,0.73551,0.92305],[0.15173,0.74472,0.91416],[0.14519,0.75381,0.90496],[0.13886,0.76279,0.89550],[0.13278,0.77165,0.88580],[0.12698,0.78037,0.87590],[0.12151,0.78896,0.86581],[0.11639,0.79740,0.85559],[0.11167,0.80569,0.84525],[0.10738,0.81381,0.83484],[0.10357,0.82177,0.82437],[0.10026,0.82955,0.81389],[0.09750,0.83714,0.80342],[0.09532,0.84455,0.79299],[0.09377,0.85175,0.78264],[0.09287,0.85875,0.77240],[0.09267,0.86554,0.76230],[0.09320,0.87211,0.75237],[0.09451,0.87844,0.74265],[0.09662,0.88454,0.73316],[0.09958,0.89040,0.72393],[0.10342,0.89600,0.71500],[0.10815,0.90142,0.70599],[0.11374,0.90673,0.69651],[0.12014,0.91193,0.68660],[0.12733,0.91701,0.67627],[0.13526,0.92197,0.66556],[0.14391,0.92680,0.65448],[0.15323,0.93151,0.64308],[0.16319,0.93609,0.63137],[0.17377,0.94053,0.61938],[0.18491,0.94484,0.60713],[0.19659,0.94901,0.59466],[0.20877,0.95304,0.58199],[0.22142,0.95692,0.56914],[0.23449,0.96065,0.55614],[0.24797,0.96423,0.54303],[0.26180,0.96765,0.52981],[0.27597,0.97092,0.51653],[0.29042,0.97403,0.50321],[0.30513,0.97697,0.48987],[0.32006,0.97974,0.47654],[0.33517,0.98234,0.46325],[0.35043,0.98477,0.45002],[0.36581,0.98702,0.43688],[0.38127,0.98909,0.42386],[0.39678,0.99098,0.41098],[0.41229,0.99268,0.39826],[0.42778,0.99419,0.38575],[0.44321,0.99551,0.37345],[0.45854,0.99663,0.36140],[0.47375,0.99755,0.34963],[0.48879,0.99828,0.33816],[0.50362,0.99879,0.32701],[0.51822,0.99910,0.31622],[0.53255,0.99919,0.30581],[0.54658,0.99907,0.29581],[0.56026,0.99873,0.28623],[0.57357,0.99817,0.27712],[0.58646,0.99739,0.26849],[0.59891,0.99638,0.26038],[0.61088,0.99514,0.25280],[0.62233,0.99366,0.24579],[0.63323,0.99195,0.23937],[0.64362,0.98999,0.23356],[0.65394,0.98775,0.22835],[0.66428,0.98524,0.22370],[0.67462,0.98246,0.21960],[0.68494,0.97941,0.21602],[0.69525,0.97610,0.21294],[0.70553,0.97255,0.21032],[0.71577,0.96875,0.20815],[0.72596,0.96470,0.20640],[0.73610,0.96043,0.20504],[0.74617,0.95593,0.20406],[0.75617,0.95121,0.20343],[0.76608,0.94627,0.20311],[0.77591,0.94113,0.20310],[0.78563,0.93579,0.20336],[0.79524,0.93025,0.20386],[0.80473,0.92452,0.20459],[0.81410,0.91861,0.20552],[0.82333,0.91253,0.20663],[0.83241,0.90627,0.20788],[0.84133,0.89986,0.20926],[0.85010,0.89328,0.21074],[0.85868,0.88655,0.21230],[0.86709,0.87968,0.21391],[0.87530,0.87267,0.21555],[0.88331,0.86553,0.21719],[0.89112,0.85826,0.21880],[0.89870,0.85087,0.22038],[0.90605,0.84337,0.22188],[0.91317,0.83576,0.22328],[0.92004,0.82806,0.22456],[0.92666,0.82025,0.22570],[0.93301,0.81236,0.22667],[0.93909,0.80439,0.22744],[0.94489,0.79634,0.22800],[0.95039,0.78823,0.22831],[0.95560,0.78005,0.22836],[0.96049,0.77181,0.22811],[0.96507,0.76352,0.22754],[0.96931,0.75519,0.22663],[0.97323,0.74682,0.22536],[0.97679,0.73842,0.22369],[0.98000,0.73000,0.22161],[0.98289,0.72140,0.21918],[0.98549,0.71250,0.21650],[0.98781,0.70330,0.21358],[0.98986,0.69382,0.21043],[0.99163,0.68408,0.20706],[0.99314,0.67408,0.20348],[0.99438,0.66386,0.19971],[0.99535,0.65341,0.19577],[0.99607,0.64277,0.19165],[0.99654,0.63193,0.18738],[0.99675,0.62093,0.18297],[0.99672,0.60977,0.17842],[0.99644,0.59846,0.17376],[0.99593,0.58703,0.16899],[0.99517,0.57549,0.16412],[0.99419,0.56386,0.15918],[0.99297,0.55214,0.15417],[0.99153,0.54036,0.14910],[0.98987,0.52854,0.14398],[0.98799,0.51667,0.13883],[0.98590,0.50479,0.13367],[0.98360,0.49291,0.12849],[0.98108,0.48104,0.12332],[0.97837,0.46920,0.11817],[0.97545,0.45740,0.11305],[0.97234,0.44565,0.10797],[0.96904,0.43399,0.10294],[0.96555,0.42241,0.09798],[0.96187,0.41093,0.09310],[0.95801,0.39958,0.08831],[0.95398,0.38836,0.08362],[0.94977,0.37729,0.07905],[0.94538,0.36638,0.07461],[0.94084,0.35566,0.07031],[0.93612,0.34513,0.06616],[0.93125,0.33482,0.06218],[0.92623,0.32473,0.05837],[0.92105,0.31489,0.05475],[0.91572,0.30530,0.05134],[0.91024,0.29599,0.04814],[0.90463,0.28696,0.04516],[0.89888,0.27824,0.04243],[0.89298,0.26981,0.03993],[0.88691,0.26152,0.03753],[0.88066,0.25334,0.03521],[0.87422,0.24526,0.03297],[0.86760,0.23730,0.03082],[0.86079,0.22945,0.02875],[0.85380,0.22170,0.02677],[0.84662,0.21407,0.02487],[0.83926,0.20654,0.02305],[0.83172,0.19912,0.02131],[0.82399,0.19182,0.01966],[0.81608,0.18462,0.01809],[0.80799,0.17753,0.01660],[0.79971,0.17055,0.01520],[0.79125,0.16368,0.01387],[0.78260,0.15693,0.01264],[0.77377,0.15028,0.01148],[0.76476,0.14374,0.01041],[0.75556,0.13731,0.00942],[0.74617,0.13098,0.00851],[0.73661,0.12477,0.00769],[0.72686,0.11867,0.00695],[0.71692,0.11268,0.00629],[0.70680,0.10680,0.00571],[0.69650,0.10102,0.00522],[0.68602,0.09536,0.00481],[0.67535,0.08980,0.00449],[0.66449,0.08436,0.00424],[0.65345,0.07902,0.00408],[0.64223,0.07380,0.00401],[0.63082,0.06868,0.00401],[0.61923,0.06367,0.00410],[0.60746,0.05878,0.00427],[0.59550,0.05399,0.00453],[0.58336,0.04931,0.00486],[0.57103,0.04474,0.00529],[0.55852,0.04028,0.00579],[0.54583,0.03593,0.00638],[0.53295,0.03169,0.00705],[0.51989,0.02756,0.00780],[0.50664,0.02354,0.00863],[0.49321,0.01963,0.00955],[0.47960,0.01583,0.01055]] 9 | 10 | # The look-up table contains 256 entries. Each entry is a floating point sRGB triplet. 11 | # To use it with matplotlib, pass cmap=ListedColormap(turbo_colormap_data) as an arg to imshow() (don't forget "from matplotlib.colors import ListedColormap"). 12 | # If you have a typical 8-bit greyscale image, you can use the 8-bit value to index into this LUT directly. 13 | # The floating point color values can be converted to 8-bit sRGB via multiplying by 255 and casting/flooring to an integer. Saturation should not be required for IEEE-754 compliant arithmetic. 14 | # If you have a floating point value in the range [0,1], you can use interpolate() to linearly interpolate between the entries. 15 | # If you have 16-bit or 32-bit integer values, convert them to floating point values on the [0,1] range and then use interpolate(). Doing the interpolation in floating point will reduce banding. 16 | # If some of your values may lie outside the [0,1] range, use interpolate_or_clip() to highlight them. 17 | 18 | def interpolate(colormap, x): 19 | x = max(0.0, min(1.0, x)) 20 | a = int(x*255.0) 21 | b = min(255, a + 1) 22 | f = x*255.0 - a 23 | return [colormap[a][0] + (colormap[b][0] - colormap[a][0]) * f, 24 | colormap[a][1] + (colormap[b][1] - colormap[a][1]) * f, 25 | colormap[a][2] + (colormap[b][2] - colormap[a][2]) * f] 26 | 27 | def interpolate_or_clip(colormap, x): 28 | if x < 0.0: return [0.0, 0.0, 0.0] 29 | elif x > 1.0: return [1.0, 1.0, 1.0] 30 | else: return interpolate(colormap, x) 31 | -------------------------------------------------------------------------------- /xmuda/data/utils/validate.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import logging 3 | import time 4 | import os 5 | 6 | import torch 7 | import torch.nn.functional as F 8 | 9 | import matplotlib.pyplot as plt 10 | 11 | from xmuda.data.utils.evaluate import Evaluator 12 | from xmuda.data.utils.visualize import * 13 | 14 | def validate(cfg, 15 | model_2d, 16 | model_3d, 17 | dataloader, 18 | val_metric_logger, 19 | pselab_path=None, 20 | img_path=None, 21 | video_path=None): 22 | logger = logging.getLogger('xmuda.validate') 23 | logger.info('Validation') 24 | 25 | if img_path or video_path: 26 | model_2d.full_img = True 27 | 28 | # evaluator 29 | class_names = dataloader.dataset.class_names 30 | evaluator_2d = Evaluator(class_names) 31 | evaluator_3d = Evaluator(class_names) if model_3d else None 32 | evaluator_ensemble = Evaluator(class_names) if model_3d else None 33 | 34 | pselab_data_list = [] 35 | 36 | end = time.time() 37 | iter = 0 38 | with torch.no_grad(): 39 | for iteration, data_batch in enumerate(dataloader): 40 | data_time = time.time() - end 41 | # copy data from cpu to gpu 42 | if 'SCN' in cfg.DATASET_TARGET.TYPE: 43 | data_batch['x'][1] = data_batch['x'][1].cuda() 44 | data_batch['seg_label'] = data_batch['seg_label'].cuda() 45 | data_batch['img'] = data_batch['img'].cuda() 46 | else: 47 | raise NotImplementedError 48 | 49 | # predict 50 | preds_2d = model_2d(data_batch) 51 | preds_3d = model_3d(data_batch) if model_3d else None 52 | 53 | pred_label_voxel_2d = preds_2d['seg_logit'].argmax(1).cpu().numpy() 54 | if model_2d.dual_head: 55 | pred_label_voxel_2d_3d = preds_2d['seg_logit2'].argmax(1).cpu().numpy() 56 | 57 | pred_label_voxel_3d = preds_3d['seg_logit'].argmax(1).cpu().numpy() if model_3d else None 58 | if model_3d.dual_head: 59 | pred_label_voxel_3d_2d = preds_3d['seg_logit2'].argmax(1).cpu().numpy() if model_3d else None 60 | 61 | # softmax average (ensembling) 62 | probs_2d = F.softmax(preds_2d['seg_logit'], dim=1) 63 | probs_3d = F.softmax(preds_3d['seg_logit'], dim=1) if model_3d else None 64 | feats_2d = preds_2d['feats'] 65 | feats_3d = preds_3d['feats'] if model_3d else None 66 | pred_label_voxel_ensemble = (probs_2d + probs_3d).argmax(1).cpu().numpy() if model_3d else None 67 | 68 | # get original point cloud from before voxelization 69 | seg_label = data_batch['orig_seg_label'] 70 | points_idx = data_batch['orig_points_idx'] 71 | # loop over batch 72 | left_idx = 0 73 | for batch_ind in range(len(seg_label)): 74 | curr_points_idx = points_idx[batch_ind] 75 | # check if all points have predictions (= all voxels inside receptive field) 76 | assert np.all(curr_points_idx) 77 | 78 | curr_seg_label = seg_label[batch_ind] 79 | right_idx = left_idx + curr_points_idx.sum() 80 | pred_label_2d = pred_label_voxel_2d[left_idx:right_idx] 81 | pred_label_3d = pred_label_voxel_3d[left_idx:right_idx] if model_3d else None 82 | pred_label_ensemble = pred_label_voxel_ensemble[left_idx:right_idx] if model_3d else None 83 | 84 | # evaluate 85 | evaluator_2d.update(pred_label_2d, curr_seg_label) 86 | if model_3d: 87 | evaluator_3d.update(pred_label_3d, curr_seg_label) 88 | evaluator_ensemble.update(pred_label_ensemble, curr_seg_label) 89 | 90 | if pselab_path is not None: 91 | assert np.all(pred_label_2d >= 0) 92 | curr_probs_2d = probs_2d[left_idx:right_idx] 93 | curr_probs_3d = probs_3d[left_idx:right_idx] if model_3d else None 94 | curr_feats_2d = feats_2d[left_idx:right_idx] 95 | curr_feats_3d = feats_3d[left_idx:right_idx] if model_3d else None 96 | pselab_data_list.append({ 97 | 'probs_2d': curr_probs_2d.cpu().numpy(), 98 | 'features_2d': curr_feats_2d.cpu().numpy(), 99 | 'probs_3d': curr_probs_3d.cpu().numpy() if model_3d else None, 100 | 'features_3d': curr_feats_3d.cpu().numpy() if model_3d else None 101 | }) 102 | 103 | #draw images, first iteration only. 104 | if (img_path and iteration==0) or video_path: 105 | print(data_batch['img_path'][batch_ind]) 106 | 107 | img = plt.imread(data_batch['img_path'][batch_ind]) 108 | img_indices = data_batch['img_indices'][batch_ind] 109 | seg_labels = pred_label_voxel_2d[left_idx:right_idx] 110 | save_pth = os.path.join(img_path, '2d_ptcloud_{}.png'.format(iter)) 111 | draw_points_image_labels(img, img_indices, seg_labels, save_pth=save_pth, color_palette_type=cfg.DATASET_TARGET.TYPE[:-3]) 112 | draw_points_pred_gt(img, img_indices, pred_label_voxel_2d[left_idx:right_idx], data_batch['seg_label'][left_idx:right_idx].cpu().numpy(), save_pth='2d_correct.png') 113 | 114 | if model_2d.dual_head: 115 | seg_labels = pred_label_voxel_2d_3d[left_idx:right_idx] 116 | save_pth = os.path.join(img_path, '2d_3d_ptcloud_{}.png'.format(iter)) 117 | draw_points_image_labels(img, img_indices, seg_labels, save_pth=save_pth, color_palette_type=cfg.DATASET_TARGET.TYPE[:-3]) 118 | 119 | seg_labels = pred_label_voxel_3d[left_idx:right_idx] 120 | save_pth = os.path.join(img_path, '3d_ptcloud_{}.png'.format(iter)) 121 | draw_points_image_labels(img, img_indices, seg_labels, save_pth=save_pth, color_palette_type=cfg.DATASET_TARGET.TYPE[:-3]) 122 | draw_points_pred_gt(img, img_indices, pred_label_voxel_3d[left_idx:right_idx], data_batch['seg_label'][left_idx:right_idx].cpu().numpy(), save_pth='3d_correct.png') 123 | 124 | if model_3d.dual_head: 125 | seg_labels = pred_label_voxel_3d_2d[left_idx:right_idx] 126 | save_pth = os.path.join(img_path, '3d_2d_ptcloud_{}.png'.format(iter)) 127 | draw_points_image_labels(img, img_indices, seg_labels, save_pth=save_pth, color_palette_type=cfg.DATASET_TARGET.TYPE[:-3]) 128 | 129 | seg_labels = pred_label_voxel_ensemble[left_idx:right_idx] 130 | save_pth = os.path.join(img_path, 'softmax_ptcloud_{}.png'.format(iter)) 131 | draw_points_image_labels(img, img_indices, seg_labels, save_pth=save_pth, color_palette_type=cfg.DATASET_TARGET.TYPE[:-3]) 132 | 133 | seg_labels = data_batch['seg_label'][left_idx:right_idx].cpu().numpy() 134 | save_pth = os.path.join(img_path, 'gt_ptcloud_{}.png'.format(iter)) 135 | draw_points_image_labels(img, img_indices, seg_labels, save_pth=save_pth, color_palette_type=cfg.DATASET_TARGET.TYPE[:-3]) 136 | 137 | draw_points_pred_agree(img, img_indices, pred_label_voxel_2d[left_idx:right_idx], pred_label_voxel_3d[left_idx:right_idx], save_pth='agreement.png') 138 | 139 | img = preds_2d['full_logit'][batch_ind].argmax(0).cpu().numpy() 140 | save_pth = os.path.join(img_path, '2d_fullimage_{}.png'.format(iter)) 141 | print(save_pth) 142 | draw_seg_image(img, save_pth=save_pth, color_palette_type=cfg.DATASET_TARGET.TYPE[:-3]) 143 | 144 | iter += 1 145 | exit() 146 | 147 | 148 | left_idx = right_idx 149 | 150 | seg_loss_2d = F.cross_entropy(preds_2d['seg_logit'], data_batch['seg_label']) 151 | seg_loss_3d = F.cross_entropy(preds_3d['seg_logit'], data_batch['seg_label']) if model_3d else None 152 | val_metric_logger.update(seg_loss_2d=seg_loss_2d) 153 | if seg_loss_3d is not None: 154 | val_metric_logger.update(seg_loss_3d=seg_loss_3d) 155 | 156 | batch_time = time.time() - end 157 | val_metric_logger.update(time=batch_time, data=data_time) 158 | end = time.time() 159 | 160 | if not video_path: 161 | model_2d.full_img = False 162 | 163 | # log 164 | cur_iter = iteration + 1 165 | if cur_iter == 1 or (cfg.VAL.LOG_PERIOD > 0 and cur_iter % cfg.VAL.LOG_PERIOD == 0): 166 | logger.info( 167 | val_metric_logger.delimiter.join( 168 | [ 169 | 'iter: {iter}/{total_iter}', 170 | '{meters}', 171 | 'max mem: {memory:.0f}', 172 | ] 173 | ).format( 174 | iter=cur_iter, 175 | total_iter=len(dataloader), 176 | meters=str(val_metric_logger), 177 | memory=torch.cuda.max_memory_allocated() / (1024.0 ** 2), 178 | ) 179 | ) 180 | 181 | val_metric_logger.update(seg_iou_2d=evaluator_2d.overall_iou) 182 | if evaluator_3d is not None: 183 | val_metric_logger.update(seg_iou_3d=evaluator_3d.overall_iou) 184 | eval_list = [('2D', evaluator_2d)] 185 | if model_3d: 186 | eval_list.extend([('3D', evaluator_3d), ('2D+3D', evaluator_ensemble)]) 187 | for modality, evaluator in eval_list: 188 | logger.info('{} overall accuracy={:.2f}%'.format(modality, 100.0 * evaluator.overall_acc)) 189 | logger.info('{} overall IOU={:.2f}'.format(modality, 100.0 * evaluator.overall_iou)) 190 | logger.info('{} class-wise segmentation accuracy and IoU.\n{}'.format(modality, evaluator.print_table())) 191 | 192 | model_2d.full_img = False 193 | if pselab_path is not None: 194 | i = 1 195 | while len(pselab_data_list): 196 | pselab_data = {} 197 | end = min(1000, len(pselab_data_list)) 198 | print(end, i, len(pselab_data_list)) 199 | for key in list(pselab_data_list[0].keys()): 200 | pselab_data[key] = np.concatenate([item[key] for item in pselab_data_list[:end]]) 201 | 202 | pselab_data_list = pselab_data_list[end:] 203 | save_path = pselab_path + '_pt{:03d}'.format(i) 204 | np.savez(save_path, **pselab_data) 205 | logger.info('Saved pseudo label data to {}'.format(save_path)) 206 | i += 1 207 | -------------------------------------------------------------------------------- /xmuda/draw_images.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import os 3 | import os.path as osp 4 | import argparse 5 | import logging 6 | import time 7 | import socket 8 | import warnings 9 | 10 | import torch 11 | 12 | from xmuda.common.utils.checkpoint import CheckpointerV2 13 | from xmuda.common.utils.logger import setup_logger 14 | from xmuda.common.utils.metric_logger import MetricLogger 15 | from xmuda.common.utils.torch_util import set_random_seed 16 | from xmuda.models.build import build_model_2d, build_model_3d 17 | from xmuda.data.build import build_dataloader 18 | from xmuda.data.utils.validate import validate 19 | from xmuda.data.utils.visualize import draw_points_image_labels 20 | 21 | 22 | def parse_args(): 23 | parser = argparse.ArgumentParser(description='xMUDA test') 24 | parser.add_argument( 25 | '--cfg', 26 | dest='config_file', 27 | default='', 28 | metavar='FILE', 29 | help='path to config file', 30 | type=str, 31 | ) 32 | parser.add_argument('ckpt2d', type=str, help='path to checkpoint file of the 2D model') 33 | parser.add_argument('ckpt3d', type=str, help='path to checkpoint file of the 3D model') 34 | parser.add_argument('--pselab', action='store_true', help='generate pseudo-labels') 35 | parser.add_argument( 36 | 'opts', 37 | help='Modify config options using the command-line', 38 | default=None, 39 | nargs=argparse.REMAINDER, 40 | ) 41 | args = parser.parse_args() 42 | return args 43 | 44 | 45 | def draw(cfg, args, output_dir=''): 46 | logger = logging.getLogger('xmuda.test') 47 | 48 | # build 2d model 49 | model_2d = build_model_2d(cfg)[0] 50 | 51 | # build 3d model 52 | model_3d = build_model_3d(cfg)[0] 53 | 54 | model_2d = model_2d.cuda() 55 | model_3d = model_3d.cuda() 56 | 57 | # build checkpointer 58 | checkpointer_2d = CheckpointerV2(model_2d, save_dir=output_dir, logger=logger) 59 | if args.ckpt2d: 60 | # load weight if specified 61 | weight_path = args.ckpt2d.replace('@', output_dir) 62 | checkpointer_2d.load(weight_path, resume=False) 63 | else: 64 | # load last checkpoint 65 | checkpointer_2d.load(None, resume=True) 66 | checkpointer_3d = CheckpointerV2(model_3d, save_dir=output_dir, logger=logger) 67 | if args.ckpt3d: 68 | # load weight if specified 69 | weight_path = args.ckpt3d.replace('@', output_dir) 70 | checkpointer_3d.load(weight_path, resume=False) 71 | else: 72 | # load last checkpoint 73 | checkpointer_3d.load(None, resume=True) 74 | 75 | # build dataset 76 | test_dataloader = build_dataloader(cfg, mode='test', domain='target') 77 | 78 | pselab_path = None 79 | if args.pselab: 80 | pselab_dir = osp.join(output_dir, 'pselab_data') 81 | os.makedirs(pselab_dir, exist_ok=True) 82 | assert len(cfg.DATASET_TARGET.TEST) == 1 83 | pselab_path = osp.join(pselab_dir, cfg.DATASET_TARGET.TEST[0] + '.npy') 84 | 85 | # ---------------------------------------------------------------------------- # 86 | # Test 87 | # ---------------------------------------------------------------------------- # 88 | 89 | set_random_seed(cfg.RNG_SEED) 90 | test_metric_logger = MetricLogger(delimiter=' ') 91 | model_2d.eval() 92 | model_3d.eval() 93 | 94 | 95 | with torch.no_grad(): 96 | for iteration, data_batch in enumerate(test_dataloader): 97 | # copy data from cpu to gpu 98 | if 'SCN' in cfg.DATASET_TARGET.TYPE: 99 | data_batch['x'][1] = data_batch['x'][1].cuda() 100 | data_batch['seg_label'] = data_batch['seg_label'].cuda() 101 | data_batch['img'] = data_batch['img'].cuda() 102 | else: 103 | raise NotImplementedError 104 | 105 | # predict 106 | preds_2d = model_2d(data_batch) 107 | preds_3d = model_3d(data_batch) if model_3d else None 108 | 109 | pred_label_voxel_2d = preds_2d['seg_logit'].argmax(1).cpu().numpy() 110 | pred_label_voxel_3d = preds_3d['seg_logit'].argmax(1).cpu().numpy() if model_3d else None 111 | 112 | # get original point cloud from before voxelization 113 | seg_label = data_batch['orig_seg_label'] 114 | points_idx = data_batch['orig_points_idx'] 115 | 116 | draw_points_image_labels(255*torch.ones(data_batch['img'][0].shape[1:]), data_batch['img_indices'][0], pred_label_voxel_2d[0:points_idx[0].shape[0]], color_palette_type='SemanticKITTI') 117 | break 118 | 119 | 120 | def main(): 121 | args = parse_args() 122 | 123 | # load the configuration 124 | # import on-the-fly to avoid overwriting cfg 125 | from xmuda.common.config import purge_cfg 126 | from xmuda.config.xmuda import cfg 127 | cfg.merge_from_file(args.config_file) 128 | cfg.merge_from_list(args.opts) 129 | purge_cfg(cfg) 130 | cfg.freeze() 131 | 132 | output_dir = cfg.OUTPUT_DIR 133 | # replace '@' with config path 134 | if output_dir: 135 | config_path = osp.splitext(args.config_file)[0] 136 | output_dir = output_dir.replace('@', config_path.replace('configs/', '')) 137 | if not osp.isdir(output_dir): 138 | warnings.warn('Make a new directory: {}'.format(output_dir)) 139 | os.makedirs(output_dir) 140 | 141 | # run name 142 | timestamp = time.strftime('%m-%d_%H-%M-%S') 143 | hostname = socket.gethostname() 144 | run_name = '{:s}.{:s}'.format(timestamp, hostname) 145 | 146 | logger = setup_logger('xmuda', output_dir, comment='test.{:s}'.format(run_name)) 147 | logger.info('{:d} GPUs available'.format(torch.cuda.device_count())) 148 | logger.info(args) 149 | 150 | logger.info('Loaded configuration file {:s}'.format(args.config_file)) 151 | logger.info('Running with config:\n{}'.format(cfg)) 152 | 153 | assert cfg.MODEL_2D.DUAL_HEAD == cfg.MODEL_3D.DUAL_HEAD 154 | draw(cfg, args, output_dir) 155 | 156 | 157 | if __name__ == '__main__': 158 | main() 159 | -------------------------------------------------------------------------------- /xmuda/models/build.py: -------------------------------------------------------------------------------- 1 | from xmuda.models.xmuda_arch import Net2DSeg, Net3DSeg 2 | from xmuda.models.metric import SegIoU, ConfMatrix 3 | 4 | 5 | def build_model_2d(cfg): 6 | model = Net2DSeg(num_classes=cfg.MODEL_2D.NUM_CLASSES, 7 | backbone_2d=cfg.MODEL_2D.TYPE, 8 | backbone_2d_kwargs=cfg.MODEL_2D[cfg.MODEL_2D.TYPE], 9 | dual_head=cfg.MODEL_2D.DUAL_HEAD 10 | ) 11 | train_metric = SegIoU(cfg.MODEL_2D.NUM_CLASSES, name='seg_iou_2d') 12 | return model, train_metric 13 | 14 | 15 | def build_model_3d(cfg): 16 | model = Net3DSeg(num_classes=cfg.MODEL_3D.NUM_CLASSES, 17 | backbone_3d=cfg.MODEL_3D.TYPE, 18 | backbone_3d_kwargs=cfg.MODEL_3D[cfg.MODEL_3D.TYPE], 19 | dual_head=cfg.MODEL_3D.DUAL_HEAD 20 | ) 21 | train_metric = SegIoU(cfg.MODEL_3D.NUM_CLASSES, name='seg_iou_3d') 22 | return model, train_metric 23 | -------------------------------------------------------------------------------- /xmuda/models/losses.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import torch 3 | import logging 4 | 5 | 6 | def entropy_loss(v): 7 | """ 8 | Entropy loss for probabilistic prediction vectors 9 | input: batch_size x classes x points 10 | output: batch_size x 1 x points 11 | """ 12 | # (num points, num classes) 13 | if v.dim() == 2: 14 | v = v.transpose(0, 1) 15 | v = v.unsqueeze(0) 16 | # (1, num_classes, num_points) 17 | assert v.dim() == 3 18 | n, c, p = v.size() 19 | return -torch.sum(torch.mul(v, torch.log2(v + 1e-30))) / (n * p * np.log2(c)) 20 | 21 | def logcoral_loss(x_src, x_trg): 22 | """ 23 | Geodesic loss (log coral loss), reference: 24 | https://github.com/pmorerio/minimal-entropy-correlation-alignment/blob/master/svhn2mnist/model.py 25 | :param x_src: source features of size (N, ..., F), where N is the batch size and F is the feature size 26 | :param x_trg: target features of size (N, ..., F), where N is the batch size and F is the feature size 27 | :return: geodesic distance between the x_src and x_trg 28 | """ 29 | # check if the feature size is the same, so that the covariance matrices will have the same dimensions 30 | assert x_src.shape[-1] == x_trg.shape[-1] 31 | assert x_src.dim() >= 2 32 | batch_size = x_src.shape[0] 33 | if x_src.dim() > 2: 34 | # reshape from (N1, N2, ..., NM, F) to (N1 * N2 * ... * NM, F) 35 | x_src = x_src.flatten(end_dim=-2) 36 | x_trg = x_trg.flatten(end_dim=-2) 37 | 38 | # subtract the mean over the batch 39 | x_src = x_src - torch.mean(x_src, 0) 40 | x_trg = x_trg - torch.mean(x_trg, 0) 41 | 42 | # compute covariance 43 | factor = 1. / (batch_size - 1) 44 | 45 | cov_src = factor * torch.mm(x_src.t(), x_src) 46 | cov_trg = factor * torch.mm(x_trg.t(), x_trg) 47 | 48 | # dirty workaround to prevent GPU memory error due to MAGMA (used in SVD) 49 | # this implementation achieves loss of zero without creating a fork in the computation graph 50 | # if there is a nan or big number in the cov matrix, use where (not if!) to set cov matrix to identity matrix 51 | condition = (cov_src > 1e30).any() or (cov_trg > 1e30).any() or torch.isnan(cov_src).any() or torch.isnan(cov_trg).any() 52 | cov_src = torch.where(torch.full_like(cov_src, condition, dtype=torch.uint8), torch.eye(cov_src.shape[0], device=cov_src.device), cov_src) 53 | cov_trg = torch.where(torch.full_like(cov_trg, condition, dtype=torch.uint8), torch.eye(cov_trg.shape[0], device=cov_trg.device), cov_trg) 54 | 55 | if condition: 56 | logger = logging.getLogger('xmuda.train') 57 | logger.info('Big number > 1e30 or nan in covariance matrix, return loss of 0 to prevent error in SVD decomposition.') 58 | 59 | _, e_src, v_src = cov_src.svd() 60 | _, e_trg, v_trg = cov_trg.svd() 61 | 62 | # nan can occur when taking log of a value near 0 (problem occurs if the cov matrix is of low rank) 63 | log_cov_src = torch.mm(v_src, torch.mm(torch.diag(torch.log(e_src)), v_src.t())) 64 | log_cov_trg = torch.mm(v_trg, torch.mm(torch.diag(torch.log(e_trg)), v_trg.t())) 65 | 66 | # Frobenius norm 67 | return torch.mean((log_cov_src - log_cov_trg) ** 2) 68 | 69 | def entropy(probs, normalize=False, reduction='mean'): 70 | if probs.dim() == 2: 71 | probs = probs.unsqueeze(0) 72 | assert probs.dim() == 3 73 | assert reduction in ['none', 'mean', 'sum'] 74 | 75 | entropy = -torch.sum(probs * torch.log(probs + 1e-30), 2) 76 | if normalize: #Entropy normalized by log of number of classes, aka efficiancy 77 | entropy = entropy / torch.log(torch.tensor(probs.shape[2])) 78 | 79 | if reduction == 'mean': 80 | entropy = torch.mean(entropy) 81 | elif reduction == 'sum': 82 | entropy = torch.sum(entropy) 83 | 84 | return entropy 85 | 86 | def curriculum_entropy(probs, alpha=0.002, gamma=3, reduction='mean'): 87 | """ 88 | Information Maximization Loss for probabilistic prediction vectors 89 | input: batch_size x classes x points 90 | output: Scalar Loss 91 | """ 92 | # (num_points, num_classes) 93 | if probs.dim() == 2: 94 | probs = probs.unsqueeze(0) 95 | # (1, num_points, num_classes) 96 | assert probs.dim() == 3 97 | assert reduction in ['none', 'mean', 'sum'] 98 | 99 | h = entropy(probs, normalize=True, reduction='none') 100 | h = alpha * (1-h) ** gamma * h 101 | 102 | if reduction == 'mean': 103 | h = torch.mean(h) 104 | elif reduction == 'sum': 105 | h = torch.sum(h) 106 | 107 | return h 108 | 109 | def diversity(probs): 110 | """ 111 | Information Maximization Loss for probabilistic prediction vectors 112 | input: batch_size x classes x points 113 | output: Scalar Loss 114 | """ 115 | # (num_points, num_classes) 116 | if probs.dim() == 2: 117 | probs = probs.unsqueeze(0) 118 | # (1, num_points, num_classes) 119 | assert probs.dim() == 3 120 | 121 | return -entropy(torch.mean(probs, (0, 1), True)) 122 | 123 | def weighted_diversity(probs, lmbda=3): 124 | """ 125 | Information Maximization Loss for probabilistic prediction vectors 126 | input: batch_size x classes x points 127 | output: Scalar Loss 128 | """ 129 | # (num_points, num_classes) 130 | if probs.dim() == 2: 131 | probs = probs.unsqueeze(0) 132 | # (1, num_points, num_classes) 133 | assert probs.dim() == 3 134 | 135 | h = entropy(probs, normalize=True, reduction='none') 136 | 137 | weights = torch.exp(-lmbda * torch.unsqueeze(h, 2)) 138 | mprobs = torch.sum(weights * probs, (0, 1), True) / torch.sum(weights) 139 | return -entropy(mprobs, normalize=True) 140 | -------------------------------------------------------------------------------- /xmuda/models/metric.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from xmuda.common.utils.metric_logger import AverageMeter 3 | 4 | 5 | class SegAccuracy(AverageMeter): 6 | """Segmentation accuracy""" 7 | name = 'seg_acc' 8 | 9 | def __init__(self, ignore_index=-100): 10 | super(SegAccuracy, self).__init__() 11 | self.ignore_index = ignore_index 12 | 13 | def update_dict(self, preds, labels): 14 | seg_logit = preds['seg_logit'] # (b, c, n) 15 | seg_label = labels['seg_label'] # (b, n) 16 | pred_label = seg_logit.argmax(1) 17 | 18 | mask = (seg_label != self.ignore_index) 19 | seg_label = seg_label[mask] 20 | pred_label = pred_label[mask] 21 | 22 | tp_mask = pred_label.eq(seg_label) # (b, n) 23 | self.update(tp_mask.sum().item(), tp_mask.numel()) 24 | 25 | 26 | class SegIoU(object): 27 | """Segmentation IoU 28 | References: https://github.com/pytorch/vision/blob/master/references/segmentation/utils.py 29 | """ 30 | 31 | def __init__(self, num_classes, ignore_index=-100, name='seg_iou'): 32 | self.num_classes = num_classes 33 | self.ignore_index = ignore_index 34 | self.mat = None 35 | self.name = name 36 | 37 | def update_dict(self, preds, labels): 38 | seg_logit = preds['seg_logit'] # (batch_size, num_classes, num_points) 39 | seg_label = labels['seg_label'] # (batch_size, num_points) 40 | pred_label = seg_logit.argmax(1) 41 | 42 | mask = (seg_label != self.ignore_index) 43 | seg_label = seg_label[mask] 44 | pred_label = pred_label[mask] 45 | 46 | # Update confusion matrix 47 | # TODO: Compare the speed between torch.histogram and torch.bincount after pytorch v1.1.0 48 | n = self.num_classes 49 | with torch.no_grad(): 50 | if self.mat is None: 51 | self.mat = seg_label.new_zeros((n, n)) 52 | inds = n * seg_label + pred_label 53 | self.mat += torch.bincount(inds, minlength=n ** 2).reshape(n, n) 54 | 55 | def reset(self): 56 | self.mat = None 57 | 58 | @property 59 | def iou(self): 60 | h = self.mat.float() 61 | union = h.sum(1) + h.sum(0) - torch.diag(h) 62 | union[union == 0] = 1 63 | iou = torch.diag(h) / union 64 | return iou 65 | 66 | @property 67 | def global_avg(self): 68 | return self.iou.mean().item() 69 | 70 | @property 71 | def avg(self): 72 | return self.global_avg 73 | 74 | def __str__(self): 75 | return '{iou:.4f}'.format(iou=self.iou.mean().item()) 76 | 77 | @property 78 | def summary_str(self): 79 | return str(self) 80 | 81 | class ConfMatrix(object): 82 | def __init__(self, name='conf_matrix', inverse_labels=False): 83 | self.name = name 84 | self.inverse_labels = inverse_labels 85 | self.TP = 0. 86 | self.FP = 0. 87 | self.TN = 0. 88 | self.FN = 0. 89 | return 90 | 91 | def update_dict(self, preds, labels): 92 | with torch.no_grad(): 93 | preds = preds.detach() 94 | labels = labels.detach() 95 | if self.inverse_labels: 96 | self.TN += torch.sum(torch.logical_and(preds > 0.5, labels > 0.5)) 97 | self.FN += torch.sum(torch.logical_and(preds > 0.5, labels <= 0.5)) 98 | self.TP += torch.sum(torch.logical_and(preds <= 0.5, labels <= 0.5)) 99 | self.FP += torch.sum(torch.logical_and(preds <= 0.5, labels > 0.5)) 100 | else: 101 | self.TP += torch.sum(torch.logical_and(preds > 0.5, labels > 0.5)) 102 | self.FP += torch.sum(torch.logical_and(preds > 0.5, labels <= 0.5)) 103 | self.TN += torch.sum(torch.logical_and(preds <= 0.5, labels <= 0.5)) 104 | self.FN += torch.sum(torch.logical_and(preds <= 0.5, labels > 0.5)) 105 | 106 | def reset(self): 107 | self.TP = 0. 108 | self.FP = 0. 109 | self.TN = 0. 110 | self.FN = 0. 111 | 112 | def F1(self): 113 | precision = self.precision() 114 | recall = self.recall() 115 | if (precision + recall) == 0: 116 | return 0 117 | else: 118 | return 2*(precision*recall)/(precision + recall) 119 | 120 | def recall(self): 121 | P = self.TP + self.FN 122 | if P == 0: 123 | return 0 124 | else: 125 | return self.TP/P 126 | 127 | def precision(self): 128 | PP = self.TP + self.FP 129 | if PP == 0: 130 | return 0 131 | else: 132 | return self.TP/PP 133 | 134 | def __str__(self): 135 | return '(f1: {:.4f}, recall: {:.4f}, precision: {:.4f})'.format(self.F1(), self.recall(), self.precision()) 136 | -------------------------------------------------------------------------------- /xmuda/models/resnet34_unet.py: -------------------------------------------------------------------------------- 1 | """UNet based on ResNet34""" 2 | import torch 3 | import torch.nn as nn 4 | import torch.nn.functional as F 5 | from torchvision.models.resnet import resnet34 6 | 7 | 8 | class UNetResNet34(nn.Module): 9 | def __init__(self, pretrained=True): 10 | super(UNetResNet34, self).__init__() 11 | 12 | # ----------------------------------------------------------------------------- # 13 | # Encoder 14 | # ----------------------------------------------------------------------------- # 15 | net = resnet34(pretrained) 16 | # Note that we do not downsample for conv1 17 | # self.conv1 = net.conv1 18 | self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=1, padding=3, bias=False) 19 | self.conv1.weight.data = net.conv1.weight.data 20 | self.bn1 = net.bn1 21 | self.relu = net.relu 22 | self.maxpool = net.maxpool 23 | self.layer1 = net.layer1 24 | self.layer2 = net.layer2 25 | self.layer3 = net.layer3 26 | self.layer4 = net.layer4 27 | 28 | # ----------------------------------------------------------------------------- # 29 | # Decoder 30 | # ----------------------------------------------------------------------------- # 31 | _, self.dec_t_conv_stage5 = self.dec_stage(self.layer4, num_concat=1) 32 | self.dec_conv_stage4, self.dec_t_conv_stage4 = self.dec_stage(self.layer3, num_concat=2) 33 | self.dec_conv_stage3, self.dec_t_conv_stage3 = self.dec_stage(self.layer2, num_concat=2) 34 | self.dec_conv_stage2, self.dec_t_conv_stage2 = self.dec_stage(self.layer1, num_concat=2) 35 | self.dec_conv_stage1 = nn.Conv2d(2 * 64, 64, kernel_size=3, padding=1) 36 | 37 | # dropout 38 | self.dropout = nn.Dropout(p=0.4) 39 | 40 | @staticmethod 41 | def dec_stage(enc_stage, num_concat): 42 | in_channels = enc_stage[0].conv1.in_channels 43 | out_channels = enc_stage[-1].conv2.out_channels 44 | conv = nn.Sequential( 45 | nn.Conv2d(num_concat * out_channels, out_channels, kernel_size=3, padding=1), 46 | nn.BatchNorm2d(out_channels), 47 | nn.ReLU(inplace=True), 48 | ) 49 | t_conv = nn.Sequential( 50 | nn.ConvTranspose2d(out_channels, in_channels, kernel_size=2, stride=2), 51 | nn.BatchNorm2d(in_channels), 52 | nn.ReLU(inplace=True) 53 | ) 54 | return conv, t_conv 55 | 56 | def forward(self, x): 57 | # pad input to be divisible by 16 = 2 ** 4 58 | h, w = x.shape[2], x.shape[3] 59 | min_size = 16 60 | pad_h = int((h + min_size - 1) / min_size) * min_size - h 61 | pad_w = int((w + min_size - 1) / min_size) * min_size - w 62 | if pad_h > 0 or pad_w > 0: 63 | x = F.pad(x, [0, pad_w, 0, pad_h]) 64 | 65 | # ----------------------------------------------------------------------------- # 66 | # Encoder 67 | # ----------------------------------------------------------------------------- # 68 | inter_features = [] 69 | x = self.conv1(x) 70 | x = self.bn1(x) 71 | x = self.relu(x) 72 | inter_features.append(x) 73 | x = self.maxpool(x) # downsample 74 | x = self.layer1(x) 75 | inter_features.append(x) 76 | x = self.layer2(x) # downsample 77 | inter_features.append(x) 78 | x = self.layer3(x) # downsample 79 | x = self.dropout(x) 80 | inter_features.append(x) 81 | x = self.layer4(x) # downsample 82 | x = self.dropout(x) 83 | 84 | # ----------------------------------------------------------------------------- # 85 | # Decoder 86 | # ----------------------------------------------------------------------------- # 87 | # upsample 88 | x = self.dec_t_conv_stage5(x) 89 | x = torch.cat([inter_features[3], x], dim=1) 90 | x = self.dec_conv_stage4(x) 91 | 92 | # upsample 93 | x = self.dec_t_conv_stage4(x) 94 | x = torch.cat([inter_features[2], x], dim=1) 95 | x = self.dec_conv_stage3(x) 96 | 97 | # upsample 98 | x = self.dec_t_conv_stage3(x) 99 | x = torch.cat([inter_features[1], x], dim=1) 100 | x = self.dec_conv_stage2(x) 101 | 102 | # upsample 103 | x = self.dec_t_conv_stage2(x) 104 | x = torch.cat([inter_features[0], x], dim=1) 105 | x = self.dec_conv_stage1(x) 106 | 107 | # crop padding 108 | if pad_h > 0 or pad_w > 0: 109 | x = x[:, :, 0:h, 0:w] 110 | 111 | return x 112 | 113 | 114 | def test(): 115 | b, c, h, w = 2, 20, 120, 160 116 | image = torch.randn(b, 3, h, w).cuda() 117 | net = UNetResNet34(pretrained=True) 118 | net.cuda() 119 | feats = net(image) 120 | print('feats', feats.shape) 121 | 122 | 123 | if __name__ == '__main__': 124 | test() 125 | -------------------------------------------------------------------------------- /xmuda/models/scn_unet.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | 4 | import sparseconvnet as scn 5 | DIMENSION = 3 6 | 7 | 8 | class UNetSCN(nn.Module): 9 | def __init__(self, 10 | in_channels, 11 | m=16, # number of unet features (multiplied in each layer) 12 | block_reps=1, # depth 13 | residual_blocks=False, # ResNet style basic blocks 14 | full_scale=4096, 15 | num_planes=7 16 | ): 17 | super(UNetSCN, self).__init__() 18 | 19 | self.in_channels = in_channels 20 | self.out_channels = m 21 | n_planes = [(n + 1) * m for n in range(num_planes)] 22 | 23 | self.sparseModel = scn.Sequential().add( 24 | scn.InputLayer(DIMENSION, full_scale, mode=4)).add( 25 | scn.SubmanifoldConvolution(DIMENSION, in_channels, m, 3, False)).add( 26 | scn.UNet(DIMENSION, block_reps, n_planes, residual_blocks)).add( 27 | scn.BatchNormReLU(m)).add( 28 | scn.OutputLayer(DIMENSION)) 29 | 30 | def forward(self, x): 31 | x = self.sparseModel(x) 32 | return x 33 | 34 | 35 | def test(): 36 | b, n = 2, 100 37 | coords = torch.randint(4096, [b, n, DIMENSION]) 38 | batch_idxs = torch.arange(b).reshape(b, 1, 1).repeat(1, n, 1) 39 | coords = torch.cat([coords, batch_idxs], 2).reshape(-1, DIMENSION + 1) 40 | 41 | in_channels = 3 42 | feats = torch.rand(b * n, in_channels) 43 | 44 | x = [coords, feats.cuda()] 45 | 46 | 47 | net = UNetSCN(in_channels).cuda() 48 | out_feats = net(x) 49 | 50 | print('out_feats', out_feats.shape) 51 | 52 | 53 | if __name__ == '__main__': 54 | test() 55 | -------------------------------------------------------------------------------- /xmuda/models/xmuda_arch.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | 4 | from xmuda.models.resnet34_unet import UNetResNet34 5 | from xmuda.models.scn_unet import UNetSCN 6 | 7 | 8 | class Net2DSeg(nn.Module): 9 | def __init__(self, 10 | num_classes: int, 11 | dual_head: bool, 12 | backbone_2d, 13 | backbone_2d_kwargs: dict, 14 | full_img: bool = False, 15 | ): 16 | super(Net2DSeg, self).__init__() 17 | 18 | # 2D image network 19 | if backbone_2d == 'UNetResNet34': 20 | self.net_2d = UNetResNet34(**backbone_2d_kwargs) 21 | feat_channels = 64 22 | else: 23 | raise NotImplementedError('2D backbone {} not supported'.format(backbone_2d)) 24 | 25 | # segmentation head 26 | self.linear = nn.Linear(feat_channels, num_classes) 27 | 28 | # 2nd segmentation head 29 | self.dual_head = dual_head 30 | if dual_head: 31 | self.linear2 = nn.Linear(feat_channels, num_classes) 32 | 33 | # Full Image Return 34 | self.full_img = full_img 35 | 36 | def forward(self, data_batch): 37 | # (batch_size, 3, H, W) 38 | img = data_batch['img'] 39 | img_indices = data_batch['img_indices'] 40 | 41 | # 2D network 42 | feats = self.net_2d(img) 43 | 44 | # 2D-3D feature lifting 45 | img_feats = [] 46 | for i in range(feats.shape[0]): 47 | img_feats.append(feats.permute(0, 2, 3, 1)[i][img_indices[i][:, 0], img_indices[i][:, 1]]) 48 | img_feats = torch.cat(img_feats, 0) 49 | 50 | preds = {'feats': img_feats} 51 | 52 | if self.full_img: 53 | full_logits = self.linear(feats.permute(0, 2, 3, 1)).permute(0, 3, 1, 2) 54 | preds['full_logit'] = full_logits 55 | 56 | logits = [] 57 | for i in range(feats.shape[0]): 58 | logits.append(full_logits.permute(0, 2, 3, 1)[i][img_indices[i][:, 0], img_indices[i][:, 1]]) 59 | preds['seg_logit'] = torch.cat(logits, 0) 60 | 61 | if self.dual_head: 62 | full_logits2 = self.linear2(feats) 63 | preds['full_logit2'] = full_logits2 64 | 65 | logits = [] 66 | for i in range(feats.shape[0]): 67 | logits.append(full_logits.permute(0, 2, 3, 1)[i][img_indices[i][:, 0], img_indices[i][:, 1]]) 68 | preds['seg_logit2'] = torch.cat(logits, 0) 69 | 70 | else: 71 | # linear 72 | preds['seg_logit'] = self.linear(img_feats) 73 | if self.dual_head: 74 | preds['seg_logit2'] = self.linear2(img_feats) 75 | 76 | return preds 77 | 78 | def classify_features(self, features): 79 | return self.linear(features) 80 | 81 | def classifier_parameters(self): 82 | return self.linear.parameters() 83 | 84 | def xmuda_classifier_parameters(self): 85 | if self.dual_head: 86 | return self.linear2.parameters() 87 | return None 88 | 89 | def feat_encoder_parameters(self): 90 | return self.net_2d.parameters() 91 | 92 | class Net3DSeg(nn.Module): 93 | def __init__(self, 94 | num_classes, 95 | dual_head, 96 | backbone_3d, 97 | backbone_3d_kwargs, 98 | ): 99 | super(Net3DSeg, self).__init__() 100 | 101 | # 3D network 102 | if backbone_3d == 'SCN': 103 | self.net_3d = UNetSCN(**backbone_3d_kwargs) 104 | else: 105 | raise NotImplementedError('3D backbone {} not supported'.format(backbone_3d)) 106 | 107 | # segmentation head 108 | self.linear = nn.Linear(self.net_3d.out_channels, num_classes) 109 | 110 | # 2nd segmentation head 111 | self.dual_head = dual_head 112 | if dual_head: 113 | self.linear2 = nn.Linear(self.net_3d.out_channels, num_classes) 114 | 115 | def forward(self, data_batch): 116 | feats = self.net_3d(data_batch['x']) 117 | x = self.linear(feats) 118 | 119 | preds = { 120 | 'feats': feats, 121 | 'seg_logit': x, 122 | } 123 | 124 | if self.dual_head: 125 | preds['seg_logit2'] = self.linear2(feats) 126 | 127 | return preds 128 | 129 | def classify_features(self, features): 130 | return self.linear(features) 131 | 132 | def classifier_parameters(self): 133 | return self.linear.parameters() 134 | 135 | def xmuda_classifier_parameters(self): 136 | if self.dual_head: 137 | return self.linear2.parameters() 138 | return None 139 | 140 | def feat_encoder_parameters(self): 141 | return self.net_3d.parameters() 142 | 143 | def test_Net2DSeg(): 144 | # 2D 145 | batch_size = 2 146 | img_width = 400 147 | img_height = 225 148 | 149 | # 3D 150 | num_coords = 2000 151 | num_classes = 11 152 | 153 | # 2D 154 | img = torch.rand(batch_size, 3, img_height, img_width) 155 | u = torch.randint(high=img_height, size=(batch_size, num_coords // batch_size, 1)) 156 | v = torch.randint(high=img_width, size=(batch_size, num_coords // batch_size, 1)) 157 | img_indices = torch.cat([u, v], 2) 158 | 159 | # to cuda 160 | img = img.cuda() 161 | img_indices = img_indices.cuda() 162 | 163 | net_2d = Net2DSeg(num_classes, 164 | backbone_2d='UNetResNet34', 165 | backbone_2d_kwargs={}, 166 | dual_head=True) 167 | 168 | net_2d.cuda() 169 | out_dict = net_2d({ 170 | 'img': img, 171 | 'img_indices': img_indices, 172 | }) 173 | for k, v in out_dict.items(): 174 | print('Net2DSeg:', k, v.shape) 175 | 176 | 177 | def test_Net3DSeg(): 178 | in_channels = 1 179 | num_coords = 2000 180 | full_scale = 4096 181 | num_seg_classes = 11 182 | 183 | coords = torch.randint(high=full_scale, size=(num_coords, 3)) 184 | feats = torch.rand(num_coords, in_channels) 185 | 186 | feats = feats.cuda() 187 | 188 | net_3d = Net3DSeg(num_seg_classes, 189 | dual_head=True, 190 | backbone_3d='SCN', 191 | backbone_3d_kwargs={'in_channels': in_channels}) 192 | 193 | net_3d.cuda() 194 | out_dict = net_3d({ 195 | 'x': [coords, feats], 196 | }) 197 | for k, v in out_dict.items(): 198 | print('Net3DSeg:', k, v.shape) 199 | 200 | 201 | if __name__ == '__main__': 202 | test_Net2DSeg() 203 | test_Net3DSeg() 204 | -------------------------------------------------------------------------------- /xmuda/test.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import os 3 | import os.path as osp 4 | import argparse 5 | import logging 6 | import time 7 | import socket 8 | import warnings 9 | 10 | import torch 11 | 12 | from xmuda.common.utils.checkpoint import CheckpointerV2 13 | from xmuda.common.utils.logger import setup_logger 14 | from xmuda.common.utils.metric_logger import MetricLogger 15 | from xmuda.common.utils.torch_util import set_random_seed 16 | from xmuda.models.build import build_model_2d, build_model_3d 17 | from xmuda.data.build import build_dataloader 18 | from xmuda.data.utils.validate import validate 19 | 20 | 21 | def parse_args(): 22 | parser = argparse.ArgumentParser(description='xMUDA test') 23 | parser.add_argument( 24 | '--cfg', 25 | dest='config_file', 26 | default='', 27 | metavar='FILE', 28 | help='path to config file', 29 | type=str, 30 | ) 31 | parser.add_argument('ckpt2d', type=str, help='path to checkpoint file of the 2D model') 32 | parser.add_argument('ckpt3d', type=str, help='path to checkpoint file of the 3D model') 33 | parser.add_argument('--pselab', action='store_true', help='generate pseudo-labels') 34 | parser.add_argument('--source', action='store_true', help='Also calculate performance on source data') 35 | parser.add_argument('--draw', action='store_true', help='draw image of segmenation') 36 | parser.add_argument('--video', action='store_true', help='Create video of segmentation') 37 | parser.add_argument( 38 | 'opts', 39 | help='Modify config options using the command-line', 40 | default=None, 41 | nargs=argparse.REMAINDER, 42 | ) 43 | args = parser.parse_args() 44 | return args 45 | 46 | 47 | def test(cfg, args, output_dir=''): 48 | logger = logging.getLogger('xmuda.test') 49 | 50 | # build 2d model 51 | model_2d = build_model_2d(cfg)[0] 52 | 53 | # build 3d model 54 | model_3d = build_model_3d(cfg)[0] 55 | 56 | model_2d = model_2d.cuda() 57 | model_3d = model_3d.cuda() 58 | 59 | # build checkpointer 60 | checkpointer_2d = CheckpointerV2(model_2d, save_dir=output_dir, logger=logger) 61 | if args.ckpt2d: 62 | # load weight if specified 63 | weight_path = args.ckpt2d.replace('@', output_dir) 64 | checkpointer_2d.load(weight_path, resume=False) 65 | else: 66 | # load last checkpoint 67 | checkpointer_2d.load(None, resume=True) 68 | checkpointer_3d = CheckpointerV2(model_3d, save_dir=output_dir, logger=logger) 69 | if args.ckpt3d: 70 | # load weight if specified 71 | weight_path = args.ckpt3d.replace('@', output_dir) 72 | checkpointer_3d.load(weight_path, resume=False) 73 | else: 74 | # load last checkpoint 75 | checkpointer_3d.load(None, resume=True) 76 | 77 | # build dataset 78 | if args.source: 79 | source_dataloader = build_dataloader(cfg, mode='test', domain='source') 80 | target_dataloader = build_dataloader(cfg, mode='test', domain='target') 81 | 82 | pselab_path = None 83 | if args.pselab: 84 | pselab_dir = osp.join(output_dir, 'pselab_data') 85 | os.makedirs(pselab_dir, exist_ok=True) 86 | assert len(cfg.DATASET_TARGET.TEST) == 1 87 | pselab_path = osp.join(pselab_dir, cfg.DATASET_TARGET.TEST[0]) 88 | 89 | img_path = None 90 | if args.draw: 91 | img_path = osp.join(output_dir, 'imgs_new') 92 | os.makedirs(img_path, exist_ok=True) 93 | 94 | video_path = None 95 | if args.video: 96 | video_path = osp.join(output_dir, 'videos') 97 | os.makedirs(video_path, exist_ok=True) 98 | 99 | # ---------------------------------------------------------------------------- # 100 | # Test 101 | # ---------------------------------------------------------------------------- # 102 | 103 | set_random_seed(cfg.RNG_SEED) 104 | test_metric_logger = MetricLogger(delimiter=' ') 105 | model_2d.eval() 106 | model_3d.eval() 107 | 108 | if args.source: 109 | logger.info('Source Test Set') 110 | validate(cfg, model_2d, model_3d, source_dataloader, test_metric_logger) 111 | logger.info('Target Test Set') 112 | validate(cfg, model_2d, model_3d, target_dataloader, test_metric_logger, pselab_path=pselab_path, img_path=img_path, video_path=video_path) 113 | 114 | 115 | def main(): 116 | args = parse_args() 117 | 118 | # load the configuration 119 | # import on-the-fly to avoid overwriting cfg 120 | from xmuda.common.config import purge_cfg 121 | from xmuda.config.xmuda import cfg 122 | cfg.merge_from_file(args.config_file) 123 | cfg.merge_from_list(args.opts) 124 | purge_cfg(cfg) 125 | cfg.freeze() 126 | 127 | output_dir = cfg.OUTPUT_DIR 128 | # replace '@' with config path 129 | if output_dir: 130 | config_path = osp.splitext(args.config_file)[0] 131 | output_dir = output_dir.replace('@', config_path.replace('configs/', '')) 132 | if not osp.isdir(output_dir): 133 | warnings.warn('Make a new directory: {}'.format(output_dir)) 134 | os.makedirs(output_dir) 135 | 136 | # run name 137 | timestamp = time.strftime('%m-%d_%H-%M-%S') 138 | hostname = socket.gethostname() 139 | run_name = '{:s}.{:s}'.format(timestamp, hostname) 140 | 141 | logger = setup_logger('xmuda', output_dir, comment='test.{:s}'.format(run_name)) 142 | logger.info('{:d} GPUs available'.format(torch.cuda.device_count())) 143 | logger.info(args) 144 | 145 | logger.info('Loaded configuration file {:s}'.format(args.config_file)) 146 | logger.info('Running with config:\n{}'.format(cfg)) 147 | 148 | assert cfg.MODEL_2D.DUAL_HEAD == cfg.MODEL_3D.DUAL_HEAD 149 | test(cfg, args, output_dir) 150 | 151 | 152 | if __name__ == '__main__': 153 | main() 154 | --------------------------------------------------------------------------------