├── .gitignore ├── .gitmodules ├── LICENSE ├── README.md ├── data_factory ├── examples ├── datasets.ipynb ├── google_map_screenshots │ ├── 00.2013.03.png │ ├── 00.2024.07.png │ ├── 01.2013.02.png │ ├── 01.2023.05.png │ └── roi.npy └── inference.ipynb ├── requirements.txt ├── scripts ├── configs │ ├── fine_tune.yml │ └── train.yml ├── evaluate.affine.py ├── evaluate.py ├── fine_tune.py ├── train.py └── visualize.py ├── setup.py ├── src └── robust_scene_change_detect │ ├── datasets │ ├── __init__.py │ ├── indices │ │ ├── pscd.test.index │ │ ├── pscd.train.index │ │ ├── pscd.val.index │ │ ├── vl-cmu-cd.train.index │ │ └── vl-cmu-cd.val.index │ ├── pscd.py │ └── vl_cmu_cd.py │ ├── evaluation.py │ ├── models │ ├── CD_model.py │ ├── TANet.py │ ├── __init__.py │ ├── backbone_dinov2.py │ ├── backbone_resnet.py │ ├── coattention.py │ ├── merge_temporal_feature.py │ └── transformer.py │ └── torch_utils.py └── test ├── __init__.py ├── test_datasets.py ├── test_evaluation.py ├── test_models.py └── test_torch_utils.py /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__/* 2 | **/.DS_Store 3 | *.pyc 4 | *.pth 5 | *.egg-info/ -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "src/py_utils"] 2 | path = thirdparties/py_utils 3 | url = https://github.com/ChadLin9596/python_utils 4 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 Chad Lin 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Robust Scene Change Detection Using Visual Foundation Models and Cross-Attention Mechanisms 2 | 3 | 4 | ### Paper 5 | 6 | * [arXiv link](http://arxiv.org/abs/2409.16850) 7 | 8 | ### An Introduction Video (3 minutes) 9 | 10 | [![](https://img.youtube.com/vi/KX2E8Q5D-Fk/0.jpg)](https://www.youtube.com/watch?v=KX2E8Q5D-Fk) 11 | 12 | 13 | ### Installation 14 | 15 | ```bash 16 | # clone main repo and corresponding submodule 17 | $ git clone https://github.com/ChadLin9596/Robust-Scene-Change-Detection --recursive 18 | 19 | # or 20 | $ git clone https://github.com/ChadLin9596/Robust-Scene-Change-Detection 21 | $ cd 22 | $ git submodule init 23 | $ git submodule update 24 | 25 | # create a Python 3.9.6 (or other env can run DinoV2) virtual environment 26 | $ source /bin/activate 27 | $ cd 28 | $ pip install -r requirements.txt 29 | 30 | # install 31 | $ pip install -e thirdparties/py_utils 32 | $ pip install -e . 33 | 34 | ``` 35 | 36 | ### Datasets 37 | 38 | * download [VL-CMU-CD](https://huggingface.co/datasets/Flourish/VL-CMU-CD/blob/main/VL-CMU-CD-binary255.zip) & [PSCD](https://kensakurada.github.io/pscd/term_of_use.html) datasets 39 | 40 | * update the both dataset directories to `datasets/data_factory` 41 | 42 | ### Example usage 43 | 44 | * unittest 45 | ``` bash 46 | $ cd /src/unittest 47 | $ python -m unittest 48 | ``` 49 | 50 | * loading a model and test (please check [inference.ipynb](examples/inference.ipynb)) 51 | ```python 52 | import torch 53 | import robust_scene_change_detect.models as models 54 | 55 | B = 1 56 | H = 504 # need to be 14 * n 57 | W = 504 # need to be 14 * n 58 | 59 | # load model 60 | model = models.get_model_from_pretrained("dino_2Cross_CMU") 61 | model = model.cuda().eval() 62 | model.module.upsample.size = (H, W) 63 | 64 | # load image 65 | t0 = torch.rand(B, 3, H, W).cuda() 66 | t1 = torch.rand(B, 3, H, W).cuda() 67 | 68 | with torch.no_grad(): 69 | pred = model(t0, t1) # B, H, W, 2 70 | pred = pred.argmax(dim=-1) # B, H, W 71 | ``` 72 | 73 | * training 74 | ```bash 75 | # modify the configuration in scripts/configs/train.yml 76 | $ python /src/scripts/train.py \ 77 | /src/scripts/configs/train.yml 78 | ``` 79 | 80 | * fine-tune 81 | ```bash 82 | # modify the configuration in scripts/configs/fine_tune.yml 83 | $ python /src/scripts/fine_tune.py \ 84 | /src/scripts/configs/fine_tune.yml 85 | ``` 86 | 87 | * evaluation 88 | ```bash 89 | $ python /src/scripts/evaluate.py \ 90 | /.pth 91 | ``` 92 | 93 | * qualitive results 94 | ```bash 95 | $ python /scripts/visualize.py \ 96 | /best.val.pth \ 97 | --option